Unmanned aerial vehicle autonomous navigation method for urban air traffic

Abstract

The invention discloses an unmanned aerial vehicle autonomous navigation method for urban air traffic. The method mainly solves the problems that an existing unmanned aerial vehicle autonomous navigation model is high in training cost and low in efficiency, and flight decision and risk assessment are not matched. The scheme comprises the following steps: 1) initializing a flight strategy population, a risk assessment population and a reinforcement learning module; 2) optimizing the flight strategy population through a dynamic fusion fitness evaluation (DFFE) mechanism; 3) carrying out evolution without additional environment interaction on the risk assessment population by using shared flight log data generated by the flight strategy population; and 4) performing gradient optimization on the reinforcement learning module by adopting a hierarchical weighted sampling HWS strategy, and realizing coevolution of the population and the reinforcement learning module through a bidirectional information guide mechanism. According to the method, the safety, training efficiency and final performance of autonomous navigation of the unmanned aerial vehicle are remarkably improved by synchronously optimizing the flight strategy population and the risk assessment population and adopting shared experience evolution and dynamic fitness assessment.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence technology, and further relates to autonomous navigation control of aircraft. Specifically, it is an autonomous navigation method and system for unmanned aerial vehicles (UAVs) for urban air mobility (UAM), which can be used to realize autonomous path planning and navigation of UAVs in complex and dynamic low-altitude airspace environments. Background Technology

[0002] With the rapid development of the "low-altitude economy," Urban Air Mobility (UAM) and drone logistics are considered key technologies for solving urban congestion and improving transportation efficiency. Drones need to safely and efficiently perform tasks such as package delivery, medical emergency supplies transport, and personnel shuttle in urban low-altitude airspace filled with tall buildings, high-voltage power lines, and signal interference. The core technological bottleneck in this application scenario lies in the autonomous navigation capability of drones. Urban airspace is an extremely complex dynamic environment, filled with static obstacles (buildings, bridges), dynamic obstacles (other drones, birds), and uncertainties (such as sudden strong winds). More importantly, drone flight missions exhibit typical "reward sparsity": during flight periods lasting several minutes or even tens of minutes, the system cannot obtain clear "good" or "bad" immediate feedback; a one-time reward or penalty is only received upon mission completion (safe arrival) or failure (collision, loss of contact). This "reward sparsity" characteristic makes traditional reinforcement learning (RL) methods difficult to apply, as they rely on dense reward signals for learning and will fail to converge under sparse rewards due to low exploration efficiency. To address this technical bottleneck, researchers have proposed various improvement methods, including reward shaping and curriculum learning. However, most of these methods rely on manually designed prior information, have weak generalization ability, and are difficult to adapt to the dynamically changing application scenarios in UAM.

[0003] In recent years, as evolutionary algorithms (EAs) have demonstrated increasingly significant advantages in gradient-free optimization and global search, researchers have begun to explore their integration with reinforcement learning, gradually developing the evolutionary reinforcement learning (ERL) framework. This framework attempts to combine the global exploration capabilities of evolutionary algorithms with the local gradient optimization capabilities of reinforcement learning to improve the stability and robustness of policy search. However, existing ERL methods still face many challenges when applied to reward-sparse environments such as UAM. First, most ERL algorithms only perform evolutionary search on the flight policy network (policy network), neglecting the important role of the risk assessment network (value function network) in evaluating flight states. This leads to an inconsistency in the optimization pace between flight decisions and risk assessment, weakening the convergence performance and sample utilization efficiency of the policy network. Second, the fitness evaluation of individuals in the population (flight policies) is usually based only on the final result of the task (long-term cumulative reward), lacking sensitivity to local advantageous states or superior obstacle avoidance behaviors (local exploration behaviors) during flight, further exacerbating the imbalance between exploration and exploitation. Furthermore, because evolutionary algorithms require a large number of individuals to interact with the environment for evaluation (i.e., extensive simulated flight), they are prone to problems such as high training costs and slow convergence speeds, which are particularly evident in extremely complex scenarios like Unified Ambient Air (UAM). There is an urgent need to research a new solution that can meet the requirements of high safety, high efficiency, and low training costs in UAM scenarios. Summary of the Invention

[0004] The purpose of this invention is to overcome the shortcomings of existing technologies and propose an autonomous navigation method and system for unmanned aerial vehicles (UAVs) in urban air traffic. This method addresses core problems encountered by existing UAV autonomous navigation models when trained in complex, reward-sparse environments such as UAM (Unmanned Aerial Navigation) systems. These problems include high training costs, low sample efficiency, safety hazards caused by mismatch between flight decisions and risk assessment, and an imbalance between route exploration and utilization. The method aims to ensure the synchronous evolution and optimization of the UAV flight strategy module and risk assessment module, fundamentally improving the reliability and safety of navigation decisions. It achieves dynamic, multi-dimensional, and accurate evaluation of flight strategies, effectively balancing the exploration of new routes with the utilization of known routes. Simultaneously, it significantly improves the utilization efficiency of training data (i.e., simulated flight samples), greatly reducing the training cost and time of UAM navigation models and accelerating commercialization. This invention can simultaneously optimize the flight strategy network and risk assessment network, and introduces multi-scale value information into the individual evaluation mechanism to improve the accuracy of fitness assessment. This enhances the robustness of the strategy to sparse feedback and its environmental generalization ability while improving sample efficiency.

[0005] To achieve the above objectives, the technical solution adopted by the present invention includes the following steps:

[0006] (1) Initialize the reinforcement learning module and the two populations:

[0007] (1a) Initialize the reinforcement learning module, including the risk assessment network, the flight strategy network, and the experience buffer R;

[0008] (1b) Set up two populations, including a flight strategy population and a risk assessment population, with sizes m and n respectively, and initialize them; set the Gaussian distributions of the flight strategy population and the risk assessment population as follows: and Set the parameters related to reinforcement learning and iteration;

[0009] (2) Flight strategy population optimization:

[0010] (2a) From A population of flight strategies is generated through sampling, where each individual flight strategy interacts in the navigation environment to collect flight log data, and is evaluated using a dynamic fusion fitness evaluation mechanism; this mechanism uses dynamically adjusted weights. The final reward of a task that reflects long-term performance Short-term exploratory value in reflecting flight process risk prediction errors Weighted fusion is performed to obtain a comprehensive individual fitness score. ;

[0011] (2b) The flight strategy population is ranked based on the comprehensive fitness score. The top K elite flight strategy individuals are selected and their parameters are used to update the mean vector and covariance matrix of the cross-entropy method (CEM) to guide the next generation of population to explore a higher performance parameter space and achieve flight strategy population optimization.

[0012] (3) Risk assessment population optimization:

[0013] (3a) From A risk assessment population is generated by sampling and evaluated using shared flight log data generated during flight strategy population interactions. The fitness function of individuals in the risk assessment population is defined as the risk prediction error, and the accuracy of the risk assessment network in approximating the true state Q value is continuously optimized through evolutionary search.

[0014] (3b) The risk assessment population is sorted according to the risk prediction error, the top K elite risk assessors are selected, and their parameters are used to update the mean vector and covariance matrix of CEM, so as to drive the risk assessment population to evolve towards a more accurate value assessment direction and achieve the optimization of the risk assessment population.

[0015] (4) Perform gradient optimization on the reinforcement learning module TD3:

[0016] (4a) TD3 is trained using a hierarchical weighted sampling (HWS) strategy. This strategy mixes empirical data generated by the evolutionary population and TD3 itself from the flight log database according to a preset mixing ratio to ensure the consistency of the training data distribution and thus improve training stability.

[0017] (4b) Using the experience batch obtained by hierarchical sampling, backpropagation and gradient update are performed on the risk assessment network and flight strategy network by the TD3 algorithm to obtain the parameter-optimized reinforcement learning risk assessment network and flight strategy network;

[0018] (5) Bidirectional information interaction and collaborative optimization between the reinforcement learning module and the evolutionary population:

[0019] (5a) The reinforcement learning risk assessment network and flight strategy network with the parameters optimized in step (4b) are injected into their respective evolutionary populations as new candidate individuals, and reinforcement learning is used to provide high-quality genes for population evolution.

[0020] (5b) Periodically, the training of the TD3 flight strategy network is guided by the elite flight strategies in the population; the elite risk assessors transfer their value assessment capabilities to the TD3 risk assessment network through strategy distillation, so as to achieve information interaction and collaborative optimization.

[0021] (6) Repeat steps (2) to (5) until the training reaches the preset total number of interaction steps or the convergence condition is met, then the algorithm terminates and outputs the flight strategy network with the best current performance as the final result to complete the UAV autonomous navigation.

[0022] Compared with the prior art, the present invention has the following advantages:

[0023] First, enhanced security and reliability:

[0024] Because this invention uses dual-population collaborative optimization DPCO as its core framework, it ensures that the "flight strategy" and "risk assessment" capabilities of the UAV evolve synchronously and are always matched. This makes the UAV more decisive and reliable in making decisions when facing complex and dynamic urban environments, and greatly reduces the risk of collisions caused by decision-making errors.

[0025] Second, higher training efficiency and lower R&D costs:

[0026] This invention significantly improves the utilization rate of data samples (simulated flight data) by using risk assessment population evolution based on shared experience. In simulation testing, when facing sparse reward tasks such as typical "urban canyons" or "dynamic obstacle courses", this invention requires far fewer simulated flight interaction steps to achieve the same performance level than existing technologies, significantly shortening the model training cycle and reducing computational resource consumption.

[0027] Third, superior final performance and path planning capabilities:

[0028] Because this invention features a unique Dynamic Fusion Fitness Evaluation (DFFE) mechanism, it can more accurately assess the comprehensive value of each flight strategy, effectively helping UAVs escape local optimal paths. Therefore, in terms of final indicators such as mission success rate and path efficiency, the navigation model trained by this invention outperforms existing benchmark models, enabling UAVs to find safer and more economical flight routes. Attached Figure Description

[0029] Figure 1 This is a schematic diagram illustrating the implementation process of the method of the present invention.

[0030] Figure 2 A schematic diagram of the existing evolutionary reinforcement learning exploration framework EvoRainbow-Exp;

[0031] Figure 3 This is a comparison diagram of ablation experiment results provided in the embodiments of the present invention;

[0032] Figure 4 This is a schematic diagram of a maze-like task environment provided in an embodiment of the present invention;

[0033] Figure 5 This is a schematic diagram of a multi-room exploration task environment provided in an embodiment of the present invention;

[0034] Figure 6 This is a schematic diagram of a multi-task composition environment provided in an embodiment of the present invention. Detailed Implementation

[0035] The present invention will now be further described with reference to the accompanying drawings.

[0036] Example 1: Refer to Appendix Figure 1 The present invention proposes an autonomous navigation method for unmanned aerial vehicles (UAVs) for urban air traffic, which specifically includes the following steps:

[0037] Step 1) Initialize the reinforcement learning module and the two populations:

[0038] (1a) Initialize the reinforcement learning module, which includes a risk assessment network, a flight strategy network, and an experience buffer R; wherein the flight strategy network and the risk assessment network in the reinforcement learning module both adopt a network structure containing 3 convolutional layers and 3 fully connected layers; the experience buffer R, i.e. the flight log database, has a capacity of 1e5.

[0039] (1b) Set up two populations, including a flight strategy population and a risk assessment population, with sizes m and n respectively, and initialize them; set the Gaussian distributions of the flight strategy population and the risk assessment population as follows: and The parameters related to reinforcement learning and iteration are set. In this embodiment, the population size m of the flight strategy is 5 to 10, and the population size n of the risk assessment is 4 to 6. The parameters related to reinforcement learning and iteration include that each strategy individual interacts independently with the environment at least 5 times, the discount factor γ of reinforcement learning is 0.97, the learning rate is 3e-4, and the batch size is 128. Any value from {0.1, 0.01, 0.001} is selected as the noise amplitude of the cross-entropy method CEM for population optimization.

[0040] Step 2) Flight strategy population optimization:

[0041] (2a) From A population of flight strategies is generated through sampling, where each individual flight strategy interacts in the navigation environment to collect flight log data, and is evaluated using a dynamic fusion fitness evaluation mechanism; this mechanism uses dynamically adjusted weights. The final reward of a task that reflects long-term performance Short-term exploratory value in reflecting flight process risk prediction errors Weighted fusion is performed to obtain a comprehensive individual fitness score. The calculation expression is as follows:

[0042] .

[0043] The above dynamic weights in this embodiment It is adaptively calculated based on the mean and standard deviation of the current population's final task reward and short-term exploration value, and specifically includes the following steps:

[0044] (2a1) Calculate the confidence level of the final reward of the task respectively. Trust in the value of short-term exploration :

[0045] ,

[0046] (2a2) will and Normalization is performed to obtain dynamic weights. :

[0047] ;

[0048] in, and These are the average return and standard deviation of the current population, respectively. and These are the mean negative temporal difference error and its standard deviation of the current population, respectively. A small constant, used to prevent the denominator from being zero or its value from being unstable, is set to 10. -6 .

[0049] (2b) The flight strategy population is ranked based on the comprehensive fitness score. The top K elite flight strategy individuals are selected and their parameters are used to update the mean vector and covariance matrix of the cross-entropy method (CEM) to guide the next generation of population to explore a higher performance parameter space and achieve flight strategy population optimization.

[0050] Step 3) Risk assessment and population optimization:

[0051] (3a) From A risk assessment population is generated through sampling, and the population is evaluated using shared flight log data generated during flight strategy population interactions. The fitness function of individuals in the risk assessment population is defined as the risk prediction error, and the approximation accuracy of the risk assessment network to the true state Q value is continuously optimized through evolutionary search. The fitness function of individuals in the risk assessment population mentioned in this step of this embodiment is expressed as follows:

[0052] ,

[0053] in, For risk assessment, the individual's fitness score, This indicates that a negative average is calculated for a batch of data sampled from shared flight log data. For instant rewards, For the current risk assessment individual's value assessment of the state-action pair, This is a risk assessment network's value assessment of the next state.

[0054] (3b) The risk assessment population is sorted according to the risk prediction error, the top K elite risk assessors are selected, and their parameters are used to update the mean vector and covariance matrix of CEM, so as to drive the risk assessment population to evolve towards a more accurate value assessment direction and achieve the optimization of the risk assessment population.

[0055] In this embodiment, the above-mentioned screening and updating are both based on the fitness scores of individuals to rank their respective populations and select the top K elite individuals. Then, using the parameters of these elite individuals, the mean vector and covariance matrix of the CEM are updated according to the following formula:

[0056] ,

[0057] ,

[0058] in, and These are the updated mean vector and covariance matrix, respectively. Let be the parameter of the i-th elite individual. These are the weighting coefficients. The mean vector before the update. This is a regularization term used to prevent the covariance matrix from degenerating. The aforementioned weight coefficients... Set according to any of the following methods:

[0059] Method 1: Assign the same weight to all elite individuals, and calculate using the following formula: ;

[0060] Method 2: Assign higher weight to top-ranked elite individuals; the calculation formula is as follows:

[0061] ,

[0062] Where i represents the rank of an elite individual after sorting by fitness from high to low.

[0063] Step 4) Perform gradient optimization on the reinforcement learning module TD3:

[0064] (4a) TD3 is trained using a hierarchical weighted sampling (HWS) strategy. This strategy mixes empirical data generated by the evolutionary population and TD3 itself from the flight log database according to a preset mixing ratio to ensure the consistency of the training data distribution and thus improve training stability.

[0065] (4b) Using the empirical batches obtained from stratified sampling, backpropagation and gradient updates are performed on the risk assessment network and flight strategy network using the TD3 algorithm to obtain the parameter-optimized reinforcement learning risk assessment network and flight strategy network. The backpropagation and gradient updates performed on the risk assessment network and flight strategy network using the TD3 algorithm include: using truncated double Q-learning, taking the smaller value of the two predictions from the risk assessment network as the target Q-value; using delayed policy updates to make the update frequency of the flight strategy network lower than that of the risk assessment network; and using target policy smoothing, adding noise to the target action when calculating the target Q-value to reduce overestimation caused by model errors.

[0066] Step 5) Two-way information interaction and collaborative optimization between the reinforcement learning module and the evolutionary population:

[0067] (5a) The reinforcement learning risk assessment network and flight strategy network with the parameters optimized in step 4b) are injected into their respective evolutionary populations as new candidate individuals, and reinforcement learning is used to provide high-quality genes for population evolution.

[0068] (5b) Periodically, the training of the TD3 flight strategy network is guided by the elite flight strategies in the population; elite risk assessors transfer their value assessment capabilities to the TD3 risk assessment network through strategy distillation, achieving information interaction and collaborative optimization; the strategy distillation method described in this embodiment is specifically implemented by minimizing a distillation loss term that includes the mean squared error (MSE), and its loss function is... The calculation formula is:

[0069] ,

[0070] in, Indicates the state and actions The mathematical expectation; The outputs for the two risk assessment networks included in the elite risk assessor team. value, These are the outputs of the two risk assessment networks in the TD3 module, respectively. value.

[0071] Step 6) Repeat steps 2) to 5) until the training reaches the preset total number of interaction steps or the convergence condition is met, at which point the algorithm terminates and outputs the flight strategy network with the best current performance as the final result, thus completing the UAV autonomous navigation.

[0072] Example 2: The overall implementation steps of the UAV autonomous navigation method proposed in this example are the same as in Example 1. Now, given the parameter settings, the implementation process of the present invention will be further described in detail with specific examples.

[0073] Step 1. Initialization Phase

[0074] (1.1) Initialize the reinforcement learning module, which includes two risk assessment networks and their corresponding target networks, a flight strategy network and its target network, and a "flight log database" with a capacity of 1e5; all networks adopt a structure containing 3 convolutional layers (hidden units 16, 32, 64) and 3 fully connected layers (hidden units 256).

[0075] (1.2) Set the flight strategy population size to m (5 to 10 members) and the risk assessment population size to n (4 to 6 members), and initialize the risk assessment population. Flight strategy population The noise amplitude of the cross-entropy method (CEM) is set to a value in {0.1, 0.01, 0.001}. Meanwhile, the discount factor γ of reinforcement learning is set to 0.97, the learning rate is 3e-4, and the batch size is 128.

[0076] Step 2. Flight Strategy Population Optimization

[0077] (2.1) Individual sampling and dynamic fusion evaluation: from the current Gaussian distribution m flight strategy individuals are generated through sampling, forming a strategy population. Each flight strategy individual interacts independently 5 times in the virtual flight environment to collect flight logs, and is evaluated using a Dynamic Fusion Fitness Evaluation (DFFE) mechanism. The final task reward, reflecting long-term performance, will be used. Error in predicting flight process risks Weighted fusion is performed to obtain a comprehensive individual fitness score. :

[0078] ,

[0079] Calculate the overall fitness score, where:

[0080] ,

[0081] Dynamic weights It is adaptively calculated using the following formula based on the mean and standard deviation of the current population return and TD error:

[0082]

[0083]

[0084] in, It is a small constant, and to avoid cases where it is non-zero or numerically unstable, it is set to 10 in this embodiment. -6 .

[0085] (2.2) Elite Selection and Distribution Update: The strategy population is ranked based on the comprehensive fitness score, and the top K "ace flight strategies" are selected. Using only the parameters of these elite individuals, the mean vector and covariance matrix of the CEM are updated according to the following formula, thereby guiding the next generation of the population to explore a higher-performance parameter space:

[0086]

[0087]

[0088] in, For the first The weighting coefficients of each individual:

[0089] ;

[0090] Step 3. Evolution and Assessment of Risk Assessment Populations

[0091] (3.1) Sampling and Experience-Based Evaluation: n individuals in the risk assessment network are generated by sampling from the current corresponding CEM distribution. To mitigate the potential distributional differences between the evolutionary population and the TD3 module, a hierarchical weighted sampling (HWS) strategy is used to extract data from the "flight log database" R when evaluating the fitness of this population. This strategy mixes the experience data generated by the evolutionary population and TD3 itself in a specific proportion to ensure the consistency of the evaluation data distribution and avoid value estimation bias. Its fitness function is defined as the risk prediction error. The aim is to continuously optimize the accuracy of the Q-value function in approximating the true state value through evolutionary search.

[0092] ,

[0093] (3.2) Elite selection and distribution update: The risk assessment population is sorted according to the negative TD error fitness, the top K "ace risk assessors" are selected, and their parameters are used to update the distribution of CEM to drive the risk assessment population to evolve towards more accurate value assessment.

[0094] Step 4. Gradient optimization of the module

[0095] (4.1) Experience sampling: A batch of experience data is routinely sampled from the flight log database R with a capacity of 1e5, and the batch size is 128.

[0096] (4.2) TD3 Gradient Update: Utilizing the empirical batches obtained from sampling, standard backpropagation and gradient updates are performed on the risk assessment network and flight policy network using the double-delay deep deterministic policy gradient (TD3) algorithm. This process leverages three core techniques of TD3 to ensure the stability of policy learning and reduce overestimation of Q-values:

[0097] Clipped Double Q-learning (TD3) involves the algorithm simultaneously learning two Q-functions by minimizing the mean squared error to ensure the stability of policy learning. and Furthermore, gradient learning is performed using the same optimization objective; during updates, the smaller of the two Q-function predictions is used as the target for updating.

[0098]

[0099] in, This represents the target Q-value of the current training sample. Indicates whether the termination state has been reached. Measuring the level of concern that intelligent agents have for the future. This represents the action the policy network will take in the current state. Using this approach can effectively reduce the overestimation of the Q-value.

[0100] Delayed policy updates occur when the policy network is fixed. The Q-value network can more accurately approximate the final result, so the policy network is updated less frequently than the Q-value network. For example, the policy network is updated only once after the Q-value network is updated twice. This allows for a more accurate risk assessment before the flight policy network is evaluated and updated.

[0101] Target Policy Smoothing (TD3) adds noise to the target action when calculating the target Q-value. This is called "target policy smoothing". The purpose of this is to make the target value less dependent on a specific or inaccurate action, thereby reducing overly optimistic estimates caused by model errors.

[0102] Step 5. Reinforcement learning network injection into the population phase

[0103] (5.1) Information injection from reinforcement learning to the population: The TD3 risk assessment network and flight strategy network optimized by each gradient are injected as new candidate individuals into their respective evolutionary populations. The aim is to utilize the rapid optimization capability of reinforcement learning to provide high-quality genes for population evolution and ensure the efficiency of its evolutionary direction.

[0104] (5.2) Elite Guidance from Population to Reinforcement Learning: Periodically, elite individuals within the population guide the training of the reinforcement learning module. Elite flight strategies influence the parameters of the TD3 flight strategy network through "genetic soft updates," with an update parameter of 0.3. Elite risk assessors transfer their precise value assessment capabilities to the TD3 risk assessment network through "strategy distillation." The loss function for this process is:

[0105]

[0106] The strategy distillation loss weight is 0.3, and the distillation loss term is:

[0107]

[0108] Step 6. Elite-led reinforcement learning phase

[0109] (6.1) Iterative execution: Execute the collaborative optimization process described in steps 2 to 5 in a complete and cyclical manner.

[0110] (6.2) Termination condition: When the training reaches the preset total number of interaction steps, preferably 2e5 or 1.75e5 in this embodiment, or when other convergence conditions are met, the algorithm terminates and outputs the flight strategy network with the best current performance as the final result.

[0111] The technical effects of the present invention will be further illustrated by the following simulation experiments:

[0112] 1. Simulation conditions and content:

[0113] The simulation experiments in this embodiment were conducted in the MiniGrid environment, a lightweight set of environments specifically designed for studying reward sparsity problems and providing an abstract representation of urban air traffic (UAM) navigation tasks. To fully verify the effectiveness of the method of this invention, the experiment selected seven abstract task scenarios of three different difficulty levels to simulate different challenges in UAM navigation:

[0114] (1) Maze-type tasks, as shown in the appendix Figure 4 As shown, the simulation includes "maze" and "lava" scenarios: simulating the navigation and risk avoidance capabilities of drones in an "urban canyon" filled with static obstacles (including tall buildings and bridges).

[0115] (2) Multi-room exploration tasks, as shown in the attached document. Figure 5 As shown, there are "6 rooms", "10 rooms", and "15 rooms": these simulate the exploration capabilities of drones in multi-regional and wide-area urban spaces, requiring the model to have stronger exploration capabilities to cope with more complex airspace structures.

[0116] (3) Combined multi-task classes, as shown in the appendix Figure 6 As shown, "Opening Door 1" and "Opening Door 2" simulate a scenario where a drone needs to fly over multiple waypoints in sequence and perform complex logical tasks. This is an extremely reward-sparse challenge, which is highly consistent with the mission characteristics of UAM, which has a long flight range and a single final reward.

[0117] The hardware platform for the simulation experiment was a server configured with an Intel(R) Core(TM) i9-10900X CPU and an NVIDIA GeForce RTX 3090 24GB GPU. The software environment was a Linux Ubuntu 20.04 LTS operating system, Python 3.9 as the programming language, and PyTorch 1.12.0 as the deep learning framework.

[0118] In terms of network structure, both the flight strategy and risk assessment networks adopt a structure containing 3 convolutional layers (hidden units 16, 32, 64) and 3 fully connected layers (hidden units 256). The key parameters for reinforcement learning are set as follows: learning rate 3e-4, discount factor γ of 0.97, and batch size of 128. The core parameters of the algorithm in this invention are set as follows: policy population size of 5 to 10, value function population size of 4 to 6, experience pool size of 1e5, and population evaluation times of 5.

[0119] To verify the advancement of this invention, several mainstream algorithms were selected for comparison, including: classic reinforcement learning algorithms: PPO, TD3; representative evolutionary reinforcement learning algorithms: ERL, VEB-RL, EvoRainbow, and EvoRainbow-Exp as the main benchmark (see Figure 2). Figure 2 The basic framework of EvoRainbow-Exp is shown in the diagram. This invention introduces innovative mechanisms such as dual-population cooperation and dynamic fitness evaluation on this basis.

[0120] 2. Simulation Result Analysis

[0121] Experimental results show that the proposed method (DPCOERL) demonstrates significant advantages in both sample efficiency and final convergence performance across all seven simulated UAM sparse reward tasks. In complex airspace exploration tasks such as "10 rooms" and "15 rooms," traditional PPO and TD3 algorithms struggle to learn effective policies, while the proposed method achieves optimal final performance. In the most challenging "Open Door 2" (simulating a complex multiwaypoint task), the sample efficiency advantage of the proposed method is particularly pronounced, with reward accumulation speed far exceeding other algorithms in the early training phase (before 125,000 interaction steps), and the final average reward is approximately 4% higher than the state-of-the-art benchmark model EvoRainbow-Exp.

[0122] The quantitative results are shown in Table 1. The method of this invention achieved the highest average reward score in all high-difficulty tasks except for "Room 6".

[0123] Table 1: Average Final Rewards for Various Tasks

[0124]

[0125] See attached document Figure 3 To verify the effectiveness of the innovative components of this invention, an ablation experiment was conducted on "Open Door 2" (simulating a complex multi-waypoint mission).

[0126] After removing the DFFE and HWS mechanisms (DPCOERL / DFFE+HWS, blue curve in the figure), the model's performance significantly outperforms the baseline model EvoRainbow-Exp (cyan curve in the figure). This demonstrates that the basic framework of introducing value function population and policy population for collaborative optimization is effective and can improve sample efficiency. Effectiveness of Dynamic Fusion Fitness (DFFE):

[0127] After adding the DFFE mechanism (DPCOERL / HWS, yellow curve in the figure) to the dual-population approach, the reward curve becomes more stable, and the final result is better. This proves that by comprehensively evaluating long-term returns and short-term value, the DFFE mechanism can effectively help the strategy escape local optima. Effectiveness of Hierarchical Weighted Sampling (HWS):

[0128] The complete method of this invention (DPCOERL, purple curve in the figure), after adopting HWS, shows further improvement in sample efficiency in the early stage of training compared to the version without HWS, a smoother reward curve, and increased policy stability. This proves that the HWS mechanism can effectively ensure the stability of the guiding signals between modules, thereby accelerating the overall optimization speed.

[0129] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the laws, regulations and standards of the relevant countries and regions, and corresponding operation portals are provided for users to choose to authorize or refuse.

[0130] The simulation analysis above demonstrates the correctness and effectiveness of the method proposed in this invention. It is evident that this invention, through a series of innovative designs such as a dual-population collaborative optimization framework, dynamic fusion fitness evaluation, and hierarchical weighted sampling, effectively solves the problems of low sample efficiency and imbalance between exploration and utilization in existing technologies applied to reward-sparse navigation environments such as UAM, significantly improving the algorithm's convergence speed, final performance, and training stability. Parts of this invention not described in detail are common knowledge to those skilled in the art.

[0131] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Obviously, those skilled in the art, after understanding the content and principle of the present invention, may make various modifications and changes in form and detail without departing from the principle and structure of the present invention. However, these modifications and changes based on the concept of the present invention are still within the scope of protection of the claims of the present invention.

Claims

1. An autonomous navigation method for unmanned aerial vehicles (UAVs) used in urban air traffic, characterized in that, Includes the following steps: (1) Initialize the reinforcement learning module and the two populations: (1a) Initialize the reinforcement learning module, including the risk assessment network, the flight strategy network, and the experience buffer R; (1b) Set up two populations, including a flight strategy population and a risk assessment population, with sizes m and n respectively, and initialize them; set the Gaussian distributions of the flight strategy population and the risk assessment population as follows: and Set the parameters related to reinforcement learning and iteration; (2) Flight strategy population optimization: (2a) From A population of flight strategies is generated through sampling, where each individual flight strategy interacts in the navigation environment to collect flight log data, and is evaluated using a dynamic fusion fitness evaluation mechanism; this mechanism uses dynamically adjusted weights. The final reward of a task that reflects long-term performance Short-term exploratory value in reflecting flight process risk prediction errors Weighted fusion is performed to obtain a comprehensive individual fitness score. ; (2b) The flight strategy population is ranked based on the comprehensive fitness score. The top K elite flight strategy individuals are selected and their parameters are used to update the mean vector and covariance matrix of the cross-entropy method (CEM) to guide the next generation of population to explore a higher performance parameter space and achieve flight strategy population optimization. (3) Risk assessment population optimization: (3a) From A risk assessment population is generated by sampling and the shared flight log data generated during the interaction of the flight strategy population is used to evaluate the population. The fitness function of the individuals in the risk assessment population is defined as the risk prediction error, and the approximation accuracy of the risk assessment network to the real state Q value is continuously optimized through evolutionary search. (3b) The risk assessment population is sorted according to the risk prediction error, the top K elite risk assessors are selected, and their parameters are used to update the mean vector and covariance matrix of CEM, so as to drive the risk assessment population to evolve towards a more accurate value assessment direction and achieve the optimization of the risk assessment population. (4) Perform gradient optimization on the reinforcement learning module TD3: (4a) TD3 is trained using a hierarchical weighted sampling (HWS) strategy. This strategy mixes empirical data generated by the evolutionary population and TD3 itself from the flight log database according to a preset mixing ratio to ensure the consistency of the training data distribution and thus improve training stability. (4b) Using the experience batch obtained by hierarchical sampling, backpropagation and gradient update are performed on the risk assessment network and flight strategy network by the TD3 algorithm to obtain the parameter-optimized reinforcement learning risk assessment network and flight strategy network; (5) Bidirectional information interaction and collaborative optimization between the reinforcement learning module and the evolutionary population: (5a) The reinforcement learning risk assessment network and flight strategy network with the parameters optimized in step (4b) are injected into their respective evolutionary populations as new candidate individuals, and reinforcement learning is used to provide high-quality genes for population evolution. (5b) Periodically, the training of the TD3 flight strategy network is guided by the elite flight strategies in the population; the elite risk assessors transfer their value assessment capabilities to the TD3 risk assessment network through strategy distillation, so as to achieve information interaction and collaborative optimization. (6) Repeat steps (2) to (5) until the training reaches the preset total number of interaction steps or the convergence condition is met, the algorithm terminates; and outputs the flight strategy network with the best current performance as the final result to complete the UAV autonomous navigation.

2. The autonomous navigation method for unmanned aerial vehicles according to claim 1, characterized in that: In step (1a), both the flight strategy network and the risk assessment network in the reinforcement learning module adopt a network structure containing 3 convolutional layers and 3 fully connected layers; the experience buffer R, i.e., the flight log database, has a capacity of 1e5.

3. The autonomous navigation method for unmanned aerial vehicles according to claim 1, characterized in that: In step (1b), the population size m of the flight strategy is 5 to 10, and the population size n of the risk assessment is 4 to 6. The reinforcement learning and iteration-related parameters include at least 5 independent interactions between each strategy individual and the environment, a discount factor γ of 0.97 for reinforcement learning, a learning rate of 3e-4, and a batch size of 128. Any value from {0.1, 0.01, 0.001} is selected as the noise amplitude of the cross-entropy method CEM for population optimization.

4. The autonomous navigation method for unmanned aerial vehicles according to claim 1, characterized in that: The comprehensive individual fitness score described in step (2a) The calculation expression is as follows: 。 5. The autonomous navigation method for unmanned aerial vehicles according to claim 1 or 4, characterized in that: The dynamic weight It is adaptively calculated based on the mean and standard deviation of the current population's final task reward and short-term exploration value, and specifically includes the following steps: (2a1) Calculate the confidence level of the final reward of the task respectively. Trust in the value of short-term exploration : ， (2a2) will and Normalization is performed to obtain dynamic weights. : ； in, and These are the average return and standard deviation of the current population, respectively. and These are the mean negative temporal difference error and its standard deviation of the current population, respectively. A tiny constant, used to prevent the denominator from being zero or its value from being unstable, is set to 10. -6 .

6. The autonomous navigation method for unmanned aerial vehicles according to claim 1, characterized in that: The screening and updating described in steps (2b) and (3b) are based on ranking the populations according to the fitness scores of individuals and selecting the top K elite individuals. Then, using the parameters of these elite individuals, the mean vector and covariance matrix of the CEM are updated according to the following formula: ， ， in, and These are the updated mean vector and covariance matrix, respectively. Let be the parameter of the i-th elite individual. These are the weighting coefficients. The mean vector before the update. It is a regularization term used to prevent the covariance matrix from degenerating.

7. The autonomous navigation method for unmanned aerial vehicles according to claim 6, characterized in that: The weighting coefficient Set according to any of the following methods: Method 1: Assign the same weight to all elite individuals, and calculate using the following formula: ; Method 2: Assign higher weight to top-ranked elite individuals; the calculation formula is as follows: ， Where i represents the rank of an elite individual after sorting by fitness from high to low.

8. The autonomous navigation method for unmanned aerial vehicles according to claim 1, characterized in that: The fitness function of individuals in the risk assessment population described in step (3a) is expressed as follows: ， in, For risk assessment, the individual's fitness score, This indicates that a negative average is calculated for a batch of data sampled from shared flight log data. For instant rewards, For the current risk assessment individual's value assessment of the state-action pair, This is a risk assessment network's value assessment of the next state.

9. The autonomous navigation method for unmanned aerial vehicles according to claim 1, characterized in that: The backpropagation and gradient update of the risk assessment network and flight strategy network by using the TD3 algorithm in step (4b) includes: using truncated double Q learning, taking the smaller value of the two predictions of the risk assessment network as the target Q value; using delayed policy update to make the update frequency of the flight strategy network lower than that of the risk assessment network; and using target policy smoothing, adding noise to the target action when calculating the target Q value to reduce overestimation caused by model error.

10. The autonomous navigation method for unmanned aerial vehicles according to claim 1, characterized in that: The strategy distillation method described in step (5b) is specifically achieved by minimizing a distillation loss term that includes the mean squared error (MSE), and its loss function is... The calculation formula is: ， in, Indicates the state and actions The mathematical expectation; The outputs for the two risk assessment networks included in the elite risk assessor team. value, These are the outputs of the two risk assessment networks in the TD3 module, respectively. value.

Images