Unmanned aerial vehicle synchronous navigation and radio mapping method based on reinforcement learning and regional adaptive ensemble learning

Abstract

The invention discloses an unmanned aerial vehicle synchronous navigation and radio mapping method based on reinforcement learning and region adaptive integrated learning, and the method comprises the following steps: S1, constructing a radio environment, and carrying out region division according to geographical location information; the method comprises the following steps: S1, detecting a flight area of an unmanned aerial vehicle by adopting an area detector, S4, exploring an environment by the unmanned aerial vehicle through interactive learning, and simultaneously collecting and learning radio frequency data, S5, carrying out strategy training by adopting an experience pool jointly constructed based on actual experience and simulation experience, and S6, storing experience, calculating a return and updating a network. According to the unmanned aerial vehicle synchronous navigation and radio mapping method based on reinforcement learning and regional adaptive ensemble learning, through synchronous navigation and mapping, the unmanned aerial vehicle can be accurately navigated in an unknown or interference environment, meanwhile, a high-precision radio coverage map is established, and the method is particularly suitable for unmanned aerial vehicle communication and navigation tasks in a complex urban environment.

Description

Technical Field

[0001] The present invention belongs to the technical field of unmanned aerial vehicles (UAVs), and in particular to a method for synchronous navigation and radio mapping of UAVs based on reinforcement learning and regional adaptive integrated learning. Background Art

[0002] With the rapid development of drone technology, cellular-connected UAVs (UAVs) have become a promising technology. By reusing existing cellular network infrastructure, they have the potential to enable truly remote drone operations. This connectivity approach offers advantages such as easier legal monitoring and management, high-capacity payload transmission, and enhanced cellular positioning. Against this backdrop, UAV navigation and radio frequency mapping technologies have become a hot topic of research.

[0003] However, achieving cellular connectivity for drones still faces key challenges. Traditional cellular networks are primarily designed for ground-based users, with base station antennas typically facing the ground, making aerial coverage unreliable. Even for 5G and future 6G networks, it remains unclear whether universal coverage in the sky is economically feasible. Furthermore, drones face more severe interference issues in the air than ground-based users, primarily due to the potential for strong line-of-sight (LoS) channels with unassociated base stations.

[0004] Therefore, drones need to perceive and adapt to complex RF environments and build accurate RF maps. However, in real-world applications, there are numerous interference factors: regional signal failures, measurement equipment misjudgments leading to incorrect label data, unstable signals in marginal areas causing systematic offsets, and random noise interference in the environment. These factors severely impact the accuracy of RF map learning and the reliability of navigation decisions. Traditional methods rely primarily on simple noise filtering or outlier detection, which cannot effectively handle complex multi-modal interference.

[0005] Traditional methods for optimizing drone trajectories have significant limitations. Formulating the corresponding optimization problem requires an accurate and analyzable end-to-end communication model, including base station antenna models, channel models, interference models, and even local environment models. However, most previous work only assumes simplified models, such as isotropic antenna radiation or free-space path loss channel models for air-to-ground links. Although some studies have considered more complex channel models, such as probabilistic line-of-sight channel models and angle-dependent channel parameters, these statistical models can only predict average performance and cannot provide performance guarantees for actual deployment environments. Offline optimization trajectory design methods rely on accurate global channel model parameters, which are difficult to obtain in practice.

[0006] Compared with traditional methods, trajectory optimization methods based on deep reinforcement learning show significant technical advantages. First, the independence of its mathematical model means that the algorithm does not need to build a precise mathematical model or obtain complete environmental prior knowledge. It can autonomously discover the optimal strategy through interactive learning with the environment and effectively adapt to various complex actual deployment scenarios. Secondly, the powerful representation learning ability of deep neural networks enables the algorithm to process high-dimensional complex state spaces, comprehensively consider influencing factors such as multi-dimensional position information and multi-channel communication quality indicators, and achieve end-to-end optimization and automatic feature extraction. However, although the current trajectory optimization method based on deep reinforcement learning shows good convergence and performance stability in a standardized simulation environment, it still shows obvious lack of robustness when facing multi-source interference in a real flight environment. Especially when encountering structured interference events (such as local communication failure caused by regional signal blockage or equipment failure), existing methods lack effective interference perception and compensation mechanisms, making it difficult to achieve rapid performance recovery and strategy reconstruction. Summary of the Invention

[0007] Purpose of the invention: To provide a method for synchronized navigation and radio mapping of UAVs based on reinforcement learning and regional adaptive ensemble learning. Aiming to solve the problem of UAV navigation in interference environments, this method enables UAVs to avoid areas with weak signal coverage during flight while establishing a high-precision radio coverage map, thereby achieving an optimal balance between signal coverage and flight efficiency.

[0008] Technical solution:

[0009] A method for synchronized navigation and radio mapping of unmanned aerial vehicles (UAVs) based on reinforcement learning and regional adaptive ensemble learning, comprising the following steps:

[0010] S1: Simulate the UAV navigation environment to build a radio map and a radio map database, divide the simulated navigation environment of the UAV into regions in the radio map according to the geographic location information, and set the UAV position and radio environment parameters;

[0011] S2: Build a deep reinforcement learning agent that uses the Dueling DQN architecture to separate value estimation from advantage function.

[0012] S3: Uses a regionally adaptive integrated radio frequency mapping model to learn the radio coverage of the UAV navigation environment in real time, detects and determines the partitioned area where the UAV is located in the UAV navigation environment, and identifies and marks abnormal data. Based on the characteristics of the partitioned area, a multi-submodel collaborative modeling mechanism is introduced to build an integrated system composed of multiple functionally complementary deep neural networks;

[0013] S4: The drone explores the environment through interactive learning combined with an epsilon-greedy strategy while simultaneously collecting and learning radio frequency data. The learning agent selects the next action to reach the new location based on the drone's current position and historical trajectory, and calculates rewards based on reaching the target, crossing the boundary, connection interruption, and path smoothness.

[0014] S5: Policy training is performed using an experience pool built from both real-world and simulated experience. The real-world experience comes from the actual interaction between the drone and the environment, while the simulated experience comes from the virtual trajectories generated by the RF mapping model.

[0015] S6: Use n-step learning mechanism to store experience, calculate cumulative rewards with discount factors and use Double Q-Learning method to update the network to obtain the optimal navigation strategy.

[0016] In a further embodiment, the division of the drone navigation environment into a normal area E1, a central interference area E2 and an edge interference area E3 in S1, and the setting of the drone position includes setting the initial drone position q I and setting the target position coordinates q0, wherein the radio environment parameters include base station configuration parameters, channel model parameters and building environment parameters;

[0017] Said S1 further comprises:

[0018] S1-1: Set the interruption probability of the central interference area E2 to 0.9 to simulate the possible interference caused by obstacles in the actual flight environment;

[0019] S1-2: Set a 10% probability of label flipping to simulate the interference of measurement equipment failure that may exist in actual flight;

[0020] S1-3: Set the interruption probability P of the edge interference area E3 out (q)=clip(P base (q)+0.08),0.1) to simulate the possible edge area deviation interference in the actual flight environment;

[0021] Among them, P base (q) is the basic outage probability calculated based on the real wireless propagation model, P out (q) is the final output interruption probability;

[0022] S1-4: Set other areas to add Gaussian noise P out (q)=clip(P base (q)+N(0,0.03),0.1) is used to simulate the random noise that may exist in the actual flight environment.

[0023] In a further embodiment, the learning agent in S2 is composed of a main network and a target network, wherein the main network is responsible for online learning and the target network is used to stabilize target prediction during training. The drone policy network is trained by the DRL method to enable it to estimate the action value function based on the current position state of the drone and the interruption probability predicted by the radio map;

[0024] Said S2 further comprises:

[0025] S2-1: Initialize Dueling DQN main network parameters θ and target network parameters θ - , get the current status of the drone Use multi-layer neural network to extract state features φ(s t ,θ);

[0026] S2-2: Separate the state features into two branches, the two branches are state value estimation V(s) and action advantage function A(s,v), the main network is responsible for the online action selection decision of the UAV, and the expression of the action value function Q value maximization criterion is Determine the optimal action choice;

[0027] in, is the UAV position state after N1 steps, are the possible actions in this state;

[0028] The target network is responsible for stable value evaluation. The expression of the expected cumulative reward value after executing the selected action is as follows:

[0029]

[0030] Where γ is the discount factor, γ∈[0,1] represents the importance weight of future rewards, and controls the balance between long-term and short-term benefits;

[0031] S2-3: Merge the two branches through a special aggregation layer and calculate the final action-value function using the Q-value reconstruction formula. The expression of the Q-value reconstruction formula is as follows:

[0032]

[0033] where the mean advantage term is subtracted.

[0034] In a further embodiment,: the radio frequency mapping model in S3 includes a region detector, an anomaly detector, and a plurality of sub-models, and different learning strategies are applied to different regions;

[0035] Said S3 further comprises:

[0036] S3-1: Build a region detector to determine the region where the drone is located based on its flight position;

[0037] S3-2: Construct an anomaly detector, assuming the location set is L = {l1,l2...,l n}, corresponding label P = {p1,p2...,p n}, calculate the distance d ij =||l i -l j ||, visit neighboring point N i ={j|rank(d j )≤K,j≠i}, calculate the anomaly score Perform outlier judgment i =(s i >T);

[0038] Where T is the abnormal threshold parameter. When the difference exceeds the preset abnormal threshold parameter T, the sample is judged as an abnormal sample and discarded.

[0039] S3-3: Generate KNN-based soft labels for the central interference area and repair the samples in the central interference area;

[0040] S3-4: Each sub-model adopts a region-weighted loss function during the training phase to achieve region-aware feature extraction and representation learning;

[0041] S3-5: Aggregate all sub-model outputs during prediction, perform post-processing adjustments based on the characteristics of the divided regions, and provide a final connection interruption probability prediction. The expression for the connection interruption probability prediction is as follows:

[0042]

[0043] Among them, G=3 is the number of models, f i (x|D i ) is the i-th model based on the data set D i The prediction of w i (x) is the region-based weight function.

[0044] In a further embodiment, repairing the central area sample in S3-3 further includes:

[0045] Using w i =1 / (d i +ε) Calculate the weighted average of the target point and normal area samples;

[0046] Among them, w i is the weight of the ith neighboring sample, d iis the distance to the ith neighboring sample, ε is a small constant to prevent division by zero, and a total of 5 neighboring samples are selected;

[0047] Normalized weights By soft=∑w inorm ×label i Calculate the KNN soft labels.

[0048] In a further embodiment, the multiple sub-models include integrated model one, integrated model two and integrated model three, where the integrated model one focuses on fine learning of normal area features; the integrated model two focuses on fitting edge area signal behavior; and the integrated model three performs unified modeling of all area samples.

[0049] In a further embodiment, the S4 further includes:

[0050] S4-1: Use the ε greedy strategy to randomly select actions to explore the environment with probability ε, or randomly select actions with probability 1- ε Select the action with the largest current Q value

[0051] S4-2: According to the current position q n and historical trajectory to select the next action to reach the new location Predicting outage probability using radio mapping M

[0052] Where Δs = V max Δt is the maximum displacement of the UAV;

[0053] S4-3: Measure the signal strength in the current state and calculate the reward based on reaching the target, crossing the boundary, connection interruption and trajectory smoothness

[0054] In a further embodiment, the S5 further includes:

[0055] S5-1: storing all visited locations and their corresponding outage probability measurements in a radio map database E as a basis for generating simulation experience;

[0056] S5-3: Using the Dyna architecture to implement hybrid training, for each update based on real experience, the system performs multiple additional updates based on simulated experience;

[0057] S5-4: Simulation experience generation strategy, randomly initialize the initial position of the simulation trajectory Set the simulation time step and current status Use ε-greedy strategy to select actions for simulation trajectories Based on the current state and radio mapping M; perform simulated actions Update the position of the simulation track

[0058] Predicting outage probability using radio mapping M

[0059] Calculating Simulation Rewards

[0060] S5-5: Dynamically adjust the ratio of simulated experience to real experience, set the number of simulation steps after each real experience step, and increase the ratio of simulated experience as training progresses until it reaches the upper limit;

[0061] S5-6: Apply the same agent update strategy in the simulation environment as in the real environment, using the same DuelingDQN and n-step learning mechanism to calculate rewards based on the predicted interruption probability.

[0062] In a further embodiment, the storing of experience includes using an n-step learning mechanism to store the current experience Store the measured data in the experience replay buffer D. Stored in the radio map database E;

[0063] in, is the selected action, R n For action rewards, is the interruption probability corresponding to the current position.

[0064] In a further embodiment, the S6 further includes:

[0065] S6-1: Construct a cyclic experience replay cache D with a capacity of N, using a FIFO strategy to store the most recent state-action-reward-next-state transition quadruple; based on the experience replay mechanism, the current experience and simulation experience Store in the sliding window queue and calculate the N1-step cumulative reward; measure the data Stored in the radio map database E;

[0066] S6-2: The expression for calculating the N1-step cumulative reward is as follows:

[0067]

[0068] in, is the N1-step cumulative discounted return starting from time n, N1 is the forward step parameter, γ is the discount factor, R n+i+1 The immediate reward obtained at time n+i+1, N1 controls the bias-variance trade-off, and γ balances the weight of immediate reward and long-term benefits;

[0069] S6-3: Randomly sample small batch training samples from the cyclic experience replay buffer D and perform gradient descent to update network parameters Update the target network parameters θ - =θ;

[0070] Randomly sample small batches of training samples from the radio map database E, update the target network parameters ξ, update the state n = n + 1, and update the exploration probability ε = εα;

[0071] S6-4: Check the termination condition, that is, the drone reaches the destination (||q n -q F ||≤D tol ) or out of bounds or reach the maximum number of steps Then terminate the iteration;

[0072] Among them, D tol is the position tolerance threshold, S is the safe flight area boundary, The maximum number of steps allowed in each round.

[0073] Beneficial effects of the invention: The regional adaptive deep reinforcement learning framework proposed in the present invention has achieved key technological breakthroughs on the basis of inheriting the advantages of traditional reinforcement learning. It not only maintains the model independence, online learning ability and high-dimensional state processing advantages of the reinforcement learning algorithm, but also fundamentally improves the robustness and adaptability of the algorithm in complex electromagnetic interference environments through innovative multi-scale data fusion processing mechanism, adaptive neural network architecture design and regionalized interference perception module.

[0074] (1) Taking into account various practical factors that may exist in the actual flight of UAVs, such as channel obstruction, equipment measurement error, environmental noise and regional interference, this method constructs a simulation environment that is more in line with the actual flight scene of UAVs. The simulation environment is set with regional structure interference, random label inversion, edge area perturbation, edge area perturbation and random Gaussian noise;

[0075] (2) This method uses an innovative regional identification and anomaly detection mechanism to accurately respond to the RF characteristics of different regions. By integrating deep reinforcement learning with regional adaptive RF mapping strategies, it can achieve synchronous learning of navigation path optimization and RF environment perception.

[0076] (3) This method innovatively introduces a soft label design based on KNN to address the serious interference problem in the central area. It generates more robust probability distribution labels through weighted averaging, which is suitable for cellular-connected UAV systems in complex urban environments. By introducing multi-model integration, K-nearest neighbor soft label compensation mechanism and real-simulated experience hybrid training framework, it significantly improves the navigation robustness and RF mapping accuracy in interference scenarios, and solves the key problem of coordinated optimization of communication quality assurance and path planning in target-oriented tasks of UAVs. BRIEF DESCRIPTION OF THE DRAWINGS

[0077] Figure 1 It is a schematic diagram of an application scenario of the present invention.

[0078] Figure 2 It is a schematic flow diagram of the present invention.

[0079] Figure 3 It is a schematic diagram of the system framework of the present invention.

[0080] Figure 4 This is a schematic diagram comparing the radio frequency maps after restoration by regional adaptive ensemble learning and traditional radio frequency mapping. (a) is the result map of traditional radio frequency mapping, and (b) is the radio frequency map after restoration by regional adaptive ensemble learning. DETAILED DESCRIPTION

[0081] In the following description, numerous specific details are provided to provide a more thorough understanding of the present invention. However, it will be apparent to those skilled in the art that the present invention may be practiced without one or more of these details. In other instances, certain technical features well known in the art are not described to avoid confusion with the present invention.

[0082] The present invention will be further described in detail below with reference to the accompanying drawings.

[0083] Reference Figure 1-4 , a method for synchronized navigation and radio mapping of unmanned aerial vehicles based on reinforcement learning and regional adaptive ensemble learning disclosed in the present invention, comprising the following steps:

[0084] S1: Based on the actual UAV flight situation, simulate the UAV navigation environment to build a radio mapping and radio map database, simulate various interference situations that the UAV may actually encounter in the simulated navigation environment, divide the UAV simulated navigation environment into areas in the radio mapping based on geographic location information, and set the UAV position and radio environment parameters.

[0085] The UAV navigation environment is divided into a normal area E1, a central interference area E2 and an edge interference area E3 in S1, and the setting of the UAV position includes setting the initial UAV position q IAnd set the target position coordinates q0. The radio environment parameters include base station configuration parameters, channel model parameters, and building environment parameters;

[0086] The S1 further includes:

[0087] S1-1: Set the outage probability of the central interference area E2 (800 < x < 1200 and 800 < y < 1200) to 0.9 to simulate the obstacle occlusion interference that may exist in the actual flight environment;

[0088] S1-2: Set a 10% probability of tag flipping to simulate the measurement device failure interference that may exist during actual flight;

[0089] S1-3: Set the outage probability P out (q) = clip(P base (q) + 0.08), 0.1) to simulate the edge area deviation interference that may exist in the actual flight environment;

[0090] Among them, P base (q) is the basic outage probability calculated based on the real wireless propagation model, and P out (q) is the final output outage probability;

[0091] S1-4: Set Gaussian noise P out (q) = clip(P base (q) + N(0, 0.03), 0.1) to simulate the random noise that may exist in the actual flight environment, with a mean of 0 and a standard deviation of 0.03.

[0092] S2: Construct a deep reinforcement learning agent. The learning agent adopts the Dueling DQN architecture to separate value estimation and the advantage function, so as to achieve optimal flight action selection.<00003​​​​​​​​​​​​​,θ);

[0096] S2-2: Separate the state features into two branches, the two branches are state value estimation V(s) and action advantage function A(s,v), the main network is responsible for the online action selection decision of the UAV, and the expression of the action value function Q value maximization criterion is Determine the optimal action choice;

[0097] in, is the UAV position state after N1 steps, are the possible actions in this state;

[0098] The target network is responsible for stable value evaluation. The expression of the expected cumulative reward value after executing the selected action is as follows:

[0099]

[0100] Where γ is the discount factor, γ∈[0,1] represents the importance weight of future rewards, and controls the balance between long-term and short-term benefits;

[0101] S2-3: Merge the two branches through a special aggregation layer and calculate the final action-value function using the Q-value reconstruction formula. The expression of the Q-value reconstruction formula is as follows:

[0102]

[0103] Among them, the average advantage term is subtracted to improve numerical stability.

[0104] S3: Use a regional adaptive integrated radio frequency mapping model to learn the radio coverage in the UAV navigation environment in real time, detect and determine the divided area where the UAV is located in the UAV navigation environment, and identify and mark abnormal data. Based on the characteristics of the divided area, a multi-submodel collaborative modeling mechanism is introduced to build an integrated system composed of multiple functionally complementary deep neural networks.

[0105] The radio frequency mapping model in S3 includes a region detector, an anomaly detector, and multiple sub-models, and applies different learning strategies to different regions;

[0106] Said S3 further comprises:

[0107] S3-1: Build a region detector to determine the region where the drone is located based on its flight position;

[0108] S3-2: Construct an anomaly detector, assuming the location set is L = {l1,l2...,l n}, corresponding label P = {p1,p2...,p n}, calculate the distance d ij=||l i -l j ||, visit neighboring point N i ={j|rank(d j )≤K,j≠i}, calculate the anomaly score Perform outlier judgment i =(s i >T);

[0109] Where T is the abnormal threshold parameter. When the difference exceeds the preset abnormal threshold parameter T, the sample is judged as an abnormal sample and discarded.

[0110] S3-3: Generate KNN-based soft labels for the central interference area and repair the samples in the central interference area;

[0111] S3-4: Each sub-model adopts a regional weighted loss function during the training phase to achieve regional perception feature extraction and representation learning. The multiple sub-models include integrated model 1, integrated model 2, and integrated model 3. Integrated model 1 focuses on fine learning of normal area features and reduces the weight of center area samples (center 0.3, edge 0.7) to reduce the impact of disturbed areas on modeling accuracy; integrated model 2 focuses on fitting edge area signal behavior and gives higher weights to edge samples during training (center 0.5, edge 1.0) to enhance the representation ability of transition areas; integrated model 3 uniformly models all area samples and is suitable for global feature learning. During training, equal weights are given to center and edge areas.

[0112] S3-5: Aggregate all sub-model outputs during prediction, perform post-processing adjustments based on the characteristics of the divided regions, and provide a final connection interruption probability prediction. The expression for the connection interruption probability prediction is as follows:

[0113]

[0114] Among them, G=3 is the number of models, f i (x|D i ) is the i-th model based on the data set D i The prediction of w i (x) is the region-based weight function.

[0115] The repair center area sample in S3-3 further includes:

[0116] Using w i =1 / (d i +ε) Calculate the weighted average of the target point and normal area samples;

[0117] Among them, w i is the weight of the ith neighboring sample, d iis the distance to the ith neighboring sample, ε is a small constant to prevent division by zero, and a total of 5 neighboring samples are selected;

[0118] Normalized weights By soft=Σw inorm ×label i The KNN soft label is calculated based on the principle that the signal characteristics of close positions should be similar, and the data of the normal area is used to "correct" the mislabeling of the central area.

[0119] S4: The drone explores the environment through interactive learning combined with an epsilon-greedy strategy while collecting and learning RF data. The learning agent selects the next action to reach the new location based on the drone's current position and historical trajectory, and calculates rewards based on reaching the target, crossing the boundary, connection interruption, and path smoothness.

[0120] Said S4 further comprises:

[0121] S4-1: Use the ε greedy strategy to randomly select actions to explore the environment with probability ε, or randomly select actions with probability 1- ε Select the action with the largest current Q value

[0122] S4-2: According to the current position q n and historical trajectory to select the next action to reach the new location Predicting outage probability using radio mapping M

[0123] Where Δs = V max Δt is the maximum displacement of the UAV;

[0124] S4-3: Measure the signal strength in the current state and calculate the reward based on reaching the target, crossing the boundary, connection interruption and trajectory smoothness

[0125] S5: Policy training is performed using an experience pool built from a combination of real-world and simulated experience. The real-world experience comes from the actual interaction between the UAV and the environment, while the simulated experience comes from virtual trajectories generated by the RF mapping model. This reduces the amount of data required for actual UAV flight and accelerates the policy learning process. It also reduces the number of actual flights, lowering operational costs and potential safety risks.

[0126] The S5 further includes:

[0127] S5-1: storing all visited locations and their corresponding outage probability measurements in the radio map database E as a basis for generating simulation experience;

[0128] S5-3: Using the Dyna architecture to implement hybrid training, for each update based on real experience, the system performs multiple additional updates based on simulated experience;

[0129] S5-4: Simulation experience generation strategy, randomly initialize the initial position of the simulation trajectory Set the simulation time step and current status Use ε-greedy strategy to select actions for simulation trajectories Based on the current state and radio mapping M; perform simulated actions Update the position of the simulation track

[0130] Predicting outage probability using radio mapping M

[0131] Calculating Simulation Rewards

[0132] S5-5: Dynamically adjust the ratio of simulated experience to real experience, set the number of simulation steps after each real experience step, and increase the ratio of simulated experience to an upper limit as training progresses, making full use of the gradually improving radio mapping accuracy;

[0133] S5-6: Apply the same agent update strategy as in the real environment in the simulation environment, using the same DuelingDQN and n-step learning mechanism, and calculate rewards based on the predicted interruption probability instead of the actual measurement value, thereby reducing the amount of data required for actual flight, reducing operating costs and safety risks.

[0134] S6: Use n-step learning mechanism to store experience, calculate the cumulative reward with discount factor and use Double Q-Learning method to update the network to obtain the optimal navigation strategy; the storage experience includes using n-step learning mechanism to store the current experience Store the measured data in the experience replay buffer D. Stored in the radio map database E;

[0135] in, is the selected action, R n For action rewards, is the interruption probability corresponding to the current position; calculate the cumulative discounted reward and use the Double Q-Learning method to update the network to reduce the Q value overestimation problem. As the training progresses, the exploration rate is gradually reduced to increase the utilization of the learned strategy.

[0136] The S6 further includes:

[0137] S6-1: Construct a cyclic experience replay cache D with a capacity of N, using a FIFO strategy to store the most recent state-action-reward-next-state transition quadruple; based on the experience replay mechanism, the current experience and simulation experience Store in the sliding window queue and calculate the N1-step cumulative reward; measure the data Stored in the radio map database E;

[0138] S6-2: The expression for calculating the N1-step cumulative reward is as follows:

[0139]

[0140] in, is the N1-step cumulative discounted return starting from time n, N1 is the forward step parameter, γ is the discount factor, R n+i+1 The immediate reward obtained at time n+i+1, N1 controls the bias-variance trade-off, and γ balances the weight of immediate reward and long-term benefits;

[0141] S6-3: Randomly sample small batch training samples from the cyclic experience replay buffer D and perform gradient descent to update network parameters Update the target network parameters θ - =θ;

[0142] Randomly sample small batches of training samples from the radio map database E, update the target network parameters ξ, update the state n = n + 1, and update the exploration probability ε = εα;

[0143] S6-4: Check the termination condition, that is, the drone reaches the destination (||q n -q F ||≤D tol ) or out of bounds or reach the maximum number of steps Then terminate the iteration;

[0144] Among them, D tol is the position tolerance threshold, S is the safe flight area boundary, The maximum number of steps allowed in each round.

[0145] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.

[0146] The preferred embodiments of the present invention are described in detail above in conjunction with the accompanying drawings. However, the present invention is not limited to the specific details in the above embodiments. Within the technical concept of the present invention, various equivalent transformations can be made to the technical solutions of the present invention, and these equivalent transformations all fall within the scope of protection of the present invention.

Claims

1. A method for synchronized navigation and radio mapping of unmanned aerial vehicles based on reinforcement learning and regional adaptive ensemble learning, characterized by: The method comprises the following steps: S1: Simulate the UAV navigation environment to build a radio map and a radio map database, divide the simulated navigation environment of the UAV into regions in the radio map according to the geographic location information, and set the UAV position and radio environment parameters; S2: Build a deep reinforcement learning agent that uses the Dueling DQN architecture to separate value estimation from advantage function. S3: Uses a regionally adaptive integrated radio frequency mapping model to learn the radio coverage of the UAV navigation environment in real time, detects and determines the partitioned area where the UAV is located in the UAV navigation environment, and identifies and marks abnormal data. Based on the characteristics of the partitioned area, a multi-submodel collaborative modeling mechanism is introduced to build an integrated system composed of multiple functionally complementary deep neural networks; S4: The drone explores the environment through interactive learning combined with an epsilon-greedy strategy while simultaneously collecting and learning radio frequency data. The learning agent selects the next action to reach the new location based on the drone's current position and historical trajectory, and calculates rewards based on reaching the target, crossing the boundary, connection interruption, and path smoothness. S5: Policy training is performed using an experience pool built from both real-world and simulated experience. The real-world experience comes from the actual interaction between the drone and the environment, while the simulated experience comes from the virtual trajectories generated by the RF mapping model. S6: Use n-step learning mechanism to store experience, calculate cumulative rewards with discount factors and use Double Q-Learning method to update the network to obtain the optimal navigation strategy.

2. The method for synchronized navigation and radio mapping of unmanned aerial vehicles based on reinforcement learning and regional adaptive ensemble learning according to claim 1, characterized in that: The UAV navigation environment is divided into a normal area E1, a central interference area E2 and an edge interference area E3 in S1, and the setting of the UAV position includes setting the initial UAV position q I and setting the target location coordinates q0, wherein the radio environment parameters include base station configuration parameters, channel model parameters and building environment parameters; Said S1 further comprises: S1-1: Set the interruption probability of the central interference area E2 to 0.9 to simulate the possible interference caused by obstacles in the actual flight environment; S1-2: Set a 10% probability of label flipping to simulate the interference of measurement equipment failure that may exist in actual flight; S1-3: Set the interruption probability P of the edge interference area E3 out (q)=clip(P base (q)+0.08),0.1) to simulate the possible edge area deviation interference in the actual flight environment; Among them, P base (q) is the basic outage probability calculated based on the real wireless propagation model, P out (q) is the final output interruption probability; S1-4: Set other areas to add Gaussian noise P out (q)=clip(P base (q)+N(0,0.03),0.1) is used to simulate the random noise that may exist in the actual flight environment.

3. The method for synchronized navigation and radio mapping of unmanned aerial vehicles based on reinforcement learning and regional adaptive ensemble learning according to claim 1, characterized in that: The learning agent in S2 consists of a main network and a target network. The main network is responsible for online learning, and the target network is used to stabilize target prediction during training. The drone policy network is trained by the DRL method to enable it to estimate the action value function based on the current position state of the drone and the interruption probability predicted by the radio mapping; Said S2 further comprises: S2-1: Initialize Dueling DQN main network parameters θ and target network parameters θ - , get the current status of the drone Use multi-layer neural network to extract state features φ(s t ,θ); S2-2: Separate the state features into two branches, the two branches are state value estimation V(s) and action advantage function A(s,v), the main network is responsible for the online action selection decision of the UAV, and the expression of the action value function Q value maximization criterion is Determine the optimal action choice; in, is the UAV position state after N1 steps, are the possible actions in this state; The target network is responsible for stable value evaluation. The expression of the expected cumulative reward value after executing the selected action is as follows: Where γ is the discount factor, γ∈[0,1] represents the importance weight of future rewards, and controls the balance between long-term and short-term benefits; S2-3: Merge the two branches through a special aggregation layer and calculate the final action-value function using the Q-value reconstruction formula. The expression of the Q-value reconstruction formula is as follows: where the mean advantage term is subtracted.

4. The method for synchronized navigation and radio mapping of unmanned aerial vehicles based on reinforcement learning and regional adaptive ensemble learning according to claim 1, characterized in that: The radio frequency mapping model in S3 includes a region detector, an anomaly detector, and multiple sub-models, and applies different learning strategies to different regions; Said S3 further comprises: S3-1: Build a region detector to determine the region where the drone is located based on its flight position; S3-2: Construct an anomaly detector, assuming the location set is L = {l1,l2...,l n }, corresponding label P = {p1,p2...,p n }, calculate the distance d ij =||l i -l j ||, visit neighboring point N i ={j|rank(d j )≤K,j≠i}, calculate the anomaly score Perform outlier judgment i =(s i >T); Where T is the abnormal threshold parameter. When the difference exceeds the preset abnormal threshold parameter T, the sample is judged as an abnormal sample and discarded. S3-3: Generate KNN-based soft labels for the central interference area and repair the samples in the central interference area; S3-4: Each sub-model adopts a region-weighted loss function during the training phase to achieve region-aware feature extraction and representation learning; S3-5: Aggregate all sub-model outputs during prediction, perform post-processing adjustments based on the characteristics of the divided regions, and provide a final connection interruption probability prediction. The expression for the connection interruption probability prediction is as follows: Among them, G=3 is the number of models, f i (x|D i ) is the i-th model based on the data set D i The prediction of w i (x) is the region-based weight function.

5. The method for synchronized navigation and radio mapping of unmanned aerial vehicles based on reinforcement learning and regional adaptive ensemble learning according to claim 4, characterized in that: The repair center area sample in S3-3 further includes: Using w i =1 / (d i +ε) Calculate the weighted average of the target point and normal area samples; Among them, w i is the weight of the ith neighboring sample, d i is the distance to the ith neighboring sample, ε is a small constant to prevent division by zero, and a total of 5 neighboring samples are selected; Normalized weights By soft=∑w inorm ×label i Calculate the KNN soft labels.

6. The method for synchronized navigation and radio mapping of unmanned aerial vehicles based on reinforcement learning and regional adaptive ensemble learning according to claim 4, characterized in that: The multiple sub-models include integrated model one, integrated model two and integrated model three. The integrated model one focuses on the fine learning of normal area features; the integrated model two focuses on fitting the signal behavior of the edge area; and the integrated model three uniformly models all area samples.

7. The method for synchronized navigation and radio mapping of unmanned aerial vehicles based on reinforcement learning and regional adaptive ensemble learning according to claim 1, characterized in that: Said S4 further comprises: S4-1: Use the ε greedy strategy to randomly select actions to explore the environment with probability ε, or select the action with the largest current Q value with probability 1-ε S4-2: According to the current position q n and historical trajectory to select the next action to reach the new location Predicting outage probability using radio mapping M Where Δs = V max Δt is the maximum displacement of the UAV; S4-3: Measure the signal strength in the current state and calculate the reward based on reaching the target, crossing the boundary, connection interruption and trajectory smoothness 8. The method for synchronized navigation and radio mapping of unmanned aerial vehicles based on reinforcement learning and regional adaptive ensemble learning according to claim 1, characterized in that: The S5 further includes: S5-1: storing all visited locations and their corresponding outage probability measurements in the radio map database E as a basis for generating simulation experience; S5-3: Using the Dyna architecture to implement hybrid training, for each update based on real experience, the system performs multiple additional updates based on simulated experience; S5-4: Simulation experience generation strategy, randomly initialize the initial position of the simulation trajectory Set the simulation time step and current status Use ε-greedy strategy to select actions for simulation trajectories Based on the current state and radio mapping M; perform simulated actions Update the position of the simulation track Predicting outage probability using radio mapping M Calculating Simulation Rewards S5-5: Dynamically adjust the ratio of simulated experience to real experience, set the number of simulation steps after each real experience step, and increase the ratio of simulated experience as training progresses until it reaches the upper limit; S5-6: Apply the same agent update strategy in the simulation environment as in the real environment, using the same Dueling DQN and n-step learning mechanism to calculate rewards based on the predicted interruption probability.

9. The method for synchronized navigation and radio mapping of unmanned aerial vehicles based on reinforcement learning and regional adaptive ensemble learning according to claim 1, characterized in that: The storage experience includes using an n-step learning mechanism to store the current experience Store the measured data in the experience replay buffer D. Stored in the radio map database E; in, is the selected action, R n For action rewards, is the interruption probability corresponding to the current position.

10. The method for synchronized navigation and radio mapping of unmanned aerial vehicles based on reinforcement learning and regional adaptive ensemble learning according to claim 9, characterized in that: The S6 further includes: S6-1: Construct a cyclic experience replay cache D with a capacity of N, using a FIFO strategy to store the most recent state-action-reward-next-state transition quadruple; based on the experience replay mechanism, the current experience and simulation experience Store in the sliding window queue and calculate the N1-step cumulative reward; measure the data Stored in the radio map database E; S6-2: The expression for calculating the N1-step cumulative reward is as follows: in, is the N1-step cumulative discounted return starting from time n, N1 is the forward step parameter, γ is the discount factor, R n+i+1 The immediate reward obtained at time n+i+1, N1 controls the bias-variance trade-off, and γ balances the weight of immediate reward and long-term benefits; S6-3: Randomly sample small batch training samples from the cyclic experience replay buffer D and perform gradient descent to update network parameters Update the target network parameters θ - =θ; Randomly sample small batches of training samples from the radio map database E, update the target network parameters ξ, update the state n = n + 1, and update the exploration probability ε = εα; S6-4: Check the termination condition, that is, the drone reaches the destination (||q n -q F ||≤D tol ) or out of bounds or reach the maximum number of steps Then terminate the iteration; Among them, D tol is the position tolerance threshold, S is the safe flight area boundary, The maximum number of steps allowed in each round.

Images