Aerial vehicle city airspace dynamic path planning method based on reinforcement learning
By constructing a three-dimensional composite environment model and a comprehensive reward function, and combining reinforcement learning and digital twin simulation, the environmental adaptability and safety issues of dynamic path planning for flying cars in urban airspace were solved. This achieved multi-dimensional optimization and safety redundancy of path planning, improving the operational efficiency and reliability of flying cars.
Patent Information
- Application Number
- CN202611598451.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-10-15
- Publication Date
- 2026-11-17
AI Technical Summary
Existing technologies struggle to effectively handle dynamic weather changes, unknown obstacles, and uncertainties in aircraft motion within urban airspace for flying cars. Furthermore, path planning does not adequately consider energy consumption and noise impacts, resulting in uneconomical and unsafe planned paths.
A three-dimensional composite environment model is constructed, and a comprehensive reward function is designed by combining static obstacles, dynamic weather fences, and moving target trajectory prediction. The policy network and configuration switching sub-network are trained through reinforcement learning, and combined with digital twin simulation and online fine-tuning, to achieve multi-objective optimization and safety redundancy of path planning.
It improves the completeness and accuracy of environmental information in path planning, optimizes energy consumption economy and noise social acceptability, extends the service life of configuration switching mechanism, and ensures flight safety and reliability.
Smart Images

Figure CN123113901A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of autonomous control technology for flying cars, and in particular to a dynamic path planning method for flying cars in urban airspace based on reinforcement learning. Background Technology
[0002] With the rapid development of urban air transportation, flying cars have attracted widespread attention as an emerging means of transportation to alleviate ground traffic congestion. Flying cars operate in the low-altitude urban environment and face multiple challenges such as dense buildings, complex weather conditions, and mutual interference between aircraft. Their path planning needs to meet multiple requirements in three-dimensional space, including collision safety, flight efficiency, energy economy, and noise social impact. However, traditional path planning methods for fixed-wing aircraft or rotary-wing UAVs are difficult to adapt to the operating mode of flying cars and their need to respond quickly to sudden weather changes and moving obstacles in the urban environment.
[0003] In recent years, reinforcement learning methods have consistently performed well in complex sequential decision-making problems, and have therefore been gradually introduced into the field of path planning for unmanned systems. Among the existing technologies, there are related studies on applying deep reinforcement learning to urban obstacle avoidance trajectory planning for UAVs. However, the environmental modeling stage usually only considers static building obstacles, and lacks effective representation of dynamic meteorological factors such as sudden changes in urban low-altitude wind fields, local precipitation, and turbulence. It also does not consider the motion uncertainty of surrounding aircraft. In addition, when designing the reward function, the existing methods do not fully consider the real-time energy consumption characteristics of the hybrid power system of electric aircraft, as well as the significant differences in the tolerance of flight noise in different urban areas, resulting in poor economic efficiency of the planned path operation.
[0004] In terms of practical deployment, existing reinforcement learning models, after being trained on simulation platforms, are difficult to adapt to sensor noise, communication delays, sudden weather changes, and unknown dynamic obstacles in real urban airspace. Furthermore, when the policy network experiences response degradation due to sudden environmental changes, there is a lack of effective safety redundancy mechanisms, which cannot meet the stringent safety and reliability requirements of flying car decision-making systems.
[0005] Furthermore, when a flying car with variable configuration switches between different configuration modes, its aerodynamic characteristics and control boundaries change significantly. Existing path planning methods usually treat configuration switching as a discrete logic rule decoupled from path decision-making, without comprehensively considering the mechanical wear and energy impact costs brought about by configuration switching, which often leads to unnecessary fatigue losses and energy consumption. Summary of the Invention
[0006] The purpose of this invention is to propose a dynamic path planning method for urban airspace of flying cars based on reinforcement learning, in order to solve the problems in the prior art.
[0007] To achieve the above objectives, the present invention adopts the following technical solution: a dynamic path planning method for urban airspace of flying cars based on reinforcement learning, comprising the following steps: S1: Acquire global low-altitude environmental data of the city, construct a three-dimensional composite environmental model including a static obstacle layer, a dynamic weather fence layer, and a moving target prediction trajectory layer, define the state space including the variable configuration mode of the flying car, and the action space including thrust control variables and configuration switching commands. S2, construct a comprehensive reward function based on the state space and action space. The comprehensive reward function includes at least an energy consumption penalty term, a noise exposure penalty term, a collision safety constraint term, and a path efficiency reward term. S3 uses a three-dimensional composite environment model as the simulation environment, defines the reinforcement learning training task by using the state space and the comprehensive reward function, and pre-trains a policy network for path planning and a configuration switching subnetwork connected in parallel to the policy network through a digital twin simulation platform. S4 acquires real-time sensor data of the surrounding environment during the actual flight of the flying car, and simultaneously outputs predicted data through digital twin model simulation, calculates the deviation between the two, and performs online fine-tuning of the strategy network and configuration switching sub-network based on the deviation. When the response speed of online fine-tuning continues to be lower than the preset lower limit, the emergency obstacle avoidance module, which is independent of the strategy network, temporarily takes over local control. S5 outputs thrust control quantities through the fine-tuned strategy network and configuration switching commands through the configuration switching subnetwork. Combining the current pose of the flying car with the target location, it generates a three-dimensional feasible path. S4 and S5 are executed repeatedly in each control cycle until the flying car reaches the target location.
[0008] The beneficial effects of the technical solution provided by this invention include at least the following: This invention constructs a three-dimensional composite environment model that includes a static obstacle layer, a dynamic weather fence layer, and a moving target prediction trajectory layer. This model integrates the static physical boundaries of urban low-altitude areas, dynamic weather threats, and the uncertainties of surrounding aircraft movements into the same three-dimensional grid space for unified representation and superposition, significantly improving the completeness and accuracy of the environmental information on which path planning decisions are based.
[0009] This invention designs a comprehensive reward function that incorporates multi-energy coupling characteristics and noise socialization constraints. Together with collision safety constraints and path efficiency rewards, it forms a multi-objective optimization framework. This framework enables the planned three-dimensional feasible path to actively avoid noise-sensitive areas such as residential areas, schools, and hospitals while ensuring safety. It also takes into account the energy consumption economy of the hybrid power system under different configuration modes, achieving multi-dimensional synergistic optimization and improvement of flight safety, social acceptance, and energy utilization efficiency.
[0010] This invention not only pre-trains the policy network and configuration switching sub-network offline through a digital twin simulation platform, but also performs online fine-tuning during actual flight, significantly improving the policy network's adaptability to the differences between the simulation and real environments. At the same time, this invention enables the configuration switching sub-network and the policy network to make decisions in parallel and collaboratively. In the decision-making process, the mechanical wear cost and transient power impact cost of configuration switching are introduced, allowing the system to autonomously learn how to make the optimal trade-off between mission requirements and configuration maintenance, effectively extending the service life of the switching mechanism and reducing transient energy consumption.
[0011] This invention constructs a safety redundancy closed-loop architecture with an emergency obstacle avoidance module at its core, enabling flying cars to autonomously switch to a safe obstacle avoidance mode that does not rely on neural networks when the policy network experiences response degradation. It also automatically recovers after the fault is cleared, balancing safety and mission completion capabilities. This fundamentally solves the credibility and reliability concerns faced when deploying reinforcement learning control in the highly safety-sensitive scenario of flying cars. Attached Figure Description
[0012] To more clearly illustrate the technical solutions and advantages in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0013] Figure 1 This is a flowchart of a method provided in an embodiment of the present invention. Detailed Implementation
[0014] To further illustrate the technical means and effects adopted by the present invention to achieve its intended purpose, the following, in conjunction with the accompanying drawings and preferred embodiments, details the specific implementation, structure, features, and effects of a reinforcement learning-based dynamic path planning method for urban airspace for flying cars proposed according to the present invention. In the following description, different "one embodiment" or "another embodiment" do not necessarily refer to the same embodiment. Furthermore, specific features, structures, or characteristics in one or more embodiments can be combined in any suitable form.
[0015] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.
[0016] The following examples are for illustrative purposes only and are not intended to limit the scope of the invention.
[0017] The following description, in conjunction with the accompanying drawings, details a specific scheme for a dynamic path planning method for urban airspace based on reinforcement learning for flying cars provided by this invention.
[0018] Please see Figure 1 The diagram illustrates a flowchart of a dynamic path planning method for urban airspace based on reinforcement learning for flying cars, according to an embodiment of the present invention. The method includes the following steps: S1: Acquire global low-altitude environmental data of the city, construct a three-dimensional composite environmental model including a static obstacle layer, a dynamic weather fence layer, and a moving target prediction trajectory layer, define the state space including the variable configuration mode of the flying car, and the action space including thrust control variables and configuration switching commands. S2, construct a comprehensive reward function based on the state space and action space. The comprehensive reward function includes at least an energy consumption penalty term, a noise exposure penalty term, a collision safety constraint term, and a path efficiency reward term. S3 uses a three-dimensional composite environment model as the simulation environment, defines the reinforcement learning training task by using the state space and the comprehensive reward function, and pre-trains a policy network for path planning and a configuration switching subnetwork connected in parallel to the policy network through a digital twin simulation platform. S4 acquires real-time sensor data of the surrounding environment during the actual flight of the flying car, and simultaneously outputs predicted data through digital twin model simulation, calculates the deviation between the two, and performs online fine-tuning of the strategy network and configuration switching sub-network based on the deviation. When the response speed of online fine-tuning continues to be lower than the preset lower limit, the emergency obstacle avoidance module, which is independent of the strategy network, temporarily takes over local control. S5 outputs thrust control quantities through the fine-tuned strategy network and configuration switching commands through the configuration switching subnetwork. Combining the current pose of the flying car with the target location, it generates a three-dimensional feasible path. S4 and S5 are executed repeatedly in each control cycle until the flying car reaches the target location.
[0019] In one embodiment of the present invention, the step of constructing a three-dimensional composite environment model includes: Acquire urban low-altitude lidar point cloud data and building vector contour data, register them to a unified geographic coordinate system and perform three-dimensional rasterization processing, mark the grids containing buildings, towers and high-voltage line corridors as occupied, forming a static obstacle layer; Acquire real-time wind field data from meteorological satellites and precipitation radar data, dynamically delineate meteorological hazard boundaries under a unified geographic coordinate system, and mark the grids within the meteorological hazard boundaries as dynamic fence status to form a dynamic meteorological fence layer. Acquire broadcast ADS-B historical trajectory data of surrounding aircraft, predict the position probability distribution of each surrounding aircraft within a preset time period in the future, mark the grid with probability density exceeding the preset probability density upper limit as potential conflict state, and form a moving target prediction trajectory layer. By overlaying and fusing a static obstacle layer, a dynamic weather fence layer, and a moving target trajectory prediction layer, a three-dimensional composite environment model is generated.
[0020] It should be noted that in this embodiment, the raster resolution of the three-dimensional rasterization process is adaptively set according to the safety level requirements and computing resource constraints of different areas of the urban airspace. For example, a resolution of 0.1m×0.1m×0.1m is used in high-risk areas such as densely built-up areas, schools and hospitals, while a resolution of 1m×1m×1m is used in low-risk areas such as suburbs and water areas. The resolution of high-risk areas should be higher than that of low-risk areas.
[0021] The method for delineating meteorological hazard boundaries is as follows: (1) Identify continuous spaces where precipitation intensity exceeds a preset precipitation intensity threshold based on precipitation radar data, and delineate them as initial danger boundaries; (2) Calculate the wind speed vector in each grid based on the wind field data, and classify the grids whose wind speed vector amplitude exceeds the preset wind speed threshold into the initial danger boundary; (3) Extend the initial danger boundary outward by a preset safety buffer distance as a meteorological danger boundary, and mark the three-dimensional spatial grid covered by the boundary as a dynamic fence state.
[0022] The probability distribution of the positions of surrounding aircraft within a preset time period is obtained by predicting historical trajectory data of ADS-B through Gaussian process regression or long short-term memory network. The upper limit of probability density is dynamically set according to the safety distance requirements between the current position of the flying car and surrounding aircraft.
[0023] If the same grid cell is marked by two or three of the static obstacle layer, dynamic weather fence layer, and moving target prediction trajectory layer during overlay and fusion, the final layer to which the grid cell belongs is determined according to the priority decision rule of static obstacle layer > dynamic weather fence layer > moving target prediction trajectory layer.
[0024] In addition, the three-dimensional composite environment model is fully loaded into the airborne storage unit before the flying car takes off. Preferably, during flight, only the dynamic weather fence layer and the moving target prediction trajectory layer are periodically updated incrementally, while the static obstacle layer remains unchanged during a single flight mission.
[0025] As one embodiment of the present invention, the step of constructing a comprehensive reward function based on the state space and action space includes: The energy consumption penalty is calculated in real time based on the current wind speed drag coefficient and the energy consumption curve per unit distance of the flying car in the current configuration mode. The noise exposure penalty is calculated based on the noise sensitivity level coefficient corresponding to the grid where the flying car is currently located. The collision safety constraints are calculated based on the nearest three-dimensional Euclidean distance between the grid where the flying car is currently located and the grids in the static obstacle layer and the dynamic weather fence layer. The path efficiency reward is calculated based on the reduction in the remaining straight-line distance between the flying car's current location and the target location at each decision step in reinforcement learning.
[0026] It should be noted that, in this embodiment, the specific calculation method for each sub-item of the comprehensive reward function is as follows: (1) Energy consumption penalty: By setting different flight speed ranges, load conditions and wind speed conditions in the digital twin simulation platform, flight simulation tests were conducted on the flying car. The amount of battery energy or hydrogen fuel consumed by the flying car to complete a unit horizontal displacement in each configuration mode was recorded, and a unit distance energy consumption data table was formed and stored in the airborne storage unit. During actual flight, the flying car obtains the corresponding energy consumption per unit distance E in real time from this data table through interpolation, based on its current configuration mode, current flight speed, current load conditions, and current wind conditions (including wind speed and direction). unit ; Energy consumption penalty R E Calculate using the following formula: R E = - E unit ·Δs; Among them, E unit Δs represents the energy consumption per unit distance, and Δs represents the distance the flying car travels within the current decision step. Those skilled in the art should know that the current wind speed conditions have been entered into the three-dimensional composite environment model in the aforementioned steps, and the flight speed can be obtained from the airborne flight control system, which will not be elaborated here.
[0027] (2) Noise exposure penalty: Referring to GB 3096-2008 "Environmental Quality Standard for Noise", the urban sound environment functional zone is divided into multiple noise sensitivity level areas, and each is assigned a noise sensitivity level coefficient β(x,y,z), where (x,y,z) are the corresponding grid coordinates; Each β(x,y,z) is pre-calibrated on a three-dimensional grid map and stored in the airborne storage unit. During actual flight, the coordinates of the center point of the grid where the flying car is located are retrieved in real time. Noise exposure penalty item R N Calculate using the following formula: R N = -β(x,y,z)·Δt; Where Δt is the dwell time of the flying car within the corresponding grid.
[0028] (3) Collision safety constraints: Calculate the Euclidean distances between the center point coordinates of the grid cell where the flying car is currently located and the center point coordinates of all grid cells occupied by the static obstacle layer and the dynamic weather fence layer, and take the minimum value as the nearest three-dimensional Euclidean distance d. safe ; Collision safety constraint R C The following piecewise function is used for calculation: R C = - (d safe - d threshold )-2,d safe ≤ d threshold ; R C = 0, d safe > d threshold ; Where, d threshold This is the preset safe distance threshold.
[0029] (4) Path efficiency bonus items: The path efficiency reward is calculated once at each decision step. Let the position of the flying car at the t-th decision step be p. t The target location is p goal Then the remaining straight-line distance d of the current decision step t = ||p t - p goal ||, the remaining straight-line distance of the previous decision step is d t-1 ; Path efficiency bonus item R T Calculate using the following formula: R T = (d t-1 - d t ); Preferably, when the flying car reaches the target location, a preset sparse reward R is given. T =R arrival .
[0030] In one embodiment of the present invention, the weighting coefficients of each penalty and reward term in the comprehensive reward function are adaptively adjusted according to the current mission type and the current flight phase. The weighting coefficients include a base weight and an adjustment weight. The adaptive adjustment includes the following steps: Based on the current mission type of the flying car, a set of basic weights corresponding to the mission type is selected from a set of preset basic weight configuration schemes. The mission types include at least safety-sensitive missions and economic-sensitive missions. Under safety-sensitive missions, the basic weight of the collision safety constraint item is greater than the basic weight of the path efficiency reward item. Under economic-sensitive missions, the basic weight of the energy consumption penalty item is greater than the value under safety-sensitive missions. Based on the current flight phase of the flying car, further adjustment weights are applied to the basic weight configuration. The flight phase includes at least the vertical take-off and landing phase and the cruise phase. In the vertical take-off and landing phase, the adjustment weight of the collision safety constraint is higher than that in the cruise phase, and the adjustment weight of the noise exposure penalty is lower than that in the cruise phase. In the cruise phase, the adjustment weights of the path efficiency reward and energy consumption penalty are higher than those in the vertical take-off and landing phase.
[0031] It should be noted that in this embodiment, the adaptive adjustment of the weighting coefficients is achieved through the multiplicative superposition of the base weights and the adjustment weights. Specifically, the expression for the comprehensive reward function R is: R = λ C ·R C + λ N ·R N + λ E ·R E + λ T ·R T ; Among them, R C For collision safety constraints, R N R is a noise exposure penalty term. E R is an energy consumption penalty term. T λ is the path efficiency reward item. C , λ N , λ E , λ T These are the weighting coefficients for each item; Each weighting coefficient is determined by both the base weight and the adjustment weight, that is: Λi = λ i_base ·λ i_adj i = C, N, E, or T; Wherein, the basic weight λ i_base Select from the preset configuration schemes based on the current task type. The preset configuration schemes include: Under safety-sensitive tasks, the basic weight λ of the collision safety constraint term C_base The base weight λ of the path efficiency reward item is greater than T_base Preferably, λ C_base =2.0, λ T_base =0.8, λE_base =0.6, λ N_base =0.5; Under economically sensitive tasks, the base weight λ of the energy consumption penalty term E_base The corresponding value for a task with higher security sensitivity is preferably λ. E_base =1.6, λ C_base =1.2, λ T_base =1.0, λ N_base =0.4.
[0032] Adjusting weight λ i_adj Select from the preset stage adjustment factor table based on the current flight stage, including: During the vertical takeoff and landing phase, the adjustment weight λ of the collision safety constraint term C_adj For values higher than the corresponding values during the cruise phase, the adjustment weight λ of the noise exposure penalty term is... N_adj Below the corresponding value during the cruise phase, preferably, during the vertical takeoff and landing phase λ. C_adj =1.5, λ N_adj =0.5, cruise phase λ C_adj =0.8, λ N_adj =1.2; During the cruise phase, the adjustment weights for path efficiency rewards and energy consumption penalties are higher than those during the vertical takeoff and landing phase. Preferably, during the cruise phase, λ... T_adj =1.5λ, λ E_adj =1.3λ, λ during vertical takeoff and landing T_adj =0.6, λ E_adj =0.8.
[0033] In addition, the mission type is automatically set by the flying car before takeoff based on the flight mission plan, or manually entered and adjusted by the operator. The flight phase is based on industry standards and is automatically determined according to the flying car's current altitude and current airspeed. Preferably, when the flying car is in the transition phase from the vertical take-off and landing phase to the cruise phase, the adjustment weights are smoothly transitioned by linear interpolation between the corresponding values of the vertical take-off and landing phase and the corresponding values of the cruise phase as the transition progresses.
[0034] In one embodiment of the present invention, the policy network and the configuration switching sub-network have the following parallel cooperative mechanism: During the pre-training process of the digital twin simulation platform, the output of the configuration switching sub-network serves as the conditional input to the action space of the policy network, thus limiting the thrust control output of the policy network to the physical control boundary under the current configuration mode. At each decision step, the configuration switching subnetwork outputs a target configuration pattern and its corresponding switching confidence based on the current state space. When the switching confidence exceeds the preset switching confidence limit, a configuration switching command is triggered. At each decision step, the policy network calculates the thrust control quantity of the flying car in each configuration mode based on the current state space. If in response to the configuration switching command, the output thrust control quantity is switched to the thrust control quantity corresponding to the target configuration mode through interpolation.
[0035] It should be noted that during the pre-training process of the digital twin simulation platform, the configuration switching subnetwork and the policy network share the underlying feature extraction layer. The output of the configuration switching subnetwork serves as the conditional input to the action space of the policy network, i.e., the thrust control quantity 'a' output by the policy network. control Must meet: a control ∈A(m) current ); Where A(m) current ) represents the feasible action space corresponding to the current configuration pattern.
[0036] In the actual flight decision-making process, within each decision step, the configuration switching subnetwork first determines the current state space s. t Output the switching confidence ρ of each candidate configuration mode m∈M m : ρ m = f switch (s t ;θ switch ); Where M represents the set of all configuration modes supported by the flying car, f switch For the function mapping of the configuration-switching subnetwork representation, θ switch Network parameters for configuration switching subnets; Policy network synchronization is based on the current state space s t Calculate the thrust control quantity a of the flying car under each candidate configuration mode m. m : a m = h(s t ,m;θ policy ); Where h is the function mapping of the policy network representation, θ policy These are the network parameters for the policy network; When a certain configuration mode m0 has ρ m Exceeding the preset switching confidence limit ρ threshold And m0 ≠ m current At that time, the configuration switching instruction c is triggered. switch = m0 indicates a request to switch from the current configuration mode to the target configuration mode m0.
[0037] In response to a configuration switching command, the thrust control output of the policy network smoothly transitions from the thrust control output corresponding to the current configuration mode to the thrust control output corresponding to the target configuration mode through interpolation. Once the target configuration is fully established, the policy network completely switches to the thrust control output of the target configuration mode.
[0038] In one embodiment of the present invention, a configuration switching cost function is introduced during the training of the configuration switching subnetwork, specifically including: Obtain the design life parameters and cumulative switching times of each configuration switching mechanism of the flying car. Based on the target configuration mode pointed to by the current configuration switching command, query the preset wear coefficient table of the switching mechanism, calculate the equivalent fatigue damage value of the switching mechanism for this switching as the mechanical wear cost. Obtain the instantaneous power increment required to perform configuration switching under the current flight speed and load conditions, and calculate the time integral of the instantaneous power increment exceeding the steady-state cruise power as the energy impact cost; The total cost of configuration switching is obtained by weighted summing of mechanical wear cost and energy impact cost. This total cost is then used as a negative reward term during the training of the configuration switching subnetwork and superimposed on the comprehensive reward function. In other words, at each decision step, the output value of the comprehensive reward function is this superimposed amount. When the configuration switching subnetwork maintains its current configuration, the total switching cost is zero.
[0039] It should be noted that, in this embodiment, the specific mechanism by which the total cost of configuration switching is superimposed as a negative reward is as follows: (1) Cost of mechanical wear: In advance, during the development phase of the flying car, accelerated life tests of the switching mechanism were conducted to obtain the mapping relationship between the equivalent fatigue damage of the switching mechanism and the cumulative number of switching operations after performing different configuration switching operations (including forward switching and reverse switching), thus forming a wear coefficient table for the switching mechanism. In actual flight, when the configuration switching subnetwork outputs a target configuration mode, the wear coefficient table of the switching mechanism is queried according to the switching path to obtain the unit switching wear coefficient corresponding to this switching. Then, the fatigue life correction of the unit switching wear coefficient is performed based on the cumulative number of switching of each switching mechanism, and the corrected unit switching wear coefficient is used as the mechanical wear cost of this switching.
[0040] (2) Cost of energy impact: Pre-calibrate the instantaneous peak power and power change curve required for the flying car to perform configuration switching under different flight speed ranges and load conditions; In actual flight, when a configuration switching command is triggered, the system queries the calibrated data based on the current flight speed and current load conditions to obtain the curve of the instantaneous power increment required for the current switching as a function of time, and calculates the power increment time integral over the duration during which the instantaneous power increment exceeds the steady-state cruise power, as the energy impact cost of this switching.
[0041] The total cost of configuration switching is obtained by weighted summation of mechanical wear cost and energy impact cost, with the weights of both being preset values and both being positive. During training, the total cost of configuration switching is superimposed on the comprehensive reward function as a negative reward term during the training of the configuration switching subnetwork. If the configuration switching subnetwork maintains the current configuration in the current decision step, the total cost of configuration switching is set to zero, and the superposition does not take effect.
[0042] As one embodiment of the present invention, the deviation between real-time sensing data and predicted data is calculated including position deviation index, velocity deviation index, heading deviation index and model mismatch index, wherein the model mismatch index is the maximum deviation between the obstacle position perceived in the real-time sensing data and the predicted position of the corresponding obstacle in the digital twin simulation. When any of the deviations exceeds its corresponding preset upper limit, continuous counting begins. Online fine-tuning is triggered only when the continuous count value reaches the preset trigger threshold. If all deviations fall below the corresponding preset upper limit of deviation in any control cycle during the continuous counting process, the continuous count value will be cleared to zero.
[0043] It should be noted that, in this embodiment, the position deviation index is the Euclidean distance between the actual three-dimensional position of the flying car at the current moment and the predicted position output by the digital twin model. The speed deviation index is the Euclidean distance between the actual three-dimensional speed vector of the flying car at the current moment and the predicted speed vector output by the digital twin model; The heading deviation index is the angle difference between the actual heading angle of the flying car at the current moment and the predicted heading angle output by the digital twin model, and it will be mapped to the interval [0,π]. The model mismatch index is the maximum Euclidean distance between the positions of each dynamic obstacle perceived in real-time sensor data and the predicted positions of the corresponding dynamic obstacles in the digital twin model. It is used to characterize the overall accuracy of the digital twin model in predicting the motion of dynamic obstacles.
[0044] Each deviation has a corresponding upper limit preset. The count value of continuous counting starts from zero. In each control cycle, the four deviations are judged in turn. If any one of them exceeds its corresponding upper limit, the count value is incremented by 1. If all four deviations fall back to below their respective upper limits within a certain control cycle, the count value is cleared to zero. Online fine-tuning is triggered only when k exceeds a preset trigger threshold (e.g., a positive integer between 3 and 10) to prevent false triggering of fine-tuning due to instantaneous sensor noise or brief communication fluctuations.
[0045] As one embodiment of the present invention, the step of online fine-tuning of the policy network and configuration switching sub-network based on the deviation includes: Freeze the feature extraction layer parameters of the policy network and the configuration switching subnetwork, and only update their decision layer parameters; The real-time sensor data collected during the current flight of the flying car, the corresponding policy network output, and the return value of the comprehensive reward function are used to construct an experience sample, which is then stored in the online experience replay pool. The degree to which each deviation exceeds the corresponding preset deviation limit is normalized and sorted. In each online fine-tuning, experience samples are extracted from the online experience replay pool using a weighted random sampling method for gradient update, where the sampling weight is proportional to the degree of deviation. The system monitors the satellite navigation signal strength and data link communication status of the flying car in real time. When the satellite navigation signal strength is detected to be lower than the preset lower limit or the data link communication is interrupted, the flying car is determined to have entered a communication denial environment. The online fine-tuning gradient update step size is reduced to a preset ratio of the normal update step size until communication is restored.
[0046] It should be noted that, in this embodiment, freezing the feature extraction layer parameters means fixing the weights of this part of the network layer during the fine-tuning process. It does not participate in gradient updates, but only updates the decision layer parameters, thereby reducing the number of parameters that need to be optimized in the online fine-tuning stage and accelerating the fine-tuning convergence speed.
[0047] The construction method of the experience sample is as follows: at each decision step, the real-time sensing data collected by the airborne sensing system at the current moment, the thrust control quantity output by the policy network at this decision step, the configuration switching command output by the configuration switching sub-network at this decision step, and the instantaneous reward value given by the comprehensive reward function for the current state-action pair are combined into a complete experience sample and stored in the online experience replay pool.
[0048] The specific implementation of the weighted random sampling method is as follows: when online fine-tuning is triggered, a preset number of experience samples are selected from the online experience replay pool, and the deviation values corresponding to each experience sample are summarized. The one with the largest deviation is taken as the deviation representative value of the experience sample. The deviation representative values of all experience samples are normalized so that the sampling probability of each experience sample is proportional to its normalized deviation representative value.
[0049] The mechanism for dealing with communication denial environments is as follows: the airborne flight control system continuously monitors the signal strength of the satellite navigation receiver and the connection status of the data link communication module. When it is determined that the flying car has entered a communication denial environment, the gradient update step size is reduced to a preset proportion of the normal value, preferably 10% to 30% of the normal value, in order to avoid increased positioning error due to loss of navigation signal or data packet loss and contamination of experience samples due to communication interruption.
[0050] In one embodiment of the present invention, the step of having an emergency obstacle avoidance module independent of the policy network temporarily take over local control includes: Real-time monitoring of the convergence speed of gradient updates in online fine-tuning, using the average rate of decrease of the loss function over N consecutive control cycles as a measure of the fine-tuning response speed; When the fine-tuning response speed is lower than the preset response speed lower limit and the duration exceeds the preset tolerance duration upper limit, the emergency obstacle avoidance module temporarily takes over the local control of the flying car. During this period, the strategy network and configuration switching sub-network continue to operate in the background. When it is detected that the average value of the loss function of online fine-tuning continues to decrease and all deviation indicators fall back to below the corresponding preset deviation upper limit within M consecutive control cycles, the local control is returned. In addition, if the emergency obstacle avoidance module takes over control for a duration exceeding the preset takeover duration limit, it will guide the flying car to land.
[0051] It should be noted that the "loss function" is a function used to measure the difference between the output of the policy network and the expected deviation during the online fine-tuning process. The smaller the value, the better the policy network performs in the current environment.
[0052] The preset values for the lower limit of response speed and the upper limit of tolerance time are pre-set and stored in the airborne flight control system according to the type of flying car, mission scenario and safety requirements.
[0053] When it is determined that the emergency obstacle avoidance module shall take over local control, the emergency obstacle avoidance module generates obstacle avoidance control commands using the artificial potential field method or the dynamic window method, and directly issues them to the attitude controller and actuators of the flying car. The priority of these obstacle avoidance control commands is higher than the priority of the output commands of the policy network and the configuration switching sub-network.
[0054] During the period when the emergency obstacle avoidance module takes over, the strategy network and configuration switching subnetwork continue to operate, continuously performing forward inference and parameter fine-tuning calculations in the background using the real-time sensor data received at the current moment. However, their output commands do not participate in the actual flight control and are only used to update network parameters and detect recovery conditions.
[0055] The recovery condition is determined as follows: During each control cycle of the takeover period, the background continuously calculates the loss function for online fine-tuning and checks whether all deviations have fallen back to below the corresponding preset deviation upper limit; When the average value of the loss function shows a continuous downward trend over M consecutive control cycles, and all deviations fall back below the corresponding preset deviation upper limit, the response speed of the policy network is determined to have recovered to a reliable level, and control is returned to the policy network and the configuration switching sub-network.
[0056] In addition, during the takeover period of the emergency obstacle avoidance module, the system continuously accumulates the duration of its takeover. If the accumulated duration exceeds the preset takeover duration limit, it indicates that the policy network cannot restore effective control capabilities within a reasonable time. The airborne flight control system immediately sends an emergency landing command to the flight control computer of the flying car, guiding the flying car to the nearest safe landing point and executing a vertical descent landing procedure to ensure flight safety.
[0057] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.
Claims
1. A dynamic path planning method for urban airspace of flying cars based on reinforcement learning, characterized in that, The method includes: S1: Acquire global low-altitude environmental data of the city, construct a three-dimensional composite environmental model including a static obstacle layer, a dynamic weather fence layer, and a moving target prediction trajectory layer, define the state space including the variable configuration mode of the flying car, and the action space including thrust control variables and configuration switching commands. S2, construct a comprehensive reward function based on the state space and action space, the comprehensive reward function includes at least an energy consumption penalty term, a noise exposure penalty term, a collision safety constraint term, and a path efficiency reward term; S3, using the three-dimensional composite environment model as the simulation environment, defining the reinforcement learning training task together with the state space and the comprehensive reward function, and pre-training a policy network for path planning and a configuration switching sub-network connected in parallel with the policy network through the digital twin simulation platform; S4. During the actual flight of the flying car, real-time sensor data of the surrounding environment is acquired. At the same time, the predicted data is output through simulation by digital twin model. The deviation between the two is calculated, and the strategy network and configuration switching sub-network are fine-tuned online based on the deviation. When the response speed of the online fine-tuning is continuously lower than the preset lower limit, the emergency obstacle avoidance module, which is independent of the strategy network, temporarily takes over the local control. S5 outputs thrust control quantities through the fine-tuned strategy network and configuration switching commands through the configuration switching subnetwork. Combining the current pose of the flying car with the target location, it generates a three-dimensional feasible path. S4 and S5 are executed repeatedly in each control cycle until the flying car reaches the target location.
2. The method for dynamic path planning in urban airspace for flying cars based on reinforcement learning according to claim 1, characterized in that: The steps involved in constructing a three-dimensional composite environment model include: The city's low-altitude lidar point cloud data and building vector contour data are acquired, registered to a unified geographic coordinate system, and subjected to three-dimensional rasterization. The raster containing buildings, towers, and high-voltage line corridors is marked as occupied, forming the static obstacle layer. Real-time wind field data from meteorological satellites and precipitation radar data are acquired, and meteorological hazard boundaries are dynamically delineated under a unified geographic coordinate system. The grids within the meteorological hazard boundaries are marked as dynamic fences, forming the dynamic meteorological fence layer. The system acquires broadcast ADS-B historical trajectory data of surrounding aircraft, predicts the position probability distribution of each surrounding aircraft within a preset time period in the future, and marks the grids with probability density exceeding the preset probability density upper limit as potential conflict states, thus forming the moving target prediction trajectory layer. The static obstacle layer, dynamic weather fence layer, and moving target prediction trajectory layer are superimposed and fused to generate the three-dimensional composite environment model.
3. The method for dynamic path planning in urban airspace for flying cars based on reinforcement learning according to claim 2, characterized in that: The steps for constructing the comprehensive reward function based on the state space and action space include: The energy consumption penalty is calculated in real time based on the current wind speed drag coefficient and the energy consumption curve per unit distance of the flying car in the current configuration mode; The noise exposure penalty is calculated based on the noise sensitivity level coefficient corresponding to the grid where the flying car is currently located. The collision safety constraint is calculated based on the nearest three-dimensional Euclidean distance between the grid where the flying car is currently located and the grids in the static obstacle layer and the dynamic weather fence layer. The path efficiency reward is calculated based on the reduction in the remaining straight-line distance between the flying car's current location and the target location at each decision step of the reinforcement learning process.
4. The method for dynamic path planning in urban airspace for flying cars based on reinforcement learning according to claim 3, characterized in that: The weighting coefficients of each penalty and reward item in the comprehensive reward function are adaptively adjusted according to the current mission type and current flight phase. The weighting coefficients include a base weight and an adjustment weight. The adaptive adjustment includes the following steps: Based on the current mission type of the flying car, a set of basic weights corresponding to the mission type is selected from a set of preset basic weight configuration schemes. The mission type includes at least safety-sensitive missions and economic-sensitive missions. Under safety-sensitive missions, the basic weight of the collision safety constraint item is greater than the basic weight of the path efficiency reward item. Under economic-sensitive missions, the basic weight of the energy consumption penalty item is greater than the value under safety-sensitive missions. Based on the current flight phase of the flying car, further adjustment weights are applied to the basic weight configuration. The flight phase includes at least a vertical take-off and landing phase and a cruise phase. In the vertical take-off and landing phase, the adjustment weight of the collision safety constraint is higher than that in the cruise phase, and the adjustment weight of the noise exposure penalty is lower than that in the cruise phase. In the cruise phase, the adjustment weights of the path efficiency reward and energy consumption penalty are higher than those in the vertical take-off and landing phase.
5. The method for dynamic path planning in urban airspace for flying cars based on reinforcement learning according to claim 1, characterized in that: The policy network and the configuration switching sub-network have the following parallel coordination mechanism: During the pre-training process of the digital twin simulation platform, the output of the configuration switching sub-network is used as the conditional input of the action space of the policy network, so that the thrust control quantity output by the policy network is limited by the physical control boundary of the current configuration mode. At each decision step, the configuration switching subnetwork outputs a target configuration pattern and its corresponding switching confidence based on the current state space. When the switching confidence exceeds a preset switching confidence upper limit, a configuration switching command is triggered. At each decision step, the policy network calculates the thrust control quantity of the flying car in each configuration mode according to the current state space. If in response to the configuration switching command, the output thrust control quantity is switched to the thrust control quantity corresponding to the target configuration mode through interpolation.
6. The method for dynamic path planning in urban airspace for flying cars based on reinforcement learning according to claim 1, characterized in that: The configuration switching subnetwork is trained using a configuration switching cost function, which includes: Obtain the design life parameters and cumulative switching times of each configuration switching mechanism of the flying car. Based on the target configuration mode pointed to by the current configuration switching command, query the preset wear coefficient table of the switching mechanism, calculate the equivalent fatigue damage value of the switching mechanism for this switching as the mechanical wear cost. Obtain the instantaneous power increment required to perform configuration switching under the current flight speed and current load conditions, and calculate the time integral of the instantaneous power increment exceeding the steady-state cruise power as the energy impact cost; The mechanical wear cost and the energy impact cost are weighted and summed to obtain the total configuration switching cost. This total cost is then used as a negative reward term during the training of the configuration switching sub-network and superimposed on the comprehensive reward function. That is, at each decision step, the output value of the comprehensive reward function is this superposition amount. When the configuration switching sub-network maintains its current configuration, the total switching cost is zero.
7. The method for dynamic path planning in urban airspace for flying cars based on reinforcement learning according to claim 1, characterized in that: The calculation of the deviation between the real-time sensing data and the predicted data includes position deviation index, velocity deviation index, heading deviation index and model mismatch index, wherein the model mismatch index is the maximum deviation between the obstacle position perceived in the real-time sensing data and the predicted position of the corresponding obstacle in the digital twin simulation. When any of the deviations exceeds its corresponding preset upper limit, continuous counting begins; The online fine-tuning is triggered only when the continuous count value reaches the preset trigger threshold. If, during any control cycle in the continuous counting process, all deviations fall back to below the corresponding preset upper limit of deviation, the continuous count value is cleared to zero.
8. The method for dynamic path planning in urban airspace for flying cars based on reinforcement learning according to claim 7, characterized in that: The steps for online fine-tuning of the policy network and configuration switching sub-network based on the aforementioned deviations include: Freeze the feature extraction layer parameters of the policy network and the configuration switching sub-network, and only update their decision layer parameters; The real-time sensor data collected during the current flight of the flying car, the corresponding policy network output, and the return value of the comprehensive reward function are used to construct an experience sample, which is then stored in the online experience replay pool. The degree to which each deviation exceeds the corresponding preset deviation upper limit is normalized and sorted. In each online fine-tuning, experience samples are extracted from the online experience replay pool using a weighted random sampling method for gradient update, wherein the sampling weight is proportional to the degree of deviation. The system monitors the satellite navigation signal strength and data link communication status of the flying car in real time. When the satellite navigation signal strength is detected to be lower than the preset lower limit or the data link communication is interrupted, the flying car is determined to have entered a communication denial environment. The online fine-tuning gradient update step size is reduced to a preset ratio of the normal update step size until communication is restored.
9. The method for dynamic path planning in urban airspace for flying cars based on reinforcement learning according to claim 8, characterized in that: The steps involving the emergency obstacle avoidance module, independent of the policy network, temporarily taking over local control include: The convergence speed of the gradient update in the online fine-tuning is monitored in real time, and the average rate of decrease of the loss function over N consecutive control cycles is used as a measure of the fine-tuning response speed. When the fine-tuning response speed is lower than the preset lower limit of response speed and the duration exceeds the preset upper limit of tolerance time, the emergency obstacle avoidance module temporarily takes over the local control of the flying car. During this period, the strategy network and configuration switching sub-network continue to operate in the background. When it is detected that the average value of the loss function of the online fine-tuning continues to decrease and all deviation indicators fall back to below the corresponding preset deviation upper limit within M consecutive control cycles, the local control is returned. Furthermore, if the duration of the emergency obstacle avoidance module taking over control exceeds the preset takeover duration limit, it will guide the flying car to land.
