A method and system for stratospheric airship path optimization by fusing time-series modeling and reinforcement learning

Abstract

This application provides a method and system for stratospheric airship path optimization that integrates temporal modeling and reinforcement learning. The method includes: acquiring three-dimensional wind field sequence data at multiple time steps and multiple altitude levels; processing the wind field sequence data using a Transformer encoder to extract wind field feature vectors describing its spatiotemporal evolution trend; acquiring the airship's current state information, including its position, velocity, and energy state; fusing the wind field feature vectors with the airship's current state information to form an enhanced state representation; and inputting the enhanced state representation into a trained deep reinforcement learning policy network to generate action commands to control the airship's motion. The advantages of this application are: it can simultaneously consider the complex temporal and spatial correlations of wind fields; it significantly improves the airship's dwell performance and robustness under dynamic wind fields; and it possesses stronger environmental adaptability and energy efficiency.

Description

Technical Field

[0001] This application belongs to the field of airship path planning, specifically involving a method and system for optimizing the path of stratospheric airships by integrating temporal modeling and reinforcement learning. Background Technology

[0002] High-altitude airships are typical near-space flight platforms, typically operating at altitudes between 19 and 22 km. They offer advantages such as low cost, flexible deployment, long endurance, and high payload capacity, making them promising for applications in communication relay, Earth remote sensing, environmental monitoring, and military reconnaissance. To meet the demands of continuous observation and information acquisition, airships need to achieve long-term stable loitering in specific areas. The ability to plan loitering paths within these areas is crucial; however, the complex and variable wind environment at high altitudes, coupled with the platform's limited energy resources, severely restricts the sustainability and reliability of their loitering missions. Airships are large in size and have a wide windward area, making them susceptible to high-altitude wind disturbances. High-altitude wind fields exhibit significant non-stationarity and inter-layer inconsistencies, with wind speed and direction showing significant variations over time and at altitude. Furthermore, airships typically rely on limited energy supplies such as solar power, and energy allocation must balance propulsion, control, and payload systems, making energy constraints particularly prominent. Therefore, there is an urgent need for intelligent path planning methods that address the spatiotemporal variations of wind fields while effectively handling energy constraints, in order to improve the stability of airships and the efficiency of mission execution.

[0003] In traditional path planning research, graph search methods (such as A*, D, and Dijkstra's algorithms), evolutionary optimization algorithms (such as genetic algorithms, simulated annealing, and particle swarm optimization), and sampled planning methods (such as RRT, RRT, and PRM) are widely used for path generation in static or simplified environments. However, these methods generally suffer from performance bottlenecks when dealing with high-dimensional, dynamically changing, and highly uncertain environments. Specifically, graph search algorithms such as A* and Dijkstra's computational cost increases dramatically in large-scale maps and struggle to meet the real-time requirements of dynamic environments. Sampled methods such as RRT and its improvements (RRT*, Informed RRT*, etc.) perform better in high-dimensional spaces, but still face problems such as slow convergence, suboptimal solutions, and sensitivity to high-density obstacles. Furthermore, PRM's efficiency drops significantly in high-obstacle-density environments and it struggles to adapt to rapid environmental changes. In dynamic environments, frequent replanning is still often required, leading to increased response latency and computational burden. In summary, while traditional path planning methods are highly reliable in simple scenarios, they suffer from high computational complexity, poor adaptability, and insufficient real-time performance in high-dimensional, dynamic, and uncertain environments. They are unable to effectively adapt to the dynamic evolution and uncertain disturbances of wind fields, leading to problems such as path deviation, excessive energy consumption, and decreased dwell performance.

[0004] The rapid development of deep learning and reinforcement learning has provided new approaches to solving path planning problems in complex dynamic environments. Reinforcement learning, as an intelligent method that learns optimal strategies through interaction with the environment, has demonstrated powerful decision-making capabilities in complex dynamic systems such as robot control and autonomous driving. In the field of stratospheric airship path planning, many reinforcement learning-based path planning algorithms have emerged. However, reinforcement learning methods generally suffer from a core limitation: the decision-making basis for the strategy still mainly relies on the current or short-term predicted wind field state, lacking systematic modeling and long-term consideration of future wind field evolution trends. This "short-sighted" strategy makes it difficult for airships to perform globally optimal path planning in complex wind fields, thus affecting their dwell performance and energy efficiency. Therefore, a novel method integrating temporal wind field modeling and reinforcement learning decision-making is urgently needed to improve the perception of future wind field changes and achieve forward-looking, global, and adaptive path planning. Summary of the Invention

[0005] The purpose of this application is to overcome the shortcomings of existing technologies, which still rely mainly on the current or short-term predicted wind field conditions for decision-making, and lack systematic modeling and long-term consideration of future wind field evolution trends.

[0006] To achieve the above objectives, this application proposes an airship path planning method that integrates temporal modeling and reinforcement learning, including: Acquire three-dimensional wind field sequence data with multiple time steps and multiple height levels; The wind field sequence data is processed using a Transformer encoder to extract wind field feature vectors that describe its spatiotemporal evolution trend; Obtain the current status information of the airship, including the airship's position, speed, and energy status; The wind field feature vector is fused with the airship's current state information to form an enhanced state representation; The enhanced state representation is input into the trained deep reinforcement learning policy network to generate action commands to control the airship's movement.

[0007] As an improvement to the above method, the processing of the wind field sequence data using a Transformer encoder includes: A time-height dual position encoding mechanism is adopted to simultaneously encode the position information of each data point in the wind field sequence data in both the time dimension and the vertical height dimension, enabling the Transformer encoder to capture the wind field co-evolution law and temporal dynamics between vertical heights. The input of the Transformer encoder Represented as: ; in, The wind field vector input to the encoder is represented as: ; in, This is obtained by flattening the temporal and spatial dimensions of the original wind field tensor into a unified sequence; T Indicates the time step; H Indicates the number of floors in the vertical height; For learnable parameters, Indicates the model dimension; For position encoding, it is represented as: ; in, Indicates the position index of the sequence; Indicates time location encoding: ; ; in, k Indicates a dimension index; t Indicates time; Spatial location encoding: ; ; in, h Indicates the vertical height of the layer.

[0008] As an improvement to the above method, the current status information of the airship also includes: the airship's three-dimensional coordinates, heading angle, vertical speed, and the coordinates of the stationary target position.

[0009] As an improvement to the above method, the fusion of the wind field feature vector with the current state information of the airship includes: The wind field feature vector is concatenated with the current state information of the airship to form a single, higher-dimensional state vector.

[0010] As an improvement to the above method, the deep reinforcement learning policy network is trained and optimized using a proximal policy optimization algorithm.

[0011] As an improvement to the above method, the training process of the deep reinforcement learning policy network employs a multi-objective reward function, wherein the reward function... Includes: Target Residency Incentive Progress rewards for moving towards the target Energy consumption penalty and security boundary penalties : ; ; ; ; ; in, This represents the reward value continuously given when the airship's main current is within the target range; Indicates the airship's current position; Indicates the target location; Indicates the acceptable target tolerance; This represents the scaling factor that balances the progress reward magnitude with other reward components; This indicates the current battery energy status of the airship. This indicates the airship's battery's maximum energy state; Indicator functions that represent boundary violation events; This refers to an excessive angular velocity or acceleration that could jeopardize the stability of the airship. and These are the weighting coefficients.

[0012] As an improvement to the above method, the action command is a multi-dimensional continuous action vector, which includes the airship's horizontal propulsion speed, heading angle, and vertical heave speed, so as to realize the dynamic adjustment of the airship in three-dimensional space.

[0013] This application also provides an airship path planning system that integrates temporal modeling and reinforcement learning, implemented based on the above method, including: The wind field modeling module receives wind field sequence data from multiple time steps and multiple altitude layers, and uses a Transformer encoder that integrates time-altitude dual position encoding mechanism to generate wind field feature vectors. The status acquisition module is used to acquire the current status information of the airship; The state fusion module is used to fuse the wind field feature vector with the airship's current state information to generate an enhanced state representation; The decision control module is used to input the enhanced state representation into the trained deep reinforcement learning-based policy network and output action commands to control the movement of the airship.

[0014] Compared with existing technologies, the advantages of this application are: 1. Temporal Feature Modeling Mechanism for Multi-Level Wind Fields: Addressing the non-stationary nature and strong cross-altitude variations of wind fields in airship missions, this application introduces a Transformer model for the first time to model multi-level wind field sequences over the next 12 hours. To effectively capture the co-evolutionary patterns and temporal dynamics of wind fields across vertical altitudes, a "time-altitude dual-position encoding mechanism" is designed. This mechanism enables the model to simultaneously consider the complex temporal and spatial relationships of wind fields, and the output wind field latent vector serves as part of the airship's state, providing richer environmental priors for reinforcement learning strategies.

[0015] 2. Coupling and Fusion of Transformer Perception and Reinforcement Learning Strategies: This application uses the wind field latent vector output by the Transformer encoder as a state enhancement vector, which is input into the policy network along with the airship's current state information. This "perception-decision integration" mechanism enables the policy network to make more forward-looking control actions based on future wind field trends, thereby significantly improving the airship's dwell performance and robustness under dynamic wind fields.

[0016] 3. A temporal wind-driven reinforcement learning framework supporting dynamic altitude control strategies: This application designs a multi-dimensional action space for airship altitude control, including horizontal movement and altitude adjustment. The reinforcement learning strategy can dynamically adjust the airship's altitude based on the Transformer's output to find the optimal altitude layer with low energy consumption, stable wind field, and strong dwell capability. This enables the airship to achieve "adaptive dwell" in a three-dimensional wind field, exhibiting stronger environmental adaptability and energy efficiency compared to traditional fixed-altitude or experience-based layer selection methods. Attached Figure Description

[0017] Figure 1 The diagram shows a flowchart of a stratospheric airship path optimization method that integrates temporal modeling and reinforcement learning. Figure 2 The figure shows the average reward curve of the policy network during the training process; Figure 3 The diagram shows a comparison of the Transformer-PPO and standard PPO strategies during a 72-hour stay under typical disturbed wind conditions. Detailed Implementation

[0018] The technical solution of this application will be described in detail below with reference to the accompanying drawings.

[0019] This application proposes a method and system for stratospheric airship path optimization that integrates temporal modeling and reinforcement learning. The Transformer model, with its superior performance in sequence modeling, particularly in capturing long-term dependencies, has achieved groundbreaking progress in fields such as natural language processing and computer vision. Combining the Transformer with reinforcement learning endows the agent with the ability to perceive future environmental changes, thereby significantly improving its control performance in dynamic, high-dimensional, and uncertain environments.

[0020] Example 1 like Figure 1 As shown, the stratospheric airship path optimization method integrating temporal modeling and reinforcement learning proposed in this application includes: 1. Environmental simulation modeling; Considering the simplified dynamics of the airship as a point mass in space, its motion is influenced by its own propulsion, directional control, and the ambient wind field. Let the current position of the airship be... ， The speed is the sum of the self-propulsion speed and the wind speed. Assume the airship's speed at any given moment... Horizontal heading angle and vertical velocity Then its position change is described by the following kinematic equations: (1) in, Represents the spatial location of the three-dimensional wind field and time The wind speed vector is shown below. This wind field is derived from a numerical weather prediction model, and it exhibits significant non-uniformity and time-varying characteristics at different altitudes and time periods.

[0021] Considering that actual wind fields contain nondeterministic disturbances, especially in the stratospheric region where they are significantly affected by factors such as temperature gradients, cyclonic activity, and gravity waves, wind speed fluctuations exhibit a non-negligible randomness. To simulate this characteristic, this method uses a base wind field... A Gaussian white noise perturbation model is introduced to construct the perturbation wind field: (2) in, This is a three-dimensional Gaussian noise term, used to characterize the effects of wind speed measurement errors, prediction errors, or small-scale disturbances. The covariance matrix is ​​set as a diagonal matrix. (3) in These represent the standard deviations of wind speed disturbances in the three spatial dimensions. In this method, based on existing literature and statistical analysis of actual meteorological data, [the following is used:] As the default setting, this perturbation model applies independent and identically distributed perturbations to the wind speed at each simulation time step to simulate stochastic wind field changes and improve the generalization performance of the path planning strategy in non-ideal environments.

[0022] Substituting the perturbed wind field, the airship motion model is as follows: (4) This modeling approach effectively integrates flight control inputs with complex environmental disturbances, forming a dynamic system with stochastic and non-stationary characteristics. This provides a more challenging training environment for reinforcement learning strategies. By training the agent in diverse, disturbed wind fields, its policy robustness under complex weather conditions can be significantly improved, thus enabling it to better achieve mission objectives such as stationary deployment.

[0023] In long-duration, continuous flight missions, the energy management mechanism of high-altitude airships differs significantly from that of traditional ground-based or short-duration flight platforms. Their path planning strategies must be closely integrated with the energy acquisition and consumption processes. Unlike most path planning missions that only consider navigation performance, airship missions often last for several days, or even more than a week, and their internal battery capacity is insufficient to cover the propulsion and system maintenance energy required for the entire mission cycle. Therefore, airships must possess a sustainable energy acquisition mechanism, typically relying on onboard solar panels for energy replenishment. The availability of solar energy exhibits significant temporal periodicity, influenced by changes in the solar altitude angle, showing marked differences at different times of the day, and is completely interrupted at night. This characteristic creates a strong coupling relationship between energy supply and demand and path selection.

[0024] To accurately characterize this energy cycle, this method constructs a dynamic model that includes energy generation and consumption. First, the airship must overcome air resistance to maintain a constant speed during flight, and its propulsion energy consumption increases cubically with speed. The ground velocity vector at the current moment is defined as... Then the motor power consumption can be modeled as: (5) in, This represents the power coefficient related to the airship's aerodynamic characteristics and propulsion system efficiency. Furthermore, the power consumption of the airship's avionics equipment (including communication, navigation, and sensor subsystems) is also considered, as these operate continuously throughout the mission lifecycle. It is approximately a constant.

[0025] In terms of energy input, the airship mainly relies on solar energy. Let the solar irradiance be... The photoelectric conversion efficiency is Then the charging power per unit time is: (6) in, It is the solar energy efficiency factor, representing the proportion of solar radiation available at the current time, and is used to introduce a periodic function approximation for simulating circadian rhythms. (7) in This indicates the length of daylight (typically 12 hours). This refers to sunrise time. The above modeling enables continuous modeling of the variation in solar input power during the day-night cycle.

[0026] (8) This energy model introduces an energy constraint mechanism into path planning, requiring reinforcement learning agents to balance path quality and energy feasibility during policy learning. This drives the evolution of flight strategies towards low-energy consumption, high efficiency, and day-night coordination, meeting the requirements of high-altitude airship stationing missions for continuity and robustness.

[0027] 2. Temporal feature modeling for multi-layered wind fields; One of the biggest challenges for airships to remain in high-altitude environments is the complex and ever-changing wind field. High-altitude wind fields not only exhibit significant non-stationarity, but their wind speed and direction also show strong variations and complex co-evolutionary relationships across different vertical altitudes. Traditional wind field modeling methods often struggle to effectively capture these complex spatiotemporal dynamics. To address this issue, this application introduces the Transformer model to model multi-layered wind field sequences over the next 12 hours. The Transformer model, with its self-attention mechanism, can effectively capture long-range dependencies within the sequence.

[0028] Original wind field tensor A series of transformations are performed to create an input representation suitable for the Transformer encoder. The tensor is first reshaped into a sequence format: (9) in, T Indicates the time step (one time step per hour for the next 12 hours). H This represents the number of vertical height layers (21 different height layers in this method). This reshaping operation flattens the temporal and spatial dimensions into a unified sequence, enabling the Transformer to treat spatiotemporal patterns as sequential dependencies. Each spatiotemporal point is then projected into a high-dimensional embedding space through a learnable linear transformation. (10) in, The wind field vector is the input encoder; These are learnable parameters. This represents the model dimension. This projection layer, acting as a feature expansion mechanism, allows the model to learn rich representations of wind field features.

[0029] 2.1 Time-altitude dual location coding mechanism; Unlike standard Transformer applications that use positional encoding to process time series data, traditional Transformer positional encoding typically only considers the time dimension, neglecting the vertical spatial correlation of multiple wind fields. This application requires the wind field data to simultaneously encode both temporal and spatial (height) positional information. This application proposes an innovative "Time-Height Dual Positional Encoding" (THDPE) mechanism, enabling the model to simultaneously capture the co-evolutionary patterns and temporal dynamics of wind fields across different vertical heights. It preserves the independent physical meaning of the time and spatial dimensions, allowing the model to learn these two dependencies decoupledly. THDPE achieves this by superimposing two independent positional encodings: Temporal location encoding: Used to encode the relative or absolute location information of wind field sequences in the time dimension, capturing the temporal evolution pattern of the wind field.

[0030] (11) (12) in, k This represents the dimension index, which is used to traverse all dimensions of the location encoding and assign different frequencies to each dimension, allowing the model to distinguish location information at different scales. t Indicates time.

[0031] Spatial location encoding: Specifically used to encode the location information of wind fields in the vertical dimension. Since there are physical correlations between wind fields at different altitudes, introducing altitude location encoding helps the model learn the co-evolutionary patterns in these vertical directions. It models the altitude-related characteristics of the atmosphere.

[0032] (13) (14) in, h Indicates the vertical height of the layer.

[0033] Complete location encoding combining temporal and spatial components: (15) in, This represents the sequence position index. The final input of the Transformer encoder contains both content and position information: (16) This dual-location encoding mechanism enables the Transformer model to not only understand the temporal trend of wind field changes when processing wind field sequences, but also to perceive the interactions between different altitude layers. This allows for more accurate extraction of feature information from future multi-layered wind fields, enabling the model to simultaneously perceive both temporal trends and spatial stratification within the wind field data.

[0034] 2.2 Transformer encoder; To accurately capture the dynamic characteristics of wind field variations over time and altitude, this application introduces a spatiotemporal feature encoder based on a Transformer architecture. This encoder aims to extract powerful latent representations suitable for policy learning from forecast wind field sequences, thereby assisting airships in robust path decision-making in dynamically disturbed environments. The employed Transformer encoder possesses strong long-term modeling capabilities and learns wind field dependency patterns at different scales through a multi-head attention mechanism.

[0035] Each encoder layer consists of two main sublayers: a multi-head self-attention mechanism and a feed-forward network (FFN), both of which employ residual connections and layer normalization for stable training. The multi-head attention mechanism allows the model to fully reference features from other locations in the sequence when constructing the representation for each spatiotemporal location, thereby capturing the global structure and long-range dependencies in the wind field sequence.

[0036] Attention calculation is performed using a scaled dot product: (17) Q, K, and V are vectors generated from the input features through independent learnable projections. They are the query vector, key vector, and value vector, respectively, and form the basis for attention computation. In self-attention, all three come from the same input. The number of attention heads indicates the number of attention heads. This multi-head design allows the model to simultaneously focus on different types of wind field patterns, such as temporal trends, spatial gradients, and local turbulence features. This multi-head design enables the model to capture different types of wind field dependencies in parallel across multiple attention subspaces, such as wind speed trends, height changes, and short-term disturbances, effectively enhancing the model's overall perception of wind field patterns.

[0037] Subsequently, the feedforward network applies a non-linear mapping to the attention output, further enhancing the model's expressive power: (18) in, ,generally Residual connections and layer normalization ensure stable training and efficient information flow within the network.

[0038] go through After processing by each encoder layer, the Transformer generates a contextualized representation for each spatiotemporal location. To generate a fixed-size feature vector suitable for the downstream policy network, global average pooling is applied: (19) in, Indicates the first Layer encoder for position The output of this aggregation strategy ensures that the final wind field representation captures the global spatiotemporal pattern while maintaining computational efficiency. The feature vectors contain rich representations of the multi-layered wind field sequence for the next 12 hours, and the latent vectors... This will be part of the airship's state, providing richer environmental priors for reinforcement learning strategies.

[0039] 3. Algorithm design for Transformer perception and reinforcement learning strategies; To achieve integrated perception and decision-making for the airship, this application fuses the wind field latent vector output by the Transformer encoder with the airship's current state information, and inputs both into a reinforcement learning policy network. This tight coupling enables the policy network to fully utilize predictive information about future wind fields, thereby making more forward-looking and robust control decisions.

[0040] 3.1 Integration Strategy and Value Network Architecture; In terms of state construction, this method fully integrates environmental prediction features with the current state information of the airship platform. The wind field encoder uses the global wind field features extracted by the Transformer to represent... This reflects the temporal wind speed structure of the area where the airship is located at the current moment. The airship's own state vector... This includes mission-related physical quantities such as its three-dimensional position, stationary target, velocity, remaining energy level, and heading. Concatenating these two values ​​yields the final complete state vector. (20) This state representation not only preserves the current observable mission state of the airship, but also introduces a structural understanding of the future environment, giving the policy network the ability to make long-term decisions in complex environments.

[0041] In terms of network architecture, the policy network and value network share the Transformer feature extractor, ensuring consistency in their understanding of the environment, and complete their respective tasks through separate output branches. This structural design not only improves the training efficiency of the model but also facilitates the transfer and generalization of feature representations between policy learning and state estimation. The policy network uses a multi-layer perceptron (MLP) with softmax output to model the complete state, thus outputting in the continuous action space. The probability distribution function on: (twenty one) in, It is a 3-layer fully connected network with a dimension of 128.

[0042] 3.2 PPO Optimization Based on Pruning Objectives; To achieve efficient and stable policy updates, this method selects the Proximal Policy Optimization (PPO) algorithm as the core optimizer for reinforcement learning. Proximal Policy Optimization (PPO), a deep reinforcement learning algorithm proposed by OpenAI in 2017, introduces a probability ratio pruning mechanism, effectively limiting the magnitude of policy changes during iteration, thus alleviating the performance degradation problem caused by over-updating in traditional policy gradient algorithms. Compared to other reinforcement learning methods in continuous action spaces, such as SAC (Soft Actor-Critic algorithm) and DDPG (Deep Deterministic Policy Gradient), PPO is more stable, solving the challenge of stable policy updates in high-dimensional action spaces. It is particularly suitable for continuous optimization tasks in high-dimensional state spaces.

[0043] The objective function for PPO optimization is defined as follows: (twenty two) in, It is a time step t The expected value is that the clip function limits the policy update magnitude to [1-]. ,1+ Within the specified range, anything exceeding this range will be truncated. This represents the probability ratio between the old and new strategies. θ It is the set of learnable parameters for the policy network. Advantage estimation measures the relative superiority of a current action compared to a mean strategy. It employs generalized advantage estimation (GAE) to reduce variance in advantage calculation and improve the stability of strategy estimation.

[0044] (twenty three) in, This represents the time difference error. Parameter As a discount factor for future returns, and The trade-off between controllable bias and variance is addressed. PPO, which combines advantage estimation with policy pruning, demonstrates excellent sample efficiency and policy stability, making it particularly suitable for long-term, multivariate coupled airship dwell path planning tasks.

[0045] 3.4 State-space design; In a reinforcement learning-based path planning framework, the design of the state space directly impacts the effectiveness of policy learning and environmental adaptability. A reasonable state representation should not only fully depict the airship's current kinematic state but also accurately capture external disturbances affecting its flight trajectory, especially the wind field environment with significant spatiotemporal coupling. Therefore, this method constructs a composite state space that integrates airship dynamics and wind field dynamics information to enhance the policy network's environmental awareness and decision-making foresight.

[0046] Airship state vector Used to describe time The basic kinematic characteristics and mission objectives of the airship are defined as follows: (twenty four) in, This indicates the current position of the airship in the global geographic coordinate system. For heading angle, This indicates the horizontal propulsion speed relative to the airflow; , Indicates energy state, This indicates the coordinates of the specified target location. This representation method provides basic information for trajectory planning while maintaining computational efficiency. It is suitable for policy search in continuous action spaces.

[0047] Wind field, as the most significant external disturbance source during airship flight, has a decisive impact on path planning strategies due to its spatiotemporal variations. To effectively model this dynamic environment, this method introduces a three-dimensional wind field tensor. Characterizing the time-series evolution and vertical hierarchical structure of wind speed. Among them, Indicates the length of the time series for the forecast wind field. This indicates the number of vertical height layers considering height-related wind variations; factor 3 corresponds to the wind speed component. .

[0048] The wind field tensor element is defined as: (25) in, Indicates a time index. Indicates the height layer index. Corresponding to the This tensor effectively captures the evolution of wind speed across different time scales and vertical structures, enabling the policy network to learn potential disturbance patterns and their impact on path planning.

[0049] Ultimately, the complete system state is determined at time step [number missing]. t Defined as the following union structure: (26) This composite state dynamically integrates the airship's current mission state with the spatiotemporal wind field, providing a highly informative input for policy gradient-based methods, which helps improve the policy's generalization ability and wind field adaptability.

[0050] 3.5 Airship maneuvering space design; Action space To achieve precise control of the airship in complex wind fields while maintaining the convergence and efficiency of reinforcement learning policy optimization, this method designs an action space based on continuous control variables, using a set of executable continuous control commands. The airship's action vector at each time step is defined as follows: (27) in, Indicates the horizontal propulsion speed relative to the airflow. This represents the heading angle relative to the global coordinate system. This represents the vertical lift velocity. The design fully covers the requirements of the airship's three-dimensional motion in terms of control dimensions, enabling the strategy to adaptively adjust its path in dynamic environments.

[0051] Considering the physical characteristics of airships, such as low thrust, high inertia, and weak maneuverability, they cannot perform high-frequency and drastic maneuvering changes in actual flight. Therefore, multiple dynamic constraints are introduced in the action space design to ensure that the generated control strategy is physically feasible and engineering practical.

[0052] (28) (29) (30) To facilitate the policy network in defining the action space boundaries during the learning phase, Table 1 summarizes the variable definitions and their value ranges for the continuous action space: Table 1. Variable definitions and their value ranges in the continuous action space.

[0053] The continuous action space not only covers the three-dimensional control degrees of freedom of the airship, but also guides the generation of physically feasible and dynamically stable control sequences through rate and acceleration constraints. This design provides structural guarantees for the stable training and generalization deployment of reinforcement learning strategies in complex wind fields.

[0054] 3.5 Design of Multi-Objective Reward Function; In reinforcement learning frameworks, the reward function is the core driving mechanism for agent learning. It defines the feedback signal obtained by the agent after taking a specific action in the environment, directly determining the target direction and convergence performance of policy optimization. For airship path planning tasks in windy environments, a reasonable reward function design should not only promote the agent's efficient movement towards the target, but also comprehensively consider factors such as energy consumption and safety.

[0055] To address this, our method constructs a multi-component weighted reward function, which jointly models multiple task indicators such as target achievement efficiency, propulsion energy consumption, safety boundary, and trajectory stability at a single-step time scale. Its mathematical form is as follows: (31) Target Retention Bonus: The target retention bonus continues to provide positive reinforcement as long as the target retention is successfully maintained within the specified range. (32) in, This represents the reward value continuously given when the airship's main current is within the target range; Indicates the current position of the airship. It is the target location. An acceptable target tolerance is defined as 40 km in this method. While the airship's main current remains within a 40 km radius of the target, continuous action is taken. Reward: If the target location is missed, the reward is 0.

[0056] Progress Bonus: When the airship is outside the loitering area, this component encourages effective movement toward the target. (33) in, It is a scaling factor that balances the progress reward magnitude with other reward components.

[0057] Energy Consumption Penalty: As the airship consumes electrical energy and charges with solar power during flight, the battery level fluctuates dynamically. To ensure the airship has sufficient electrical power to guarantee the normal operation of avionics and the ability to respond to emergencies, a certain energy reserve must be maintained. A penalty is imposed when the airship's energy is too low: (34) The current battery energy state is: , This represents the battery's maximum energy state.

[0058] Safety and Constraint Penalties: To ensure safe operation, penalties are included for boundary violations and path irregularities. (35) in, It is the indicator function for boundary violation events. Penalties may include excessive angular velocity or acceleration that could jeopardize the stability of the airship. and These are the weighting coefficients, which are 0.5 and 0.2 respectively.

[0059] The following are the results of verification using the method described in this application with actual data: The proposed method was implemented using the PyTorch framework and trained in a simulation environment employing real wind fields. Wind fields from East Asia between 2021 and 2023 were used for training, capturing wind fields within the latitude range of 0-40°N and longitude range of 100-140°E in different seasons. The real wind field data were derived from ERA5 reanalysis data, represented using a discrete 160*160 grid, with each grid representing 0.25 degrees of longitude and latitude, a height resolution of 100m, and a time resolution of 1 hour. The wind field window size in the state space was 32*32, with a time step of 15 minutes.

[0060] Table 2 summarizes the parameters of the learning algorithm. The deployed policy network was trained using a GPU for 30 hours with 5 million simulation runs, and data was collected using 10 parallel simulation environments. Training was conducted on Windows 10 with a training environment of 64GB RAM and an RTX 4090 GPU.

[0061] Table 2 Training Parameters

[0062] like Figure 2As shown, the average reward curve of the policy network during training is illustrated. The horizontal axis represents the number of training steps, and the vertical axis represents the average cumulative reward obtained after each round of interaction in all parallel environments. In the early training stage (the first 20 million steps), the agent has not yet established an effective correlation between the wind field structure and the control policy. The policy output is approximately random, exhibiting large fluctuations in rewards and a generally low overall level. The airship frequently deviates from the target area and even exhibits irrational behaviors such as reversal of direction or rapid energy consumption under strong wind disturbances. As training progresses, the policy network gradually learns how to identify wind direction patterns in the wind field structure that are conducive to propulsion and rationally adjusts its speed and heading to counteract wind interference. After approximately 20 million steps, the average reward begins to increase significantly and tends to stabilize, indicating that the policy network gradually converges and possesses stable path generation and energy control capabilities. At this stage, the agent can stably achieve stationary control of the target area and maintain good performance even under complex disturbance scenarios such as strong winds.

[0063] To verify the path planning and target dwell performance of the proposed improved policy algorithm transformer-PPO under complex wind field conditions, this application conducted comparative experiments with the standard Proximal Policy Optimization (PPO) algorithm. The standard PPO algorithm maintains the same structure and training mechanism as the proposed method, thus serving as a good comparison to reflect the effectiveness of the improved module. Both algorithms are implemented based on the PyTorch framework, using the same hyperparameter configuration and training process.

[0064] like Figure 3 As shown, this paper presents a comparison of the Transformer-PPO and standard PPO strategies during a 72-hour stay mission under typical disturbed wind conditions, comprehensively evaluating the performance differences between the two algorithms in path control, altitude adjustment, energy management, and propulsion efficiency. Figure 3 The top left image shows a two-dimensional geographic trajectory map based on latitude and longitude, with green circular areas marking the target's dwell range. It can be seen that although both strategies can maintain relatively stable regional dwell over long time scales, the Transformer-PPO (blue trajectory) exhibits a more concentrated trajectory distribution, with the dwell center highly overlapping with the target area, and less path volatility, demonstrating superior regional coverage performance and location control stability. In contrast, while the standard PPO strategy (red trajectory) can maintain basic dwell, its trajectory shows significant drift and loop characteristics, indicating weaker robustness against disturbances under complex wind fields.

[0065] Example 2 This application also provides an airship path planning system that integrates temporal modeling and reinforcement learning, implemented based on the above method, including: The wind field modeling module receives wind field sequence data from multiple time steps and multiple altitude layers, and uses a Transformer encoder that integrates time-altitude dual position encoding mechanism to generate wind field feature vectors. The status acquisition module is used to acquire the current status information of the airship; The state fusion module is used to fuse the wind field feature vector with the airship's current state information to generate an enhanced state representation; The decision control module is used to input the enhanced state representation into the trained deep reinforcement learning-based policy network and output action commands to control the movement of the airship.

[0066] This application may also provide a computer device, including: at least one processor, memory, at least one network interface, and a user interface. The various components in this device are coupled together via a bus system. It is understood that the bus system is used to implement communication between these components. In addition to a data bus, the bus system also includes a power bus, a control bus, and a status signal bus.

[0067] The user interface can include a display, keyboard, or clicking device. Examples include a mouse, trackball, touchpad, or touchscreen.

[0068] It is understood that the memory in the embodiments disclosed in this application may be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory may be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDRSDRAM), Enhanced Synchronous DRAM (ESDRAM), Synchlink DRAM (SLDRAM), and Direct Rambus RAM (DRRAM). The memories described herein are intended to include, but are not limited to, these and any other suitable types of memory.

[0069] In some implementations, the memory stores elements such as executable modules or data structures, or subsets thereof, or extended sets thereof: operating systems and applications.

[0070] The operating system includes various system programs, such as the framework layer, core library layer, and driver layer, used to implement various basic business functions and handle hardware-based tasks. The application programs include various applications, such as media players and browsers, used to implement various application functions. Programs implementing the methods of the embodiments of this disclosure can be included in the application programs.

[0071] In the above embodiments, the processor can also invoke programs or instructions stored in memory, specifically programs or instructions stored in an application program, for the following purposes: Follow the steps described above.

[0072] The above methods can be applied to or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above methods can be completed by integrated logic circuits in the processor's hardware or by software instructions. The processor can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic diagrams disclosed above. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the disclosed methods can be directly implemented by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. The software modules can reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. This storage medium is located in memory; the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above methods.

[0073] It is understood that the embodiments described in this application can be implemented using hardware, software, firmware, middleware, microcode, or a combination thereof. For hardware implementation, the processing unit can be implemented in one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), general-purpose processors, controllers, microcontrollers, microprocessors, other electronic units for performing the functions described in this application, or combinations thereof.

[0074] For software implementation, the technology of this application can be implemented by executing the functional modules (e.g., procedures, functions, etc.) of this application. The software code can be stored in memory and executed by a processor. The memory can be implemented in the processor or outside the processor.

[0075] This application may also provide a non-volatile storage medium for storing a computer program. When the computer program is executed by a processor, it can implement the steps in the above method embodiments.

[0076] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application and are not intended to limit it. Although this application has been described in detail with reference to the embodiments, those skilled in the art should understand that modifications or equivalent substitutions to the technical solutions of this application do not depart from the spirit and scope of the technical solutions of this application, and should all be covered within the scope of the claims of this application.

Claims

1. A method for airship path planning that integrates temporal modeling and reinforcement learning, comprising: Acquire three-dimensional wind field sequence data with multiple time steps and multiple height levels; The wind field sequence data is processed using a Transformer encoder to extract wind field feature vectors that describe its spatiotemporal evolution trend; Obtain the current status information of the airship, including the airship's position, speed, and energy status; The wind field feature vector is fused with the airship's current state information to form an enhanced state representation; The enhanced state representation is input into the trained deep reinforcement learning policy network to generate action commands to control the airship's movement.

2. The airship path planning method integrating temporal modeling and reinforcement learning according to claim 1, characterized in that, The process of processing the wind field sequence data using a Transformer encoder includes: A time-height dual position encoding mechanism is adopted to simultaneously encode the position information of each data point in the wind field sequence data in both the time dimension and the vertical height dimension, enabling the Transformer encoder to capture the wind field co-evolution law and temporal dynamics between vertical heights. The input of the Transformer encoder Represented as: ； in, The wind field vector input to the encoder is represented as: ； in, This is obtained by flattening the temporal and spatial dimensions of the original wind field tensor into a unified sequence; T Indicates the time step; H Indicates the number of floors in the vertical height; For learnable parameters, Indicates the model dimension; For position encoding, it is represented as: ； in, Indicates the position index of the sequence; Indicates time location encoding: ； ； in, k Indicates a dimension index; t Indicates time; Spatial location encoding: ； ； in, h Indicates the vertical height of the layer.

3. The airship path planning method integrating temporal modeling and reinforcement learning according to claim 1, characterized in that, The current status information of the airship also includes: the airship's three-dimensional coordinates, heading angle, vertical speed, and the coordinates of the stationary target position.

4. The airship path planning method integrating temporal modeling and reinforcement learning according to claim 1, characterized in that, The process of fusing the wind field feature vector with the airship's current state information includes: The wind field feature vector is concatenated with the current state information of the airship to form a single, higher-dimensional state vector.

5. The airship path planning method integrating temporal modeling and reinforcement learning according to claim 1, characterized in that, The deep reinforcement learning policy network is trained and optimized using a proximal policy optimization algorithm.

6. The airship path planning method integrating temporal modeling and reinforcement learning according to claim 1, characterized in that, The training process of the deep reinforcement learning policy network employs a multi-objective reward function, the reward function... Includes: Target Residency Incentive Progress rewards for moving towards the target Energy consumption penalty and security boundary penalties : ； ； ； ； ； in, This represents the reward value continuously given when the airship's main current is within the target range; Indicates the airship's current position; Indicates the target location; Indicates the acceptable target tolerance; This represents the scaling factor that balances the progress reward magnitude with other reward components; This indicates the current battery energy status of the airship. This indicates the airship's battery's maximum energy state; Indicator functions that represent boundary violation events; This refers to an excessive angular velocity or acceleration that could jeopardize the stability of the airship. and These are the weighting coefficients.

7. The airship path planning method integrating temporal modeling and reinforcement learning according to claim 1, characterized in that, The action command is a multi-dimensional continuous action vector, which includes the airship's horizontal propulsion speed, heading angle, and vertical heave speed, so as to realize the dynamic adjustment of the airship in three-dimensional space.

8. An airship path planning system integrating temporal modeling and reinforcement learning, implemented based on the method described in any one of claims 1-7, characterized in that, include: The wind field modeling module receives wind field sequence data from multiple time steps and multiple altitude layers, and uses a Transformer encoder that integrates time-altitude dual position encoding mechanism to generate wind field feature vectors. The status acquisition module is used to acquire the current status information of the airship; The state fusion module is used to fuse the wind field feature vector with the airship's current state information to generate an enhanced state representation; and The decision control module is used to input the enhanced state representation into the trained deep reinforcement learning-based policy network and output action commands to control the movement of the airship.

Images