A control method for autonomous flight of stratospheric airship
By applying soft actor-critician algorithm and attention mechanism-long and short-term memory network in stratospheric airships, the reward function and control strategy are optimized, and the problem of insufficient autonomy and immunity of airships is solved, and more efficient and stable Flight control.
Patent Information
- Application Number
- CN202411156573.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-22
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2044-08-22
AI Technical Summary
The autonomy of stratospheric airships is weak and has poor anti-wind ability. Existing control methods such as PID controllers perform poorly in complex environments and cannot effectively adapt to nonlinear and highly coupled flight control problems.
Using a deep reinforcement learning method based on the soft actor-critician algorithm (SAC), combining attention mechanism and long and short-term memory network (ALSTM), design reward functions based on current and past states, optimize control strategies, and pass experience playback pools Update network parameters to improve airship autonomy and immunity.
It significantly improves the adaptability of the dynamic characteristics of the airship and the handling capacity of complex wind farm environments, realizes end-to-end control from position, speed and attitude to actuator, improves system response and robustness, improves control response speed and stability.
Smart Images

Figure CN118672294B_ABST
Abstract
Description
Technical Field
[0001] The invention discloses a control method for autonomous flight of a stratospheric airship, belonging to the technical field of flight control. Background Art
[0002] The stratospheric environment places strict requirements on the operation of airships, requiring them to adapt to the significant differences between the stratosphere and the ground or lower atmosphere. Secondly, the action range of the actuators of stratospheric airships is effective. Controllers designed by traditional methods, such as backstepping, will have actuator saturation problems and cannot achieve good control effects. In addition, due to the complexity of the stratospheric environment, including the uncertainty of factors such as wind, temperature and atmospheric pressure, the control algorithm of stratospheric airships needs to have strong adaptability and robustness. Other control methods such as proportional-integral-derivative (PID) controllers, although they perform well in some simple environments, have limited performance in dynamic and complex environments. This is mainly because PID controllers rely on preset control parameters and cannot be adjusted adaptively, resulting in poor performance when dealing with nonlinear and highly coupled flight control problems. In recent years, deep reinforcement learning technology has shown great potential in the field of flight control. Reinforcement learning algorithms continuously optimize strategies by interacting with the environment to achieve the best control effect. The soft actor-critic (SAC) algorithm, as a deep reinforcement learning algorithm based on entropy regularization, has good stability and convergence, but has limitations when processing long time series data. Summary of the invention
[0003] The object of the present invention is to provide a control method for autonomous flight of a stratospheric airship, so as to solve the problems in the prior art that the stratospheric airship has weak autonomy and poor anti-wind disturbance capability.
[0004] A control method for autonomous flight of a stratospheric airship, comprising:
[0005] S1 establishes the airship flight environment, adds the actual wind field data as disturbance to the airship dynamics model, and collects state information and stores it in the experience playback pool;
[0006] S2 improves the reward function by referring to the proportional-integral-derivative controller PID, designs a reward function based on the current state and the past state, and measures and optimizes the control effect from multiple angles;
[0007] S3 will state information Input to the policy network based on the soft actor-critic algorithm SAC, and improve the policy network to the attention mechanism-long short-term memory network ALSTM, ALSTM continuously updates the learning optimal control strategy ,according to Select the optimal control action ;
[0008] S4 inputs the current state and past state, optimal control action and reward function into the evaluation network based on SAC, and improves the evaluation network into a long short-term memory network LSTM, updates the network according to the reward information and predicted Q value, and guides the update of ALSTM, so that ALSTM generates more Better action.
[0009] The airship flight environment includes the establishment of a four-dimensional space-time wind field regarding longitude, latitude, altitude and time.
[0010] S2 includes:
[0011] ;
[0012] ; ;
[0013] ; ;
[0014] ; ;
[0015] In the formula, is the reward function, , , They are horizontal speed reward, altitude reward and pitch angle attitude reward. , , is the state error corresponding to the three rewards, , , These are the task instructions corresponding to the three rewards. , , is the state error of the previous moment corresponding to the three rewards, , , are the horizontal velocity, altitude and pitch angle attitude respectively, , , are the proportional coefficients corresponding to the three rewards, , , is the integral coefficient corresponding to the three rewards, , , are the error responses corresponding to the three rewards, and the error responses include the proportional response of the error, the cumulative error impact and the response of the error change rate.
[0016] A stability reward is set based on the horizontal speed reward, altitude reward, and pitch angle attitude reward. When the state error is less than a given value, the stability reward is used to supplement the horizontal speed reward, altitude reward, and pitch angle attitude reward:
[0017] ;
[0018] ;
[0019] ;
[0020] ;
[0021] In the formula, , , They are the stability rewards corresponding to the horizontal speed reward, altitude reward and pitch angle attitude reward.
[0022] ALSTM includes a state information preprocessing unit, an ALSTM policy network unit, a policy updating unit and an action selection unit.
[0023] The state information preprocessing unit processes the state information stored in the experience replay pool. Perform normalization to adapt to the input requirements of the ALSTM network;
[0024] The ALSTM strategy network unit adopts a network structure that combines the LSTM network with the attention mechanism and receives the preprocessed , including input layer, hidden layer 1, hidden layer 2 and output layer;
[0025] The strategy update unit continuously updates the parameters of the ALSTM strategy network unit during the training process, so that the ALSTM strategy network unit learns and gradually approaches ;
[0026] The action selection unit selects the current moment according to the output of the ALSTM strategy network unit. , and Actuators used in airships.
[0027] The input layer of the ALSTM strategy network unit is an LSTM layer with a dimension of 6×128; hidden layer 1 is an attention mechanism with a dimension of 128×128; hidden layer 2 is a fully connected layer with a dimension of 128×256; and the output layer is a fully connected layer with a dimension of 256×4.
[0028] LSTM includes a state-action information integration unit, an LSTM evaluation network unit, a Q-value calculation unit, an evaluation network update unit and a feedback mechanism unit.
[0029] The state-action information integration unit integrates the current state and past state, the optimal control action and the reward function at the current moment, and performs normalization processing;
[0030] The LSTM evaluation network unit adopts the LSTM network structure, receives the normalized current state and past state, the optimal control action and the reward function, and captures the dynamic change characteristics in the time series, including the input layer, hidden layer 1, hidden layer 2 and output layer;
[0031] The Q-value calculation unit calculates the predicted Q-value of the current state-action pair based on the output of the LSTM evaluation network unit;
[0032] The evaluation network update unit uses the gradient descent optimization method to update the parameters of the LSTM evaluation network unit based on the actual reward value and predicted Q value to improve the prediction accuracy;
[0033] The feedback mechanism unit feeds back the predicted Q value of the LSTM evaluation network unit to the ALSTM to guide the parameter update and optimization of the ALSTM strategy network unit.
[0034] The input layer of the LSTM evaluation network unit is an LSTM layer with a dimension of 9×256; the hidden layer 1 is a fully connected layer with a dimension of 256×128; the hidden layer 2 is a fully connected layer with a dimension of 128×64; the output layer is a fully connected layer with a dimension of 64×1;
[0035] The feedback mechanism unit includes a policy gradient calculation unit and a policy optimization unit. The policy gradient calculation unit calculates the policy gradient according to the feedback result of the LSTM evaluation network unit to guide the optimization direction of the ALSTM policy network unit.
[0036] The policy optimization unit uses the policy gradient to update the parameters of the ALSTM policy network unit, so that the ALSTM policy network unit generates better control actions and improves the performance of the overall control system.
[0037] Compared with the prior art, the present invention has the following beneficial effects: the present invention can more effectively capture the dynamic characteristics of the airship and the complex and changeable wind field environment, realize end-to-end control from the state information such as the position, speed and attitude of the airship to the specific actuators such as propellers, and improve the system response and robustness; utilize the SAC algorithm in deep reinforcement learning to improve the autonomy and anti-interference ability of the airship; combine the LSTM network with the attention mechanism to optimize the network structure, so that the algorithm is more efficient and stable when processing time series data and dynamically changing environments; by designing a reward function based on the current state and the past state, the control effect is measured and optimized from multiple angles, and the control response speed and stability during the flight are improved. BRIEF DESCRIPTION OF THE DRAWINGS
[0038] Figure 1 is a schematic diagram of a dynamic model of a stratospheric airship of the present invention;
[0039] Figure 2 It is the ALSTM structure diagram;
[0040] Figure 3 It is the ALSTM-SAC structure diagram;
[0041] Figure 4 It is a curve chart of reward value changes during the training process;
[0042] Figure 5 It is the curve diagram of the change of horizontal speed;
[0043] Figure 6 It is the curve diagram of vertical velocity variation;
[0044] Figure 7 is the variation curve of pitch angular velocity;
[0045] Figure 8 It is a graph of the change of horizontal position;
[0046] Fig. 9 It is a graph of the change in vertical position;
[0047] Fig.10 It is a curve graph of the change of pitch angle. DETAILED DESCRIPTION
[0048] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention is described clearly and completely below. Obviously, the described embodiments are part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0049] A control method for autonomous flight of a stratospheric airship, comprising:
[0050] S1 establishes the airship flight environment, adds the actual wind field data as disturbance to the airship dynamics model, and collects state information and stores it in the experience playback pool;
[0051] S2 improves the reward function by referring to the proportional-integral-derivative controller PID, designs a reward function based on the current state and the past state, and measures and optimizes the control effect from multiple angles;
[0052] S3 will state information Input to the policy network based on the soft actor-critic algorithm SAC, and improve the policy network to the attention mechanism-long short-term memory network ALSTM, ALSTM continuously updates the learning optimal control strategy ,according to Select the optimal control action ;
[0053] S4 inputs the current state and past state, optimal control action and reward function into the evaluation network based on SAC, and improves the evaluation network into a long short-term memory network LSTM, updates the network according to the reward information and predicted Q value, and guides the update of ALSTM, so that ALSTM generates more Better action.
[0054] The airship flight environment includes the establishment of a four-dimensional space-time wind field regarding longitude, latitude, altitude and time.
[0055] S2 includes:
[0056] ;
[0057] ; ;
[0058] ; ;
[0059] ; ;
[0060] In the formula, is the reward function, , , They are horizontal speed reward, altitude reward and pitch angle attitude reward. , , is the state error corresponding to the three rewards, , , These are the task instructions corresponding to the three rewards. , , is the state error of the previous moment corresponding to the three rewards, , , are the horizontal velocity, altitude and pitch angle attitude respectively, , , are the proportional coefficients corresponding to the three rewards, , , is the integral coefficient corresponding to the three rewards, , , are the error responses corresponding to the three rewards, and the error responses include the proportional response of the error, the cumulative error impact and the response of the error change rate.
[0061] A stability reward is set based on the horizontal speed reward, altitude reward, and pitch angle attitude reward. When the state error is less than a given value, the stability reward is used to supplement the horizontal speed reward, altitude reward, and pitch angle attitude reward:
[0062] ;
[0063] ;
[0064] ;
[0065] ;
[0066] In the formula, , , They are the stability rewards corresponding to the horizontal speed reward, altitude reward and pitch angle attitude reward.
[0067] ALSTM includes a state information preprocessing unit, an ALSTM policy network unit, a policy updating unit and an action selection unit.
[0068] The state information preprocessing unit processes the state information stored in the experience replay pool. Perform normalization to adapt to the input requirements of the ALSTM network;
[0069] The ALSTM strategy network unit adopts a network structure that combines the LSTM network with the attention mechanism and receives the preprocessed , including input layer, hidden layer 1, hidden layer 2 and output layer;
[0070] The strategy update unit continuously updates the parameters of the ALSTM strategy network unit during the training process, so that the ALSTM strategy network unit learns and gradually approaches ;
[0071] The action selection unit selects the current moment according to the output of the ALSTM strategy network unit. , and Actuators used in airships.
[0072] The input layer of the ALSTM strategy network unit is an LSTM layer with a dimension of 6×128; hidden layer 1 is an attention mechanism with a dimension of 128×128; hidden layer 2 is a fully connected layer with a dimension of 128×256; and the output layer is a fully connected layer with a dimension of 256×4.
[0073] LSTM includes a state-action information integration unit, an LSTM evaluation network unit, a Q-value calculation unit, an evaluation network update unit and a feedback mechanism unit.
[0074] The state-action information integration unit integrates the current state and past state, the optimal control action and the reward function at the current moment, and performs normalization processing;
[0075] The LSTM evaluation network unit adopts the LSTM network structure, receives the normalized current state and past state, the optimal control action and the reward function, and captures the dynamic change characteristics in the time series, including the input layer, hidden layer 1, hidden layer 2 and output layer;
[0076] The Q-value calculation unit calculates the predicted Q-value of the current state-action pair based on the output of the LSTM evaluation network unit;
[0077] The evaluation network update unit uses the gradient descent optimization method to update the parameters of the LSTM evaluation network unit based on the actual reward value and predicted Q value to improve the prediction accuracy;
[0078] The feedback mechanism unit feeds back the predicted Q value of the LSTM evaluation network unit to the ALSTM to guide the parameter update and optimization of the ALSTM strategy network unit.
[0079] The input layer of the LSTM evaluation network unit is an LSTM layer with a dimension of 9×256; the hidden layer 1 is a fully connected layer with a dimension of 256×128; the hidden layer 2 is a fully connected layer with a dimension of 128×64; the output layer is a fully connected layer with a dimension of 64×1;
[0080] The feedback mechanism unit includes a policy gradient calculation unit and a policy optimization unit. The policy gradient calculation unit calculates the policy gradient according to the feedback result of the LSTM evaluation network unit to guide the optimization direction of the ALSTM policy network unit.
[0081] The policy optimization unit uses the policy gradient to update the parameters of the ALSTM policy network unit, so that the ALSTM policy network unit generates better control actions and improves the performance of the overall control system.
[0082] In recent years, deep reinforcement learning technology has shown great potential in the field of flight control. Reinforcement learning algorithms continuously optimize strategies by interacting with the environment to achieve the best control effect. As a deep reinforcement learning algorithm based on entropy regularization, the soft actor-critic (SAC) algorithm has good stability and convergence, but has limitations in processing long time series data. In order to further improve the flight control performance of stratospheric airships, the present invention proposes a flight controller based on attention mechanism, long short-term memory network and soft actor-critic algorithm (ALSTM-SAC). Long short-term memory (LSTM) network is a special type of recurrent neural network (RNN) designed to process and predict time-based sequence data. By introducing a gating mechanism, LSTM can effectively solve the gradient vanishing and gradient exploding problems in long sequence data, making it perform better when processing long-term dependent tasks. LSTM networks perform particularly well in fields such as natural language processing, time series prediction and control systems. Attention mechanism is a deep learning model based on self-attention, which was originally used for natural language processing tasks. Compared with traditional RNN and LSTM, the attention mechanism can better capture global context information and has higher efficiency and better performance when processing long sequence data. The Attention mechanism has significant advantages in processing complex spatiotemporal data. Combining LSTM with the Attention mechanism can more comprehensively capture long-term and short-term dependencies in time series data. LSTM is good at capturing long-term dependencies, while the Attention mechanism can capture global context information. The ALSTM network composed of the two enables the algorithm to learn control strategies more effectively when processing time series data and dynamically changing environments, and has stronger expressiveness and stability. The ALSTM-SAC algorithm designed in the present invention is based on the SAC algorithm, and uses LSTM and the Attention mechanism to improve the policy network and the evaluation network therein.
[0083] In order to accurately describe the flight process, the body coordinate system is established with the volume center of the airship as the origin. ,in Along the axis of the airship, pointing toward the head of the airship, In the longitudinal plane of the airship, Pointing vertically downward, perpendicular to the longitudinal symmetry plane of the airship, , Construct a right-handed coordinate system, such as Figure 1 As shown. Navigation coordinate system It is the reference coordinate system for modeling the kinematic equations of the airship, and the "North-East" geographic coordinate system is used. The origin is the volume center at the launch point of the airship. The local horizontal plane points to due north. Axis along The plumb line of the point points downward. Axis and , A right-handed coordinate system is formed.
[0084] The state space of the environment is the state variable of the airship dynamics, including the position, altitude, attitude angle, speed and angular velocity of the airship. This 6-dimensional continuous state space provides complete state information of the airship in the environment, taking into account the influence of the airship's position in the air, motion state and external environment. The action space of the environment defines the available actions for controlling the airship, using a continuous representation method to flexibly adjust parameters. This 4-dimensional continuous space includes the propeller thrust component T, μ and the rudder deflection angle .
[0085] This continuous action space allows the agent to output different control quantities under different flight conditions, achieving a more flexible and accurate flight control strategy. The reward function is used to evaluate the quality of the control strategy. In the flight control of the stratospheric airship, it is very important to design a reasonable reward function. The reward function can take into account factors such as altitude error and attitude stability. The instructions obtained according to the task are .
[0086] The LSTM network is a special recurrent neural network (RNN) that is good at processing and predicting time-based sequence data. By introducing a gating mechanism, LSTM can effectively solve the gradient vanishing and gradient exploding problems in traditional RNNs, making it perform better when processing long-term dependent tasks. The core of the LSTM unit consists of three gating mechanisms: the input gate, the forget gate, and the output gate. The input gate controls how new input information affects the cell state, the forget gate determines whether the information in the cell state is retained, and the output gate determines how the information in the cell state affects the output.
[0087] The attention mechanism allows the model to dynamically focus on different parts of the input data as it processes information. It does this by assigning a weight (the attention weight) to each input to emphasize the most relevant information for the current task. In the self-attention model, the weights are calculated using a learnable function that evaluates the relevance of each part of the input data to the current task.
[0088] The ALSTM network is a network structure that combines the Attention mechanism and the long short-term memory network. Its structure is as follows: Figure 2As shown in the figure. LSTM is good at processing sequence data and can capture long-term dependencies in time series, while the attention mechanism can help the network better focus on the most important parts of the input sequence, thereby improving the performance and efficiency of the model. The ALSTM network enables the model to not only capture long-term dependencies in time series data, but also identify and prioritize those pieces of information that are most critical to the current task. This feature is particularly helpful in processing rich and complex time dynamics and improves decision-making capabilities in complex environments.
[0089] The ALSTM-SAC algorithm combines the reinforcement learning algorithm of the ALSTM network and the SAC algorithm. Its basic principle is to control the environment through alternating optimization of the policy network (Actor) and the value network (Critic). Figure 3 The overall architecture of the ALSTM-SAC algorithm is shown. The Actor network (ALSTM) is mainly responsible for strategy generation. Generate Action The ALSTM network improves the accuracy and efficiency of decision making. Based on the feedback obtained from the environment and the experience replay pool, the policy network is continuously adjusted and optimized to achieve a better decision strategy. The Critic network (LSTM) is responsible for evaluating the value of the state-action pair. The Critic network consists of two independent LSTM networks, which calculate two value functions respectively. , which helps to reduce valuation bias. The target value network maintains stability through a soft update mechanism, thereby helping the Critic network to more accurately evaluate the long-term value of actions. The experience replay pool is a key module in the algorithm, which is used to store data samples such as state, action, reward, and next state generated by interaction with the environment. By randomly extracting samples from the experience replay pool for training and breaking the time correlation between samples, the training effect and the stability of the algorithm can be significantly improved. The environment module represents the actual interaction environment. At each time step, the Actor network generates Act on the environment, and the environment returns a new state as well as These interaction data are stored in the experience replay pool and used for subsequent policy and value updates. The training process consists of two main parts: policy update and value update. In the policy update process, the policy network is optimized by maximizing the value of the actions generated by the policy network in the critic network. In the value update process, the critic network is updated by minimizing the error between the predicted value and the target value.
[0090] In order to verify the effectiveness of the present invention, an experimental platform based on a simulation environment was built. The hardware environment uses a high-performance computing server equipped with an NVIDIA GeForce RTX 4080 GPU for training to accelerate the training process of the deep learning model. The wind field data was selected from the era5 reanalysis wind speed data set in a certain place in July 2023, and the parameter settings are shown in Table 1.
[0091] Table 1 Parameter settings
[0092] .
[0093] The experimental results are as follows Figure 4 As shown in the figure, the average reward of the ALSTM-SAC algorithm is significantly higher than that of the LSTM-SAC algorithm. This shows that the overall performance of the algorithm has been significantly improved after adding the attention mechanism. The attention mechanism can more effectively capture and utilize key information in the environment, thereby improving the performance of the strategy.
[0094] In terms of convergence speed and training efficiency, the convergence steps of the ALSTM-SAC algorithm are 5171, which is significantly lower than the 5904 of LSTM-SAC. This means that ALSTM-SAC reaches the predetermined reward level in fewer training steps, showing faster convergence speed and higher training efficiency. The introduction of the attention mechanism may enable the model to capture effective features faster and accelerate the optimization process of the strategy.
[0095] Regarding the stability of the algorithm, ALSTM-SAC has a volatility of 2377.2781, which is higher than LSTM-SAC. Despite the higher volatility, this may be due to the higher overall reward of ALSTM-SAC. Higher reward levels are usually accompanied by greater volatility, as the strategy may generate larger fluctuations when exploring high-yield areas.
[0096] The maximum drawdown of ALSTM-SAC is 5.6796, which is significantly lower than LSTM-SAC's 8.1063. This shows that in the worst case, ALSTM-SAC performs more robustly and has a smaller maximum drop in reward value. A lower maximum drawdown means that the algorithm is better able to maintain the stability and robustness of the strategy when facing extreme situations.
[0097] In summary, the ALSTM-SAC algorithm with the introduction of the attention mechanism outperforms the LSTM-SAC algorithm in terms of performance, convergence speed, and worst-case stability. Despite the higher volatility, this does not significantly affect its overall advantage. These results show that the ALSTM-SAC algorithm is an effective and robust improvement scheme that is more suitable for the flight environment of stratospheric airships during ascent.
[0098] Since the reward value of the LSTM-SAC algorithm is significantly lower than that of the ALSTM-SAC algorithm, and it was found during the test that the LSTM-SAC algorithm failed to effectively control the flight altitude, the PID algorithm was selected for comparison with the ALSTM-SAC algorithm. The PID algorithm is a traditional and widely used model-free control algorithm that performs well in flight control, so it is relatively reasonable to use it as a benchmark comparison.
[0099] The experimental results show that the ALSTM-SAC algorithm has superior performance in many aspects compared to the PID controller. The success criterion is that the height error does not exceed 2% of the specified change height. The success rate of the ALSTM-SAC algorithm in multiple tests reached 98%, while the success rate of the PID algorithm was 87%.
[0100] set up In a specific ascending flight, the change process of each variable is as follows: Figure 5 , Figure 6 , Figure 7 , Figure 8 , Fig. 9 and Fig.10 As shown. In horizontal control, the PID controller has obvious overshoot in the initial response stage of horizontal speed, then gradually decreases and stabilizes near the target value, but shows a certain downward trend. In contrast, the initial response of the ALSTM-SAC algorithm is relatively smooth, with smaller overshoot and oscillation amplitude, and stabilizes near the target value faster. The performance of the two algorithms in horizontal displacement is basically the same, both cruising at a speed of about 5m / s, but it can be observed that the cruising speed under PID control is slightly higher than 5m / s.
[0101] In vertical control, in terms of speed performance, the PID controller has a significant overshoot in the initial stage, and then drops rapidly. Although it eventually stabilizes, the whole process fluctuates greatly. The ALSTM-SAC algorithm also has overshoot, but the amplitude and duration are smaller than the PID controller, showing better initial control accuracy. In terms of displacement, the ALSTM-SAC algorithm has a relatively stable control of vertical displacement, with less overshoot and faster recovery.
[0102] In terms of attitude control, the initial response phase of the PID controller has significant overshoot and large oscillations. In contrast, the initial response of the ALSTM-SAC algorithm is smoother and has less oscillations. Although the PID controller responds faster, its significant overshoot and oscillations may affect stability and precise control. Although the ALSTM-SAC algorithm has a slower initial response, it can reach the target value more stably and quickly, providing a more consistent control effect. This is particularly important in scenarios that require high stability and precise control.
[0103] Based on the above analysis, the ALSTM-SAC algorithm shows superior performance in multiple control dimensions compared to the PID controller, showing a higher success rate and more stable control response. This shows that the ALSTM-SAC algorithm has stronger adaptability and robust performance in the face of complex and changing environmental conditions.
[0104] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some or all of the technical features thereof may be replaced by equivalents, and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.
Claims
1. A control method for autonomous flight of a stratospheric airship, characterized in that: include: S1 establishes the airship flight environment, adds the actual wind field data as disturbance to the airship dynamics model, and collects state information and stores it in the experience playback pool; S2 improves the reward function by referring to the proportional-integral-derivative controller PID, designs a reward function based on the current state and the past state, and measures and optimizes the control effect from multiple angles; S3 will state information Input to the policy network based on the soft actor-critic algorithm SAC, and improve the policy network to the attention mechanism-long short-term memory network ALSTM, ALSTM continuously updates the learning optimal control strategy ,according to Select the optimal control action ; S4 inputs the current state and past state, optimal control action and reward function into the evaluation network based on SAC, and improves the evaluation network into a long short-term memory network LSTM, updates the network according to the reward information and predicted Q value, and guides the update of ALSTM, so that ALSTM generates more Better action; The airship flight environment includes the establishment of a four-dimensional space-time wind field with respect to longitude, latitude, altitude, and time; S2 includes: ; ; ; ; ; ; ; In the formula, is the reward function, , , They are horizontal speed reward, altitude reward and pitch angle attitude reward. , , is the state error corresponding to the three rewards, , , These are the task instructions corresponding to the three rewards. , , is the state error of the previous moment corresponding to the three rewards, , , are the horizontal velocity, altitude and pitch angle attitude respectively, , , are the proportional coefficients corresponding to the three rewards, , , is the integral coefficient corresponding to the three rewards, , , are error responses corresponding to the three rewards, wherein the error responses include proportional responses of errors, cumulative error impacts, and responses of error change rates; A stability reward is set based on the horizontal speed reward, altitude reward, and pitch angle attitude reward. When the state error is less than a given value, the stability reward is used to supplement the horizontal speed reward, altitude reward, and pitch angle attitude reward: ; ; ; ; In the formula, , , They are the stability rewards corresponding to the horizontal speed reward, altitude reward, and pitch angle attitude reward; ALSTM includes a state information preprocessing unit, an ALSTM policy network unit, a policy updating unit, and an action selection unit; The state information preprocessing unit processes the state information stored in the experience replay pool. Perform normalization to adapt to the input requirements of the ALSTM network; The ALSTM strategy network unit adopts a network structure that combines the LSTM network with the attention mechanism and receives the preprocessed , including input layer, hidden layer 1, hidden layer 2 and output layer; The strategy update unit continuously updates the parameters of the ALSTM strategy network unit during the training process, so that the ALSTM strategy network unit learns and gradually approaches ; The action selection unit selects the current moment according to the output of the ALSTM strategy network unit. , and Actuators used in airships; The input layer of the ALSTM strategy network unit is an LSTM layer with a dimension of 6×128; the hidden layer 1 is an attention mechanism with a dimension of 128×128; the hidden layer 2 is a fully connected layer with a dimension of 128×256; the output layer is a fully connected layer with a dimension of 256×4; LSTM includes a state-action information integration unit, an LSTM evaluation network unit, a Q-value calculation unit, an evaluation network update unit, and a feedback mechanism unit; The state-action information integration unit integrates the current state and past state, the optimal control action and the reward function at the current moment, and performs normalization processing; The LSTM evaluation network unit adopts the LSTM network structure, receives the normalized current state and past state, the optimal control action and the reward function, and captures the dynamic change characteristics in the time series, including the input layer, hidden layer 1, hidden layer 2 and output layer; The Q-value calculation unit calculates the predicted Q-value of the current state-action pair based on the output of the LSTM evaluation network unit; The evaluation network update unit uses the gradient descent optimization method to update the parameters of the LSTM evaluation network unit based on the actual reward value and predicted Q value to improve the prediction accuracy; The feedback mechanism unit feeds back the predicted Q value of the LSTM evaluation network unit to the ALSTM to guide the parameter update and optimization of the ALSTM strategy network unit; The input layer of the LSTM evaluation network unit is an LSTM layer with a dimension of 9×256; the hidden layer 1 is a fully connected layer with a dimension of 256×128; the hidden layer 2 is a fully connected layer with a dimension of 128×64; the output layer is a fully connected layer with a dimension of 64×1; The feedback mechanism unit includes a policy gradient calculation unit and a policy optimization unit. The policy gradient calculation unit calculates the policy gradient according to the feedback result of the LSTM evaluation network unit to guide the optimization direction of the ALSTM policy network unit. The policy optimization unit uses the policy gradient to update the parameters of the ALSTM policy network unit, so that the ALSTM policy network unit generates better control actions and improves the performance of the overall control system.
