Airship autonomous guidance flight method based on deep reinforcement learning algorithm

Abstract

The invention provides an airship autonomous guidance flight method based on a deep reinforcement learning algorithm. The method comprises the following steps: constructing an environment model by using reanalysis meteorological data and an airship kinematics model; a Markov decision process model is established, and the action space of the Markov decision process model is designed according to the environment model; based on the environment model and the Markov decision process model, adopting a deep reinforcement learning algorithm to carry out offline training on a pre-constructed agent to obtain an airship autonomous guidance flight agent; and determining an autonomous guidance flight strategy based on the airship autonomous guidance flight intelligent agent. Based on an environment model and a Markov decision process model, an intelligent agent trained through a deep reinforcement learning algorithm can consider various environment factors and airship dynamic constraints, so that a safer and more reliable flight strategy is formulated. The decision-making mode does not depend on experience of engineers, but is based on a large amount of simulation training data and real-time observation data, so that reliability and accuracy of decision-making are improved.

Description

Technical Field

[0001] The present invention relates to the field of airship flight control technology, and in particular to an airship autonomous guided flight method based on a deep reinforcement learning algorithm. Background Art

[0002] Current airship flight planning relies primarily on the traditional model of "meteorological data analysis and manual remote control flight." Engineers rely on forecast weather information and their flight control experience to remotely control unmanned airships. Manual remote control flight relies heavily on vast amounts of measurement and control data and meteorological data, and flight decisions are based on the engineers' experience. Due to the inherent errors in meteorological data predictions and the data's spatial and temporal sparsity, flight analysis based on meteorological data is limited in reliability and accuracy. Furthermore, this empirical approach may perform well in the short term, but as flight durations increase, engineers face significant challenges in both energy and physical capacity. Summary of the Invention

[0003] This invention provides an autonomous airship guided flight method based on a deep reinforcement learning algorithm, which aims to address the low reliability and accuracy of flight analysis based on meteorological data in the existing technology and improve the reliability and accuracy of decision-making. The technical solutions proposed by this invention are as follows: In a first aspect, the present invention provides an airship autonomous guided flight method based on a deep reinforcement learning algorithm, comprising: Obtain reanalysis meteorological data and airship kinematic models, and use them to construct an environmental model; Establishing a Markov decision process model, wherein the action space of the Markov decision process model is designed according to the environment model; Based on the environment model and the Markov decision process model, a pre-built intelligent agent is trained offline using a deep reinforcement learning algorithm to obtain an airship autonomous guided flight intelligent agent; An autonomous guided flight strategy is determined based on the airship autonomous guided flight agent.

[0004] Optionally, obtaining reanalyzed meteorological data and an airship kinematic model, and constructing an environmental model using the reanalyzed meteorological data and the airship kinematic model includes: Obtaining reanalyzed meteorological data, and extracting wind speed data and wind direction data within a preset time period and geographical range from the reanalyzed meteorological data; interpolating the wind speed data and wind direction data within the preset time period and geographical range to obtain updated reanalysis data; Train the pre - constructed airship kinematic model based on the updated re - analysis data to obtain the environmental model; the input of the airship kinematic model is the speed and angular velocity of the airship, and the output is the position and attitude of the airship.

[0005] Optionally, the Markov decision process model includes a state space, an action space, a state transition probability matrix, a reward function, and a discount factor; the state space includes airship state parameters, airship environmental parameters, and prior information; The airship state parameters include the airship position, the airship attitude, and the airship airspeed, the airship environmental parameters include the wind speed and the wind direction, and the prior information includes whether the airship has reached near the target point at present, the distance from the airship to the target point, and the angle difference between the airship flight heading and its angle to the flight target point.

[0006] Optionally, based on the environmental model and the Markov decision process model, use the deep reinforcement learning algorithm to offline train a pre - constructed agent to obtain an airship autonomous guidance flight agent, including: Based on the Markov decision process model, construct a framework for the interaction between the agent and the environmental model; Initialize the environmental model and the agent; the agent includes a policy network and a value network; In the framework, the airship randomly samples an action using the policy network, executes the action, interacts with the environmental model, and obtains the next state and the current reward; where the next state includes the airship state parameters and airship environmental parameters at the next moment; Perform calculation and processing on the airship state parameters and airship environmental parameters at the next moment to obtain prior information; After fusing, normalizing the airship state parameters, airship environmental parameters, and prior information at the next moment, input them into the policy network to obtain the Gaussian probability action distribution parameters output by the policy network, and sample the next action according to the Gaussian probability action distribution parameters; The agent stores the current state, the current action, the current reward, and the next state as an experience tuple in the data experience pool; Randomly sample a batch of experience tuples from the data experience pool, and use the sampled experience tuples to update the policy network and the value network; The agent continues to interact with the environmental model and continuously updates its policy network and value network until convergence or other stop conditions are met to obtain the airship autonomous guidance flight agent.

[0007] Optionally, the value network adopts the learning method of temporal difference residuals and is updated according to the loss function of the following formula: Where, To evaluate the loss function of the network, is the reward value at time is the discount factor, is the evaluation of the network for the state at time is the evaluation of the network for the state at time

[0008] Optionally, the reward function includes the potential energy reward of the airship to the target point, the penalty for the airship having too high an airspeed, the reward for flying in the target residence area, and the reward for the airship continuously flying within the legal range; The potential energy reward of the airship to the target point, the penalty for the airship having too high an airspeed, the reward for flying in the target residence area, and the reward for the airship continuously flying within the legal range are determined by the following formula: where is the potential energy reward of the airship to the target point, is the distance from the current position to the target point, is the penalty for the airship having too high an airspeed, is the airspeed of the airship, is the reward for flying in the target residence area, is the reward for the airship continuously flying within the legal range.

[0009] In a second aspect, the present invention further provides an airship autonomous guidance flight device based on a deep reinforcement learning algorithm, including the following modules: An acquisition module, configured to acquire reanalysis meteorological data and an airship kinematic model, and construct an environment model by using the reanalysis meteorological data and the airship kinematic model; A modeling module, configured to establish a Markov decision process model, wherein the action space of the Markov decision process model is designed according to the environment model; A training module, configured to offline train a pre-constructed agent based on the environment model and the Markov decision process model by using a deep reinforcement learning algorithm to obtain an airship autonomous guidance flight agent; A determination module, configured to determine an autonomous guidance flight strategy based on the airship autonomous guidance flight agent.

[0010] In a third aspect, the present invention further provides an electronic device, including a memory, a processor, and a computer program stored on the memory and running on the processor, where when the processor executes the computer program, it implements the airship autonomous guidance flight method based on the deep reinforcement learning algorithm as described in the first aspect above.

[0011] Fourthly, the present invention further provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it implements the autonomous guidance flight method of an airship based on a deep reinforcement learning algorithm as described in the first aspect above.

[0012] Fifthly, the present invention further provides a computer program product, including a computer program. When the computer program is executed by a processor, it implements the autonomous guidance flight method of an airship based on a deep reinforcement learning algorithm as described in the first aspect above.

[0013] Based on the above technical solutions, the beneficial effects of the present invention compared with the prior art are as follows: Manual remote control flight often requires relying on a large amount of measurement and control data and meteorological data, and these data may be sparse in terms of time and space. The autonomous guidance flight method of an airship based on a deep reinforcement learning algorithm provided by the present invention can make up for the lack of data through a learning algorithm. The agent will learn the transfer relationship between different states and the reward function during the training process, so as to be able to make reasonable decisions in the absence of specific data. There are certain errors in the prediction of meteorological data, which will have an adverse impact on flight analysis. However, the autonomous guidance flight method based on deep reinforcement learning does not completely rely on the prediction results of meteorological data. The agent will consider various possible weather conditions during training and learn the optimal flight strategies under different weather conditions. Therefore, even if there are errors in the prediction of meteorological data, the agent can still formulate a reasonable flight plan. Moreover, based on the environmental model and the Markov decision process model, the agent trained by the deep reinforcement learning algorithm can consider various environmental factors and the dynamic constraints of the airship, so as to formulate a safer and more reliable flight strategy. This method trains the agent through the deep reinforcement learning algorithm, enabling it to make autonomous flight decisions. This decision-making method does not rely on the experience of engineering personnel, but is based on a large amount of simulation training data and real-time observation data, thus improving the reliability and accuracy of the decision-making.

[0014] Other features and advantages of the present invention will be described in the following specification, and part of them will become obvious from the specification, or will be understood by implementing the present invention. The objectives and other advantages of the present invention are achieved and obtained by the structures specifically pointed out in the specification, claims and drawings.

[0015] To make the above objectives, features and advantages of the present invention more obvious and understandable, the following preferred embodiments are specifically given, and detailed descriptions are made in conjunction with the accompanying drawings as follows. Description of the Drawings

[0016] To more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the accompanying drawings required for the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description are some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other accompanying drawings can be obtained based on these drawings.

[0017] Figure 1 It is a schematic flow chart of the airship autonomous guidance flight method based on the deep reinforcement learning algorithm provided by the present invention.

[0018] Figure 2 It is a schematic framework diagram of the airship kinematic model provided by the present invention.

[0019] Figure 3 It is a schematic diagram of the airship kinematic model provided by the present invention Figure 1 .

[0020] Figure 4 It is a schematic diagram of the airship kinematic model provided by the present invention Figure 2 .

[0021] Figure 5 It is a schematic diagram of the airship speed in the wind field movement provided by the present invention.

[0022] Figure 6 It is a schematic diagram of the airship autonomous guidance flight in an unknown wind field provided by the present invention.

[0023] Figure 7 It is a schematic diagram of the agent training process provided by the present invention.

[0024] Figure 8 It is a schematic structural diagram of the airship autonomous guidance flight device based on the deep reinforcement learning algorithm provided by the present invention.

[0025] Figure 9 It is a schematic structural diagram of the electronic device provided by the present invention. Detailed implementation manners

[0026] To make the objectives, technical solutions and advantages of the present invention clearer, the following will clearly and completely describe the technical solutions in the present invention with reference to the accompanying drawings in the present invention. Obviously, the described embodiments are some but not all of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art without creative efforts based on the embodiments in the present invention belong to the scope of protection of the present invention.

[0027] The following combines Figures 1-7 to describe the airship autonomous guidance flight method based on the deep reinforcement learning algorithm of the present invention.

[0028] Refer toFigure 1 As shown, the method includes the following: Step S110: Obtain reanalysis meteorological data and an airship kinematic model, and construct an environmental model using the reanalysis meteorological data and the airship kinematic model.

[0029] The above reanalysis meteorological data includes wind speed, wind direction, air pressure, temperature, etc., and these data have a direct impact on the flight state of the airship. By obtaining real-time meteorological data, a more accurate environmental model can be constructed. The above airship kinematic model is used to describe the motion states of the airship in space, such as position, speed, acceleration, etc., and the relationships between them. This model is the basis for understanding how the airship responds to control inputs.

[0030] Combining the reanalysis meteorological data and the airship kinematic model, a comprehensive environmental model containing various environmental factors can be constructed. This model will be used to simulate the flight state of the airship under different conditions and provide a training environment for the deep reinforcement learning algorithm.

[0031] Step S120: Establish a Markov decision process model, where the action space of the Markov decision process model is designed according to the environmental model.

[0032] The Markov decision process (MDP) is a mathematical framework for describing decision-making in an uncertain environment. It includes elements such as a state space, an action space, a state transition probability, and a reward function. In this scenario, the state space can include the position, speed, attitude, etc. of the airship; the action space is designed according to the environmental model and includes various control instructions that the airship can execute (such as adjusting thrust, controlling the rudder, etc.). The state transition probability describes the transition relationship between states, and the reward function is used to evaluate the quality of the actions taken by the agent in each state.

[0033] The present invention realizes the autonomous guidance flight of an unmanned airship based on a deep reinforcement learning algorithm. The airship generally flies on an isobaric surface, so the research problem of airship autonomous guidance is defined as the guidance problem of an aircraft on a two-dimensional plane. The method implementation mainly includes three parts: establishing an environmental model, establishing a Markov decision process (MDP) model, and training a deep reinforcement learning algorithm. Among them, the environmental model includes an airship kinematic model and meteorological data, the MDP model includes an action space, a state space, and a reward mechanism, and the reinforcement learning algorithm is a proximal policy optimization (PPO) algorithm.

[0034] Specifically, in constructing the deep reinforcement learning framework for airship autonomous guidance flight, establishing the Markov decision process model is specifically divided into the following sub-steps: 1) Establish A five-tuple: is the state space, and the state space contains the full - range state information of the airship, mainly composed of the airship state parameters (position, attitude, airspeed), the airship environmental parameters (wind speed, wind direction), and the prior information (whether it has reached near the target point, the distance to the target point, the angle difference between the flight heading and the target point).

[0035] is the action space, and the action space is designed according to the characteristics of the environmental model, including the airship airspeed and the airship angular velocity (i.e., the change amount of the heading angle ). The value ranges of these two control commands are determined by the physical capabilities of the airship, and there is a calculation relationship between them: where, is the turning radius.

[0036] By adjusting these two parameters, the agent can control the flight speed and turning of the airship.

[0037] The airship airspeed and the turning radius The value ranges of these two input quantities are determined by the capabilities of the airship, and the results trained by different ranges through the reinforcement learning method will be completely different.

[0038] is the state - transition probability matrix, and the state - transition probability describes the probability distribution of the airship transferring from the current state to the next state.

[0039] is the reward function, and the reward function is used to evaluate the quality of the actions taken by the agent in each state. In this scenario, the reward function adopts a mixed design, including the potential energy reward of the airship to the target point , the penalty for excessive airspeed , the reward for flying in the target residence area and the reward for continuously flying within the legal range . The complete reward function is: where, is the potential energy reward of the airship to the target point, is the distance from the current position to the target point, is the penalty for excessive airspeed of the airship, is the airspeed of the airship, is the reward for flying in the target residence area, The reward for the airship to continuously fly within the legal range.

[0040] Among them, the coefficients , , , are respectively set to 10, -0.25, 0.1, and 10 (when the airship is in the target residence area = 10, otherwise 0).

[0041] is the discount factor, and the discount factor is used to control the present value of future rewards and affect the agent's emphasis on long-term interests. Figure 6 is the schematic diagram of the autonomous guidance flight of the airship provided by the present invention in an unknown wind field. The target residence area is a circular area with the target flight point as the center and the residence radius as the radius.

[0042] 2) Coordinated design of the action space and the state space: When designing the action space and the state space, it is necessary to ensure that they can work together to jointly support the decision-making process of the agent. The design of the action space should consider the physical limitations and control requirements of the airship, while the state space should contain sufficient information to support the agent in making reasonable decisions.

[0043] 3) Design of the reward function: The design of the reward function is one of the key elements in the MDP model. In this scenario, the reward function is designed through a hybrid approach to balance the requirements in different aspects, including quickly reaching the target point, avoiding excessive airspeed, flying in the target residence area, and maintaining legal flight. By adjusting the coefficients and forms of the reward function, the agent can be guided to optimize its behavior towards the desired goal.

[0044] Step S130: Based on the environmental model and the Markov decision process model, use a deep reinforcement learning algorithm to offline train a pre-constructed agent to obtain an airship autonomous guidance flight agent.

[0045] The deep reinforcement learning algorithm combines the feature extraction ability of deep learning and the decision optimization ability of reinforcement learning. It uses a deep neural network to approximate the policy or value function of the agent. During the training process, the agent interacts with the environment, selects actions according to the current state, and observes the results (i.e., the next state and the reward). Then, the agent updates its policy or value function based on this information to maximize the cumulative reward. Through iteration, the agent can learn the strategy of taking optimal actions in different states.

[0046] Step S140: Determine the autonomous guidance flight strategy based on the airship autonomous guidance flight agent.

[0047] During the training process, the agent continuously optimizes its flight strategy to maximize the cumulative reward. This means that the agent can continuously learn and adapt to new flight environments and mission requirements, thus providing more intelligent and efficient flight solutions.

[0048] After training, the agent will become an agent capable of autonomously guiding the airship flight. It can quickly calculate the optimal control commands based on real-time airship environmental parameters (such as wind speed and wind direction). This agent can be embedded in the airship's flight control system to achieve real-time autonomous guidance flight.

[0049] Manual remote control flight often relies on a large amount of measurement and control data and meteorological data, and these data may be sparse in time and space. The autonomous guidance flight method based on deep reinforcement learning can make up for the lack of data through learning algorithms. The agent will learn the transition relationships and reward functions between different states during the training process, so as to be able to make reasonable decisions in the absence of specific data. There are certain errors in the prediction of meteorological data, which will have an adverse impact on flight analysis. However, the autonomous guidance flight method based on deep reinforcement learning does not completely rely on the prediction results of meteorological data. The agent will consider various possible weather conditions during training and learn the optimal flight strategies under different weather conditions. Therefore, even if there are errors in the prediction of meteorological data, the agent can still formulate reasonable flight plans. Moreover, based on the environmental model and Markov decision process model, the agent trained by the deep reinforcement learning algorithm can take into account various environmental factors and the dynamic constraints of the airship, enabling it to make autonomous flight decisions. This decision-making method does not rely on the experience of engineering personnel, but is based on a large amount of simulation training data and real-time observation data, thus improving the reliability and accuracy of decision-making. The agent can learn and adapt to different flight environments and mission requirements. When facing complex and changeable meteorological conditions, the agent can dynamically adjust according to real-time meteorological data and airship states, and formulate more reasonable flight strategies. This method realizes the autonomous flight of the airship. The agent can quickly calculate the optimal control commands based on real-time airship states and meteorological data, and autonomously execute flight tasks. This reduces the need for manual intervention and improves the autonomy and safety of flight.

[0050] The agent trained by the deep reinforcement learning algorithm can take into account various environmental factors and the dynamic constraints of the airship, thus formulating a safer and more reliable flight strategy. In case of emergencies (such as bad weather, mechanical failures, etc.), the agent can quickly adjust the flight plan to ensure the safety of the airship. The agent can make dynamic adjustments based on real-time meteorological data and the state of the airship, adapting to different flight environments and mission requirements. This enables the airship to perform flight missions in a wider geographical area and climate conditions. By optimizing the flight path and control strategy, the agent can reduce the energy consumption and flight time of the airship, improving flight efficiency. This is particularly important for missions that require long flights (such as scientific exploration, cargo transportation, etc.). The realization of the autonomous guidance flight strategy reduces the dependence on pilots and reduces the risks and uncertainties brought by human operations. This enables the airship to perform flight missions in a more autonomous and intelligent manner.

[0051] The airship autonomous guidance flight method based on the deep reinforcement learning algorithm provided by the present invention uses an agent to simulate the decision-making process of the airship, enabling it to learn and adapt to various situations in a complex environment. Through continuous training and optimization, the airship can learn to make the best decisions in case of emergencies, thus improving its self-adaptive ability. During the training process, safety constraints and penalty mechanisms can be added so that the airship can consider safety when making decisions. For example, when the airship approaches a dangerous area or makes an operation error, corresponding penalties can be given, thus prompting the airship to learn to avoid these dangerous behaviors. Clear mission objectives are set for the airship, and its ability to complete tasks is continuously optimized during the training process. Through deep reinforcement learning, the airship can learn how to efficiently perform tasks such as monitoring, communication relay, meteorological observation, etc. The real-time data update and dynamic adjustment strategy are adopted to ensure that the algorithm can make the best decision within a limited time. By real-time monitoring the state of the airship and environmental information, and quickly adjusting the flight trajectory and control strategy, it is ensured that the airship can quickly respond to emergencies and make the best response.

[0052] The present invention obtains real-time environmental information, such as wind speed, wind direction, etc., through sensors and meteorological data. An agent that can adapt to different environments is trained using a deep reinforcement learning algorithm. The agent can dynamically adjust the flight trajectory according to the real-time environmental information, so as to ensure that the airship can maintain an optimal flight state in different environments. Thus, it is ensured that the airship can cover the target area to perform tasks such as monitoring, communication relay, meteorological observation, etc., and can maximize the monitoring time of the area of interest and improve the area coverage rate. Moreover, the flight control strategy of the airship is optimized through the deep reinforcement learning algorithm to ensure its good flight stability. For example, the airship can be trained to quickly adjust its attitude and maintain flight stability when encountering disturbances such as gusts. Combining the real-time environmental information and safety constraints, a flight trajectory that avoids dangerous areas is planned. At the same time, the state of the airship is monitored in real time during flight to ensure that it does not enter dangerous areas or make operational mistakes. By dynamically adjusting the flight trajectory, unnecessary energy consumption is reduced, thereby increasing the stay time of the airship in the air. For example, the airship can be trained to adjust its flight speed and altitude according to conditions such as wind speed and wind direction during flight, so as to save energy. Subsequently, the energy system and flight control strategy of the airship can be combined to optimize its flight trajectory to maximize the stay time in the air. The deep reinforcement learning algorithm and the trajectory planning algorithm can also be designed as a modular structure, making it convenient to apply to different models of airships. At the same time, by adjusting the algorithm parameters and training data, it can adapt to airships of different designs and sizes.

[0053] In an optional embodiment, a wind field model approximating the real environment is generated by linearly interpolating the reanalysis meteorological data. The steps of obtaining the reanalysis meteorological data and the airship kinematic model described in step S110 and constructing an environmental model using the reanalysis meteorological data and the airship kinematic model include: S1101. Obtain reanalysis meteorological data, and extract wind speed data and wind direction data within a preset time period and geographical range from the reanalysis meteorological data.

[0054] Reanalysis meteorological data is a type of data that combines observational data and the output data of numerical weather prediction models, providing long-term, high-resolution global meteorological information. These data can include various meteorological elements such as wind speed, wind direction, temperature, humidity, etc. In this scenario, the focus is mainly on wind speed and wind direction data because these data have a direct impact on the flight state and trajectory of the airship.

[0055] According to the flight plan and mission requirements of the airship, a preset time period and geographical range are determined. This time period usually covers the entire process of the airship's expected takeoff, flight, and landing, while the geographical range includes all areas that the airship may pass through. The wind speed and wind direction data within this time period and geographical range are extracted from the reanalysis meteorological data.

[0056] S1102. Interpolate between the wind speed data and wind direction data within the preset time period and geographical range to obtain updated reanalysis data.

[0057] Since the resolution of the reanalysis meteorological data may not be sufficient to meet the requirements of airship autonomous guidance, it is necessary to interpolate the extracted data. Interpolation is a method of estimating unknown data points from known data points, which can improve the resolution and accuracy of the data. In this scenario, a combination of spatial interpolation and temporal interpolation can be used to interpolate the wind speed and wind direction data to obtain updated reanalysis data. Estimate the wind speed and direction at positions not directly obtained to achieve a smooth reconstruction of the wind field within the entire simulation area based on the reanalysis data.

[0058] S1103. Train a pre-constructed airship kinematic model based on the updated reanalysis data to obtain the environment model; the input of the airship kinematic model is the speed and angular velocity of the airship, and the output is the position and attitude of the airship.

[0059] The airship kinematic model describes the motion state and trajectory of the airship in space. In this scenario, the input of the model is the speed and angular velocity of the airship (i.e., control commands), and the output is the position and attitude of the airship (i.e., state information). Use the updated reanalysis data as the input to train the pre-constructed airship kinematic model. The training process includes adjusting and optimizing the model parameters to ensure that the model can accurately simulate the flight state of the airship under different meteorological conditions. The trained airship kinematic model combined with the meteorological data constitutes an environment model for training the deep reinforcement learning algorithm. This environment model can simulate the flight state of the airship under different meteorological conditions such as wind speed and wind direction, providing real-time feedback for the decision-making of the intelligent agent.

[0060] The present invention can more accurately simulate the flight state of the airship under different meteorological conditions by using the environment model constructed with reanalysis meteorological data and the airship kinematic model. This helps the intelligent agent to formulate safer and more reliable flight strategies and avoid flight accidents under adverse weather conditions. The environment model can take into account the influence of various meteorological factors, including wind speed and wind direction. This enables the intelligent agent to adjust the flight plan and control strategy according to real-time meteorological data to adapt to different flight environments and mission requirements. By optimizing the flight path and control strategy, the intelligent agent can reduce the energy consumption and flight time of the airship and improve the flight efficiency. This is particularly important for tasks that require long-term flight (such as scientific exploration, cargo transportation, etc.). Using interpolation processing on the reanalysis meteorological data can obtain meteorological data with higher resolution and accuracy. This reduces the dependence on real-time meteorological data and reduces the uncertainty caused by data sparsity or prediction errors.

[0061] The above kinematic model of the airship is specifically described as follows: The research results on the airship trajectory planning will be applied to the guidance layer of the control system. The dynamic model is provided by the flight control system. Therefore, the present invention only models the kinematic model of the airship. The inputs of the kinematic model are the velocity and angular velocity of the airship, and the outputs are the position and attitude of the airship, as shown in Figure 2 .

[0062] During the level flight stage of the stratospheric airship, it flies on an isobaric surface. This model is established based on isobaric meteorological data, so it is considered that the airship flies in a two-dimensional plane environment. Therefore, as shown in Figure 3 , the research object in the present invention is abstracted as a particle in a two-dimensional plane.

[0063] As shown in Figure 3 , its pose can be described by three variables, including the two-dimensional plane coordinates relative to the external coordinate system and its heading angle (Heading Direction), which is expressed as: Among them, the position is obtained by converting through the center point of the airship's residence. The calculation method is described below. The origin is the coordinate of the center point of the residence flight area; the heading angle is the angle between the airship and the axis (geographical east). According to the right-hand rule, counterclockwise is positive, and the value range is .

[0064] The velocity motion model believes that the flight of the airship can be controlled by two velocities: 1) The velocity of the airship: represents the translational velocity at moment; 2) The angular velocity of the airship: represents the turning velocity at moment.

[0065] Therefore, the velocity at moment, where forward motion is positive and clockwise direction is positive. The angular velocity of the airship is calculated according to the turning radius r of the airship, .

[0066] Assume that the airship particle moves without noise. Let represent Motion control at a moment, and remains unchanged within the time period, then the airship moves along a circular arc as the Figure 4 trajectory, as .

[0067] denotes the state of the airship at the moment, and the calculation of the position of its turning center is as follows: Therefore, after one time unit, the state of the airship at the moment is: As Figure 5 shown, the motion of the airship is determined by its own airspeed and the wind speed at its location , and the two are in a relationship of vector addition: wherein, is the airship speed.

[0068] In an optional embodiment, the present invention uses the Proximal Policy Optimization algorithm to train and obtain an autonomous guidance policy network. The Proximal Policy Network is an algorithm based on the Actor-Critic framework, which is essentially a policy-based algorithm. While training and optimizing a policy network, an additional evaluation network is learned to help the policy learn better. Therefore, two deep neural networks need to be established in the initialization stage: the Actor network (i.e., the policy network) and the Critic network (i.e., the evaluation network). Among them, the Gaussian probability distribution parameters of the airship action space are obtained by processing the current state using the Actor network, and the actions generated by the Actor are evaluated using the Critic network. The above-mentioned step S130, based on the environment model and the Markov decision process model, uses the deep reinforcement learning algorithm to offline train a pre-constructed agent to obtain an airship autonomous guidance flight agent, including: S1301. Based on the Markov decision process model, construct a framework for the interaction between the agent and the environment model.

[0069] The Markov Decision Process (MDP) is a mathematical model that describes how an agent makes decisions in an environment. In this model, the agent selects an action based on the current state, and the environment returns the next state and a reward based on the action. In this embodiment, a framework for the interaction between the agent and the environment model is constructed based on the MDP. This framework defines how the agent observes the environment, selects actions, receives rewards, and updates its state.

[0070] S1302. Initialize the environment model and the agent; the agent includes a policy network and a value network.

[0071] The environment model was previously constructed by reanalyzing meteorological data and the airship kinematic model, and is used to simulate the flight state of the airship under different meteorological conditions. The agent includes a policy network (Actor Network) and a value network (Critic Network). The policy network is responsible for outputting the probability distribution of actions based on the current state, while the value network is responsible for estimating the value of the current state or the state-action pair.

[0072] S1303. In the said framework, the airship randomly samples an action using the policy network, executes the action, interacts with the environment model, and obtains the next state and the current reward; wherein, the next state includes the airship state parameters and the airship environment parameters at the next moment.

[0073] In the interaction framework, the airship (as the entity of the agent) randomly samples an action using the policy network. This action is the airship's speed, angular velocity, or other control commands. After executing the action, the agent interacts with the environment model to obtain the next state and the current reward. The next state includes the airship state parameters (such as position, speed, attitude, etc.) and the airship environment parameters (such as wind speed, wind direction, etc.) at the next moment.

[0074] S1304. Perform calculation processing on the airship state parameters and the airship environment parameters at the next moment to obtain prior information.

[0075] Perform calculation processing on the airship state parameters and the airship environment parameters at the next moment to obtain prior information. These prior information specifically include: whether the airship has approached the target point, the straight-line distance from the airship to the target point, and the angular difference between the current flight heading of the airship and the direction of the target point.

[0076] After fusing and normalizing the airship state parameters, the airship environment parameters, and the prior information at the next moment, input them into the policy network.

[0077] The process of generating prior information according to the airship state parameters and the airship environment parameters at the next moment in the present invention is as follows: The input information for this calculation process includes the airship state parameters and airship environment parameters at the next moment. The airship state parameters include the position coordinates (x_next, y_next, z_next), velocity v_next, and heading angle at the next moment. The airship environment parameters refer to the position coordinates (x_target, y_target, z_target) of the target point.

[0078] The output information is prior information and consists of the following three parts: Whether approaching the target point: A boolean value used to indicate whether the airship is close enough to the target point.

[0079] Distance to the target point: The straight-line distance between the airship and the target point.

[0080] Angle difference between the heading and the target point: The angular difference between the current heading of the airship and the direction of the target point.

[0081] The calculation process is as follows: First, using the Euclidean distance formula in three-dimensional space, based on the position coordinates (x_next, y_next, z_next) of the airship at the next moment and the position coordinates (x_target, y_target, z_target) of the target point, calculate the straight-line distance between them.

[0082] Secondly, preset a distance threshold. If the calculated straight-line distance is less than or equal to this threshold, it is determined that the airship is close to the target point, and the boolean value True is output; otherwise, False is output.

[0083] Then, based on the relative coordinates of the target point and the airship's position at the next moment, calculate the azimuth angle of the target point relative to the airship. Calculate the difference between the airship's heading angle and the calculated azimuth angle, that is, the angular difference. To ensure that the angular difference value is within a reasonable range (such as - to ), angle normalization processing is required. For example, in a two-dimensional plane, through the relative change amounts (delta_x and delta_y) of the target point and the airship's position, use the atan2 function to calculate the azimuth angle. And determine the azimuth angle as / 2, - / 2 or 0 according to the sign of delta_y (theoretically, delta_x and delta_y cannot be zero at the same time). After calculating the azimuth angle, subtract it from the airship's heading angle to obtain the angular difference value. Subsequently, use the normalization function to adjust the angular difference value to the range of - to range.

[0084] Through the above steps, an output containing key prior information can be generated, which is crucial for the agent to evaluate the current state and adjust the flight strategy during subsequent decision-making processes.

[0085] S1305. After fusing and normalizing the airship state parameters, airship environment parameters, and prior information at the next moment, input them into the policy network to obtain the Gaussian probability action distribution parameters output by the policy network, and sample the next action according to the Gaussian probability action distribution parameters.

[0086] The policy network outputs a Gaussian probability action distribution parameter, which describes the probabilities of different actions. Sample the next action according to the Gaussian probability action distribution parameters, and the agent will continue to execute this action.

[0087] S1306. The agent stores the current state, current action, current reward, and next state as an experience tuple in the data experience pool.

[0088] The agent stores the current state, current action, current reward, and next state as an experience tuple in the data experience pool. This experience pool is used for subsequent updates of the policy network and the evaluation network. When each round of interaction (data collection) between the airship and the environment model is completed, the policy network and the evaluation network are iteratively updated. After learning multiple times, clear the data experience pool and conduct the next round of interaction (data collection).

[0089] S1307. Randomly sample a batch of experience tuples from the data experience pool and use the sampled experience tuples to update the policy network and the evaluation network.

[0090] Randomly sample a batch of experience tuples from the data experience pool. Use the sampled experience tuples to update the policy network and the evaluation network. This usually involves optimization algorithms such as gradient descent to minimize the loss functions (such as the policy gradient loss of the policy network and the mean squared error loss of the evaluation network).

[0091] S1308. The agent continues to interact with the environment model and continuously updates its policy network and evaluation network until convergence is reached or other stopping conditions are met, obtaining the airship autonomous guidance flight agent.

[0092] The agent continues to interact with the environment model, execute the above steps S1303 to S1308, continuously sample new experience tuples and store them in the experience pool. At the same time, the agent continuously samples experience tuples from the experience pool to update the policy network and the evaluation network. This process will continue until the policy network and the evaluation network reach convergence (i.e., the performance is stable and no longer improves significantly) or other stopping conditions are met (such as reaching the maximum number of iterations, training time, etc.).

[0093] Under the constraints of the airship system, the present invention establishes a continuous action space and a continuous observation space that conform to the motion capabilities of the airship, and trains a policy network that can achieve autonomous guidance of the airship based on the proximal policy optimization algorithm. The state space includes airship state parameters, airship environment parameters, and prior information, and the prior information is calculated from the airship state parameters and flight target information (i.e., the position coordinates of the above target points). Adding prior information to the state information greatly improves the convergence speed of the policy network and the training efficiency. Action space sampling and constraint are carried out to sample the action space generated by the policy network and truncate the actions according to the physical characteristics of the airship to ensure that the output actions conform to the actual operation capabilities of the airship.

[0094] The airship autonomous guidance flight intelligent agent trained by the present invention through the deep reinforcement learning algorithm can autonomously make decisions on actions according to the current state and reward signals, realizing the autonomous flight control of the airship. Since the intelligent agent is trained in a simulated environment, various extreme situations and fault scenarios can be simulated, improving the robustness and fault tolerance of the intelligent agent. The intelligent agent can learn flight strategies under different meteorological conditions and adapt to different flight environments and mission requirements. The training process of the intelligent agent can be easily extended to more complex flight tasks and environmental conditions, only by adjusting the environmental model and the reward function.

[0095] In the process of policy learning, the update of the policy network follows the principle of policy gradient. The policy network has two fully connected hidden layers, each layer containing 64 nodes, and the activation function uses the Relu function. The policy network produces two output values: the mean and the standard deviation , where uses tanh as the activation function output, uses Softplus as the activation function output, and these two values determine the action space distribution (Gaussian distribution, as shown in the following formula). In each interaction between the airship and the environment, the airship randomly samples actions from this distribution.

[0096] Specifically, the policy network includes an input layer, hidden layers, and an output layer. The input of the input layer is the current state information of the airship, such as position, speed, heading, etc. There are two fully connected hidden layers, each layer containing 64 nodes. These nodes use the ReLU function as the activation function, which helps to introduce non-linear factors and enhance the expression ability of the network. The output layer outputs two values, the mean and the standard deviation , which together determine the Gaussian distribution of the action space.

[0097] The mean : Use the tanh function as the activation function. The tanh function limits the output between -1 and 1, which makes sense for many physical control tasks because actions often need to be within a certain finite range. The mean represents the central position of the Gaussian distribution, i.e., the most likely action value.

[0098] Standard deviation : Use the Softplus function as the activation function. The Softplus function is a smoothed version of the ReLU function, which ensures that the output is always positive, which is necessary for the standard deviation because the standard deviation cannot be negative. The standard deviation represents the width of the Gaussian distribution, i.e., the uncertainty or exploration degree of the action.

[0099] The above represents the probability density function of the Gaussian distribution, where is the possible action value.

[0100] is the mean, representing the most likely action value. is the standard deviation, representing the uncertainty of the action.

[0101] represents the probability of taking action under the given state , which is given by the probability density function of the Gaussian distribution , where corresponds to action , and has been appropriately scaled and offset according to and (if the action space is not -1 to 1, then needs to be scaled and offset).

[0102] In each round of interaction between the airship and the environment, the Actor network first calculates the mean and the standard deviation based on the current state, and then generates a Gaussian distribution according to these parameters. Then, a random action value is sampled from this distribution, and this action value will be executed to affect the state and reward of the environment.

[0103] The parameters of the Actor network (i.e., the weights and biases of the two hidden layers) will be updated by the policy gradient method. This involves calculating the policy gradient (i.e., the derivative of the action probability with respect to the network parameters) and a reward signal (such as the discounted cumulative reward), and then using these gradients to update the network parameters to maximize the expected reward.

[0104] In the present invention, the proximal policy optimization algorithm uses a truncated method to calculate the loss function of the Actor network.

[0105] Among them, is the probability ratio of the new policy to the old policy at time t. represents the policy function, that is, the probability of taking action under state . is the parameter of the policy function and is optimized through learning. is the parameter of the old policy.

[0106] The loss function of the Actor network is expressed as: Among them, is the advantage function, representing the advantage of taking action under state relative to the current value function estimate. is used to limit the value of the ratio function within a specific range , . is the truncation constant, which is used to help set the range of policy updates, specifically to limit the range of to avoid too large a policy update step size.

[0107] The PPO algorithm optimizes the policy by maximizing this loss function.

[0108] The advantage function (Advantage Function) is calculated as follows: In the formula, and are pre-set hyperparameters. represents the value function, that is, the estimated future cumulative reward under state . is the discount factor, which is used to control the weight of future rewards in the current value evaluation. The discount factor usually takes values between 0 and 1, and the closer it is to 1, the more importance is attached to future rewards. represents the immediate reward obtained at time. The immediate reward is the direct feedback obtained by the agent from the environment after executing an action. represents the value of the state value function at state at time. It is used to calculate the temporal difference error together with the immediate reward to update the estimate of the advantage function. Indicates the number of steps to be pushed backward starting from the current time step

[0109] During the entire training process of proximal policy optimization, the Actor network is updated to maximize the mean value

[0110] In an optional embodiment, the evaluation network adopts a learning method of temporal difference residuals and is updated according to the loss function of the following formula: Where is the loss function of the evaluation network, indicating the loss function of the evaluation network at time step are the parameters of the evaluation network is The reward value at time represents the reward value obtained by the agent from the environment at time step is the discount factor used to calculate the present value of future rewards. It determines the importance of future rewards in the current decision-making is the state at time The state value function value under the state, which is the evaluation of the evaluation network on the state at time is The state at time The state value function value under the state, which is the evaluation of the evaluation network on the state at time step and represents the value estimate of the evaluation network on the state at time step

[0111] During the training process, the parameters of the evaluation network are updated by the gradient descent method to minimize the mean value of the loss function This includes the following steps: 1) Collect data: The agent interacts with the environment to collect data pairs of states, actions, rewards, and next states ( , , , )

[0112] 2) Calculate the target value: For each time step , calculate the target value This is the value predicted by the evaluation network

[0113] 3) Calculate the loss: Use the collected data and the target value to calculate the loss function ​​​​​

[0114] 4) Update parameters: Update the parameters of the evaluation network by gradient descent methods (such as stochastic gradient descent, Adam, etc.) to minimize the mean of the loss function.

[0115] 5) Repeat: Repeat the above steps until the performance of the evaluation network converges or reaches the predetermined number of training rounds.

[0116] In an optional embodiment, the reward function includes the potential energy reward of the airship reaching the target point, the penalty for the airship having too high an airspeed, the reward for flying in the target residence area, and the reward for the airship continuously flying within the legal range; The potential energy reward of the airship reaching the target point, the penalty for the airship having too high an airspeed, the reward for flying in the target residence area, and the reward for the airship continuously flying within the legal range are determined by the following formula: Refer to Figure 7 shown, the training process of the intelligent agent of the present invention includes the following: Initialize the environment: Set the environment for reinforcement learning, including the state space, action space, reward function, etc. This is the place where learning occurs.

[0117] Initialize multiple airships: The airship can be regarded as an instance of the intelligent agent. Initialization means setting the initial state or position for each intelligent agent.

[0118] The airship obtains observation information: The intelligent agent (airship) obtains the state information of the current environment through its sensors or observation devices. This information is the basis for the intelligent agent to make decisions.

[0119] Calculate and obtain prior knowledge: The intelligent agent uses the observation information (i.e., the next state above, including the airship state parameters and airship environment parameters at the next moment) to calculate the prior knowledge (i.e., the prior information above). The prior knowledge helps the intelligent agent better understand the environment and make more effective decisions.

[0120] After fusing the data, input it into the policy network: After the intelligent agent fuses the observation information and the prior knowledge, it inputs them into the policy network. The policy network outputs the action probability distribution according to the input information.

[0121] The policy network calculates the action and inputs it into the environment: The policy network calculates the action according to the action probability distribution and sends the action to the environment for execution.

[0122] The airship obtains feedback rewards and data experience: The environment gives feedback rewards according to the actions executed by the intelligent agent and provides the observation data at the next moment. This information is used to update the policy and prior knowledge of the intelligent agent.

[0123] Check whether there is an airship that meets the interaction termination condition: After each interaction, check whether the policy convergence condition is met (such as the reward reaching a certain threshold, the policy performance being stable, etc.) or whether the preset maximum number of training rounds has been reached. If one of these conditions is met, the interaction process terminates. If no airship meets the interaction termination condition, re-acquire the observation information and repeat the above steps.

[0124] End of training: If all airships meet the interaction termination condition or reach the maximum number of training rounds, the reinforcement learning process ends. At this time, the agent has learned a policy that can maximize the cumulative reward in the given environment.

[0125] The present invention uses the real-time state parameters of the airship, environmental parameters, and the prior information calculated in real time as the input of the policy network, and calculates the optimal flight policy in the current situation in real time, improving the autonomy and adaptability of guidance. Based on the deep reinforcement learning algorithm, in actual flight, the policy can be optimized according to real-time flight data, helping the unmanned airship to become smarter and smarter. The present invention establishes an agent with a continuous action space according to the actual flight ability of the unmanned airship, and has the following advantages compared with the model with a discrete action space: (1) The continuous action space can provide higher action accuracy, enabling the airship to perform more delicate operations. Its angle adjustment, speed control, etc. can be accurate to a very small range, thus realizing a smoother and more accurate flight path; (2) The continuous action space allows actions to be selected within a larger range, enabling the algorithm to explore more possibilities, and thus find a better guidance strategy. The discrete action space may limit the optimization of the policy due to limited actions; (3) The control of the airship is usually a continuous process, and the adjustment of parameters such as speed and flight angle changes continuously. Using a continuous action space can more naturally model these physical processes, avoiding the errors and complexities brought by discretizing continuous control; (4) In some cases, the deep reinforcement learning algorithm using a continuous action space will have better convergence and training effects because they can perform more detailed policy updates through gradient information; (5) In complex tasks, discretizing the action space may require combining multiple continuous control variables into discrete actions, which will lead to an explosion in the dimension of the action space, increasing the complexity of learning and calculation. The continuous action space can avoid this situation. The present invention does not require the import of weather forecast data. The input of the policy network is only the state parameters of the airship and the real-time airship environmental parameters that can be measured by the airship, and can realize the edge deployment of the policy network to achieve autonomous guided flight of the airship.

[0126] The autonomous guidance flight device of an airship based on a deep reinforcement learning algorithm provided by the present invention will be described below. The autonomous guidance flight device of an airship based on a deep reinforcement learning algorithm described below can be correspondingly referred to the autonomous guidance flight method of an airship based on a deep reinforcement learning algorithm described above.

[0127] The autonomous guidance flight device of an airship based on a deep reinforcement learning algorithm provided by the present invention, referring to Figure 8 as shown, includes: An acquisition module 210, configured to acquire reanalysis meteorological data and an airship kinematic model, and construct an environment model by using the reanalysis meteorological data and the airship kinematic model; A modeling module 220, configured to establish a Markov decision process model, wherein an action space of the Markov decision process model is designed according to the environment model; A training module 230, configured to offline train a pre-constructed agent by using the deep reinforcement learning algorithm based on the environment model and the Markov decision process model, to obtain an autonomous guidance flight agent of the airship; A determination module 240, configured to determine an autonomous guidance flight strategy based on the autonomous guidance flight agent of the airship.

[0128] Figure 9 An entity structure diagram of an electronic device is exemplified, as Figure 9 shown. The electronic device may include: a processor 310, a communication interface 320, a memory 330, and a communication bus 340. Among them, the processor 310, the communication interface 320, and the memory 330 complete communication with each other through the communication bus 340. The processor 310 may call logical instructions in the memory 330 to execute the autonomous guidance flight method of an airship based on a deep reinforcement learning algorithm.

[0129] In addition, when the logical instructions in the above-mentioned memory 330 are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), magnetic disks, or optical discs that can store program codes.

[0130] On the other hand, the present invention also provides a computer program product. The computer program product includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the airship autonomous guidance flight method based on the deep reinforcement learning algorithm provided by the above-mentioned various methods.

[0131] On another aspect, the present invention also provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it is implemented to execute the airship autonomous guidance flight method based on the deep reinforcement learning algorithm provided by the above-mentioned various methods.

[0132] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art can understand and implement it without creative labor.

[0133] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on such an understanding, the technical solution, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disks, optical discs, etc., and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments.

[0134] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements for some of the technical features. These modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An autonomous guidance flight method for an airship based on a deep reinforcement learning algorithm, characterized in that, Including: Obtain reanalysis meteorological data and an airship kinematic model, and construct an environmental model using the reanalysis meteorological data and the airship kinematic model; Establish a Markov decision process model, wherein the action space of the Markov decision process model is designed according to the environmental model; Based on the environmental model and the Markov decision process model, use a deep reinforcement learning algorithm to offline train a pre-constructed agent to obtain an airship autonomous guidance flight agent; Determine an autonomous guidance flight strategy based on the airship autonomous guidance flight agent.

2. The autonomous guidance flight method of an airship based on a deep reinforcement learning algorithm according to claim 1, wherein The obtaining reanalysis meteorological data and an airship kinematic model, and constructing an environmental model using the reanalysis meteorological data and the airship kinematic model includes: Obtain reanalysis meteorological data, and extract wind speed data and wind direction data within a preset time period and geographical range from the reanalysis meteorological data; Interpolate between the wind speed data and the wind direction data within the preset time period and geographical range to obtain updated reanalysis data; Train a pre-constructed airship kinematic model based on the updated reanalysis data to obtain the environmental model; the input of the airship kinematic model is the speed and angular velocity of the airship, and the output is the position and attitude of the airship.

3. The autonomous guidance flight method of an airship based on a deep reinforcement learning algorithm according to claim 1, characterized in that, The Markov decision process model includes a state space, an action space, a state transition probability matrix, a reward function, and a discount factor; the state space includes airship state parameters, airship environmental parameters, and prior information; The airship state parameters include the airship position, the airship attitude, and the airship airspeed, the airship environmental parameters include the wind speed and the wind direction, and the prior information includes whether the airship has reached near the target point at present, the distance from the airship to the target point, and the angle difference between the airship flight heading and its angle to the flight target point.

4. The autonomous guidance flight method of an airship based on a deep reinforcement learning algorithm according to claim 3, characterized in that, The based on the environmental model and the Markov decision process model, using a deep reinforcement learning algorithm to offline train a pre-constructed agent to obtain an airship autonomous guidance flight agent includes: Based on the Markov decision process model, construct a framework for the interaction between the agent and the environmental model; Initialize the environmental model and the agent; the agent includes a policy network and a value network; In the framework, the airship randomly samples an action using the policy network, executes the action, and interacts with the environmental model to obtain the next state and the current reward; wherein, the next state includes the airship state parameters and the airship environmental parameters at the next moment; Perform calculation processing on the airship state parameters and the airship environmental parameters at the next moment to obtain prior information; After fusing and normalizing the airship state parameters, the airship environmental parameters, and the prior information at the next moment, input them into the policy network to obtain the Gaussian probability action distribution parameters output by the policy network, and sample the next action according to the Gaussian probability action distribution parameters; The agent stores the current state, the current action, the current reward, and the next state as an experience tuple in the data experience pool; Randomly sample a batch of experience tuples from the data experience pool, and use the sampled experience tuples to update the policy network and the value network; The agent continues to interact with the environmental model and continuously updates its policy network and evaluation network until convergence is achieved or other stopping conditions are met, obtaining the autonomous guidance flight agent for the airship.

5. The autonomous guidance flight method of an airship based on the deep reinforcement learning algorithm according to claim 4, characterized in that, The evaluation network adopts a learning method of temporal difference residuals and is updated according to the loss function in the following formula: Among them, is the loss function of the evaluation network, is the reward value at time is the discount factor, is the evaluation of the evaluation network for the state at time is the evaluation of the evaluation network for the state at time 6. The autonomous guidance flight method of an airship based on the deep reinforcement learning algorithm according to claim 3, wherein The reward function includes potential energy reward for the airship to reach the target point, penalty for excessive airspeed of the airship, reward for flying in the target residence area, and reward for the airship continuously flying within the legal range; The potential energy reward for the airship to reach the target point, penalty for excessive airspeed of the airship, reward for flying in the target residence area, and reward for the airship continuously flying within the legal range are determined by the following formula: Among them, is the potential energy reward for the airship to reach the target point, is the distance from the current position to the target point, is the penalty for the excessive airspeed of the airship, is the airspeed of the airship, is the reward for flying in the target residence area, is the reward for the airship to continuously fly within the legal range.

7. An airship autonomous guidance flight device based on a deep reinforcement learning algorithm, characterized in that, Including: An acquisition module for acquiring reanalysis meteorological data and the kinematic model of the airship, and constructing an environmental model using the reanalysis meteorological data and the kinematic model of the airship; A modeling module for establishing a Markov decision process model, wherein the action space of the Markov decision process model is designed according to the environmental model; A training module for offline training a pre-constructed agent based on the environmental model and the Markov decision process model using a deep reinforcement learning algorithm to obtain an autonomous guidance flight agent for the airship; A determination module for determining an autonomous guidance flight strategy based on the autonomous guidance flight agent for the airship.

8. An electronic device, comprising a memory, a processor, and a computer program stored on the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the airship autonomous guidance flight method based on the deep reinforcement learning algorithm according to any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the airship autonomous guidance flight method based on the deep reinforcement learning algorithm according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the airship autonomous guidance flight method based on the deep reinforcement learning algorithm according to any one of claims 1 to 6.

Images