Intelligent control method for high-speed aircraft

Abstract

The invention relates to an intelligent control method for a high-speed aircraft, and the method comprises the steps: building an aircraft control architecture, carrying out the decision-making of a neural network model, training a decision-making neural network, and carrying out the real-time compensation of a control instruction according to the flight state of the aircraft, so as to achieve the control of a variable-configuration high-speed aircraft. An aircraft controller adopts a model + learning double-layer architecture, a decision neural network is designed to be of a Double Network structure, overload errors of pitching, yawing and rolling channels are considered in a reward function design, training of a decision neural network model is based on a double-delay deterministic strategy gradient algorithm, a training process is optimized by adopting a prior experience playback technology and a target strategy smoothing mechanism, and a decision neural network model is obtained. The control system can achieve real-time intelligent adjustment of control instructions, overload quantities and attitude angles of three channels of the aircraft and corresponding rudder deflection instructions output by the controller at the last moment are obtained from the environment, the intelligent body outputs control instructions of pitching, yawing and rolling channels, and therefore intelligent control over the variable-configuration high-speed aircraft is achieved.

Description

Technical Field

[0001] The present invention relates to the technical field of high-speed aircraft control, and in particular to an intelligent control method for a high-speed aircraft, which is mainly applicable to the control of a variable-configuration high-speed aircraft under conditions of configuration changes and rapid changes in environmental parameters. Background Art

[0002] Variable-configuration high-speed aircraft have the characteristics of complex flight environment, large flight envelope, drastic changes in aerodynamic characteristics, and serious coupling of motion models, which lead to significant changes in the quality of the control object, the characteristics of the control actuators, and the control method during the flight. This also increases the difficulty of aircraft control and puts forward strict requirements for the control system design. Therefore, it is of great significance to study the intelligent control method suitable for variable-configuration high-speed aircraft to ensure that it can maintain good flight performance and complete the attack on the predetermined target when the configuration changes and the flight environment and other factors change.

[0003] Deep reinforcement learning technology has been widely studied and applied in the fields of artificial intelligence and automatic control, and is considered to be one of the core technologies for designing intelligent systems. It can process uncertain information and solve uncertain faults, making the aircraft control system suitable for changing flight environments and having the ability to adapt to harsh environments. Therefore, how to combine deep reinforcement learning algorithms with traditional controllers to achieve intelligent control of variable-configuration high-speed aircraft is an urgent problem to be solved. Summary of the invention

[0004] The technical problems to be solved by the present invention are:

[0005] In order to avoid the shortcomings of the prior art, the present invention provides a high-speed aircraft intelligent control method for solving the problem of unstable control under the conditions of configuration changes and rapid changes in environmental parameters in a high-speed aircraft penetration scenario.

[0006] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0007] A high-speed aircraft intelligent control method is characterized by adopting a two-layer controller architecture, the first layer is a PID nominal controller, the input of which is a reference control instruction, and the output is a three-channel rudder deflection control instruction [δ′ z ;δ′ y ;δ′ x ]; The second layer is an intelligent controller based on the TD3 algorithm, which is used to compensate for the control instructions of the PID nominal controller of the first layer. The input is the overload, attitude angle and three-channel control instructions output at the last moment of the aircraft. The output is a three-channel compensation rudder deflection control command [δ″ z ; δ″ y ; δ″x ]; the control instruction [δ′ output by the first layer controller z ;δ′ y ;δ′ x ] and the control instructions output by the second-layer controller [δ″ z ; δ″ y ; δ″ x ] is superimposed as the output instruction of the entire double-layer controller [δ z ; δ y ; δ x ], where δ z is the pitch rudder control command, δ y is the yaw control command, δ x It is the roll rudder control command.

[0008] A further technical solution of the present invention is as follows: the intelligent controller based on the TD3 algorithm is a TD3 algorithm decision neural network model, which adopts a Double Network network structure, that is, the entire network structure is composed of six deep neural networks, namely, Actor network, Critic1 network, Critic2 network, TargetActor network, Target Critic1 network, and TargetCritic2 network;

[0009] The actions generated by the Actor network are used as inputs to the Critic1 and Critic2 networks to estimate the corresponding Q values. The Q values ​​provided by the two Critic networks are used to evaluate the strategies generated by the Actor network. Backpropagation updates the parameters of the Actor network to adjust the strategies in a more optimal direction.

[0010] The Target Actor network is copied from the main Actor network and provides actions for calculating the target value. The TargetCritic1 and Target Critic2 networks are copied from the main Critic1 and Critic2 networks and use the actions and next states generated by the TargetActor to estimate the target Q value.

[0011] The weights of the Actor network, Critic1 network, and Critic2 network are updated in real time according to the double-delayed deterministic policy gradient theorem during the learning process, denoted as Actor_e, Critic1_e, and Critic2_e, and the network weight coefficients are θ μ , and The Target Actor network, Target Critic1 network, and Target Critic2 network update their parameters by tracking the parameters of the previous pair of networks, denoted as Actor_t, Critic1_t, and Critic2_t, with weights θ μ′ , and

[0012] The decision neural network uses two sets of Critic networks. When calculating the target value, the smaller value of the two is taken to suppress the network over-estimation problem caused by bootstrapping and maximization. In order to calculate the expected future reward, the Actor target network is first used to estimate the action in the next state:

[0013] a′=μ′(s′|θ μ′ )

[0014] Add noise ε to the action to increase the exploratory nature of the algorithm. The noise follows a normal distribution:

[0015] a′=a′+ε

[0016] The TD3 algorithm uses two Critic target networks to estimate the Q value of the action, as shown in the following formula:

[0017]

[0018]

[0019] To avoid overestimation, the minimum of the two is selected as the target Q value, as shown in the following formula:

[0020]

[0021] A further technical solution of the present invention is as follows: the reward function of the TD3 algorithm decision neural network model is the superposition of the reward functions of the pitch channel, the yaw channel and the roll channel, and the reward function of the pitch channel is:

[0022]

[0023] in, is the pitch overload error, M t It is the logic judgment value. When the pitch overload error When M t =0, otherwise M t =1;

[0024] The reward function of the yaw channel:

[0025]

[0026] The reward function of the rolling channel:

[0027]

[0028] in, and They are yaw overload error and roll overload error respectively.

[0029] A further technical solution of the present invention is to train the TD3 algorithm decision neural network model based on a double-delayed deterministic policy gradient algorithm, and the deterministic strategy is denoted as π θ (s), which means that the action a in state s is obtained s→a under the policy of parameter θ;

[0030] The network weight coefficient θ of the Actor network μ Update according to the deterministic policy gradient principle, that is,

[0031]

[0032]

[0033] Among them, α is the learning rate of the network weight coefficient of the network Actor network, J(θ μ ) is the expected reward;

[0034] Both Critic1 and Critic2 networks are updated by minimizing the error between the evaluation value and the target value. The specific method is to use the gradient descent algorithm to minimize the error L between the evaluation value and the target value. ci :

[0035]

[0036] After Critic1 and Critic2 networks are updated d steps later, the Actor network is updated; first, the Actor network is used to calculate the action in state s:

[0037] a new =μ(s|θ μ )

[0038] Then use Critic1 or Critic2 network to calculate the state action pair (s, a new ), assuming that the Critic1 network is used:

[0039]

[0040] Finally, the gradient ascent algorithm is used to maximize q new , thereby completing the update of the Actor network.

[0041] A further technical solution of the present invention: During the training process,

[0042] The "soft update" strategy is adopted, and a small weight factor τ∈(0,1) is used to weight the current network and the target network. The specific method is to update the Critic first after the training and learning begins, and then update the Actor after the Critic is updated several times, and then soft update the target Actor and the two target Critic:

[0043]

[0044] A further technical solution of the present invention: During the training process,

[0045] The target strategy smoothing method is adopted. In order to reduce the variance and improve the accuracy of the estimated value of Q(s,a) and the stability of the network, a small amount of random noise is added to the action, and the noise obeys the normal distribution;

[0046] ε~N(0,σ)

[0047] Among them, N(0,σ) is a normal distribution with mean zero and variance σ;

[0048] And find the average value in the small batch sampling to approximate the expectation of the action; calculate the target y value as:

[0049]

[0050] A further technical solution of the present invention: During the training process,

[0051] The sample data set is established by using the priority experience replay mechanism, starting from the most recent sample data set (s t ,a t ,r t ,s t+1 ) randomly extracts a minibatch data set as a learning sample, thereby eliminating the data association between samples.

[0052] A computer system, characterized in that it comprises: one or more processors, and a computer-readable storage medium for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the above-mentioned method.

[0053] A computer-readable storage medium, characterized in that it stores computer-executable instructions, which are used to implement the above method when executed.

[0054] A computer program product, characterized by comprising computer executable instructions, wherein the instructions are used to implement the above method when executed.

[0055] The beneficial effects of the present invention are:

[0056] The invention provides a high-speed aircraft intelligent control method, which applies the deep reinforcement learning algorithm to the traditional PID control architecture, designs the Double Network decision neural network structure and reward function, and uses the priority experience replay technology and the target strategy smoothing mechanism to optimize the training process of the variable configuration high-speed aircraft intelligent control model, forming a new type of intelligent control method suitable for variable configuration high-speed aircraft. Compared with the prior art, the advantages are as follows:

[0057] 1. Improvements based on the classic controller structure can maximize the use of existing design experience and design ideas to ensure the stability of the controller. The TD3 algorithm avoids the problem of network overestimation, making value assessment more accurate and training more stable. The design idea of ​​using the dual-drive architecture of classic control + deep reinforcement learning can combine existing achievements with advanced algorithms, give full play to the advantages of both, and achieve the best compromise between stability and dynamic performance.

[0058] 2. Unlike adaptive control, the design method using deep reinforcement learning can avoid the problems of too many parameters and complex forms in traditional controller design. It can achieve control parameter design and optimization through training, thereby greatly reducing the designer's workload, improving design efficiency, and improving the automation and intelligence level of the entire system. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] The drawings are only for the purpose of illustrating particular embodiments and are not to be considered limiting of the present invention. Like reference symbols denote like components throughout the drawings.

[0060] Figure 1 It is a block diagram of a dual-layer architecture of aircraft intelligent control provided by an embodiment of the present invention;

[0061] Figure 2 It is a TD3 decision neural network structure diagram provided by an embodiment of the present invention;

[0062] Figure 3 It is a framework diagram of the principle of deep reinforcement learning provided by the aircraft example of the present invention;

[0063] Figure 4 This is a TD3 algorithm decision neural network training process diagram provided by an embodiment of the present invention;

[0064] Figure 5 This is a comparison diagram of TD3 and PID control effects under a +20% aerodynamic coefficient deviation condition provided by an embodiment of the present invention;

[0065] Figure 6This is a comparison diagram of TD3 and PID control effects under a -20% aerodynamic coefficient positive pull condition provided by an embodiment of the present invention;

[0066] Figure 7 This is a graph of 100 groups of Monte Carlo simulation results under the condition of random deviation of aerodynamic coefficients of [-20%, +20%] provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0067] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0068] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchangeable where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0069] The present invention provides a high-speed aircraft intelligent control method, including constructing an aircraft control architecture, constructing a decision neural network model, training the decision neural network, and performing real-time compensation for control instructions according to the flight state of the aircraft to achieve control of a variable-configuration high-speed aircraft. The aircraft controller adopts a model + learning double-layer architecture, the decision neural network is designed as a Double Network structure, the reward function design takes into account the overload errors of the pitch, yaw, and roll channels, the training of the decision neural network model is based on a double-delay deterministic policy gradient algorithm, and the training process is optimized by using a priority experience replay technology and a target policy smoothing mechanism, so that the control system can achieve real-time intelligent adjustment of the control instructions, obtain the overload of the three channels of the aircraft, the attitude angle and the corresponding rudder deviation instruction output by the controller at the previous moment from the environment, and the intelligent body outputs the control instructions of the pitch, yaw, and roll channels to achieve intelligent control of the variable-configuration high-speed aircraft.

[0070] The specific steps are as follows:

[0071] (1) Construct a variable-configuration high-speed aircraft controller architecture;

[0072] (2) Construct the TD3 algorithm decision neural network model;

[0073] (3) training the decision neural network and optimizing the training process of deep reinforcement learning by using a double-delayed deep deterministic policy gradient mechanism, so that the control system can realize intelligent adjustment of the three-channel rudder deflection commands of pitch, yaw and roll;

[0074] (4) The intelligent controller based on the TD3 algorithm controls the flight of the aircraft and adjusts the three-channel rudder commands of pitch, yaw, and roll in real time according to the flight status of the aircraft to achieve control of the variable-configuration high-speed aircraft.

[0075] Specifically, constructing a variable-configuration high-speed aircraft controller architecture includes:

[0076] Construct a two-layer controller architecture for a high-speed aircraft with variable configuration: the first layer is a PID nominal controller, whose input is the reference control command and output is the three-channel rudder deflection control command [δ′ z ;δ′ y ;δ′ x ], used to control the attitude of the aircraft; the second layer is an intelligent controller based on the TD3 algorithm, which is used to compensate for the control instructions of the PID nominal controller of the first layer. The input is the overload of the aircraft, the attitude angle and the three-channel control instructions output at the last moment. The output is a three-channel compensation rudder deflection control command [δ″ z ; δ″ y ; δ″ x ]. The control instruction [δ′ output by the first layer controller z ;δ′ y ;δ′ x ] and the control instructions output by the second-layer controller [δ″ z ; δ″ y ; δ″ x ] is superimposed as the output instruction of the entire double-layer controller [δ z ; δ y ; δ x ], where δ z is the pitch rudder control command, δ y is the yaw rudder control command, δ x It is the rolling rudder control command;

[0077] Specifically, building a decision neural network model includes:

[0078] Construct the TD3 algorithm decision neural network model, using the Double Network network structure, its structure diagram is as follows Figure 2As shown in the figure, the entire network structure consists of six deep neural networks, namely Actor network, Critic1 network, Critic2 network, Target Actor network, Target Critic1 network, and Target Critic2 network. The actions generated by the Actor network are used as inputs to the Critic1 network and the Critic2 network to estimate the corresponding Q value, while the Q values ​​provided by the two Critic networks are used to evaluate the strategies generated by the Actor network, and the parameters of the Actor network are updated by back propagation to adjust the strategies in a more optimal direction. The Target Actor network is copied from the main Actor network and provides actions for calculating the target value. The Target Critic1 and Target Critic2 networks are copied from the main Critic1 and Critic2 networks, and the target Q value is estimated using the actions generated by the TargetActor and the next state. For the attitude control problem of variable-configuration high-speed aircraft, the input of the TD3 decision neural network is the overload of the aircraft, the attitude angle of the aircraft, and the three-channel control instructions output at the previous moment, and the output is the three-channel compensation rudder deviation control instruction.

[0079] During the operation of the algorithm, the weights of the Actor network, Critic1 network, and Critic2 network are updated in real time according to the double-delay deterministic policy gradient theorem during the learning process, denoted as Actor_e, Critic1_e, and Critic2_e, and the network weight coefficients are θ μ , and The Target Actor network, Target Critic1 network, and Target Critic2 network update their parameters by tracking the parameters of the previous pair of networks, denoted as Actor_t, Critic1_t, and Critic2_t, with weights θ μ′ , and

[0080] Specifically, the network structure of Double Network includes:

[0081] The decision neural network described above uses two sets of Critic networks, and takes the smaller value of the two when calculating the target value, thereby suppressing the network over-estimation problem caused by bootstrapping and maximization. In order to calculate the expected future reward, the Actor target network is first used to estimate the action in the next state:

[0082] a′=′(s′|θ μ′ ) (1)

[0083] Add noise ε to the action to increase the exploratory nature of the algorithm. The noise follows a normal distribution:

[0084] a′=a′+ε (2)

[0085] The TD3 algorithm uses two critic target networks to estimate the Q value of the action, as shown in formula (3):

[0086]

[0087] In order to avoid overestimation, the minimum value of the two is selected as the target Q value, as shown in formula (4)

[0088]

[0089] Specifically, the reward function includes:

[0090] The reward function design, taking the pitch channel as an example, takes into account both pitch overload and pitch overload error. The optimization goal is to stabilize the tracking signal within a certain period of time and minimize overshoot, response time and other indicators. Based on this, the reward function of the pitch channel is:

[0091]

[0092] in, is the pitch overload error, M t It is the logic judgment value. When the pitch overload error When M t =0, otherwise M t =1.

[0093] Similar to the pitch channel, the reward functions of the yaw channel and the roll channel are

[0094]

[0095] in, and They are the yaw overload error and the roll overload error respectively. The reward functions of the pitch channel, yaw channel and roll channel are superimposed to obtain the final reward function:

[0096] r=r x +r y +r z (8)

[0097] Specifically, the training steps of the decision neural network include:

[0098] The decision neural network model is trained based on the TD3 algorithm, and the deterministic strategy is denoted as π θ(s), indicating that the action a in state s is obtained s→a under the policy of parameter θ.

[0099] The network weight coefficient θ of the Actor network μ Update according to the deterministic policy gradient principle, that is,

[0100]

[0101] Among them, α is the learning rate of the network weight coefficient of the network Actor network, J(θ μ ) is the expected reward.

[0102] Both Critic1 and Critic2 networks are updated by minimizing the error between the evaluation value and the target value. The specific method is to minimize the error L between the evaluation value and the target value using the gradient descent algorithm. ci :

[0103]

[0104] After Critic1 and Critic2 networks are updated d steps later, the Actor network is updated. First, the Actor network is used to calculate the action in state s:

[0105] a new =μ(s|θ μ ) (11)

[0106] Then use Critic1 or Critic2 network to calculate the state action pair (s, a new ), assuming that the Critic1 network is used:

[0107]

[0108] Finally, the gradient ascent algorithm is used to maximize q new , thereby completing the update of the Actor network.

[0109] Specifically, the decision neural network parameter update strategy includes:

[0110] In order to reduce the error, reduce the variance and improve the stability of the algorithm, a "soft update" strategy is adopted, using a small weight factor τ∈(0,1) to weighted average the current network and the target network to ensure the smoothness and stability of the update. The specific method is to update the Critic first after starting training, and then update the Actor after the Critic is updated several times, and then soft update the target Actor and the two target Critic:

[0111]

[0112] Specifically, the target strategy smoothing method includes:

[0113] The target strategy smoothing method is adopted. Target strategy smoothing is a regularization strategy for deep value learning. In the actual training process, in order to reduce the variance and improve the accuracy of the estimated value of Q(s,a) and the stability of the network, a small amount of random noise is added to the action, and the noise obeys the normal distribution:

[0114] ε~N(0,σ)(14)

[0115] Where N(0,σ) is a normal distribution with mean zero and variance σ.

[0116] And find the average value in the small batch sampling to approximate the expectation of the action. The target y value can be calculated as:

[0117]

[0118] Specifically, the priority experience playback mechanism includes:

[0119] Using the priority experience replay mechanism, during the training process of the TD3 algorithm, each sample is sequential, and there is a correlation between the previous and next samples. In order to eliminate the correlation between samples, the priority experience replay mechanism is used to establish a sample data set. During the training process, the most recent sample data set (s t ,a t ,r t ,s t+1 ) randomly extracts a minibatch data set as a learning sample, thereby eliminating the data association between samples.

[0120] Specifically, the steps of controlling the aircraft using the intelligent controller include:

[0121] When introducing deep reinforcement learning algorithms for design, it is necessary to model the interaction between the original problem environment and the intelligent agent, including the selection of observations, the adjustment of network structure and parameters, and the setting of various reinforcement learning hyperparameters. The observations obtained from the environment are designed as three-channel overload [n z ;n y ;n x ], attitude angle [α; β; γ] and the corresponding controller output rudder deflection command at the previous moment observation The output action of the intelligent agent is designed as the compensation control instruction of the pitch channel, yaw channel and roll channel. The output action of the intelligent agent is [δ″ z ; δ″ y ; δ″x ].

[0122] The effects of this application are described in detail below in conjunction with simulation.

[0123] In one embodiment of the present invention, the TD3 algorithm is used to train the decision neural network 300 times. Under the conditions of ±20% aerodynamic coefficient deviation and random deviation, the pitch channel and roll channel of the aircraft are controlled.

[0124] In one embodiment of the present invention, the PID controller gain coefficient is used as the baseline, the optimized Δδ, i.e., the range of Action, is normalized to [-1, 1], and the training sampling time is set to t s = 0.01s, the observation obtained from the environment is designed to be the overload of the three channels [n z ;n y ;n x ], attitude angle [α; β; γ] and the corresponding controller output rudder deflection command at the previous moment observation The output action of the intelligent agent is designed as the control instruction of the pitch channel, yaw channel and roll channel. The output action of the intelligent agent is [δ″ z ; δ″ y ; δ″ x ],like Figure 3 shown.

[0125] In one embodiment of the present invention, both the controller trained by TD3 and the traditional PID controller can stabilize the aircraft, and the control effect is as follows: Figure 5 , Figure 6 As shown in Figure 2, the error between the overload command and the true value of the trained intelligent controller is smaller, and the control accuracy is better. Figure 7 As shown in the figure, 100 sets of Monte Carlo simulation results are statistically analyzed. Compared with the PID controller, the average control accuracy of the pitch channel command tracking of the TD3 training results is improved by 51.8%, and the average control accuracy of the pitch channel command tracking is improved by 17.39%, and the control performance is improved.

[0126] The above description is only a specific implementation mode of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should be included in the protection scope of the present invention.

Claims

1. A high-speed aircraft intelligent control method, characterized in that: A two-layer controller architecture is adopted. The first layer is a PID nominal controller, whose input is the reference control command and output is the three-channel rudder deflection control command [δ z ′;δ y ′;δ x The second layer is an intelligent controller based on the TD3 algorithm, which is used to compensate for the control instructions of the PID nominal controller of the first layer. The input is the overload, attitude angle and three-channel control instructions output at the last moment of the aircraft. The output is a three-channel compensation rudder deflection control command [δ z ″;δ y ″;δ x ″]; the control instruction output by the first layer controller [δ z ′;δ y ';δ x ′] and the control instructions output by the second-layer controller [δ z ″;δ y ″;δ x ″] is superimposed as the output instruction of the entire double-layer controller [δ z ; δ y ; δ x ], where δ z is the pitch rudder control command, δ y is the yaw control command, δ x It is the roll rudder control command.

2. The high-speed aircraft intelligent control method according to claim 1, characterized in that: The intelligent controller based on the TD3 algorithm is a TD3 algorithm decision neural network model, which adopts a Double Network network structure, that is, the entire network structure is composed of six deep neural networks, namely, Actor network, Critic1 network, Critic2 network, TargetActor network, Target Critic1 network, and Target Critic2 network; The actions generated by the Actor network are used as inputs to the Critic1 and Critic2 networks to estimate the corresponding Q values. The Q values ​​provided by the two Critic networks are used to evaluate the strategies generated by the Actor network. Backpropagation updates the parameters of the Actor network to adjust the strategies in a more optimal direction. The Target Actor network is copied from the main Actor network and provides actions for calculating the target value. The TargetCritic1 and Target Critic2 networks are copied from the main Critic1 and Critic2 networks and use the actions and next states generated by the Target Actor to estimate the target Q value. The weights of the Actor network, Critic1 network, and Critic2 network are updated in real time according to the double-delayed deterministic policy gradient theorem during the learning process, denoted as Actor_e, Critic1_e, and Critic2_e, and the network weight coefficients are θ μ , and The Target Actor network, Target Critic1 network, and Target Critic2 network update their parameters by tracking the parameters of the previous pair of networks, denoted as Actor_t, Critic1_t, and Critic2_t, with weights θ μ′ , and The decision neural network uses two sets of Critic networks. When calculating the target value, the smaller value of the two is taken to suppress the network over-estimation problem caused by bootstrapping and maximization. In order to calculate the expected future reward, the Actor target network is first used to estimate the action in the next state: a′=μ′(s′|θ μ′ ) Add noise ε to the action to increase the exploratory nature of the algorithm. The noise follows a normal distribution: a′=a′+ε The TD3 algorithm uses two Critic target networks to estimate the Q value of the action, as shown in the following formula: To avoid overestimation, the minimum of the two is selected as the target Q value, as shown in the following formula:

3. The high-speed aircraft intelligent control method according to claim 2, characterized in that: The reward function of the TD3 algorithm decision neural network model is the superposition of the reward functions of the pitch channel, yaw channel and roll channel. The reward function of the pitch channel is: in, is the pitch overload error, M t It is the logic judgment value. When the pitch overload error When M t =0, otherwise M t =1; The reward function of the yaw channel: The reward function of the rolling channel: in, and They are yaw overload error and roll overload error respectively.

4. The high-speed aircraft intelligent control method according to claim 3, characterized in that: The TD3 algorithm decision neural network model is trained based on the double-delay deterministic policy gradient algorithm, and the deterministic strategy is recorded as π θ (s), which means that the action a in state s is obtained s→a under the policy of parameter θ; The network weight coefficient θ of the Actor network μ Update according to the deterministic policy gradient principle, that is, Among them, α is the learning rate of the network weight coefficient of the network Actor network, J(θ μ ) is the expected reward; Both Critic1 and Critic2 networks are updated by minimizing the error between the evaluation value and the target value. The specific method is to use the gradient descent algorithm to minimize the error L between the evaluation value and the target value. ci : After Critic1 and Critic2 networks are updated d steps later, the Actor network is updated; first, the Actor network is used to calculate the action in state s: a new =μ(s|θ μ ) Then use Critic1 or Critic2 network to calculate the state action pair (s, a new ), assuming that the Critic1 network is used: Finally, the gradient ascent algorithm is used to maximize q new , thereby completing the update of the Actor network.

5. The high-speed aircraft intelligent control method according to claim 4, characterized in that: During training, The "soft update" strategy is adopted, and a small weight factor τ∈(0,1) is used to weight the current network and the target network. The specific method is to update the Critic first after the training and learning starts, and then update the Actor after the Critic is updated several times, and then soft update the target Actor and the two target Critic:

6. A high-speed aircraft intelligent control method according to claim 5, characterized in that: During training, The target strategy smoothing method is adopted. In order to reduce the variance and improve the accuracy of the estimated value of Q(s,a) and the stability of the network, a small amount of random noise is added to the action, and the noise obeys the normal distribution; ε~N(0,σ) Among them, N(0,σ) is a normal distribution with mean zero and variance σ; And find the average value in the small batch sampling to approximate the expectation of the action; calculate the target y value as:

7. The high-speed aircraft intelligent control method according to claim 6, characterized in that: During training, The sample data set is established by using the priority experience replay mechanism, starting from the most recent sample data set (s t ,a t ,r t ,s t+1 ) randomly extracts a minibatch data set as a learning sample, thereby eliminating the data association between samples.

8. A computer system, characterized in that include: One or more processors, and a computer-readable storage medium for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors are enabled to implement the method of claim 1.

9. A computer-readable storage medium, characterized in that Computer executable instructions are stored, and when the instructions are executed, they are used to implement the method of claim 1.

10. A computer program product, characterized in that The invention comprises computer executable instructions, which are used to implement the method of claim 1 when being executed.

Images