A go-around control method based on an improved TD3 algorithm

Abstract

The application provides a go-around control method based on an improved TD3 algorithm, and belongs to the technical field of aerospace. The application comprises the following steps: S1, building a carrier-based aircraft model; S2, constructing a neural network model of a segmented activation structure, using the neural network model to construct a target policy network and a policy network, and further obtaining an improved TD3 reinforcement learning algorithm; S3, performing parameter configuration on the improved TD3 reinforcement learning algorithm, and selecting parameters from the carrier-based aircraft model to configure state observation, action output and a reward function; S4, designing a go-around control model based on the improved TD3 reinforcement learning algorithm; S5, calculating go-around area parameters; and S6, determining a go-around decision method according to the calculated go-around area parameters. The application can output optimal elevator deflection instructions, realizes angle of attack keeping, does not need to depend on complex preset control laws or a large number of manual parameter adjustment, and meets the constraints and requirements of actual go-around operation.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of aerospace technology, and particularly relates to a complex flight control method based on an improved TD3 algorithm. BACKGROUND

[0002] When the carrier platform sails in the sea, it is affected by the sea wind, sea waves and ocean currents, and will produce six degrees of freedom motion of roll, pitch, yaw, sway, surge and heave. At the same time, the sea surface atmospheric environment will form a complex and changeable ship wake field at the tail of the carrier platform, and in addition, the deck space of the carrier platform is limited, so the carrier aircraft approach and landing is a very difficult and dangerous task. According to statistics, the carrier aircraft approach and landing stage accounts for only one fourth of the total task time, but the accident rate is as high as 50%. How to ensure that the carrier aircraft can quickly take off after a failed landing to achieve safe complex flight is one of the important research directions to improve flight safety.

[0003] Therefore, it is of great significance to provide a complex flight control method based on an improved TD3 algorithm, which can effectively reduce the complex flight area of the carrier aircraft, improve the complex flight performance, reduce the accident rate during the complex flight of the carrier aircraft, ensure the safety of the pilot and the aircraft, improve the adaptability of the carrier aircraft and the complex flight safety of the carrier aircraft. SUMMARY

[0004] Therefore, the present application provides a complex flight control method based on an improved TD3 algorithm, which can effectively reduce the complex flight area, improve the complex flight performance, and provide protection for the complex flight safety of the carrier aircraft.

[0005] The present application provides a complex flight control method based on an improved TD3 algorithm, comprising the following steps:

[0006] S1: building a carrier aircraft model;

[0007] S2: constructing a neural network model with a segmented activation structure, using the neural network model to construct a target policy network and a policy network, and obtaining an improved TD3 reinforcement learning algorithm;

[0008] S3: parameter configuration of the improved TD3 reinforcement learning algorithm, selecting parameters from the carrier aircraft model to configure state observation, action output and reward function;

[0009] S4: constructing a complex flight control model based on the improved TD3 reinforcement learning algorithm;

[0010] S5: calculating the complex flight area parameters;

[0011] S6: determining the complex flight decision method according to the calculated complex flight area parameters.

[0012] On the basis of the above technical scheme, preferably, the content of step S1 is to perform mathematical modeling on the longitudinal motion of the carrier aircraft according to the principle of flight dynamics, and obtain a longitudinal nonlinear model of the carrier aircraft.

[0013] Preferably, the specific content of step S2 is to first build a TD3 reinforcement learning algorithm framework, including a first value network , a second value network , a first target value network , a second target value network , a policy network , and a target policy network ; the first target value network , the second target value network , and the target policy network respectively obtain evaluation values, and the smaller value in the evaluation values is taken as a TD target, according to the TD target, the first value network and the second value network are used to update the first target value network and the second target value network , after the first target value network and the second target value network are updated for several times, the policy network updates the target policy network by maximizing the cumulative expected return.

[0014] Then a neural network model with a segmented activation structure is constructed, which is obtained by adding coefficients b ( x ) and weighted sum of spline functions spline ( x ) to obtain an activation function. w b 、w s

[0015] Based on the neural network model with a segmented activation structure, the first value network , the second value network , the first target value network , and the second target value network in the TD3 reinforcement learning algorithm framework are constructed, the policy network and the target policy network in the TD3 reinforcement learning algorithm framework are constructed by using a fully connected neural network model, and an improved TD3 algorithm is obtained.

[0016] ​Preferably, the content of step S3 comprises: S31: setting a state observation quantity, adopting an angle of attack holding strategy, and controlling the carrier-based aircraft to perform a missed approach maneuver by controlling the elevator of the carrier-based aircraft; and designing a value network q s a wq , wherein s is a state quantity, a is an action, wq is a parameter of the value network, and the deviation value between the actual angle of attack of the carrier-based aircraft and the ideal angle of attack is observed , the deviation value, the integral of the deviation value, and the differential of the deviation value are taken as the state observation quantity ; S32: setting the rudder deflection of the elevator of the carrier-based aircraft as an action output quantity; S33: comprehensively considering the state observation quantity, the action output quantity, and the change in the angle of attack of the carrier-based aircraft, and designing a reward function r ; S34: setting the hyperparameters of an activation function in a TD3 reinforcement learning algorithm framework, and setting the hyperparameters of a first value network , a second value network , a first target value network , a second target value network , and the hyperparameters of a reward function r .

[0017] Preferably, the content of step S33 is: comprehensively considering the state observation quantity, the action output quantity, and the change in the angle of attack of the carrier-based aircraft, designing a reward unit r 1, and introducing a penalty unit r 2; and the expression of the reward function is .

[0018] Preferably, in step S34, the base function b x is a Silu function, and the spline function spline x is a linear combination of B-spline functions.

[0019] Preferably, the content of step S4 comprises: S41: determining a missed approach lower boundary criterion; S42: obtaining a missed approach trajectory of the carrier-based aircraft; and S43: building a building material and reasoning missed approach maneuver model, introducing an improved TD3 algorithm to control the elevator of the carrier-based aircraft based on the missed approach trajectory of the carrier-based aircraft by adopting an angle of attack holding strategy, constructing a missed approach maneuver model based on a reinforcement learning algorithm, and controlling the carrier-based aircraft to perform a missed approach maneuver.

[0020] ​​​​​Preferably, the contents of step S41 are: (1) when the carrier-based aircraft passes the stern, the distance from the hook of the carrier-based aircraft to the stern is at least 3.0m; (2) the maximum allowable response time for the pilot after receiving the go-around signal is 0.7s; (3) the go-around control method is to use the military thrust control method to control the aircraft throttle and the improved TD3 algorithm to control the aircraft elevator.

[0021] Preferably, step S5 includes: specifying the initial position, initial speed, initial track angle, speed change, track angle change, and disturbance sinking rate of the carrier-based aircraft according to the lower boundary criterion for go-around, and obtaining the go-around trajectory of the carrier-based aircraft; translating the go-around trajectory of the carrier-based aircraft so that the go-around trajectory passes 3m above the tail of the carrier-based aircraft platform to obtain the critical go-around trajectory; and calculating the go-around envelope and the go-around zone.

[0022] Preferably, step S6 involves: after obtaining the go-around safety and go-around zone, comparing the real-time altitude of the carrier-based aircraft during the landing process. h ordinate of the go-around envelope h f The relationship between the altitude and the flight path will influence the decision to proceed: if the real-time altitude... h <Re-flight envelope ordinate> h f If the carrier-based aircraft fails to successfully go around the gap, there is a risk of hitting the ship; if the real-time altitude... h >Re-flight envelope vertical axis h f The go-around control model based on the improved TD3 reinforcement learning algorithm designed in step S4 is used to perform the go-around.

[0023] The present invention provides a go-around control method based on an improved TD3 algorithm, which has the following advantages compared with the prior art:

[0024] (1) The improved TD3-based go-around control model can adaptively and continuously output the optimal elevator deflection command according to the real-time status (mainly angle of attack deviation information) to maintain the angle of attack. It does not rely on complex preset control laws or a large amount of manual parameter tuning. It uses military thrust to control the throttle, which meets the constraints and requirements of actual go-around operations. Based on the strict go-around lower boundary criterion, the critical go-around trajectory is calculated by specifying initial conditions and disturbances. The go-around envelope is obtained by translating the critical trajectory, and then the go-around zone is determined, providing a clear and quantitative safety boundary basis for go-around decision-making.

[0025] (2) The neural network with a segmented activation structure (combined with the Silu basis function and the B-spline spline function) has better local adaptability and smoothness, and can more accurately fit the complex nonlinear dynamic characteristics of the carrier-based aircraft, especially in the key task of angle of attack control, thereby improving the evaluation accuracy of the value network for the state-action value, and the improved TD3 algorithm framework effectively improves the overestimation problem of the value function; the double Q learning and target policy smoothing regularization inherent in the TD3 algorithm, combined with the improved activation function, enhances the robustness of the model to state observation noise, model uncertainty and environmental disturbance, and improves the generalization ability in complex and dynamic landing and recovery scenarios;

[0026] (3) The decision is made by comparing the real-time height of the carrier-based aircraft with the longitudinal coordinate of the recovery envelope, which is directly related to the physical safety boundary, and has high reliability. BRIEF DESCRIPTION OF DRAWINGS

[0027] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or the prior art description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0028] Figure 1 The step flow chart of the landing and recovery control method based on the improved TD3 algorithm of the present application;

[0029] Figure 2 The TD3 reinforcement learning algorithm framework schematic diagram of the landing and recovery control method based on the improved TD3 algorithm of the present application;

[0030] Figure 3 The neural network model structure with segmented activation structure of the landing and recovery control method based on the improved TD3 algorithm of the present application;

[0031] Figure 4 The schematic diagram of the carrier-based aircraft thrust recovery control model of the landing and recovery control method based on the improved TD3 algorithm of the present application;

[0032] Figure 5 The landing and recovery control model based on the improved TD3 algorithm of the landing and recovery control method based on the improved TD3 algorithm of the present application;

[0033] Figure 6 The schematic diagram of the landing and recovery trajectory, the landing and recovery envelope and the landing and recovery area of the landing and recovery control method based on the improved TD3 algorithm of the present application;

[0034] Figure 7A schematic diagram of a PID attack angle control-military thrust combined control go-around simulation model in an embodiment of a go-around control method based on an improved TD3 algorithm of the present application;

[0035] Figure 8 A schematic diagram of a go-around area of three different go-around control methods in an embodiment of a go-around control method based on an improved TD3 algorithm of the present application;

[0036] Figure 9 Go-around trajectories under three go-around control methods when the sea surface is calm in an embodiment of a go-around control method based on an improved TD3 algorithm of the present application. DETAILED DESCRIPTION

[0037] The technical solutions in the embodiments of the present application will be described in detail below with reference to the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of the present application.

[0038] Due to the limited deck space of the ship, the approach and landing of the carrier aircraft is very difficult and dangerous, and it is very important to ensure that the carrier aircraft can quickly pull up after a failed landing to achieve safe go-around. In view of this, the present application provides a go-around control method based on an improved TD3 algorithm, comprising the following steps:

[0039] S1: Build a carrier aircraft model.

[0040] The content of step S1 is to mathematically model the longitudinal motion of the carrier aircraft according to the principles of flight dynamics, obtaining a longitudinal nonlinear model of the carrier aircraft:

[0041] , wherein the parameters m , T , D , L and M are the mass, thrust, drag, lift and pitching moment of the carrier aircraft, respectively; the parameters α , β , γ , θ , and μ are the attack angle, sideslip angle, track angle, pitch angle, roll angle and track roll angle of the carrier aircraft, respectively; the dot above the parameter indicates the first derivative operation; the parameters V , p , q , r are the airspeed and three axial angular velocities of the carrier aircraft, respectively; Ix 、 I y 、 I z and I xz are respectively the three-axis rotational inertia and the inertia moment of the body coordinate system of the carrier-based aircraft.

[0042] S2: Construct a neural network model of a segmented activation structure, use the neural network model to construct a target policy network and a policy network, and further obtain an improved TD3 reinforcement learning algorithm.

[0043] The specific content of step S2 is as shown in Figure 2 , first build a TD3 reinforcement learning algorithm framework, including a first value network , a second value network , a first target value network , a second target value network , a policy network and a target policy network ; the first target value network , the second target value network and the target policy network respectively obtain evaluation values, and the smaller value in the evaluation values is taken as a TD target, according to the TD target, the first value network and the second value network are used to update the first target value network and the second target value network , after the first target value network and the second target value network are updated for several times, the policy network updates the target policy network by maximizing the cumulative expected return.

[0044] Then as shown in Figure 3 , a neural network model of a segmented activation structure is constructed, which is obtained by adding coefficients b ( x ) and a spline function spline ( x ) to the base function w b 、w s and weighted sum to obtain an activation function ; the activation function is a learnable activation function.

[0045] Based on the neural network model of the segmented activation structure, the first value network , second value network , first target value network , second target value network , adopt a fully connected neural network model to construct a policy network in the TD3 reinforcement learning algorithm framework And target policy network , get improved TD3 algorithm.

[0046] S3: parameter configuration is performed on the improved TD3 reinforcement learning algorithm, and parameters are selected from a carrier-based aircraft model to configure state observation, action output and a reward function.

[0047] The content of step S3 includes:

[0048] S31: set the state observation, adopt an angle of attack maintaining strategy, realize the missed approach control of the carrier-based aircraft by controlling the elevator of the carrier-based aircraft, and design a value network q ( s , a , wq ), wherein s is a state variable, a is an action, wq is a parameter of the designed value network, the deviation value between the actual angle of attack of the carrier-based aircraft and the ideal angle of attack , , the deviation value, the integral of the deviation value and the differential of the deviation value are taken as the state observation .

[0049] S32: set the rudder deflection of the elevator of the carrier-based aircraft as the action output.

[0050] The application sets the action output as the rudder deflection of the elevator of the carrier-based aircraft, wherein K P , K I , K D are correlation coefficients, which can be determined after the training is completed.

[0051] S33: comprehensively consider the state observation, the action output and the change of the angle of attack of the carrier-based aircraft, and design a reward function r .

[0052] Comprehensively consider the state observation, the action output and the change of the angle of attack of the carrier-based aircraft, and design a reward unit as , n and s are artificially specified parameters, is the action output; a penalty unit is introduced, and when the deviation value exceeds a threshold value mThen the training session ended early for the current round. threshold m and parameters p All settings are manually configured; the expression for the reward function is: .

[0053] S34: Define the activation function in the TD3 reinforcement learning algorithm framework and define the first value network. Second value network First Target Value Network Second target value network Hyperparameters and reward function r Hyperparameters.

[0054] Set the hyperparameters for each value network and the target value network. Set the number of input layer neurons and the number of output layer neurons for both the value network and the target value network to 1. Set the number of hidden layers to 2, with the first hidden layer having 128 neurons and the second hidden layer having 64 neurons.

[0055] basis functions b ( x ) is the Silu function. ;

[0056] spline function spline ( x () is a linear combination of B-spline functions.

[0057] , B i ( x ) is a B-spline function. i The number and coefficients of the spline function. c i According to the normal distribution Perform initialization. The standard deviation of the normal distribution is typically set to 0.1; the coefficient... w b The Xavier method is used for initialization, and the coefficients are... w s It is typically initialized to 1.

[0058] Set the hyperparameters for the policy network and the target policy network. Set the number of input layer neurons to 4, the number of output layer neurons to 1, the number of hidden layers to 2, and the number of neurons in the first and second hidden layers to 64. Choose the ReLU function as the activation function for both the policy network and the target policy network.

[0059] reward function r Parameters in n ,s , m and p are respectively 20, 0.01, 0.055 and 10.

[0060] The remaining hyperparameters of the improved TD3 algorithm are set. The improved TD3 algorithm adopts the experience replay method, the size of the entire experience replay pool is set to 10 6 , the size of the random mini-batch experience replay pool is set to 64; the reward discount factor is set to 0.99; the update frequency of the policy network and the value network is set to 2, and the smoothing factor of the target network is set to 0.05; the Gaussian noise is selected as the exploration noise of the action space and the target smoothing noise.

[0061] S4: Construct a missed approach control model based on the improved TD3 reinforcement learning algorithm.

[0062] The content of step S4 includes:

[0063] S41: Determine the missed approach boundary criterion: (1) the distance between the carrier-based aircraft and the stern of the aircraft carrier is at least 3.0 m when the carrier-based aircraft passes through the stern; (2) the maximum response time allowed by the pilot after receiving the missed approach signal is 0.7 s; (3) the missed approach control method is to control the aircraft throttle by using the military thrust control method and to control the aircraft elevator by using the improved TD3 algorithm.

[0064] S42: Obtain the missed approach trajectory of the carrier-based aircraft: the missed approach area of the carrier-based aircraft is calculated from the missed approach trajectory, and the missed approach trajectory of the carrier-based aircraft can be calculated by the following formula:

[0065] , , , , are respectively the initial speed, the initial track angle, and the initial horizontal position and the initial height of the carrier-based aircraft when the carrier-based aircraft receives the missed approach instruction; , are respectively the speed change and the track angle change of the carrier-based aircraft; is the interference sink rate; , are respectively the horizontal position and the height of the carrier-based aircraft on the missed approach track. According to the above formula and further considering the missed approach boundary criterion in step S41, a carrier-based aircraft thrust missed approach control model can be built, as shown in Figure 4 .

[0066] S43: Build a missed approach control model based on the improved TD3 algorithm. Based on the missed approach trajectory of the carrier-based aircraft constructed in step S42, an angle of attack maintaining strategy is adopted, the improved TD3 algorithm is introduced to control the elevator of the carrier-based aircraft, a missed approach control model based on the reinforcement learning algorithm is constructed, and the carrier-based aircraft is further controlled to miss approach. The structure of the missed approach control model based on the reinforcement learning algorithm is as follows:Figure 5 As shown.

[0067] S5: Calculate the parameters of the bypass area.

[0068] Step S5 includes: based on the lower boundary criterion for go-around, specifying the initial position, initial speed, initial track angle, speed change, track angle change, and disturbance sinking rate of the carrier-based aircraft to obtain the go-around trajectory of the carrier-based aircraft; translating the go-around trajectory of the carrier-based aircraft so that the go-around trajectory passes 3m above the tail of the carrier-based aircraft platform to obtain the critical go-around trajectory. x i , h i ]; Calculate the go-around envelope and go-around area according to the following formula:

[0069] ;in This is the value after translating the initial horizontal position. These are the critical points of the flight path. A series of critical points smoothly connected together yields the go-around envelope, and the envelope region between the go-around envelope and the coordinate region is the go-around zone. The relationship between the critical go-around trajectory, the go-around envelope, and the go-around zone is as follows: Figure 6 As shown.

[0070] S6: Determine the go-around decision method based on the calculated go-around zone parameters.

[0071] Step S6 involves obtaining the go-around safety and go-around zone, and then comparing the real-time altitude of the carrier-based aircraft during the landing process. h ordinate of the go-around envelope h f The relationship between the flight path and the flight path will influence the decision to resume flights: if the real-time altitude... h <Re-flight envelope ordinate> h f If the carrier-based aircraft fails to successfully go around the gap, there is a risk of hitting the ship; if the real-time altitude... h >Re-flight envelope vertical axis h f The go-around control model based on the improved TD3 reinforcement learning algorithm designed in step S4 is used to perform the go-around.

[0072] To verify the effectiveness of this embodiment, the following comparative experiment was conducted.

[0073] According to step S4, build a go-around control model based on the improved TD3 algorithm in Simulink. According to step S3, write the corresponding script file in Matlab and set the response hyperparameters.

[0074] Build a military thrust-assisted go-around control model in Simulink, such as Figure 7 A PID-military thrust combined go-around control model was constructed. The structure of the PID controller is as follows:

[0075] , y PID For PID controller output, parameters K P =2.341, K D =1.5, K I =0.13, That is, the deviation value .

[0076] The initial position of the carrier-based aircraft ( x 0, h 0) is set to (575, 0), interference subsidence rate Setting the value to 0, a comparison is made between the military thrust-controlled go-around method, the PID-military thrust combined go-around control method, and the go-around control method based on the improved TD3 algorithm designed in this invention. The go-around area calculated by the three methods can be obtained as follows: Figure 8 As shown in the figure, RL, PID, and Tra represent the go-around control method based on the improved TD3 algorithm, the PID-military thrust combined go-around control method, and the military thrust go-around control method, respectively. The figure shows that the go-around control method based on the improved TD3 algorithm designed in this invention yields a smaller go-around zone, approximately 10%, compared to the other two methods. Therefore, the safe flight area for carrier-based aircraft during landing is larger, and the go-around decision made using the method designed in this invention is more accurate and better ensures flight safety.

[0077] To interfere with the subsidence rate Set to -4.5 m / s, assuming the carrier-based aircraft receives a go-around command during its approach and landing, at which point the aircraft's speed is 69.3 m / s, the initial position of the carrier-based aircraft is ( x 0, h 0) is (600, 10). Three different go-around control methods were used to perform the go-around maneuvers, and the resulting go-around trajectories are as follows. Figure 9 As shown. According to Figure 9 When the initial position of the carrier-based aircraft is (600, 10) and the aircraft speed is 69.3 m / s, the go-around trajectories obtained using the two traditional go-around control methods are already below the stern when passing the ship, which could lead to a collision. However, the go-around control method based on the improved TD3 algorithm designed in this paper yields a go-around trajectory approximately 4.2 m above the stern when passing it. Furthermore, throughout the entire go-around process, the lowest point of the carrier-based aircraft is 14.5 m below the deck of the aircraft carrier platform, which is less than the freeboard height of the aircraft carrier platform (17 m~18 m), eliminating the risk of crashing into the sea. Therefore, the go-around control method based on the improved TD3 algorithm designed in this invention has better go-around performance.

[0078] The above description is only the preferred embodiment of the present application, and is not intended to limit the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A go-around control method based on an improved TD3 algorithm, characterized in that, Comprise the following steps: S1: build a carrier-based aircraft model; S2: construct a neural network model of segmented activation structure, use the neural network model to construct a target policy network and a policy network, and obtain an improved TD3 reinforcement learning algorithm; The specific content of step S2 is to first build a TD3 reinforcement learning algorithm framework, including a first value network , a second value network , a first target value network , a second target value network , a policy network , and a target policy network ; the first target value network , the second target value network , and the target policy network respectively obtain evaluation values, and the smaller value in the evaluation values is taken as a TD target, according to the TD target, the first value network and the second value network are used to update the first target value network and the second target value network , after the first target value network and the second target value network are updated for several times, the policy network updates the target policy network by maximizing the cumulative expected return. Then the neural network model of piecewise activation structure is constructed by adding coefficients b ( x ) to the basis function spline ( x ) and weighted sum to get the activation function w b 、w s ​ The neural network model based on the segmented activation structure is used to construct a first value network in a TD3 reinforcement learning algorithm framework , a second value network , a first target value network , a second target value network , a fully connected neural network model is used to construct a policy network in a TD3 reinforcement learning algorithm framework and a target policy network , to obtain an improved TD3 algorithm S3: parameter configuration of the improved TD3 reinforcement learning algorithm, select parameters from the carrier-based aircraft model to configure state observation, action output and reward function; S4: build a missed approach control model based on the improved TD3 reinforcement learning algorithm; S5: calculate the missed approach area parameters; S6: determine the missed approach decision method according to the calculated missed approach area parameters.

2. The go-around control method based on the improved TD3 algorithm according to claim 1, characterized in that, The content of step S1 is to model the longitudinal motion of the carrier-based aircraft according to the principle of flight dynamics, and obtain the longitudinal nonlinear model of the carrier-based aircraft.

3. The go-around control method based on the improved TD3 algorithm according to claim 1, characterized in that, The content of step S3 includes: S31: setting state observation, adopting an angle of attack maintaining strategy, realizing the recovery control of the carrier-based aircraft by controlling the elevator of the carrier-based aircraft; designing a value network q ( s , a , wq ), wherein s is a state quantity, a is an action, wq is a parameter of the value network, and the deviation value between the actual angle of attack of the carrier-based aircraft and the ideal angle of attack is observed , the deviation value, the integral of the deviation value and the differential of the deviation value are taken as the state observation ; S32: setting the rudder deflection of the elevator of the carrier-based aircraft as the action output quantity; S33: comprehensively designing a reward function based on the state observation, the action output quantity and the change of the angle of attack of the carrier-based aircraft r ; S34: setting the activation function in the TD3 reinforcement learning algorithm framework, and setting the hyperparameters of the first value network , the second value network , the first target value network , the second target value network and the hyperparameters of the reward function r .

4. The go-around control method based on the improved TD3 algorithm according to claim 3, characterized in that, The content of step S33 is: considering the state observation, the action output and the change of the angle of attack of the carrier aircraft, designing a reward unit r 1; and introducing a penalty unit r 2; the expression of the reward function is .

5. The go-around control method based on the improved TD3 algorithm according to claim 3, characterized in that, In step S34: basis functions b x are Silu functions, spline functions spline x are linear combinations of B-spline functions.​​ 6. The go-around control method based on the improved TD3 algorithm according to claim 3, characterized in that, The content of step S4 includes: S41: determine the missed approach lower boundary criterion; S42: obtain the missed approach trajectory of the carrier-based aircraft; S43: build the building material and reasoning missed approach control model, based on the missed approach trajectory of the carrier-based aircraft, adopt the angle of attack keeping strategy, introduce the improved TD3 algorithm to control the elevator of the carrier-based aircraft, and construct the missed approach control model based on the reinforcement learning algorithm to control the missed approach of the carrier-based aircraft.

7. The go-around control method based on the improved TD3 algorithm according to claim 6, characterized in that, The content of step S41 is: (1) when the carrier-based aircraft passes through the stern of the carrier, the distance between the carrier-based aircraft and the stern of the carrier is at least 3.0m; (2) the maximum response time allowed by the pilot after receiving the missed approach signal is 0.7s; (3) the missed approach control method is to control the aircraft throttle by using the military thrust control method, and to control the aircraft elevator by using the improved TD3 algorithm.

8. The go-around control method based on the improved TD3 algorithm according to claim 6, characterized in that, The content of step S5 includes: according to the missed approach lower boundary criterion, the initial position, initial speed, initial track angle, speed variation, track angle variation and disturbance sinking rate of the carrier-based aircraft are specified to obtain the missed approach trajectory of the carrier-based aircraft; the missed approach trajectory of the carrier-based aircraft is translated so that the missed approach trajectory passes through 3m above the stern of the carrier-based aircraft platform to obtain the critical missed approach trajectory; the missed approach envelope and the missed approach area are calculated.

9. The go-around control method based on the improved TD3 algorithm according to claim 8, characterized in that, The content of step S6 is: after obtaining the missed approach insurance and the missed approach area, the missed approach decision is made by comparing the real-time height h of the carrier-based aircraft during the landing process with the longitudinal coordinate h of the missed approach envelope f : if the real-time height h of the carrier-based aircraft is less than the longitudinal coordinate h of the missed approach envelope f , the carrier-based aircraft cannot successfully perform the missed approach and there is a risk of touching the ship; if the real-time height h of the carrier-based aircraft is greater than the longitudinal coordinate h of the missed approach envelope f , the missed approach is performed using the missed approach control model based on the improved TD3 reinforcement learning algorithm designed in step S4.

Images