Shipboard aircraft routing path deep reinforcement learning planning method based on fusion guidance point reward mechanism

Abstract

This invention relates to a deep reinforcement learning planning method for carrier-based aircraft movement paths, incorporating a guide point reward mechanism, and falls within the field of carrier-based aviation support. The invention generates guide point paths using third-order Bézier curves, guides carrier-based aircraft learning using a dense reward function with dynamically adjusted weights, and employs the Proximal Policy Optimization (PPO) algorithm for end-to-end training of the policy-value network. This enables efficient and smooth path planning schemes that simultaneously satisfy kinematic and obstacle avoidance constraints. By quantifying complex scheduling objectives into dense reward signals and combining this with the stable policy update mechanism of the PPO algorithm, the invention achieves end-to-end reinforcement learning planning for carrier-based aircraft movement paths. This generates efficient, safe, and smooth carrier-based aircraft movement trajectories in complex deck environments, significantly improving the efficiency and quality of path planning for carrier-based aircraft in such environments.

Description

Technical Field

[0001] This invention belongs to the field of shipborne aviation support and involves a deep reinforcement learning planning method for shipborne aircraft dispatching paths that integrates a guidance point reward mechanism. Background Technology

[0002] In modern naval warfare, carrier-based aircraft serve as the core strike force of an aircraft carrier battle group, and their efficient and safe deployment capabilities directly impact the carrier's overall combat effectiveness and survivability. The confined space and dense obstacles on an aircraft carrier deck, coupled with the highly dynamic and time-constrained nature of missions, present three main challenges to the planning of carrier-based aircraft deployment routes.

[0003] First, manual scheduling makes it difficult to monitor the movement of carrier-based aircraft in real time, and collisions with each other or fixed obstacles are highly likely during aircraft movement. Second, existing algorithms are mostly focused on heuristic search algorithms and intelligent biomimetic algorithms, which can generate feasible paths in static environments, but the computation time is long and cannot meet the requirements of rapid relocation. Third, the smoothness of the generated carrier-based aircraft paths is usually not fully considered in path planning.

[0004] For example, Chinese invention patent (application number 201910818148.9) provides an optimal control method for carrier-based aircraft deck path planning based on the NSP algorithm. It establishes an optimal control model by combining motion constraints and obstacle constraints, and uses Newton's iteration method, symplectic algorithms, and pseudospectral methods to solve the problem, achieving high-precision and high-efficiency carrier-based aircraft deck path planning and control, ensuring that the path strictly meets the end-point constraints. Chinese invention patent (application number 202211128429.X) proposes a carrier-based aircraft deck taxiing trajectory planning method based on a safe transport corridor. It generates an initial path using a hybrid A* algorithm and then constructs a safe corridor, transforming complex obstacle avoidance constraints into linear constraints. Finally, it solves the optimal control problem to generate a safe and smooth transport trajectory. While existing solutions can plan a transport path that meets obstacle avoidance requirements while satisfying the kinematic constraints of the carrier-based aircraft, the solution scale increases dramatically when the number or shape of obstacles increases, significantly increasing computation time. Furthermore, the smoothness depends on the initial path and the quality of the safe corridor construction, and may be conservative in narrow areas.

[0005] Therefore, developing carrier-based aircraft path planning under the background of deep reinforcement learning is of great significance for improving the efficiency of carrier-based aircraft transportation. Summary of the Invention

[0006] To address the aforementioned issues, this invention proposes a deep reinforcement learning planning method for carrier-based aircraft movement paths that integrates a guide point reward mechanism. This invention generates guide point paths using third-order Bézier curves, guides carrier-based aircraft learning using a dense reward function with dynamically adjusted weights, and employs the Proximal Policy Optimization (PPO) algorithm for end-to-end training of the policy-value network. This method can efficiently achieve smooth path planning schemes that simultaneously satisfy kinematic and obstacle avoidance constraints.

[0007] To achieve the above objectives, the technical solution adopted by the present invention is as follows: A deep reinforcement learning planning method for carrier-based aircraft relocation paths, incorporating a guidance point reward mechanism, includes the following steps: Step 1: Construct the kinematic model, multi-dimensional state model, and neural network model of the carrier-based aircraft; Step 1-1: Model the kinematics of the carrier-based aircraft; When a carrier-based aircraft is taxiing at low speed on the deck, its motion is described using a bicycle kinematics model. In a global Cartesian coordinate system, let the position of the rear wheel center of the carrier-based aircraft at time step t be... ,in, Let x be the x-coordinate of the center of the rear wheel of the carrier-based aircraft at time step t. Let be the ordinate of the rear wheel center of the carrier-based aircraft at time step t. Let the taxiing speed of the carrier-based aircraft at time step t be... The heading angle of the carrier-based aircraft is The front wheel steering angle is Let the control variables for carrier-based aircraft be... ,in, Let t be the longitudinal acceleration of the carrier-based aircraft at time step t. Let be the front wheel steering angular velocity at time step t. Let the carrier-based aircraft's kinematic state vector be... The carrier-based aircraft then obtains its kinematic state for the next time step based on Euler discretization. for: (1) in, , Let t be the coordinates of the center of the rear wheel of the carrier-based aircraft at time step t; , The coordinates of the rear wheel center of the carrier-based aircraft at time step t+1; Let be the glide speed at time step t; Let be the glide speed at time step t+1; Let t be the front wheel steering angle at time step t; Let t+1 be the front wheel steering angle; Let t be the heading angle of the carrier-based aircraft at time step t; The orientation angle of the carrier-based aircraft at time step t+1; Let t be the longitudinal acceleration at time step t; Let t be the front wheel steering angular rate at time step t; This refers to the wheelbase between the front and rear wheels; This is for simulating step size; , These are the minimum and maximum permissible taxiing speeds, respectively. , These are the minimum and maximum permissible front wheel steering angles, respectively; clip(·) is the clipping function.

[0008] Steps 1-2: Modeling a multi-dimensional state model of the carrier-based aircraft; To improve the efficiency of reinforcement learning training, the observation information of carrier-based aircraft is uniformly constructed as a multi-dimensional state vector. The multidimensional state vector It is composed of the carrier-based aircraft's own status, target-related status, and obstacle perception status.

[0009] Step 1-2-1: The carrier-based aircraft's own status; The carrier-based aircraft's own state is directly adopted from the carrier-based aircraft kinematic state vector defined in step 1-1. .

[0010] Step 1-2-2: Target-related status; Let the location of the target point be ,in, Let x be the x-coordinate of the target point. Let be the ordinate of the target point. Let the target heading angle be... Define the target-related state of the carrier-based aircraft relative to the target point at time step t as follows: ,in, Let be the Euclidean distance from the center of the rear wheel of the carrier-based aircraft to the target point at time step t. Let t be the target direction angle from the current position of the carrier-based aircraft to the target point at time step t. Let t be the heading deviation at time step t, as shown in formula (2): (2) in, , These are the x and y coordinates of the target point, respectively; wrap(·) is the angle normalization function.

[0011] Steps 1-2-3: Obstacle perception status; An obstacle perception state is constructed using a double-circle envelope model and a multi-directional ray detection method. First, a minimum envelope rectangle is used to describe the outer contour of the carrier-based aircraft. Let the width of this rectangle be... , length is .

[0012] (3) in, , and These represent the distance from the rear wheel of the carrier-based aircraft to the rear edge of the rectangle, the longitudinal distance between the front and rear wheels of the carrier-based aircraft, and the distance from the front wheel of the carrier-based aircraft to the front edge of the rectangle, respectively. For the minimum envelope rectangle, two circles of the same radius are used to completely cover it. The circle closer to the front of the carrier-based aircraft is defined as the front circle, and the circle closer to the rear of the aircraft is defined as the rear circle. Let the common radius of the front and rear circles be . The center of the front circle is The center of the back circle is The coordinates of the center of the front circle and the center of the back circle are calculated as follows: (4) Starting from the center of the front circle respectively and the center of the circle K detection rays are emitted outward. Here, K is the total number of rays, and the normalized detection distance corresponding to the k-th ray is denoted as . , For each ray, calculate its intersection point with the obstacle boundary, and take the normalized distance corresponding to the nearest intersection point as the ray's detection result. Construct the obstacle sensing state as follows: .

[0013] Finally, the status of the carrier-based aircraft itself. Target-related status and obstacle perception state By concatenating them sequentially, we obtain the multidimensional state vector at time step t. .

[0014] Steps 1-3: Modeling a neural network model; Establish a policy network and value network , where subscript For policy network parameters, These are the parameters of the value network. The policy network... Used based on the current multidimensional state vector Output the probability distribution of discrete action indices; the value network Used based on the current multidimensional state vector Estimate the state value.

[0015] Both the policy network and the value network employ a multilayer perceptron structure. The policy network includes a feature extraction layer and an action output layer, along with a multidimensional state vector. After being mapped to latent feature vectors by the feature extraction layer, the discrete action probability distribution is output by the action output layer. The value network structure is symmetric to the policy network, and its output is the state value estimate.

[0016] The neural network weights are initialized orthogonally, and the bias parameters are initialized with preset biases to improve gradient stability in the early stages of training.

[0017] Step 2: Model the reward function model that integrates the guide point reward mechanism; A Bezier curve is used to generate the guide point path, and a dynamic weight adjustment mechanism is designed. A dense reward function is constructed that integrates guide point tracking, obstacle avoidance, heading alignment, progress reward, turning penalty, and trajectory smoothness reward to guide the carrier-based aircraft to learn efficiently. Finally, a reward function model is constructed. Specifically: Step 2-1: Guiding Point Rewards; Step 2-1-1: Generate the guide point path; To achieve progressive guidance to the target point and ensure path smoothness, a third-order Bézier curve is used to generate the guide point path from the starting point to the target point. The specific generation method is as follows: Let the initial pose be ,in Starting position Let be the initial heading angle. Assume the target pose is... ,in For the target location, Let be the target heading angle. Let the control point extension length factor be... Define the four control points of the third-order Bézier curve as follows: , , , Specifically: ; To be based on the starting position and starting heading angle , elongation factor The calculated starting control point; To determine the target location and target heading angle , elongation factor The calculated endpoint control point; The coordinates of the corresponding control points are as follows: (5) Then any point on the third-order Bézier curve is denoted as: (6) in, It is the leading point on the Bézier curve; These are the parameters of the Bézier curve.

[0018] Step 2-1-2: Guiding Point Reward and Dynamic Weight Adjustment Mechanism; Let the position of the rear wheel center of the carrier-based aircraft at time step t be... The guiding point corresponding to time step t is ,in The parameters for the Bézier curve correspond to the mission progress. Define the distance between the current position of the carrier-based aircraft and the guide point. for: (7) in, This is the guiding point corresponding to time step t; , Let x and y be the x and y coordinates of the guide point corresponding to time step t.

[0019] Let the effective radius of the guide point foundation be... To accommodate varying guidance intensities during mission advancement, an effective guidance radius after time decay is introduced. As shown in formula (8): (8) in, This is the time decay coefficient; Ideal remaining path length at time step t.

[0020] The reward for the guiding point is recorded as shown in formula (9): (9) in, The reward is given for the guiding point at time step t; The weights are dynamic, comprehensively reflecting both environmental conditions and task priorities; The first product coefficient is awarded as a guide point reward; The second product coefficient is awarded as a guide point reward.

[0021] The dynamic weights are: (10) in, Basic weights; For obstacle avoidance factors; For position factors; For velocity factor; Angle factor.

[0022] Step 2-2: Design multi-dimensional reward components; In addition to the guide point reward, multiple reward components are designed to jointly constrain the behavior of carrier-based aircraft, including obstacle avoidance reward, artificial potential field gravity reward, differential distance reward, heading alignment reward, turning penalty, position constraint reward, trajectory smoothness reward, and curvature reward. Specifically: The obstacle avoidance reward Obstacle-based state perception The normalized detection distance of each ray in the middle is used to construct an obstacle avoidance reward.

[0023] (11) in, K represents the obstacle avoidance reward at time step t; K is the total number of rays. Let be the normalized detection distance of the k-th ray returning at time step t.

[0024] The artificial potential field gravitational reward Simulates a gravitational potential field to guide carrier-based aircraft toward the target.

[0025] (12) in, The gravitational reward for the artificial potential field at time step t; The distance between the carrier-based aircraft and the target point is expressed in Euclidean form. For capture distance; The product coefficient of gravitational reward for artificial potential field; This is a reward constant term.

[0026] The differential distance reward Reward progress based on the change in distance to the target point between the previous and next steps.

[0027] (13) in, The difference distance reward at time step t; and These are the Euclidean distances between the carrier-based aircraft and the target point at the previous and current time steps, respectively. This is the product coefficient for the differential distance reward.

[0028] The heading alignment reward When a carrier-based aircraft approaches the target point, a heading alignment reward is generated based on the deviation between the aircraft's current heading and the target heading.

[0029] (14) in, The heading alignment reward is given at time step t. The target heading angle; This is the current facing angle; The first heading alignment bonus product factor; The product factor for alignment bonus in the second course; The maximum allowable heading deviation threshold.

[0030] The turning penalty To suppress frequent and large-amplitude turns by carrier-based aircraft, a nonlinear penalty term is constructed for the front wheel steering angle.

[0031] (15) in, Penalty for turning at time step t; Let t be the front wheel steering angle at time step t; The product coefficient for the penalty of turning; The maximum allowable heading deviation threshold.

[0032] The location constraint reward : If the fore and aft wheels exceed the deck boundary, enter a restricted area, or the island superstructure, a large negative reward will be given.

[0033] The trajectory smoothness negative reward A negative reward for trajectory smoothness is constructed based on the statistical characteristics of the recent changes in velocity and front wheel steering angle over several time steps.

[0034] (16) in, The reward is the trajectory smoothness at time step t; The standard deviation of the velocity change over a recent number of time steps; The standard deviation of the change in the front wheel steering angle over a recent period of time; The first product coefficient is awarded as a reward for trajectory smoothness; A second product coefficient is awarded as a reward for trajectory smoothness.

[0035] The curvature reward The local curvature of the trajectory is estimated using the three-point method, rewarding straight-line motion and penalizing large-curvature turns.

[0036] (17) in, The curvature reward at time step t; This represents the change in the direction angle of adjacent trajectory segments; The first product coefficient is awarded for curvature reward; The second product coefficient is awarded as a curvature bonus; The maximum allowable heading deviation threshold.

[0037] Steps 2-3: Synthesize the total reward function; The guiding point reward obtained in step 2-1 The reward is then weighted and summed together with the reward components obtained in step 2-2 to obtain the instantaneous reward at time step t. : (18) Based on this, combined with environmental interaction step size And add time penalty Collision penalty and rewards for successful missions Obtain the total reward function at time step t. As shown in formula (19): (19) Total reward function Complex scheduling objectives are transformed into continuous numerical signals, driving carrier-based aircraft to autonomously learn smooth paths that satisfy kinematic and obstacle avoidance constraints during reinforcement learning.

[0038] Step 3: Train the policy network and value network based on the PPO algorithm; Step 3-1: The carrier-based aircraft completes one interaction with the environment; At time step t, based on the current multidimensional state vector Output discrete action index Indexing discrete actions Converted into actual control quantities through motion mapping The kinematic state of the carrier-based aircraft at the next time step is obtained by updating the kinematic model established in step 1-1. Subsequently, collision detection was performed on the dual-circle envelope model of the carrier-based aircraft and the obstacle boundary, and ray detection data was generated based on the center of the front and rear circles to obtain the obstacle perception state at the next time step. Then, the total reward for the current time step is calculated using the reward function model established in step 2. When the Euclidean distance between the carrier-based aircraft and the target point is less than the acquisition distance... Furthermore, the heading deviation between the carrier-based aircraft and the target heading shall not exceed the maximum permissible heading deviation threshold. At that point, it is determined that the carrier-based aircraft has successfully reached the target. Finally, the updated carrier-based aircraft's own state, target-related state, and obstacle perception state are concatenated to obtain the multi-dimensional state vector for the next time step. .

[0039] Step 3-2: Store trajectory samples; After completing one step of environmental interaction, the data generated at the current time step is stored in the trajectory buffer as a trajectory sample. Each trajectory sample includes the current multidimensional state vector. Discrete Action Index Under the old strategy, the log probability of the discrete action is... Total reward at current time step Termination mark The old value network is related to the state. Value estimation and the multidimensional state vector at the next time step .in, This is the old strategy.

[0040] Step 3-3: Calculate the advantage function and discounted return steps; After a training cycle ends, the advantage function and discount reward are calculated based on the trajectory samples in the trajectory buffer, and then the samples are shuffled and divided into batches.

[0041] First, the value estimate of the next state at the last time step is calculated using the current value network. Then, the temporal difference error for each time step is calculated in reverse chronological order. .

[0042] (20) in, Let t be the timing difference error at time step t; Discount factor; For the old value network to state Value estimation; For the old value network to state The value estimate;

[0043] Secondly, the generalized advantage estimation method is used to calculate the advantage function. : (twenty one) in, Let t be the dominance function at time step t; This is the estimate of the dominance function for the next time step; Discount factor; Smoothing parameters are estimated for generalized dominance.

[0044] Finally, based on the advantage function and the old value network, the state is... The value estimate is discounted. As shown in formula (22): (twenty two) Steps 3-4: Construct mini-batch training data; After completing the advantage function and discount reward calculation in step 3-3, the trajectory samples in the trajectory buffer are randomly shuffled and divided into several mini-batches for subsequent PPO updates.

[0045] Steps 3-5: Update the policy network; The mini-batch samples constructed in steps 3-4 are input into the policy network, and the logarithmic probability of actions and the policy entropy under the current policy are calculated forward. The policy entropy is denoted as... As shown in formula (23): (twenty three) in, Let M be the policy entropy at time step t; M be the total number of discrete actions. In the state The probability of taking discrete action m; These are the policy network parameters.

[0046] Secondly, calculate the probability ratio between the current strategy and the old strategy. : (twenty four) in, The probability ratio of the new and old strategies at time step t; This is the current strategy; This is the old strategy; This is the index of the discrete action at time step t; Let be the multidimensional state vector at time step t.

[0047] Next, a pruning substitution objective function is constructed using the PPO pruning objective. As shown in formula (25): (25) in, Replace the objective function with a pruning function; Let t be the dominance function at time step t; PPO trimming parameters; This represents the expectation of the sample time step.

[0048] Based on the pruning and substitution objective function, and combined with the entropy regularization term, the total policy loss is constructed. As shown in formula (26): (26) in, This represents the total strategic loss. It is the entropy coefficient; Let be the policy entropy at time step t.

[0049] Finally, regarding the total strategy loss Perform backpropagation and base it on the maximum gradient norm. The gradients are pruned, and then the optimizer is stepped to obtain the updated policy network parameters. .

[0050] Steps 3-6: Update the value network; Input the batch samples constructed in steps 3-4 into the value network to obtain the value network's state for the current state. Value estimation To avoid excessively large value network updates, a value pruning method is used to construct the pruned value prediction, as shown in formula (27): (27) in, For the value prediction after trimming; For the current state, the value network is used to evaluate the state. Value estimation; For the old value network to state Value estimation; For value network parameters; These are the PPO trimming parameters.

[0051] First, calculate the original value loss and the value loss after trimming separately, and take the larger of the two as the final value loss. Then, perform regression using Huber loss. Next, multiply the value loss by the value loss coefficient. Then backpropagation is performed, also based on the maximum gradient norm. The gradients are pruned, and finally the optimizer is run to obtain the updated value network parameters. .

[0052] Step 4: Utilize the trained strategy network to plan the carrier-based aircraft movement routes; Load the policy network parameters trained in step 3. And construct the corresponding environment based on the deck scene to be planned. During the path planning process, the multi-dimensional state vector of the current time step is... Input the trained policy network To obtain the discrete action index Then index the discrete actions. Converted into control variables through action mapping The carrier-based aircraft's pose is updated using the kinematic model from step 1-1. If the termination condition is not met, the multidimensional state vector for the next time step is generated, and the above process is repeated until the carrier-based aircraft's Euclidean distance to the target point is less than the acquisition distance. Furthermore, the heading deviation between the carrier-based aircraft and the target heading shall not exceed the maximum permissible heading deviation threshold. Ultimately, the aircraft position and attitude sequences recorded at each time step during the planning process are output as the aircraft transport trajectories for the deck control system to use.

[0053] The beneficial effects of this invention are as follows: (1) This invention divides the carrier-based aircraft deployment path planning process into four consecutive stages: “guide point path generation, reward function construction, strategy training, and online planning output”, thereby realizing end-to-end reinforcement learning planning of carrier-based aircraft deployment paths. Specifically, the guide point path is generated using a third-order Bézier curve, which makes the guide trajectory have good smoothness. By constructing a total reward number that integrates guide point reward, obstacle avoidance reward, artificial potential field gravity reward, differential distance reward, heading alignment reward, turning penalty, position constraint negative reward, trajectory smoothness reward, and curvature reward, the carrier-based aircraft can simultaneously take into account target approach, safe obstacle avoidance, and trajectory smoothness during the training process. (2) Further, the present invention uses a multi-dimensional state vector consisting of the carrier-based aircraft's own state, target-related state and obstacle perception state as input to the policy network and value network, and uses the PPO algorithm for end-to-end training, so that the policy network after training can stably output control quantities that satisfy bicycle kinematic constraints, thereby generating efficient, safe and smooth carrier-based aircraft transport trajectories in complex deck environments. Attached Figure Description

[0054] Figure 1 This is a flowchart of the method of the present invention; Figure 2 This is a schematic diagram representing the double circles of a carrier-based aircraft in this invention; Figure 3 This is a schematic diagram of an aircraft carrier deck in an example of the present invention; Figure 4 This is the sliding trajectory calculated in the example of the present invention. Detailed Implementation

[0055] The present invention will be further described below with reference to specific implementation examples.

[0056] Consider the trajectory planning problem for a carrier-based fighter jet on an aircraft carrier deck. The dimensional parameters of the carrier-based aircraft and related algorithm parameters are shown in Table 1. The shape and dimensions of the aircraft carrier deck and the distribution of obstacles on the deck are shown in... Figure 3 As shown in Table 2, consider the trajectory planning problem for a carrier-based aircraft from a point on the deck to the catapult T1. The specific boundary conditions are shown in Table 2.

[0057] Table 1: Carrier-based aircraft size parameters and algorithm parameters

[0058] Table 2: Initial Boundary Conditions and Terminal Target Points for Carrier-based Aircraft

[0059] A deep reinforcement learning planning method for carrier-based aircraft movement routes that integrates a guidance point reward mechanism, such as... Figure 1 As shown, it includes the following steps: Step 1: Construct the kinematic model, multi-dimensional state model, and neural network model of the carrier-based aircraft; Step 1-1: Model the kinematics of the carrier-based aircraft; When a carrier-based aircraft is taxiing at low speed on the deck, its motion is described using a bicycle kinematics model. In a global Cartesian coordinate system, let the position of the rear wheel center of the carrier-based aircraft at time step t be... ,in, Let x be the x-coordinate of the center of the rear wheel of the carrier-based aircraft at time step t. Let be the ordinate of the rear wheel center of the carrier-based aircraft at time step t. Let the taxiing speed of the carrier-based aircraft at time step t be... The heading angle of the carrier-based aircraft is The front wheel steering angle is Let the control variables for carrier-based aircraft be... ,in, Let t be the longitudinal acceleration of the carrier-based aircraft at time step t. Let be the front wheel steering angular velocity at time step t. Let the carrier-based aircraft's kinematic state vector be... The carrier-based aircraft then obtains its kinematic state for the next time step based on Euler discretization. for: (1) in, , Let t be the coordinates of the center of the rear wheel of the carrier-based aircraft at time step t; , The coordinates of the rear wheel center of the carrier-based aircraft at time step t+1; Let be the glide speed at time step t; Let be the glide speed at time step t+1; Let t be the front wheel steering angle at time step t; Let t+1 be the front wheel steering angle; Let t be the heading angle of the carrier-based aircraft at time step t; The orientation angle of the carrier-based aircraft at time step t+1; Let t be the longitudinal acceleration at time step t; Let t be the front wheel steering angular rate at time step t; This refers to the wheelbase between the front and rear wheels; This is for simulating step size; , These are the minimum and maximum permissible taxiing speeds, respectively. , These are the minimum and maximum permissible front wheel steering angles, respectively; clip(·) is the clipping function.

[0060] Steps 1-2: Modeling a multi-dimensional state model of the carrier-based aircraft; To improve the efficiency of reinforcement learning training, the observation information of carrier-based aircraft is uniformly constructed as a 26-dimensional state vector. The 26-dimensional state vector It is composed of the carrier-based aircraft's own status, target-related status, and obstacle perception status.

[0061] Step 1-2-1: The carrier-based aircraft's own status; The carrier-based aircraft's own state is directly adopted from the carrier-based aircraft kinematic state vector defined in step 1-1. .

[0062] Step 1-2-2: Target-related status; Let the location of the target point be ,in, Let x be the x-coordinate of the target point. Let be the ordinate of the target point. Let the target heading angle be... Define the target-related state of the carrier-based aircraft relative to the target point at time step t as follows: ,in, Let be the Euclidean distance from the center of the rear wheel of the carrier-based aircraft to the target point at time step t. Let t be the target direction angle from the current position of the carrier-based aircraft to the target point at time step t. Let t be the heading deviation at time step t, as shown in formula (2): (2) in, , These are the x and y coordinates of the target point, respectively; wrap(·) is the angle normalization function.

[0063] Steps 1-2-3: Obstacle perception status; An obstacle perception state is constructed using a dual-circle envelope model and a multi-directional ray detection method, such as... Figure 2 As shown. First, the outer contour of the carrier-based aircraft is described using a minimum envelope rectangle. Let the width of this rectangle be... , length is .

[0064] (3) in, , and These represent the distance from the rear wheel of the carrier-based aircraft to the rear edge of the rectangle, the longitudinal distance between the front and rear wheels of the carrier-based aircraft, and the distance from the front wheel of the carrier-based aircraft to the front edge of the rectangle, respectively. For the minimum envelope rectangle, two circles of the same radius are used to completely cover it. The circle closer to the front of the carrier-based aircraft is defined as the front circle, and the circle closer to the rear of the aircraft is defined as the rear circle. Let the common radius of the front and rear circles be . The center of the front circle is The center of the back circle is The coordinates of the center of the front circle and the center of the back circle are calculated as follows: (4) Starting from the center of the front circle respectively and the center of the circle It emits 18 detection rays outward. Here, 18 represents the total number of rays, and the normalized detection distance corresponding to the k-th ray is denoted as . , For each ray, calculate its intersection point with the obstacle boundary, and take the normalized distance corresponding to the nearest intersection point as the ray's detection result. Construct the obstacle sensing state as follows: .

[0065] Finally, the status of the carrier-based aircraft itself. Target-related status and obstacle perception state By concatenating them sequentially, we obtain the multidimensional state vector at time step t. .

[0066] Steps 1-3: Modeling a neural network model; Establish a policy network and value network , where subscript For policy network parameters, These are the parameters of the value network. The policy network... Used based on the current multidimensional state vector Output the probability distribution of discrete action indices; the value network Used based on the current multidimensional state vector Estimate the state value.

[0067] Both the policy network and the value network employ a multilayer perceptron structure. The policy network includes a feature extraction layer and an action output layer, with a 26-dimensional state vector. After being mapped to latent feature vectors by the feature extraction layer, the discrete action probability distribution is output by the action output layer. The value network structure is symmetric to the policy network, and its output is the state value estimate.

[0068] The neural network weights are initialized orthogonally, and the bias parameters are initialized with preset biases to improve gradient stability in the early stages of training.

[0069] Step 2: Model the reward function model that integrates the guide point reward mechanism; A Bezier curve is used to generate the guide point path, and a dynamic weight adjustment mechanism is designed. A dense reward function is constructed that integrates guide point tracking, obstacle avoidance, heading alignment, progress reward, turning penalty, and trajectory smoothness reward to guide the carrier-based aircraft to learn efficiently. Finally, a reward function model is constructed. Specifically: Step 2-1: Guiding Point Rewards; Step 2-1-1: Generate the guide point path; To achieve progressive guidance to the target point and ensure path smoothness, a third-order Bézier curve is used to generate the guide point path from the starting point to the target point. The specific generation method is as follows: Let the initial pose be ,in Starting position Let be the initial heading angle. Assume the target pose is... ,in For the target location, Let be the target heading angle. Let the control point extension length factor be... Define the four control points of the third-order Bézier curve as follows: , , , Specifically: ; To be based on the starting position and starting heading angle , elongation factor The calculated starting control point; To determine the target location and target heading angle , elongation factor The calculated endpoint control point; The coordinates of the corresponding control points are as follows: (5) Then any point on the third-order Bézier curve is denoted as: (6) in, It is the leading point on the Bézier curve; These are the parameters of the Bézier curve.

[0070] Step 2-1-2: Guiding Point Reward and Dynamic Weight Adjustment Mechanism; Let the position of the rear wheel center of the carrier-based aircraft at time step t be... The guiding point corresponding to time step t is ,in The parameters for the Bézier curve correspond to the mission progress. Define the distance between the current position of the carrier-based aircraft and the guide point. for: (7) in, This is the guiding point corresponding to time step t; , Let x and y be the x and y coordinates of the guide point corresponding to time step t.

[0071] Let the effective radius of the guide point foundation be... To accommodate varying guidance intensities during mission advancement, an effective guidance radius after time decay is introduced. As shown in formula (8): (8) in, This is the time decay coefficient; Ideal remaining path length at time step t.

[0072] The reward for the guiding point is recorded as shown in formula (9): (9) in, The reward is given for the guiding point at time step t; The weights are dynamic, comprehensively reflecting both environmental conditions and task priorities; The first product coefficient is awarded as a guide point reward; The second product coefficient is awarded as a guide point reward.

[0073] The dynamic weights are: (10) in, Basic weights; For obstacle avoidance factors; For position factors; For velocity factor; Angle factor.

[0074] Step 2-2: Design multi-dimensional reward components; In addition to the guide point reward, multiple reward components are designed to jointly constrain the behavior of carrier-based aircraft, including obstacle avoidance reward, artificial potential field gravity reward, differential distance reward, heading alignment reward, turning penalty, position constraint reward, trajectory smoothness reward, and curvature reward. Specifically: The obstacle avoidance reward Obstacle-based state perception The normalized detection distance of 18 rays in the middle is used to construct an obstacle avoidance reward.

[0075] (11) in, The obstacle avoidance reward is given at time step t; K=18 represents the total number of rays. Let be the normalized detection distance of the k-th ray returning at time step t.

[0076] The artificial potential field gravitational reward Simulates a gravitational potential field to guide carrier-based aircraft toward the target.

[0077] (12) in, The gravitational reward for the artificial potential field at time step t; The distance between the carrier-based aircraft and the target point is expressed in Euclidean form. For capture distance; The product coefficient of gravitational reward for artificial potential field; This is a reward constant term.

[0078] The differential distance reward Reward progress based on the change in distance to the target point between the previous and next steps.

[0079] (13) in, The difference distance reward at time step t; and These are the Euclidean distances between the carrier-based aircraft and the target point at the previous and current time steps, respectively. This is the product coefficient for the differential distance reward.

[0080] The heading alignment reward When a carrier-based aircraft approaches the target point, a heading alignment reward is generated based on the deviation between the aircraft's current heading and the target heading.

[0081] (14) in, The heading alignment reward is given at time step t. The target heading angle; This is the current facing angle; The first heading alignment bonus product factor; The product factor for alignment bonus in the second course; The maximum allowable heading deviation threshold.

[0082] The turning penalty To suppress frequent and large-amplitude turns by carrier-based aircraft, a nonlinear penalty term is constructed for the front wheel steering angle.

[0083] (15) in, Penalty for turning at time step t; Let t be the front wheel steering angle at time step t; The product coefficient for the penalty of turning; The maximum allowable heading deviation threshold.

[0084] The location constraint reward : Determine if the fore and aft wheels exceed the deck boundary, enter a restricted area, or the island superstructure, in order to provide... Negative rewards.

[0085] The trajectory smoothness negative reward A negative reward for trajectory smoothness is constructed based on the statistical characteristics of the recent changes in velocity and front wheel steering angle over several time steps.

[0086] (16) in, The reward is the trajectory smoothness at time step t; The standard deviation of the velocity change over a recent number of time steps; The standard deviation of the change in the front wheel steering angle over a recent period of time; The first product coefficient is awarded as a reward for trajectory smoothness; A second product coefficient is awarded as a reward for trajectory smoothness.

[0087] The curvature reward The local curvature of the trajectory is estimated using the three-point method, rewarding straight-line motion and penalizing large-curvature turns.

[0088] (17) in, The curvature reward at time step t; This represents the change in the direction angle of adjacent trajectory segments; The first product coefficient is awarded for curvature reward; The second product coefficient is awarded as a curvature bonus; The maximum allowable heading deviation threshold.

[0089] Steps 2-3: Synthesize the total reward function; The guiding point reward obtained in step 2-1 The reward is then weighted and summed together with the reward components obtained in step 2-2 to obtain the instantaneous reward at time step t. : (18) Based on this, combined with environmental interaction step size And add time penalty Collision penalty and rewards for successful missions Obtain the total reward function at time step t. As shown in formula (19): (19) Total reward function Complex scheduling objectives are transformed into continuous numerical signals, driving carrier-based aircraft to autonomously learn smooth paths that satisfy kinematic and obstacle avoidance constraints during reinforcement learning.

[0090] Step 3: Train the policy network and value network based on the PPO algorithm; Step 3-1: The carrier-based aircraft completes one interaction with the environment; At time step t, based on the current 26-dimensional state vector Output discrete action index Indexing discrete actions Converted into actual control quantities through motion mapping The kinematic state of the carrier-based aircraft at the next time step is obtained by updating the kinematic model established in step 1-1. Subsequently, collision detection was performed on the dual-circle envelope model of the carrier-based aircraft and the obstacle boundary, and ray detection data was generated based on the center of the front and rear circles to obtain the obstacle perception state at the next time step. Then, the total reward for the current time step is calculated using the reward function model established in step 2. When the Euclidean distance between the carrier-based aircraft and the target point is less than the acquisition distance... Furthermore, the heading deviation between the carrier-based aircraft and the target heading shall not exceed the maximum permissible heading deviation threshold. At that point, it is determined that the carrier-based aircraft has successfully reached the target. Finally, the updated carrier-based aircraft's own state, target-related state, and obstacle perception state are concatenated to obtain the multi-dimensional state vector for the next time step. .

[0091] Step 3-2: Store trajectory samples; After completing one step of environmental interaction, the data generated at the current time step is stored in the trajectory buffer as a trajectory sample. Each trajectory sample includes the current 26-dimensional state vector. Discrete Action Index Under the old strategy, the log probability of the discrete action is... Total reward at current time step Termination mark The old value network is related to the state. Value estimation and the multidimensional state vector at the next time step .in, This is the old strategy.

[0092] Step 3-3: Calculate the advantage function and discounted return steps; After a training cycle ends, the advantage function and discount reward are calculated based on the trajectory samples in the trajectory buffer, and then the samples are shuffled and divided into batches.

[0093] First, the value estimate of the next state at the last time step is calculated using the current value network. Then, the temporal difference error for each time step is calculated in reverse chronological order. .

[0094] (20) in, Let t be the timing difference error at time step t; Discount factor; For the old value network to state Value estimation; For the old value network to state The value estimate;

[0095] Secondly, the generalized advantage estimation method is used to calculate the advantage function. : (twenty one) in, Let t be the dominance function at time step t; This is the estimate of the dominance function for the next time step; Discount factor; Smoothing parameters are estimated for generalized dominance.

[0096] Finally, based on the advantage function and the old value network, the state is... The value estimate is discounted. As shown in formula (22): (twenty two) Steps 3-4: Construct batch training data; After completing the advantage function and discount reward calculation in step 3-3, the trajectory samples in the trajectory buffer are randomly shuffled and divided into mini-batch=1 batches for subsequent PPO updates.

[0097] Steps 3-5: Update the policy network; The mini-batch samples constructed in steps 3-4 are input into the policy network, and the logarithmic probability of actions and the policy entropy under the current policy are calculated forward. The policy entropy is denoted as... As shown in formula (23): (twenty three) in, Let M be the policy entropy at time step t; M=46 is the total number of discrete actions; In the state The probability of taking discrete action m; These are the policy network parameters.

[0098] Secondly, calculate the probability ratio between the current strategy and the old strategy. : (twenty four) in, The probability ratio of the new and old strategies at time step t; This is the current strategy; This is the old strategy; This is the index of the discrete action at time step t; Let be the multidimensional state vector at time step t.

[0099] Next, a pruning substitution objective function is constructed using the PPO pruning objective. As shown in formula (25): (25) in, Replace the objective function with a pruning function; Let t be the dominance function at time step t; PPO trimming parameters; This represents the expectation of the sample time step.

[0100] Based on the pruning and substitution objective function, and combined with the entropy regularization term, the total policy loss is constructed. As shown in formula (26): (26) in, This represents the total strategic loss. It is the entropy coefficient; Let be the policy entropy at time step t.

[0101] Finally, regarding the total strategy loss Perform backpropagation and base it on the maximum gradient norm. The gradients are pruned, and then the optimizer is stepped to obtain the updated policy network parameters. .

[0102] Steps 3-6: Update the value network; Input the batch samples constructed in steps 3-4 into the value network to obtain the value network's state for the current state. Value estimation To avoid excessively large value network updates, a value pruning method is used to construct the pruned value prediction, as shown in formula (27): (27) in, For the value prediction after trimming; For the current state, the value network is used to evaluate the state. Value estimation; For the old value network to state Value estimation; For value network parameters; These are the PPO trimming parameters.

[0103] First, calculate the original value loss and the value loss after trimming separately, and take the larger of the two as the final value loss. Then, perform regression using Huber loss. Next, multiply the value loss by the value loss coefficient. Then backpropagation is performed, also based on the maximum gradient norm. The gradients are pruned, and finally the optimizer is run to obtain the updated value network parameters. .

[0104] Step 4: Utilize the trained strategy network to plan the carrier-based aircraft movement routes; Load the policy network parameters trained in step 3. And construct the corresponding environment based on the deck scene to be planned. During the path planning process, the multi-dimensional state vector of the current time step is... Input the trained policy network To obtain the discrete action index Then index the discrete actions. Converted into control variables through action mapping The carrier-based aircraft's pose is updated using the kinematic model from step 1-1. If the termination condition is not met, the multidimensional state vector for the next time step is generated, and the above process is repeated until the carrier-based aircraft's Euclidean distance to the target point is less than the acquisition distance. Furthermore, the heading deviation between the carrier-based aircraft and the target heading shall not exceed the maximum permissible heading deviation threshold. Finally, the aircraft position and attitude sequences recorded at each time step during the planning process are output as the aircraft transport trajectories for the deck control system to access. Figure 4 The gliding trajectory calculated in the example.

[0105] The above embodiments are merely illustrative of the implementation methods of the present invention, but should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the protection scope of the present invention.

Claims

1. A deep reinforcement learning planning method for carrier-based aircraft movement routes that integrates a guidance point reward mechanism, characterized in that... The deep reinforcement learning planning method for carrier-based aircraft movement paths includes the following steps: Step 1: Construct the kinematic model, multi-dimensional state model, and neural network model of the carrier-based aircraft; Step 1-1: Model the kinematics of the carrier-based aircraft; When a carrier-based aircraft is taxiing at low speed on the deck, its motion is described using a bicycle kinematics model. In a global Cartesian coordinate system, let the position of the rear wheel center of the aircraft at time step t be... ,in, Let x be the x-coordinate of the center of the rear wheel of the carrier-based aircraft at time step t. Let be the ordinate of the rear wheel center of the carrier-based aircraft at time step t; let the taxiing speed of the carrier-based aircraft at time step t be . The heading angle of the carrier-based aircraft is The front wheel steering angle is Assume the control variables for carrier-based aircraft are ,in, Let t be the longitudinal acceleration of the carrier-based aircraft at time step t. Let be the front wheel steering angular velocity at time step t; let the aircraft's kinematic state vector be... The carrier-based aircraft then obtains its kinematic state for the next time step based on Euler discretization. for: （1） in, , Let t be the coordinates of the center of the rear wheel of the carrier-based aircraft at time step t; , The coordinates of the rear wheel center of the carrier-based aircraft at time step t+1; Let be the glide speed at time step t; Let be the glide speed at time step t+1; Let t be the front wheel steering angle at time step t; Let t+1 be the front wheel steering angle; Let t be the heading angle of the carrier-based aircraft at time step t; The orientation angle of the carrier-based aircraft at time step t+1; Let t be the longitudinal acceleration at time step t; Let be the front wheel steering angular velocity at time step t; This refers to the wheelbase between the front and rear wheels; This is for simulating step size; , These are the minimum and maximum permissible taxiing speeds, respectively. , These are the minimum and maximum permissible front wheel steering angles, respectively; clip(·) is the clipping function; Steps 1-2: Modeling a multi-dimensional state model of the carrier-based aircraft; To improve the efficiency of reinforcement learning training, the observation information of carrier-based aircraft is uniformly constructed as a multi-dimensional state vector. The multidimensional state vector It is composed of the carrier-based aircraft's own status, target-related status, and obstacle perception status; Steps 1-3: Modeling a neural network model; Establish a policy network and value network , where subscript For policy network parameters, The value network parameters; the policy network Used based on the current multidimensional state vector Output the probability distribution of discrete action indices; the value network Used based on the current multidimensional state vector Estimate the value of the state; Step 2: Model the reward function model that integrates the guide point reward mechanism; A Bezier curve is used to generate the guide point path, and a dynamic weight adjustment mechanism is designed. A dense reward function is constructed that integrates guide point tracking, obstacle avoidance, heading alignment, progress reward, turning penalty, and trajectory smoothness reward to guide the carrier-based aircraft to learn efficiently. Finally, a reward function model is constructed. Specifically: Step 2-1: Receive the guiding point reward; Step 2-2: Design multi-dimensional reward components; Multiple reward components are designed to jointly constrain the behavior of carrier-based aircraft, including obstacle avoidance reward, artificial potential field gravity reward, differential distance reward, heading alignment reward, turning penalty, position constraint reward, trajectory smoothness reward, and curvature reward. Steps 2-3: Synthesize the total reward function; Step 3: Train the policy network and value network based on the PPO algorithm; Step 3-1: The carrier-based aircraft completes one interaction with the environment; Step 3-2: Store trajectory samples; Step 3-3: Calculate the advantage function and discounted return steps; Steps 3-4: Construct mini-batch training data; Steps 3-5: Update the policy network; Steps 3-6: Update the value network; Step 4: Use the trained strategy network to plan the carrier-based aircraft movement path.

2. The deep reinforcement learning planning method for carrier-based aircraft movement paths based on the integrated guidance point reward mechanism as described in claim 1, characterized in that, Steps 1-2 are specifically as follows: Step 1-2-1: The carrier-based aircraft's own status; The carrier-based aircraft's own state is directly adopted from the carrier-based aircraft kinematic state vector defined in step 1-1. ; Step 1-2-2: Target-related status; Let the location of the target point be ,in, Let x be the x-coordinate of the target point. Let be the ordinate of the target point; let the target heading angle be . Define the target-related state of the carrier-based aircraft relative to the target point at time step t as follows: ,in, Let be the Euclidean distance from the center of the rear wheel of the carrier-based aircraft to the target point at time step t. Let t be the target direction angle from the current position of the carrier-based aircraft to the target point at time step t. Let t be the heading deviation at time step t, as shown in formula (2): （2） in, , These are the x and y coordinates of the target point, respectively; wrap(·) is the angle normalization function; Steps 1-2-3: Obstacle perception status; An obstacle perception state is constructed using a double-circle envelope model and a multi-directional ray detection method. First, a minimum envelope rectangle is used to describe the outer contour of the carrier-based aircraft; let the width of this rectangle be... , length is ; （3） in, , and These represent the distance from the rear wheel of the carrier-based aircraft to the rear edge of the rectangle, the longitudinal distance between the front and rear wheels of the carrier-based aircraft, and the distance from the front wheel of the carrier-based aircraft to the front edge of the rectangle, respectively. For the minimum envelope rectangle, two circles of the same radius completely cover it. The circle closer to the front of the carrier-based aircraft is defined as the front circle, and the circle closer to the rear of the aircraft is defined as the rear circle. Let the common radius of the front and rear circles be . The center of the front circle is The center of the back circle is The coordinates of the center of the front circle and the center of the back circle are calculated as follows: （4） Starting from the center of the front circle respectively and the center of the circle K detection rays are emitted outward; where K is the total number of rays, and the normalized detection distance corresponding to the k-th ray is denoted as . , For each ray, calculate its intersection point with the obstacle boundary, and take the normalized distance corresponding to the nearest intersection point as the ray detection result; construct the obstacle sensing state as follows: ; Finally, the status of the carrier-based aircraft itself. Target-related status and obstacle perception state By concatenating them sequentially, we obtain the multidimensional state vector at time step t. .

3. The deep reinforcement learning planning method for carrier-based aircraft movement paths based on the integrated guidance point reward mechanism as described in claim 2, characterized in that, In steps 1-3: Both the policy network and the value network employ a multilayer perceptron structure; the policy network includes a feature extraction layer and an action output layer, with a multidimensional state vector. After being mapped to latent feature vectors by the feature extraction layer, the discrete action probability distribution is output by the action output layer; the value network structure is symmetrical with the policy network, and its output is the state value estimate; The neural network weights are initialized orthogonally, and the bias parameters are initialized using preset biases.

4. The deep reinforcement learning planning method for carrier-based aircraft movement paths based on the integrated guidance point reward mechanism as described in claim 3, characterized in that, Step 2-1 specifically involves: Step 2-1-1: Generate the guide point path; A guide point path from the starting point to the target point is generated using a third-order Bézier curve; the specific generation method is as follows: Let the initial pose be ,in Starting position Let the initial heading angle be ; let the target pose be . ,in For the target location, Let the target heading angle be ; let the control point extension length factor be . Define the four control points of the third-order Bézier curve as follows: , , , Specifically: ; To be based on the starting position and starting heading angle , Extension length factor The calculated starting control point; To determine the target location and target heading angle , Extension length factor The calculated endpoint control point; The coordinates of the corresponding control points are as follows: （5） Then any point on the third-order Bézier curve is denoted as: （6） in, It is the leading point on the Bézier curve; These are the parameters of the Bézier curve; Step 2-1-2: Guiding Point Reward and Dynamic Weight Adjustment Mechanism; Let the position of the rear wheel center of the carrier-based aircraft at time step t be... The guiding point corresponding to time step t is ,in Define the Bézier curve parameters corresponding to the mission progress; define the distance between the current position of the carrier-based aircraft and the guide point. for: （7） in, This is the guiding point corresponding to time step t; , Let x and y coordinates be the x and y coordinates of the guide point corresponding to time step t; Let the effective radius of the guide point foundation be... To accommodate varying guidance intensities during mission advancement, an effective guidance radius after time decay is introduced. As shown in formula (8): （8） in, This is the time decay coefficient; Ideal remaining path length at time step t; The reward for the guiding point is recorded as shown in formula (9): （9） in, The reward is given for the guiding point at time step t; The weights are dynamic, comprehensively reflecting both environmental conditions and task priorities; The first product coefficient is awarded as a guide point reward; The second product coefficient is awarded as a guide point; The dynamic weights are: （10） in, Basic weights; For obstacle avoidance factors; For position factors; The velocity factor; Angle factor.

5. The deep reinforcement learning planning method for carrier-based aircraft movement paths based on the integrated guidance point reward mechanism as described in claim 4, characterized in that, Step 2-2 specifically involves: The obstacle avoidance reward Obstacle-based state perception The normalized detection distance of each ray in the middle is used to construct an obstacle avoidance reward; （11） in, K represents the obstacle avoidance reward at time step t; K is the total number of rays. Let be the normalized detection distance returned by the k-th ray at time step t; The artificial potential field gravitational reward Simulates a gravitational potential field to guide carrier-based aircraft toward the target; （12） in, The gravitational reward for the artificial potential field at time step t; The distance between the carrier-based aircraft and the target point is expressed in Euclidean form. For capture distance; The product coefficient of gravitational reward for artificial potential field; For reward constant terms; The differential distance reward Reward progress based on the change in distance to the target point between two consecutive steps; （13） in, The difference distance reward at time step t; and These are the Euclidean distances between the carrier-based aircraft and the target point at the previous and current time steps, respectively. The differential distance reward product coefficient; The heading alignment reward When a carrier-based aircraft approaches the target point, a heading alignment reward is generated based on the deviation between the aircraft's current heading and the target heading. （14） in, The heading alignment reward is given at time step t. The target heading angle; This is the current facing angle; The first heading alignment bonus product factor; The product factor for alignment bonus in the second course; The maximum allowable heading deviation threshold; The turning penalty To suppress frequent and large-amplitude turns by carrier-based aircraft, a nonlinear penalty term is constructed for the steering angle of the front wheels; （15） in, Penalty for turning at time step t; Let t be the front wheel steering angle at time step t; The product coefficient for the penalty of turning; The maximum allowable heading deviation threshold; The location constraint reward : If the fore and aft wheels are found to have exceeded the deck boundary, entered a restricted area, or entered the island superstructure, a large negative reward will be given; The trajectory smoothness negative reward A negative reward for trajectory smoothness is constructed based on the statistical characteristics of the velocity changes and front wheel steering angle changes over several recent time steps. （16） in, The reward is the trajectory smoothness at time step t; The standard deviation of the velocity change over a recent number of time steps; The standard deviation of the change in the front wheel steering angle over a recent period of time; The first product coefficient is awarded as a reward for trajectory smoothness; The second product coefficient is awarded as a reward for trajectory smoothness; The curvature reward The local curvature of the trajectory is estimated using the three-point method, rewarding straight-line motion and penalizing large-curvature turns. （17） in, The curvature reward at time step t; This represents the change in the direction angle of adjacent trajectory segments; The first product coefficient is awarded for curvature reward; The second product coefficient is awarded as a curvature bonus; The maximum allowable heading deviation threshold.

6. The deep reinforcement learning planning method for carrier-based aircraft movement paths based on the integrated guidance point reward mechanism as described in claim 5, characterized in that, Steps 2-3 are specifically as follows: The guiding point reward obtained in step 2-1 The reward is then weighted and summed together with the reward components obtained in step 2-2 to obtain the instantaneous reward at time step t. : （18） Combined with environmental interaction step size And add time penalty Collision penalty and rewards for successful missions Obtain the total reward function at time step t. As shown in formula (19): （19）。 7. The deep reinforcement learning planning method for carrier-based aircraft movement paths based on the integrated guidance point reward mechanism as described in claim 6, characterized in that, Step 3 specifically involves: Step 3-1 specifically involves: at time step t, based on the current multidimensional state vector... Output discrete action index Indexing discrete actions Converted into actual control quantities through motion mapping The kinematic state of the carrier-based aircraft at the next time step is obtained by updating the kinematic model established in step 1-1. Subsequently, collision detection is performed on the dual-circle envelope model of the carrier-based aircraft and the obstacle boundary, and ray detection data is generated based on the center of the front and rear circles to obtain the obstacle perception state at the next time step. ; The total reward at the current time step is then calculated using the reward function model established in step 2. When the Euclidean distance between the carrier-based aircraft and the target point is less than the acquisition distance... Furthermore, the heading deviation between the carrier-based aircraft and the target heading shall not exceed the maximum permissible heading deviation threshold. At that point, it is determined that the carrier-based aircraft has successfully reached the target; finally, the updated carrier-based aircraft's own state, target-related state, and obstacle perception state are concatenated to obtain the multi-dimensional state vector for the next time step. ; Step 3-2 specifically involves: after completing one step of environmental interaction, storing the data generated at the current time step into the trajectory buffer as a trajectory sample; each trajectory sample includes the current multidimensional state vector. Discrete Action Index Under the old strategy, the log probability of the discrete action is... Total reward at current time step Termination mark The old value network is related to the state. Value estimation and the multidimensional state vector at the next time step ;in, This is the old strategy; Step 3-3 specifically involves: After a training cycle ends, the advantage function and discounted reward are calculated based on the trajectory samples in the trajectory buffer, and then the samples are shuffled and divided into batches. First, the value estimate of the next state at the last time step is calculated using the current value network; then, the temporal difference error of each time step is calculated in reverse chronological order. ; （20） in, Let t be the timing difference error at time step t; Discount factor; For the old value network to state Value estimation; For the old value network to state Value estimation; Secondly, the generalized advantage estimation method is used to calculate the advantage function. : （21） in, Let t be the dominance function at time step t; This is the estimate of the dominance function for the next time step; Discount factor; To estimate the smoothing parameters for generalized dominance; Finally, based on the advantage function and the old value network, the state is... The value estimate is discounted. As shown in formula (22): （22） Steps 3-4 specifically refer to: After completing the calculation of the advantage function and discount return in step 3-3, the trajectory samples in the trajectory buffer are randomly shuffled and divided into several batches. Steps 3-5 specifically refer to: The mini-batch samples constructed in steps 3-4 are input into the policy network, and the log probability of actions and the policy entropy under the current policy are calculated in the forward pass; the policy entropy is denoted as... As shown in formula (23): （23） in, Let M be the policy entropy at time step t; M be the total number of discrete actions. In the state The probability of taking discrete action m; For policy network parameters; Secondly, calculate the probability ratio between the current strategy and the old strategy. : （24） in, The probability ratio of the new strategy to the old strategy at time step t; This is the current strategy; This is the old strategy; This is the index of the discrete action at time step t; Let t be the multidimensional state vector at time step t; Next, a pruning substitution objective function is constructed using the PPO pruning objective. As shown in formula (25): （25） in, Replace the objective function with a pruning function; Let t be the dominance function at time step t; PPO trimming parameters; This represents the expectation over the sample time step; Based on the pruning and substitution objective function, and combined with the entropy regularization term, the total policy loss is constructed. As shown in formula (26): （26） in, This represents the total strategic loss. It is the entropy coefficient; Let t be the policy entropy at time step t; Finally, regarding the total strategy loss Perform backpropagation and base it on the maximum gradient norm. The gradients are pruned, and then the optimizer is stepped to obtain the updated policy network parameters. ; Steps 3-6 specifically refer to: Input the batch samples constructed in steps 3-4 into the value network to obtain the value network's state for the current state. Value estimation To avoid excessively large value network updates, a value pruning method is used to construct the pruned value prediction, as shown in formula (27): （27） in, For the value prediction after trimming; For the current state, the value network is used to evaluate the state. Value estimation; For the old value network to state Value estimation; For value network parameters; PPO trimming parameters; First, calculate the original value loss and the value loss after trimming separately, take the larger of the two as the final value loss, and perform regression using Huber loss; then, multiply the value loss by the value loss coefficient. Then backpropagation is performed, also based on the maximum gradient norm. The gradients are pruned; finally, the optimizer is stepped to obtain the updated value network parameters. .

8. The deep reinforcement learning planning method for carrier-based aircraft movement paths based on the integrated guidance point reward mechanism as described in claim 7, characterized in that, Step 4 specifically involves: Load the parameters of the policy network trained in step 3. And construct the corresponding environment based on the deck scene to be planned; during the path planning process, the multi-dimensional state vector of the current time step is... Input the trained policy network To obtain the discrete action index Then index the discrete actions. Converted into control variables through action mapping The carrier-based aircraft's pose is updated using the kinematic model from step 1-1. If the termination condition is not met, the multidimensional state vector for the next time step is generated and the above process is repeated until the carrier-based aircraft's Euclidean distance to the target point is less than the acquisition distance. Furthermore, the heading deviation between the carrier-based aircraft and the target heading shall not exceed the maximum permissible heading deviation threshold. Finally, the aircraft position and attitude sequences recorded at each time step during the planning process are output as the aircraft transport trajectories for the deck dispatch system to use.

Images