An aircraft dynamic recovery sequencing method and device based on reinforcement learning

Abstract

The application discloses an aircraft dynamic recovery sequencing method and device based on reinforcement learning, and relates to the technical field of air traffic control.The method comprises the following steps: constructing a dynamic recovery environment model containing a go-around scene, designing a multi-dimensional reward function and a safety cost function, and realizing intelligent decision-making of a reinforcement learning agent on aircraft recovery scheduling; the multi-dimensional reward function comprehensively considers four core indexes, i.e., fuel consumption penalty, fault priority reward, task priority reward and go-around penalty; the safety cost function quantifies the fuel safety risk of an unlanded aircraft; and the reinforcement learning agent optimizes the scheduling strategy under the constraint of the dynamic balance of maximizing the reward function and minimizing the safety cost.The application realizes closed-loop optimization of the aircraft recovery process by periodically updating the environment state and advancing the decision-making stage, effectively solves the coordination problem among fuel economy, task priority and safety risk, and significantly improves the aircraft recovery efficiency and safety.

Description

Technical Field

[0001] This application relates to the field of aviation scheduling technology, and in particular to a method and device for dynamic aircraft recovery and sorting based on reinforcement learning. Background Technology

[0002] In the field of air traffic control, the Aircraft Recovery and Scheduling Problem (ARSP) is a complex dynamic decision-making problem. Traditional methods, mainly based on heuristic rules or mathematical programming, struggle to cope with dynamically changing environments (such as sudden failures and weather changes). In recent years, reinforcement learning has been introduced into this field, but its performance is highly dependent on the design of the reward function. Existing reward function designs are often limited to a single objective (such as minimizing total time), lacking comprehensive consideration of fuel safety, task priority, and failure handling. Especially in aircraft recovery scenarios, multiple objectives such as go-around risk, fuel constraints, and task urgency are mutually constraining. The scheduling strategy in the ARSP needs to be transformed into an executable recovery and sequencing sequence, which current technologies struggle to effectively balance. Furthermore, traditional methods often employ binary penalty mechanisms to handle safety constraints, leading to low learning efficiency. Therefore, there is an urgent need for a reinforcement learning reward function design method that can comprehensively quantify multiple objectives and dynamically adjust strategies. Summary of the Invention

[0003] The purpose of this application is to provide a method and device for dynamic aircraft recovery and sorting based on reinforcement learning. By periodically updating the environmental state and advancing the decision-making stage, the closed-loop optimization of the aircraft recovery process is achieved, which effectively solves the coordination problem between fuel economy, task priority and safety risk, and significantly improves the efficiency and safety of aircraft recovery.

[0004] To achieve the above objectives, this application provides the following solution:

[0005] Firstly, this application provides a reinforcement learning-based method for dynamic aircraft recovery and ranking, including:

[0006] Construct a dynamic recovery environment model that includes go-around scenarios;

[0007] Based on the dynamic recovery environment model, a multidimensional reward function is constructed; the multidimensional reward function is used to describe the fuel consumption penalty, fault priority reward, mission priority reward and go-around penalty in the go-around scenario;

[0008] Based on the dynamic recovery environment model and the multidimensional reward function, a safety cost function is constructed; the safety cost function is used to quantify the total fuel safety risk of all non-landing aircraft in the aircraft recovery sorting sequence.

[0009] Based on the multidimensional reward function and the safety cost function, the optimal aircraft recovery sorting sequence is obtained by optimizing the scheduling strategy through reinforcement learning agent.

[0010] The optimal aircraft recovery sorting sequence is converted into executable instructions and transmitted to the airport dispatch system.

[0011] Optionally, the multidimensional reward function is:

[0012] ;

[0013] in, ;

[0014] ;

[0015] In the formula, r For multidimensional reward functions; The maximum number of decision-making stages; For the stage The reward function; This is an adjustable reward weighting coefficient; This is a fuel consumption penalty item; It is a rectified linear activation function; For airplane In the stage The landing status indicator variable, This indicates a successful landing; Represents a sequence Index of aircraft awaiting landing during the mid-stage τ; Prioritize rewarding fault-related issues; This is a reward item for task priority; This is a penalty for resuming flights.

[0016] Optionally, the fuel consumption penalty is:

[0017] ;

[0018] In the formula, For the collection of aircraft awaiting recovery Represents the cardinality of a set; Representation phase Time; Representation phase Time; Indicates that aircraft i is the type of aircraft waiting to land. Fuel consumption rate at that time.

[0019] Optionally, the fault priority reward item is:

[0020] ;

[0021] In the formula, Indicates an aircraft waiting to land Failure probability assessment value; Indicates an aircraft waiting to land The failure probability assessment value.

[0022] Optionally, the task priority reward item is:

[0023] ;

[0024] In the formula, Indicates an aircraft waiting to land The mission urgency ranking is as follows: the mission urgency ranking of the aircraft waiting to land is 1.

[0025] Optionally, the go-around penalty item is:

[0026] ;

[0027] in, ;

[0028] In the formula, The delay caused by the resumption of flights Time decay factor; This is the penalty intensity coefficient; The aircraft selected for the current decision The planned recovery phase.

[0029] Optionally, the security cost function is:

[0030] ;

[0031] in, ;

[0032] In the formula, For safety cost function; For the stage The single-stage security cost; For the stage Assemble the aircraft that have not yet landed; Indicates the safe fuel level threshold for the aircraft model; Indicates airplane exist Remaining fuel level at the stage; in the aircraft exist Stage Remaining Oil Less than the safe fuel level threshold for the model Time, stage Single-stage security cost It is directly proportional to the amount of fuel shortage; This is the safety cost conversion coefficient.

[0033] Optionally, based on the multidimensional reward function and the safety cost function, an optimal aircraft recovery sorting sequence is obtained by optimizing the scheduling strategy through a reinforcement learning agent, specifically including:

[0034] By optimizing the scheduling strategy through reinforcement learning agents, a dynamic balance is achieved between maximizing the multidimensional reward function and minimizing the safety cost function, resulting in the output of the optimal sequence.

[0035] The multidimensional reward function value corresponding to the optimal sequence is determined to be the actual cumulative reward.

[0036] Determine the safety cost function value corresponding to the optimal sequence as the actual safety cost;

[0037] The ratio of actual cumulative rewards to actual security costs is determined as the effectiveness factor;

[0038] Determine whether the validity factor is lower than a preset validity threshold, and obtain the determination result;

[0039] If the judgment result is yes, then adjust the adjustable reward weight coefficient in the multidimensional reward function and return to the step "Optimize the scheduling strategy through reinforcement learning agent to achieve a dynamic balance between maximizing the multidimensional reward function and minimizing the safety cost function, and output the optimal sequence";

[0040] If the judgment result is negative, then the optimal sequence is determined to be the optimal aircraft recovery sorting sequence.

[0041] Optionally, after converting the optimal aircraft recovery sorting sequence into executable instructions and transmitting them to the airport dispatching system, the method further includes:

[0042] Execute the executable instructions and record fuel consumption, risk indicators, and go-around events;

[0043] The fuel status is updated using the dynamic fuel status update formula, and the process returns to the step "Construct a dynamic recovery environment model that includes the go-around scenario".

[0044] The formula for dynamically updating the fuel status is:

[0045] ;

[0046] In the formula, Indicates airplane exist Remaining oil volume at the stage; Representation phase and stage The time increment between; For indicator functions, when exist Indication function when the phase has not landed The value is 1.

[0047] Secondly, this application provides a computer device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the above-described reinforcement learning-based aircraft dynamic recovery sorting method.

[0048] According to the specific embodiments provided in this application, the following technical effects are disclosed:

[0049] This application provides a reinforcement learning-based method and device for dynamic aircraft recovery and sorting, constructing a dynamic environment model S containing a go-around scenario, and defining a set of execution stages for the sorting sequence. Design a multidimensional cumulative reward function *r* and a safety cost function for the overall sorted sequence. This application outputs the optimal aircraft recovery sorting sequence A* through a reinforcement learning agent; establishes a sorting effectiveness evaluation mechanism and dynamic adjustment strategy to form a closed-loop learning mechanism of "environmental state perception - action decision-making - reward feedback - safety assessment - strategy update", ultimately achieving global optimization of recovery efficiency and safety; this application outputs the optimal aircraft recovery sorting sequence through a reinforcement learning agent, realizing the coordinated optimization of fuel consumption minimization, task priority execution, emergency fault handling and safety risk control during the aircraft recovery process, providing an intelligent solution for modern aircraft dynamic scheduling, providing an effective optimization objective for the reinforcement learning agent, and realizing the generation of the optimal scheduling strategy under multi-objective constraints. Attached Figure Description

[0050] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0051] Figure 1 This is a flowchart of a reinforcement learning-based aircraft dynamic recovery and sorting method in one embodiment of this application. Detailed Implementation

[0052] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0053] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0054] In one exemplary embodiment, such as Figure 1 As shown, a reinforcement learning-based method for dynamic aircraft recovery and ranking is provided, including:

[0055] Step 101: Construct a dynamic recovery environment model that includes go-around scenarios. .

[0056] Environmental model initialization: Input the aircraft type parameters k∈K of the set of aircraft to be recovered I, and the initial fuel quantity. and safety threshold Set the maximum number of decision stages. Construct the state space S. Define the set of execution stages for the sorted sequence. .

[0057] Step 102: Construct a multi-dimensional reward function based on the dynamic recovery environment model. The multi-dimensional reward function is used to evaluate the overall scheduling effect of the sorting sequence and describes the fuel consumption penalty, fault priority reward, task priority reward, and go-around penalty in the go-around scenario.

[0058] The multidimensional reward function is: .

[0059] in, .

[0060] .

[0061] In the formula, r For multidimensional reward functions; The maximum number of decision-making stages; For the stage The reward function; The adjustable reward weight coefficient is initialized before step 104 and then evaluated in step 104 based on the actual accumulated reward. Compared to actual security costs The ratio is dynamically adjusted; This is a fuel consumption penalty item; It is a rectified linear activation function; For airplane In the stage The landing status indicator variable, This indicates a successful landing; Represents a sequence Index of aircraft awaiting landing during the mid-stage τ; Prioritize rewarding fault-related issues; This is a reward item for task priority; This is a penalty for resuming flights.

[0062] The fuel consumption penalty items are: .

[0063] In the formula, For the collection of aircraft awaiting recovery Represents the cardinality of a set; Representation phase Time; Representation phase Time; Indicates that aircraft i is the type of aircraft waiting to land. Fuel consumption rate at that time. For model category identification, ; This is a set of aircraft model categories.

[0064] The fault priority reward items are: .

[0065] In the formula, Indicates an aircraft waiting to land Failure probability assessment value; Indicates an aircraft waiting to land The failure probability assessment value; the higher the failure probability, the better. The higher the weight.

[0066] The task priority reward items are: .

[0067] In the formula, Indicates an aircraft waiting to land The mission urgency is ranked; the aircraft with the highest urgency and waiting to land has the largest weight (mission urgency ranking), with a value of 1.

[0068] The penalty for going back to flight is: .

[0069] in, .

[0070] In the formula, This is the time decay factor; Indicates the number of delay phases caused by the resumption of flights; The delay caused by the resumption of flights Time decay factor; This is the penalty intensity coefficient; The aircraft selected for the current decision The planned recovery phase, This indicates the aircraft selected for the current decision.

[0071] Step 103: Construct a safety cost function based on the dynamic recovery environment model and multidimensional reward function. The safety cost function is used to quantify the total fuel safety risk of all non-landed aircraft in the aircraft recovery sequence.

[0072] The safety cost function is: .

[0073] in, .

[0074] In the formula, For safety cost function; For the stage The single-stage security cost; For the stage Assemble the aircraft that have not yet landed; Indicates the safe fuel level threshold for the aircraft model; Indicates airplane exist Remaining fuel level at the stage; in the aircraft exist Stage Remaining Oil Less than the safe fuel level threshold for the model Time, stage Single-stage security cost It is directly proportional to the amount of fuel shortage; This is the safety cost conversion coefficient.

[0075] Step 104: Based on the multidimensional reward function and the safety cost function, the optimal aircraft recovery sorting sequence is obtained by optimizing the scheduling strategy through reinforcement learning agent.

[0076] Reinforcement learning phase: Optimize scheduling strategies through reinforcement learning agents. ,in Given the set of all possible aircraft sorting sequences, and under the constraints of a multidimensional reward function and a safety cost function, achieve a dynamic balance between maximizing the multidimensional reward function and minimizing the safety cost function, and output the optimal sequence. ;in Indicates the first Phased landing aircraft index. Dynamic recovery environment model. This is the state space, containing all aircraft state information.

[0077] Evaluation of the effectiveness of the optimal sequence: Based on the environment model Simulate execution of optimal sequence Calculate the actual cumulative reward Compared to actual security costs ,like ( If the preset validity threshold is met, then the output is executed; otherwise, the strategy parameters are adjusted and re-optimized. Specifically: the multi-dimensional reward function value corresponding to the optimal sequence is determined to be the actual cumulative reward; the safety cost function value corresponding to the optimal sequence is determined to be the actual safety cost; and the ratio of the actual cumulative reward to the actual safety cost is determined to be the validity factor. ; Determine whether the effectiveness factor is lower than the preset effectiveness threshold and obtain the judgment result; If the judgment result is yes, adjust the adjustable reward weight coefficient in the multidimensional reward function and return to the reinforcement learning stage; If the judgment result is no, determine the optimal sequence as the optimal aircraft recovery sorting sequence.

[0078] The formula for dynamically adjusting the weighting coefficients is: .

[0079] in, For learning rate, express The differential, , , They correspond to as , , Updated coefficients.

[0080] Step 105: Convert the optimal aircraft recovery sorting sequence into executable instructions and transmit them to the airport dispatching system.

[0081] Step 106: Execute the executable instructions and record the fuel consumption. Risk indicators and the resumption of flights ;

[0082] Step 107: Update the fuel status using the fuel status dynamic update formula, and return to step 101.

[0083] Dynamically update fuel status during sequence execution; original Expand to , indicating that the action space is the set of all ordered sequences of aircraft. The dynamic update formula for fuel status is: .

[0084] In the formula, Indicates airplane exist Remaining oil volume at the stage; Seconds indicate a stage. and The time increment between; For indicator functions, when exist Indication function when the phase has not landed The value is 1.

[0085] This application constructs a dynamic environment model that includes a go-around scenario. Define the set of execution stages for the sorted sequence. Design a multidimensional cumulative reward function for the overall sorted sequence. r Among them, fuel consumption penalty item Based on engine model fuel consumption rate Dynamic calculation of dwell time and priority reward for faults. Based on failure probability To achieve normalized weighting, the task priority reward P is quantified using a logarithmic function to rank urgency. ; Penalty for resuming flights Based on the number of delay stages Establish an exponential decay model; simultaneously construct a safety cost function. Quantify the total fuel safety risk of all non-landing aircraft in the sorted sequence; by measuring fuel shortages. The conversion coefficient transforms security risks into continuously optimizable variables; the reinforcement learning agent uses policy π: S → A Under the dynamic balance constraint of maximizing rewards and minimizing safety costs, the decision-making is continuously optimized, forming a closed-loop learning mechanism of "environmental state perception - action decision - reward feedback - safety assessment - policy update", ultimately achieving global optimization of recycling efficiency and safety.

[0086] In one exemplary embodiment, a computer device is provided, which may be a server or a terminal. The computer device includes a processor, memory, input / output interfaces (I / O), and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is connected to the system bus via the I / O interfaces. The processor of the computer device provides computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The I / O interfaces of the computer device are used for exchanging information between the processor and external devices. The communication interface of the computer device is used for communicating with external terminals via a network connection. When the computer program is executed by the processor, it implements a reinforcement learning-based aircraft dynamic recovery sorting method.

[0087] In one exemplary embodiment, a computer-readable storage medium is provided storing a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0088] In one exemplary embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0089] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0090] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM).

[0091] The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0092] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0093] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method for dynamic aircraft recovery and sorting based on reinforcement learning, characterized in that, include: Construct a dynamic recovery environment model that includes go-around scenarios; Based on the dynamic recovery environment model, a multidimensional reward function is constructed; the multidimensional reward function is used to describe the fuel consumption penalty, fault priority reward, mission priority reward and go-around penalty in the go-around scenario; Based on the dynamic recovery environment model and the multidimensional reward function, a safety cost function is constructed; the safety cost function is used to quantify the total fuel safety risk of all non-landing aircraft in the aircraft recovery sorting sequence. The multidimensional reward function is: ； in, ; ； In the formula, r For multidimensional reward functions; The maximum number of decision-making stages; For the stage The reward function; This is an adjustable reward weighting coefficient; This is a fuel consumption penalty item; It is a rectified linear activation function; For airplane In the stage The landing status indicator variable, This indicates a successful landing; Represents a sequence Index of aircraft awaiting landing during the mid-stage τ; Prioritize rewarding fault-related issues; This is a reward item for task priority; This is a penalty for resuming flights; Based on the multidimensional reward function and the safety cost function, the optimal aircraft recovery sorting sequence is obtained by optimizing the scheduling strategy through reinforcement learning agent. The optimal aircraft recovery sorting sequence is converted into executable instructions and transmitted to the airport dispatch system.

2. The aircraft dynamic recovery and sorting method based on reinforcement learning according to claim 1, characterized in that, The fuel consumption penalty item is: ； In the formula, For the collection of aircraft awaiting recovery Represents the cardinality of a set; Representation phase Time; Representation phase Time; Indicates that aircraft i is the type of aircraft waiting to land. Fuel consumption rate at that time.

3. The aircraft dynamic recovery and sorting method based on reinforcement learning according to claim 2, characterized in that, The fault priority reward item is as follows: ； In the formula, Indicates an aircraft waiting to land Failure probability assessment value; Indicates an aircraft waiting to land The failure probability assessment value.

4. The aircraft dynamic recovery and sorting method based on reinforcement learning according to claim 3, characterized in that, The task priority reward item is: ； In the formula, Indicates an aircraft waiting to land The mission urgency ranking is as follows: the mission urgency ranking of the aircraft waiting to land is 1.

5. The aircraft dynamic recovery and sorting method based on reinforcement learning according to claim 4, characterized in that, The penalty for re-flight is as follows: ； in, ; In the formula, The delay caused by the resumption of flights Time decay factor; This is the penalty intensity coefficient; The aircraft selected for the current decision The scheduled recovery phase.

6. The aircraft dynamic recovery and sorting method based on reinforcement learning according to claim 5, characterized in that, The security cost function is: ； in, ; In the formula, For safety cost function; For the stage The single-stage security cost; For the stage Assemble the aircraft that have not yet landed; Indicates the safe fuel level threshold for the aircraft model; Indicates airplane exist Remaining fuel level at the stage; in the aircraft exist Stage Remaining Oil Less than the safe oil level threshold for the model Time, stage Single-stage security cost It is directly proportional to the amount of fuel shortage; This is the safety cost conversion factor.

7. The aircraft dynamic recovery and sorting method based on reinforcement learning according to claim 1, characterized in that, Based on the multidimensional reward function and the safety cost function, the optimal aircraft recovery sorting sequence is obtained by optimizing the scheduling strategy through reinforcement learning agents, specifically including: By optimizing the scheduling strategy through reinforcement learning agents, a dynamic balance is achieved between maximizing the multidimensional reward function and minimizing the safety cost function, resulting in the output of the optimal sequence. The multidimensional reward function value corresponding to the optimal sequence is determined to be the actual cumulative reward. Determine the safety cost function value corresponding to the optimal sequence as the actual safety cost; The ratio of actual cumulative rewards to actual security costs is determined as the effectiveness factor; Determine whether the validity factor is lower than a preset validity threshold, and obtain the determination result; If the judgment result is yes, then adjust the adjustable reward weight coefficient in the multidimensional reward function and return to the step "Optimize the scheduling strategy through reinforcement learning agent to achieve a dynamic balance between maximizing the multidimensional reward function and minimizing the safety cost function, and output the optimal sequence"; If the judgment result is negative, then the optimal sequence is determined to be the optimal aircraft recovery sorting sequence.

8. The aircraft dynamic recovery and sorting method based on reinforcement learning according to claim 6, characterized in that, After converting the optimal aircraft recovery sorting sequence into executable instructions and transmitting them to the airport dispatching system, the process also includes: Execute the executable instructions and record fuel consumption, risk indicators, and go-around events; The fuel status is updated using the dynamic fuel status update formula, and the process returns to the step "Construct a dynamic recovery environment model that includes the go-around scenario". The formula for dynamically updating the fuel status is: ； In the formula, Indicates airplane exist Remaining oil volume at the stage; Representation phase and stage The time increment between; For indicator functions, when exist Indication function when the phase has not landed The value is 1.

9. A computer device, comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor executes the computer program to implement the reinforcement learning-based aircraft dynamic recovery sorting method according to any one of claims 1-8.

Images