Joint scheduling method and network model construction method for multi-agent reinforcement learning

Abstract

The application relates to the technical field of airport ground scheduling optimization, and particularly discloses a joint scheduling method and a network model construction method of multi-agent reinforcement learning. The method comprises the following steps: constructing a joint scheduling network model comprising a policy network, a value network, a stand conflict evaluation network and a taxiway conflict evaluation network; constructing a training sample set based on historical flight data and precomputing a benchmark taxi time; running the model in a simulation environment to collect experience data; and updating network parameters based on the sampling data by using a hybrid advantage function which simultaneously optimizes an instant reward, a stand conflict cost and a taxiway conflict cost. Through multi-agent collaborative learning and a double conflict cost constraint mechanism, the joint dynamic optimization of stand allocation and taxiway planning is realized, and the overall efficiency and safety of airport surface operation are effectively improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of airport ground dispatching optimization, in particular to a multi-agent reinforcement learning joint scheduling method and a network model construction method. BACKGROUND

[0002] In the prior art, the optimization methods for parking space allocation and taxiway planning can be mainly divided into two categories: one is independent optimization method, that is, local optimization is performed on parking space allocation and taxiway scheduling respectively, such as the parking space dynamic allocation system based on rule engine proposed in CN120449979B, which can improve the allocation efficiency, but cannot fully consider the taxiway resource constraints and conflict coordination; the other is joint scheduling method, such as the two-stage hybrid optimization method based on improved quantum differential evolution algorithm (QDE) and ant colony algorithm (ACO) proposed in CN118863436A, which realizes the preliminary coordinated scheduling of runway, parking space and taxiway by constructing a joint planning objective function model. This kind of method improves the resource integration ability to some extent, but still has the following limitations:

[0003] Static optimization dominates, and dynamic response capability is insufficient: most of the existing methods are based on historical data and preset rules for offline optimization, which is difficult to deal with real-time disturbances such as flight dynamic delay and temporary occupation of parking space, and lacks online rescheduling capability.

[0004] Low module coupling degree, and deep cooperation mechanism: although CN118863436A tries to integrate multiple scheduling links, the two-stage serial optimization structure still has feedback lag problem, and cannot realize real-time interaction and closed-loop optimization in a true sense.

[0005] Conflict processing relies on heuristic rules, and the degree of intelligence is limited: the resolution of taxiway conflicts is mostly based on predefined priority or first-come-first-served strategy, which lacks deep integration of multi-agent collaborative decision-making and safety constraints, and it is difficult to achieve dynamic balance between reward maximization and conflict minimization.

[0006] Therefore, the present application introduces the multi-agent reinforcement learning (MARL) method from the perspective of deep integration of parking space allocation and taxiway planning, constructs a joint scheduling scheme with dynamic response capability, safety constraint guarantee and real-time cooperation mechanism, in order to overcome the shortcomings of the existing methods in dynamic adaptability, cooperation efficiency and conflict resolution capability, and further improve the intelligent level and overall operation efficiency of airport surface scheduling. SUMMARY

[0007] To solve the above technical problems, the technical scheme adopted by the present application is as follows:

[0008] According to a first aspect of the present application, a method for constructing a joint scheduling network model of multi-agent reinforcement learning is provided, comprising the following steps:

[0009] S100, constructing a joint scheduling network model and initializing model parameters; the joint scheduling network model comprises a policy network, a value network, a stand conflict evaluation network and a taxiway conflict evaluation network.

[0010] S200, constructing a training sample set based on historical flight data, and precomputing a benchmark taxi time based on airport layout information.

[0011] S300, based on the training sample set and the precomputed benchmark taxi time, running the joint scheduling network model in a simulation environment, collecting experience data and storing it in an experience replay buffer.

[0012] S400, after a training round ends, sampling batch data from the experience replay buffer, calculating a hybrid advantage function, and updating the parameters of the joint scheduling network model based on the hybrid advantage function, the hybrid advantage function being constructed to optimize immediate rewards, stand conflict costs and taxiway conflict costs simultaneously.

[0013] According to a second aspect of the present application, a joint scheduling method of multi-agent reinforcement learning is provided, comprising the following steps:

[0014] Obtaining airport infrastructure information and flight plan data of a target airport, and loading the joint scheduling network model of the first aspect;

[0015] Based on the planned arrival time in the flight plan data, dividing the arrival flight sequence into consecutive decision time windows.

[0016] Modeling each flight as an agent and generating a state observation vector for each agent for a plurality of to-be-allocated flights in the current decision time window.

[0017] Inputting each state observation vector into the policy network of the joint scheduling network model to output the stand allocation action of each agent and form a joint decision vector.

[0018] Detecting conflicts in the joint decision vector and reallocating the agents that have detected conflicts to generate a conflict-free joint scheduling result for the current decision window.

[0019] The method for constructing a multi-agent reinforcement learning stand taxi joint scheduling network provided by the embodiment of the application, by constructing a double-constraint optimization architecture including a stand conflict evaluation network and a taxi path conflict evaluation network, and using a hybrid advantage function that fuses instant rewards and double conflict metrics to guide the training of a multi-agent reinforcement learning model, the model can simultaneously perceive and evaluate the taxi path conflict risk that may be caused by different stand selections during the decision-making process, thereby prepositioning the avoidance of taxi conflicts to the stand allocation decision-making stage, realizing the deep cooperation and joint optimization of the originally isolated stand allocation and taxi path planning scheduling links, and this mechanism finally drives the model to output a scheduling scheme with high bridge approach rate and low global taxi conflict, thereby improving the airport surface resource utilization efficiency and operation safety level as a whole.

[0020] It should be understood that the content described in this part is not intended to identify key or important features of the embodiments of the application, nor is it intended to limit the scope of the application. Other features of the application will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the application, the following will briefly introduce the drawings needed to be used in the embodiment description. Obviously, the drawings in the following description are only some embodiments of the application, and other drawings can be obtained by those skilled in the art without creative labor on the basis of these drawings.

[0022] Figure 1 The flowchart of the method for constructing a multi-agent reinforcement learning stand taxi joint scheduling network model provided by the embodiment of the application;

[0023] Figures 2 to 6 The experimental effect diagram. DETAILED DESCRIPTION

[0024] The technical solutions in the embodiments of the application will be described clearly and completely in the following with reference to the drawings in the embodiments of the application. Obviously, the described embodiments are only some embodiments of the application, not all the embodiments. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the application.

[0025] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used in the description of the application herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items.

[0026] It is noted that some example embodiments are described as a process or method depicted as a flowchart. Although the flowchart describes the steps of the process as occurring in a sequence, many of the steps can be performed concurrently, in parallel, or simultaneously. In addition, the order of the steps can be re-arranged. A process can terminate when its operations are completed, but could also terminate in the middle depending on desired outcomes to the process. Processes can correspond with methods, functions, procedures, subroutines, subprograms, etc.

[0027] The present application mainly solves the problem of considering the airport taxiway conflict in the gate assignment by improving the multi-agent reinforcement learning method. It can not only solve the dynamic scheduling of gate assignment, that is, when a sudden situation occurs, it can quickly calculate and give real-time allocation results, but also jointly optimize the design of gate assignment and taxiway planning, so that the gate assignment and taxiway planning reach the joint optimal value.

[0028] In a multi-agent system, the strategy of each agent can affect the behavior of other agents, leading to a non-stationary training process and increasing the difficulty of learning. Especially in the case of strict safety constraints, how to ensure the collaborative work of multiple agents while maintaining the stability of the system is a difficulty. In multi-agent tasks, the reward maximization goal of the agent often conflicts with safety constraints. For example, in order to obtain a higher reward, the agent may try to take high-risk behavior. In the airport environment, there are also the above problems. In order to maximize the reward of allocating the gate to the flight, it may cause collision and other problems in the process of flight planning. Therefore, how to effectively guarantee safety while maximizing the reward and minimize the conflict process is a key problem in multi-agent systems.

[0029] Reinforcement learning is a computational method for agents to achieve goals by interacting with the environment, which is often used to solve Markov Decision Process (MDP) problems.

[0030] In a multi-agent environment, multiple agents learn strategies simultaneously, and the behavior of each agent affects the decision-making of other agents, so a robust and stable strategy optimization method is needed. MAPPO (Multi-Agent Proximal Policy Optimization) realizes the collaborative training of multiple agents through a centralized value network (Critic) and a decentralized policy network (Actor), and uses the advantages of PPO (Proximal Policy Optimization) to improve the learning efficiency and stability in a multi-agent environment. MAPPO-Lagrangian (Multi-Agent Proximal Policy Optimization-Lagrangian) introduces a Lagrange multiplier, uses a safety constraint mechanism, and gradually converges to a strategy that satisfies the constraint by dynamically adjusting the Lagrange multiplier, thereby achieving a flexible balance between reward maximization and safety constraints.

[0031] The method proposed in the application is improved based on the multi-agent reinforcement learning MAPPO-Lagrangian method and adapted to the real airport environment to realize the joint scheduling of stand allocation and taxiway planning. The main process includes: modeling each arriving flight as an agent, assigning a stand to each agent to determine its taxi path and plan the taxiway, then adjusting the stand allocation and taxiway scheduling results of the agent, and feeding back to the next round of learning process through the idea of multi-agent reinforcement learning, so as to continuously optimize the scheduling results and realize the joint scheduling of stand allocation and taxiway planning for arriving flights. Each agent not only needs to maximize its cumulative reward, i.e. the optimization result of stand allocation, but also must ensure that it does not violate its safety constraints, i.e. the optimization result of taxi planning. The improved method meets the optimal number of all arriving flights allocated to the near stand, and also meets the minimum conflict between arriving and departing flights, i.e. the optimal total taxi time of arriving flights.

[0032] (Example 1)

[0033] This embodiment provides a joint scheduling network model construction method of multi-agent reinforcement learning, as shown in Figure 1 The method comprises the following steps:

[0034] S100, a joint scheduling network model containing a policy network, a value network, a stand conflict evaluation network and a taxi path conflict evaluation network is constructed, and the model parameters are initialized.

[0035] In the embodiment of the present application, the joint scheduling network model adopts a decentralized execution and centralized training architecture, comprising a policy network, a value network, a stand conflict evaluation network and a taxiway conflict evaluation network, wherein the policy network is executed in a decentralized manner, and the value network and the conflict evaluation network are trained in a centralized manner.

[0036] (1) Policy network

[0037] The policy network is configured to output a probability distribution of stand allocation actions according to the state observation vector of the agent. The state observation vector of each agent at least fuses the state features of the agent itself and the global stand state features, i.e. obtained by splicing the state features of the agent itself and the global stand state features. Wherein, the state features of the agent itself include but are not limited to taxiway speed, planned arrival time, planned departure time, conflict number, passenger number, selected stand number, etc. The global stand state features are an N x K matrix, wherein N is the total number of stands, and K is the feature dimension of each stand, including current occupancy state, geographic coordinates, capacity specification, stand type (such as near stand, far stand) and other information.

[0038] In the embodiment of the present application, the policy network is composed of the following layers in turn:

[0039] First multi-layer perception: a fully connected layer containing 128 neurons, using ReLU activation function, receiving state observation vector and outputting 128-dimensional features.

[0040] Graph attention network: a single-head graph attention layer with input dimension of 128 and output dimension of 64. It regards the stand as a graph node, calculates the attention weight of the current flight agent and each stand node, and outputs the weighted stand features.

[0041] Second multi-layer perception: a fully connected layer containing 64 neurons, using ReLU activation function, used to fuse the output of the graph attention network.

[0042] Action logic layer: a fully connected layer containing N+1 neurons (plus 1 represents the "not assigned" action). This layer receives the output of the second multi-layer perception, which receives the output of the second multi-layer perception as input. This layer maps high-level features to the action space, and outputs a raw, non-normalized score, called action logic value, for each possible stand (and "not assigned" option).

[0043] An action mask layer: based on the preset airport operation constraints, a feasible action space is constructed for each agent, which is constructed through an action mask mechanism. The action mask mechanism filters out non-feasible parking options according to at least one of the following constraints: aircraft type and parking space matching rules, parking space occupancy time conflict rules, and adjacent parking space safety interval rules.

[0044] Through the action mask mechanism, it can be ensured that each agent can only choose an idle parking space when selecting its own parking space. The core idea of the action mask mechanism is to directly filter out illegal actions at the decision layer, ensuring that the action space always satisfies the constraint conditions. In order to satisfy the hard constraint conditions in the parking space allocation process, the algorithm designs a constraint guarantee mechanism for the invalid action mask module. When the agent selects a parking space, the algorithm detects the state of all parking spaces, automatically deletes the parking space options that violate the airport operation constraints, generates a binary mask vector, forces the invalid action to be negative infinity, and makes the normalized probability zero, ensuring that each agent selects in the parking space that satisfies the constraints, satisfying the hard constraint conditions of the parking space allocation.

[0045] The specific execution process of the action mask layer is as follows:

[0046] Aircraft type matching check: according to the aircraft type of the agent and the capacity specification of each parking space, the matching degree is calculated. If the aircraft type is compatible with the parking space capacity, the mask bit is set to 1 (feasible), otherwise it is set to 0 (infeasible), and the aircraft type mask is generated.

[0047] Time conflict check: query the planned occupancy time period of the agent and the already occupied time period of each parking space. If there is no overlap, the mask bit is set to 1 (feasible), otherwise it is set to 0 (infeasible), and the time mask is generated.

[0048] Safety interval check: according to the airport safety operation regulations, check whether there is a conflict between the wing span of the agent and the wing span of the aircraft type already assigned to the adjacent parking space if the agent is assigned to the target parking space. If there is no conflict, the mask bit is set to 1 (feasible), otherwise it is set to 0 (infeasible), and the safety interval mask is generated.

[0049] Mask synthesis: perform a logical AND operation on the above aircraft type mask, time mask, and safety interval mask to generate the final binary action mask. The position marked as 1 in this mask represents a feasible parking space that satisfies all hard constraints.

[0050] Subsequently, the action mask layer sets the action logic value corresponding to the position marked as 0 (invalid) in the action mask to negative infinity, while the action logic value of the valid position remains unchanged.

[0051] Action selection layer: receives the action logic values after applying the mask, and converts them into a probability distribution over all feasible gates by using a Softmax function. During the training phase, actions are selected by a sampling mechanism to encourage exploration; during the application phase, the gate with the highest probability is directly selected as the final allocation decision output.

[0052] (2) Value network

[0053] The input of the value network is the joint state. The joint state is obtained by aggregating the state observation vectors of all agents in the current time window and concatenating them with the global gate state features.

[0054] In the embodiment of the application, the value network comprises:

[0055] Feature extraction layer: a fully connected layer containing 256 neurons, using ReLU activation function, for processing the input joint state.

[0056] Feature compression layer: a fully connected layer containing 64 neurons, using ReLU activation function.

[0057] Output layer: a fully connected layer containing 1 neuron, without using activation function, outputting the state value prediction V(s).

[0058] (3) Conflict evaluation network

[0059] The input of the conflict evaluation network is the joint state. The gate conflict evaluation network and the taxiway conflict evaluation network have the same structure, and have similar structures with the value network. The difference is that the output layer of the gate conflict evaluation network and the taxiway conflict evaluation network is used to output the gate conflict state cost prediction value Cg(s) and the taxiway conflict state cost prediction value Ct(s) respectively.

[0060] In the embodiment of the application, the weight matrix of all fully connected layers and graph attention layers is initialized using Xavier uniform distribution, and the bias term is initialized to 0.

[0061] S200, based on historical flight data, a training sample set is constructed, and a reference taxi time is pre-calculated based on airport layout information.

[0062] In the embodiment of the present application, the training sample set is derived from the historical flight data of sample airports after verification, cleaning and standardization processing. The structured fields of the historical flight data at least include: flight number, planned arrival time, planned departure time, runway used, aircraft type, passenger quantity and flight status (such as normal, delay, cancellation). The time length and scale of the data can be configured according to the model training requirements, for example, the data of a typical business cycle (such as a week) or a complete season can be selected to ensure coverage of different traffic patterns and running scenarios.

[0063] The specific construction process of the training sample set is as follows:

[0064] Firstly, according to the planned arrival time of the flight, the operation time of a day (such as 06:00 to 02:00 of the next day) is divided into multiple continuous and fixed time length (such as 10 minutes) decision time windows.

[0065] Subsequently, according to the order from early to late of the flight planned arrival time, each arriving flight is allocated to its corresponding decision time window.

[0066] Each decision time window and the flight set contained therein constitute a basic sample unit for model training. All sample units in the entire historical data set collectively constitute the training sample set.

[0067] In the embodiment of the present application, the pre-computed reference taxi time based on airport layout information aims to build a simulated airport environment for the reinforcement learning agent and provide accurate quantitative benchmarks for subsequent conflict assessment. The specific steps are as follows: Step 1, airport map loading and topology modeling:

[0068] Load the digital layout information of the airport, which at least includes:

[0069] Parking stand attributes: unique number, geographic coordinates, capacity specification (such as the code of the aircraft type that can be accommodated), parking stand type (such as near stand, far stand) of all parking stands.

[0070] Runway information: unique identification, start and end point coordinates, direction of the runway.

[0071] Taxiway network: topology structure of the taxiway, including the connection relationship of all nodes (intersection points, waiting points) and edges (taxiway segments).

[0072] Based on the above information, a weighted directed graph G=(V, E) is constructed for path planning, where the vertex set V represents the key points (nodes) in the airport, and the edge set E represents the passable taxiway segments. Each edge e in the graph is assigned a weight we, which represents the time (seconds) required for an aircraft to pass through the segment at the standard taxi speed. At the same time, a global time window data structure is initialized to dynamically record the occupancy status of each edge at different times during the simulation process.

[0073] Step 2, benchmark taxi time pre-computation and caching:

[0074] Based on the above graph model, an efficient path planning algorithm (e.g., QPPTW algorithm designed specifically for airport environment) is used to pre-compute the shortest taxi time under ideal conditions of no conflict and no interference from other traffic flows for each combination of runway entrance and parking stand, which is used as the benchmark taxi time.

[0075] The core of the QPPTW algorithm is to ensure that the planned use time of each segment on the path does not conflict with the existing occupancy time window while finding the shortest path.

[0076] All the computed (runway, parking stand) -> benchmark taxi time mapping relationships are stored in an efficient key-value pair cache (e.g., Redis or in-memory hash table). This cache can provide instant queries with constant time complexity for any (runway, parking stand) combination during subsequent training processes, which is used to quickly calculate the deviation of the actual taxi path from the ideal situation.

[0077] In this invention, the simulation process advances in time steps. A time step corresponds to a fixed decision time window (e.g., 10 minutes). Within each time step, all incoming flights in the window are processed; to resolve resource allocation conflicts within this time window, the algorithm internally iterates multiple logical time steps until a conflict-free scheduling solution is generated. Unless otherwise specified, 'time step' in the following refers to the simulation time step corresponding to the decision time window.

[0078] S300, based on the training sample set and the pre-computed benchmark taxi time, runs the joint scheduling network model in the simulation environment, collects experience data and stores it in the experience replay buffer.

[0079] S300 is the experience collection phase, which is executed in a training round cycle, and each round simulates a whole day of airport operation.

[0080] In embodiments of the present application, one training round is the basic unit of a complete, start-to-end, sequential decision-making and learning of the jointly scheduled network model in the simulation environment. Specifically, in the present application, one training round simulates the full-day airport flight scheduling task of a complete operating day (e.g., from 06:00 of the day to 02:00 of the next day) in the real world.

[0081] At the beginning of each round, first perform global environment reset:

[0082] Clear the occupancy records of all parking positions, and reset the occupancy schedule of each parking position to be available all day;

[0083] Reset the time window data structure of the taxiway network, and all taxiway segments return to the initial available state;

[0084] Clear the reward, cost and historical action records of all agents;

[0085] Reset the current simulation clock to the start time of the day.

[0086] Subsequently, a plurality of consecutive decision-making time windows in a day are processed in time sequence. For each decision-making time window, the following steps are performed:

[0087] S301, instantiate each arriving flight in the current decision-making time window as an agent, and update the environment state.

[0088] In order to maintain the dynamics and authenticity of the simulation environment, according to the advancement of the simulation clock, the following update operations are performed:

[0089] Release the resources that have already departed: check and update the parking position occupancy state, and release the parking positions occupied by the flights whose planned departure time has passed to the idle state.

[0090] Update the traffic flow state: according to the taxiing rules and time advancement, update the space-time occupancy state of each road segment in the taxiway network, release the occupancy that has passed time, and provide accurate basis for new taxi path planning.

[0091] S302, generate a state observation vector for each agent.

[0092] In embodiments of the present application, the state observation vector is a structured data representation that integrates local and global information, aiming to provide sufficient decision-making context for the agent, which is specifically constructed in the following way:

[0093] S3021, agent self-feature extraction:

[0094] The static and dynamic attributes of the agent itself are extracted from the flight data, normalized, and spliced into a fixed-dimensional vector. The self-features at least include:

[0095] Temporal and spatial features: current taxiing speed, planned arrival time, and planned departure time.

[0096] Operational features: number of passengers, flight type code (converted to a One-hot vector).

[0097] Interaction history features: the cumulative number of historical conflicts caused by the agent due to allocation conflicts within the current training round.

[0098] S3022, global gate status encoding:

[0099] The global gate resource status is encoded into a format that can be understood by the agent. Specifically, the system maintains a global gate status matrix, where each row represents a gate, and the columns represent the features of the gate. The features at least include:

[0100] Basic attributes: geographic coordinates, capacity specifications, and gate type (e.g., near gate or far gate).

[0101] Real-time status: current occupancy status (free / occupied), and if occupied, the departure time of the occupying flight.

[0102] S3023, observation vector construction:

[0103] The above agent self-feature vector and global gate status vector are spliced to form a final fixed-dimensional state observation vector.

[0104] In one illustrative embodiment, the agent self-feature vector and the global gate status information after flattening or aggregation are directly spliced to form a high-dimensional vector as the final state observation vector.

[0105] In another illustrative embodiment, the agent self-feature is used as a query vector, and the global gate status matrix is used as a key and value. An attention layer is used to interact. The attention layer calculates the association between the agent and each gate and outputs a fixed-dimensional context vector that integrates global information. The context vector is then spliced with the agent self-feature to form the final state observation vector. This approach can more efficiently capture the complex relationship between the agent and the numerous gates.

[0106] The state observation vector generated in the above manner enables each agent to not only perceive its own situation but also understand the competitive pattern of global resources when making distributed decisions, thereby laying the foundation for efficient collaborative scheduling.

[0107] Further, in generating the state observation vector for each agent, a fixed dimension encoding mechanism is adopted to convert the variable number of agent state information into a fixed length feature vector through weighted summation. S303, input each state observation vector into the policy network respectively, output the gate assignment action of each agent, form a joint decision vector.

[0108] S304, the joint decision vector is subjected to conflict detection, when the conflict is detected, the priority and logical time step reallocation mechanism is started, and the conflict-free scheduling result is generated through multiple iterations.

[0109] In the embodiment of the application, the priority and logical time step reallocation mechanism is a closed loop process for solving resource conflicts through multiple iterations within a single decision time window. Among them, the logical time step is a virtual time unit used to iteratively solve conflicts within a physical decision time window, and its advancement does not depend on the elapse of real time.

[0110] The specific execution process of the priority and logical time step reallocation mechanism is as follows:

[0111] (1) Conflict detection:

[0112] At the current logical time step, the system detects the joint decision vector output by the policy network.

[0113] Traverse all the gates, if the same gate is selected by two or more than two agents (flights) at the same time, it is determined that a gate assignment conflict occurs.

[0114] (2) Priority sorting:

[0115] For all conflicting agents competing for the same gate, the system sorts them according to a predefined static priority rule.

[0116] The priority rule is that the flight with earlier planned arrival time has higher priority. That is, the principle of "first come, first served" is followed.

[0117] (3) Resource allocation and state marking:

[0118] The conflicting gate is allocated to the agent with the highest priority in the conflict group.

[0119] The state of the agent is updated to "allocated" state, and the action selected by the agent is confirmed.

[0120] All other agents in the conflict group are marked as "unallocated" state. Their selection at this logical time step is considered invalid and will enter the reallocation process.

[0121] (4) Conflict cost imposition and environment update:

[0122] For all agents marked as "unassigned", their corresponding conflict counter n conflict is increased by 1.

[0123] The global environment state is updated, marking the assigned gate as "occupied" for the corresponding time period.

[0124] (5) Loop iteration and termination condition:

[0125] After the above steps are completed, the system enters the next logical time step.

[0126] In the new logical time step, only those agents in the "unassigned" state re-execute the decision-making process: generate the state observation vector, and select a new gate through the policy network (combined with the action mask).

[0127] Repeat steps (1) to (5) for a new round of conflict detection and resolution.

[0128] The termination condition of this iteration loop is one of the following two cases:

[0129] a) Ideal case: all agents in the current decision-making time window are successfully assigned to a gate, and no conflict occurs.

[0130] b) Actual case: after multiple iterations, there are still agents that cannot be assigned due to insufficient resources. At this time, the system will forcibly end the processing of the current time window, mark these agents as "assignment failed", and proceed to the next decision-making time window.

[0131] S305, execute the conflict-free scheduling result to obtain the immediate reward, immediate gate conflict cost, and immediate taxiway conflict cost of each agent.

[0132] In the embodiment of the application, the immediate reward is calculated according to the assignment result, and the goal is to guide the model to achieve efficient assignment. The immediate reward r satisfies the following condition: R = w1 × Rg + w2 × Rp + w3 × Rs - w4 × Rc. Wherein, Rg is the gate type reward, such as assigning a near gate as a positive reward of 20, and no reward for a far gate, for example, 0. Rp is the passenger reward, which is positively correlated with the number of passengers P. In one illustrative embodiment, Rp = P / 500 × 10. Rs is the stay time reward, which is positively correlated with the planned stay time S of the flight, which is equal to the interval between the planned departure time and the planned arrival time. In one illustrative embodiment, Rs = S / 60 × 0.5. Rc is the conflict penalty, which is negatively correlated with the historical number of conflicts n of the agent in the current round. In one illustrative embodiment, Rc = -n conflict × 2. w1 to w4 are weight coefficients of each reward component.

[0133] In the embodiment of the present application, the instant parking stand conflict cost is used to quantify the resource competition conflict, and the value thereof is the cumulative number n of conflicts of the agent in the current conflict resolution conflict The positive correlation is in a non-linear growth to inhibit repeated conflicts, that is, the non-linear growth with the increase of the number of conflicts of the agent.

[0134] Further, the instant parking stand conflict cost satisfies the following condition: Cost g =cost base +k×(n conflict ) α , wherein Cost g is the instant parking stand conflict cost, cost base is the benchmark parking stand conflict cost, n conflict is the cumulative number of conflicts, k is a penalty coefficient, and a is a penalty increase index greater than 1.

[0135] In the embodiment of the present application, the instant taxiway conflict cost is used to quantify the traffic flow conflict, and the value thereof is equal to the difference between the actual taxiway time of the agent and the benchmark taxiway time, reflecting the additional time loss caused by the path conflict and congestion.

[0136] S306, the joint state before the execution of the action is input into the value network, the parking stand conflict evaluation network and the taxiway conflict evaluation network respectively, and the state value prediction value, the parking stand conflict cost prediction value and the taxiway conflict cost prediction value corresponding to the current decision time window are obtained.

[0137] In the embodiment of the present application, since the value network, the parking stand conflict evaluation network and the taxiway conflict evaluation network all adopt the centralized training mode, the input thereof needs to be a joint state representation with fixed dimensions covering all agent information. The present application generates a joint state by using a fixed dimension encoding mechanism, and aggregates a variable number of agent state information into a fixed length global feature vector. In this way, it is ensured that the input dimension of the neural network remains consistent regardless of the number of flights in the time window, and the network structure does not need to be adjusted. The specific steps are as follows:

[0138] S3061, individual feature extraction

[0139] Suppose there are N flight agents in the current time window, and N is a variable that changes over time.

[0140] Each agent i has its own state observation vector o i , which has been subjected to preliminary feature engineering (such as normalization, embedding coding, etc.), and the dimension thereof is d ind .

[0141] Therefore, the input of the entire window can be regarded as a vector with dimensions [N, dind ] matrix.

[0142] S3062, feature projection and weighted sum

[0143] The goal of this process is to convert a matrix of [N, d ind ] into a fixed d global -dimensional vector. This is done in two steps:

[0144] a. Linear projection:

[0145] First, the d ind -dimensional feature vector o i of each agent i is projected through a shared fully connected layer (linear layer) into a higher dimensional or more suitable common feature space h i . h i = Wp·o i + b p , where: W p is a [d hidden , d nd ] dimensional weight matrix. b p is a d hidden -dimensional bias vector. h i is the projected feature vector of agent i, of dimension d hidden , and · denotes the dot product.

[0146] This step ensures that all agent features are mapped into the same aggregable space.

[0147] b. Attention weighted aggregation:

[0148] Subsequently, an adaptive weight is computed for the weighted sum of all agent features, instead of a simple average. This allows the network to pay more attention to the agents that are more relevant for the current decision.

[0149] Compute weight: Compute a scalar attention weight a i for each agent i. a i = exp(e i ) / ∑ N j=1 exp(e j ), where e i = ua T · tanh(Wa·h i + ba), Wa is a weight matrix, ba is a bias vector, ua is a context vector, all are learnable parameters. a i is the weight after softmax normalization, the sum of all a i is 1, tanh() is the hyperbolic tangent function.

[0150] Weighted sum: use the calculated weights to sum the projection features h of all agents i Sum to generate the final fixed-length global feature vector g. g = ∑ N i=1 a i ·h i . At this point, regardless of N, the dimension of the output g is fixed to d hidden .

[0151] S3063, output the concatenation of the global gate vector g and the global stand vector s

[0152] The generated fixed-length global feature vector g will be concatenated with the global stand vector (dimension d gate ).

[0153] The concatenated vector [g, s] (total dimension d gate +d hidden +d gate ) is the fixed-dimension input to the value network, the stand conflict evaluation network, and the taxiway conflict evaluation network.

[0154] S307, for each agent in the current decision time window, generate an experience tuple respectively, and store it in the value experience replay buffer.

[0155] In the embodiment of the present application, each experience tuple includes the specific data (state observation vector, action, immediate reward, immediate stand conflict cost, and immediate taxiway conflict cost) of the agent it belongs to, and the state value prediction, stand conflict cost prediction, and taxiway conflict cost prediction corresponding to the decision time window it belongs to.

[0156] In the embodiment of the present application, each experience tuple is accompanied by the following key metadata when stored in the experience replay buffer:

[0157] Round ID: a unique identifier for the current training round.

[0158] Time step index t: the time step position of the experience in the round.

[0159] The buffer is logically organized by rounds, and the data of each round is a list arranged in time step order. Under the index of each time step t, the set of experience tuples of all agents in that time step is stored.

[0160] Through this structure, although the physical storage is discrete tuples, the time sequence of the global state and the joint action information of each time step are logically maintained.

[0161] S400, after one training round, sample batch data from the experience replay buffer, calculate a hybrid advantage function, and update the parameters of the joint scheduling network model based on the hybrid advantage function, the hybrid advantage function being constructed to simultaneously optimize the immediate reward, the stand conflict metric, and the taxiway conflict metric.

[0162] Further, S400 specifically includes:

[0163] S401, after one training round, sample a batch of experience trajectory segments from the experience replay buffer.

[0164] The embodiment of the present application adopts a hierarchical sampling strategy to obtain segments containing continuous time steps and experience of all related agents:

[0165] Sample time step sequence: first, randomly select a complete training round from the experience replay buffer. Then, in the time step sequence of the round, randomly select a starting time step t and cut off a continuous time step sequence [t, t+1,..., t+L-1] of length L to ensure continuity in time sequence.

[0166] Assemble joint experience segments: for each time step k in the continuous time step sequence sampled in the previous step, take out all agent experience tuples in the time step k from the buffer.

[0167] Build training batch: combine the data obtained in the above steps (L time steps, each containing experience of N(k) agents) to form a centralized batch for training. The batch data logically contains the time sequence evolution of the joint state, joint action, and joint reward of the agent group.

[0168] S402, based on the sampled experience trajectory segments, calculate the value advantage function, the stand conflict state advantage function, and the taxiway conflict state advantage function through the generalized advantage estimation method.

[0169] In the embodiment of the present application, the generalized advantage estimation method is used to calculate the advantage functions corresponding to the reward and the two types of conflict costs respectively. This method balances the bias and variance of the estimation by combining the multi-step time difference error, thereby obtaining a better and more stable advantage estimation signal. The value advantage function is calculated based on the state value prediction value and the immediate reward, the stand conflict state advantage function is calculated based on the stand conflict cost prediction value and the immediate stand conflict cost, and the taxiway conflict state advantage function is calculated based on the taxiway conflict cost prediction value and the immediate taxiway conflict cost.

[0170] In the embodiment of the present application, in order to calculate the above advantage function, the batch processing capability of the network is utilized. The sampled experience trajectory segment is logically organized as a tensor with dimensions [batch size, segment length, feature dimension]. The tensor is input into the value network, the gate conflict evaluation network and the taxiway conflict evaluation network, and through one forward propagation, all state value prediction values, gate conflict cost prediction values and taxiway conflict cost prediction values corresponding to the entire segment are synchronously output. These prediction values and the real interaction data (immediate reward and cost) stored in the experience segment jointly constitute the basis for calculating the temporal difference error and the subsequent advantage function.

[0171] Further, S402 specifically comprises the following steps:

[0172] S4021, calculating the temporal difference error

[0173] For each time step k and each agent i (1≤i≤N) in an experience trajectory segment, three types of temporal difference errors are calculated respectively:

[0174] (1) The temporal difference error δ of the value network ik V : δ ik V = r ik + γ × V(s k+1 ) - V(s k ). Wherein, r ik is the immediate reward fed back by the environment to the agent i at time step k. V(s k ) is the state value prediction value of the value network for the state s k at time step k. V(s k+1 ) is the state value prediction value of the value network for the state s k+1 at time step k+1. γ represents a discount factor, which is a hyperparameter between 0 and 1 (0<γ≤1), used to measure the importance of future rewards relative to immediate rewards.

[0175] (2) The temporal difference error δ of the gate conflict evaluation network ik g : δ ik g = c ik g + γ × Cg(s k+1 ) - Cg(s k ). Wherein, c ik g is the immediate gate conflict cost fed back by the environment to the agent i at time step k. Cg(s k ) is the state value prediction value of the gate conflict evaluation network for the state s kThe predicted cost of parking space conflict. Cg(s) k+1 ) for parking space conflict assessment network for state s k+1 The predicted cost of parking space conflict.

[0176] (3) Temporal difference error δ of the taxiing path conflict assessment network ik t δ ik t =c ik t +γ×Ct(s k+1 )-Cs(s k ), where c ik t Let Ct(s) be the instantaneous glide path conflict cost fed back to agent i by the environment at time step k. k ) for the taxiing path conflict assessment network for state s k The predicted cost of taxiing path conflict. Ct(s) k+1 ) for the taxiing path conflict assessment network for state s k+1 The predicted cost of skidding path conflict.

[0177] S4022, Calculate the dominance function using GAE.

[0178] For each time step k and each agent i within the segment, the advantage function of each agent is calculated using the temporal difference error sequence calculated for that segment:

[0179] (1) Value advantage function A ik V A ik V =∑ L-1-k h=0 (γλ) h δ i（k+h） V λ represents the tradeoff parameter in generalized advantage estimation, a hyperparameter between 0 and 1 (0 ≤ λ ≤ 1), used to adjust the bias and variance of advantage estimation. ik V Used to evaluate agent i in state s k Next, execute action a k Compared to the average strategy, it has advantages in avoiding future parking space allocation conflicts.

[0180] (2) Advantage function A of parking position conflict state ik g A ik g =∑ L-1-k h=0 (γλ) h δ i（k+h）g A ik g Used to evaluate agent i in state s k Next, execute action a k Advantages in reducing future gliding congestion and conflicts compared to the average strategy.

[0181] (3) Advantage function A of taxiing path conflict state ik t A ik y =∑ L-1-k h=0 (γλ) h δ i（k+h） t A ik t Used to evaluate agent i in state s k Next, execute action a k Advantages in reducing future gliding congestion and conflicts compared to the average strategy.

[0182] S403 constructs a hybrid advantage function based on the calculated advantage functions and dynamically adjusted Lagrange multipliers.

[0183] In this embodiment of the invention, the mixed dominance function is specifically: AM = AV - λ1 × Ag g -λ2×At, where AM is the mixed advantage function, AV is the value advantage function, Ag is the parking position conflict state advantage function, At is the taxiing path conflict state advantage function, and λ1 and λ2 are Lagrange multipliers.

[0184] AV quantifies the effectiveness of the current action in improving operational efficiency (such as successful docking at the gate or obtaining passenger bonuses) relative to the average level. A positive value indicates that the action is more efficient than usual, while a negative value indicates that it is less efficient than usual. Ag is the dominance function value calculated based on the parking space conflict metric, used to quantify the parking space resource competition risk caused by the current action. A higher value indicates a greater likelihood or historically accumulated severity of parking space conflict caused by the action. At is the dominance function value calculated based on the taxiing path conflict metric, used to quantify the future taxiing path conflict risk implied by the current action. A higher value indicates that the action may lead to more severe taxiing congestion or greater taxiing time deviation.

[0185] λ1 and λ2 are dynamic adjustment Lagrange multipliers. During the training process, λ1 and λ2 are dynamically adjusted according to the corresponding constraint violation degree (i.e. the expected value of Ag and At). Generally, gradient ascent method is used for dynamic adjustment. When the stand or taxiway conflict is frequent (the constraint is severely violated), λ1 or λ2 will automatically increase, thereby amplifying the weight of the conflict cost in the mixed advantage function AM, forcing the policy network to preferentially avoid such high-risk behaviors; on the contrary, when the operation is safe, λ1 and λ2 will decrease, relaxing the safety constraint, allowing the policy network to explore more freely to maximize the main reward (efficiency).

[0186] For each time step k and each agent i within an experience trajectory segment, the mixed advantage function is AM ik =A ik V -λ1×A ik g -λ2×A ik t .

[0187] The present application has the following technical effects through the design of the above-mentioned mixed advantage function:

[0188] (1) Convert multi-objective optimization into single-objective optimization: successfully integrate the competing efficiency and safety objectives into a unified scalar signal AM, so that the standard policy gradient optimization method (such as PPO) can be directly applied, solving the complexity problem of multi-objective decision-making.

[0189] (2) Realize dynamic and adaptive trade-off between safety and efficiency: Compared with the method of fixed penalty weight in the prior art, the dynamic Lagrange multiplier mechanism enables the system to intelligently adjust its "risk preference" according to the real-time learning state and self-evaluation. This not only avoids the disadvantages of repeated manual parameter tuning of fixed weight, but also adapts to the running characteristics of different airports and different traffic, showing stronger robustness and universality.

[0190] (3) Guide to generate globally near-optimal collaborative strategy: The mixed advantage function AM provides a common optimization direction for all agents, taking into account global efficiency and safety. In the process of pursuing the maximization of their own AM, agents naturally learn to avoid and optimize the whole, and finally make the trained joint scheduling policy network able to stably output a global scheduling scheme with high performance and high safety.

[0191] S404, based on the mixed advantage function, update the parameters of the policy network, the value network, the stand conflict evaluation network and the taxiway conflict evaluation network, and the Lagrange multiplier, respectively.

[0192] In the embodiment of the present application, the individual mixed advantage function AM ik is calculated for each agent, and the parameters of the policy network, the value network, the conflict evaluation network and the Lagrange multiplier are optimized respectively through a gradient update rule.

[0193] Further, S404 specifically comprises the following steps:

[0194] S4041, updating the parameters θ of the policy network

[0195] The update target of the policy network is to maximize the expected individual mixed advantage function. The clipping objective function of the proximal policy optimization (PPO) is used to ensure the stability of the policy update. For a batch of data sampled from the experience replay buffer, the loss function L π (θ) of the policy network is defined as:

[0196] L π (θ)=-(1 / N)∑ N i=1 ∑ L k=1 min(ρ ik (θ)·AM ik ,clip(ρ ik (θ),1-ε,1+ε)·AM ik )。

[0197] Wherein, ρ ik (θ) is the probability ratio of the new and old policies, ρ ik (θ)=πθ(a ik |o ik ) / πθ old (a ik |o ik ),πθ(a ik |o ik )is the probability of the policy network selecting action a ik under state observation vector o ik , πθ old (a ik |o ik )is the probability of the old policy when collecting experience, ε is a hyperparameter, and a typical value is 0.1 or 0.2, which is used to limit the step length of each update. clip(ρ ik (θ), 1-ε, 1+ε) means limiting ρ ik (θ) to the interval [1-ε, 1+ε]. N is the number of agents in the batch.

[0198] The policy network parameters are updated by gradient ascent: θ←θ+α π ·▽ θ L π (θ), wherein απ The learning rate for the policy network.

[0199] S4042, update the value network and conflict evaluation network parameters φV, φg and φt.

[0200] The update goal of these centralized networks is to make their predicted values more accurately fit the true rewards or costs.

[0201] The loss function L V of the value network (φV) is the mean squared error between the predicted value and the target value: L V (φV) = - (1 / N)∑ N i=1 ∑ L k=1 (V φV (s k ) - R ik ) 2 . Where R ik is the discounted cumulative reward target value of agent i starting from time step k, R ik =∑ L-1-k h=0 (γ h r i（k+h） +γ L-k V φV (s L ) ), V φV (s k ) is the state value prediction in joint state s k computed by the value network with parameter φV.

[0202] The loss function L g of the gate conflict evaluation network (φg) is: L N i=1 ∑ L k=1 (C gφg (s k ) - G ik ) 2 . G ik is the discounted cumulative gate conflict cost target value of agent i starting from time step k, G ik =∑ L-1-k h=0 (γ h c gi（k+h） +γ L-kCg φg (s L ) ), C gφg (s L ) is the gate conflict cost prediction in joint state s k computed by the gate conflict evaluation network with parameter φg.

[0203] Loss function L of the taxiway conflict assessment network t (φt) = (1 / N)∑ N i=1 ∑ L k=1 (C tφt (s k ) - T ik ) 2 . T ik is the discounted cumulative taxiway conflict cost target value of the agent i starting from time step k, T ik =∑ L-1-k h=0 (γ h c ti（k+h） +γ L-kCt φt (s L ) ), C tφt (s k ) is the taxiway conflict cost prediction value at joint state s k under the taxiway conflict assessment network with parameter φt.

[0204] The parameters of these networks are updated by gradient descent method:

[0205] φV←φV-α V ·▽ φV L V (φV); φg←φg-α g ·▽ φg L g (φg); φt←φt-α t ·▽ φt L t (φt); where α V is the learning rate of the value network, α g is the learning rate of the stand conflict assessment network, and α t is the learning rate of the taxiway conflict assessment network.

[0206] S4043, update Lagrange multipliers λ1 and λ2

[0207] The update target of the Lagrange multiplier is to dynamically balance between meeting safety constraints and maximizing efficiency, and the update is as follows:

[0208] λ1←λ1+α λ ·((1 / (N·L))∑ N i=1 ∑ L k=1 (C gφg (s k) - dg); λ2 <- λ2 + α λ • ((1 / (N*L))∑ N i=1 ∑ L k=1 (C tφt (s k ) - dg).

[0209] where dg is the acceptable gate conflict average cost threshold, dt is the acceptable taxiway conflict average cost threshold, and α λ is the learning rate of the Lagrange multiplier. ((1 / (N*L))∑ N i=1 ∑ L k=1 (C gφg (s k ) - dg) and ((1 / (N*L))∑ N i=1 ∑ L k=1 (C tφt (s k ) - dt) are the average predicted cost calculated on the sampled batch.

[0210] Through the above steps, the parameters of all networks are updated once after one training round. This process is repeated until the policy converges, and finally an intelligent model that can achieve efficient and safe joint scheduling is obtained.

[0211] Further, the method further comprises the following steps:

[0212] S500, determining whether the current joint scheduling network model has reached a convergence state, and when it is determined that the model has converged, exiting the training process; otherwise, repeating the steps of S300 to S400 to continue training.

[0213] After completing the network parameter update of S400, it is evaluated whether the current joint scheduling network model has reached a convergence state. The convergence judgment is based on one or more of the following conditions, and when any condition is met, it is determined that the model has converged and the training process is exited.

[0214] Condition 1: Based on performance stability judgment:

[0215] Run the model on an independent validation set periodically (e.g., after completing X training rounds) and calculate the sliding average of the key performance indicators. If the average value changes by less than a preset threshold a in consecutive Y evaluation periods, the model is determined to have converged.

[0216] Key performance indicators: including but not limited to average bridge rate, total taxi time, conflict rate, etc.

[0217] Example: If the average bridge-reliance rate fluctuation range of the moving average value is less than 1% (a = 0.01) within 10 consecutive evaluation cycles (Y = 10), it is considered stable performance.

[0218] Condition 2: Performance target judgment:

[0219] Evaluate the model on the validation set. If its key performance indicators meet or exceed the preset target threshold, the model is considered to have converged.

[0220] Example: If the average bridge-reliance rate of the model on the validation set is stable at 90% or above, and the total taxi time is optimized by more than 10% compared to the baseline, it is considered to have achieved the performance target.

[0221] Condition 3: Maximum resource limit judgment:

[0222] As a safeguard, when the number of training rounds reaches the preset maximum limit, the training process is forcibly terminated, and the best model currently trained is output.

[0223] Through the above multi-dimensional and quantifiable judgment mechanism, the system can automatically and reliably identify the model convergence point, ensuring the performance of the model while effectively avoiding the unlimited consumption of training resources.

[0224] (Example 2)

[0225] The embodiment provides a joint scheduling method for multi-agent reinforcement learning, including the following steps:

[0226] S10, obtain the airport infrastructure information and flight plan data of the target airport, and load the joint scheduling network model described in Example 1.

[0227] This step is the initialization and preparation stage of the method in the actual application environment, and the core is to adapt the pre-trained general intelligent scheduling capability to the specific target operating environment. Among them, the target airport is a specific airport specified by the user, such as a certain civil aviation airport.

[0228] Airport infrastructure information refers to a static data set that describes the physical layout, resource attributes, and operating rules of the airport scene, and is the basis for building a scheduling simulation environment and making real-time decisions. Specifically, it includes but is not limited to:

[0229] Parking space resource information: unique number, geographic coordinates, capacity specification (such as maximum accommodated aircraft wing span), type classification (such as passenger near gate, far gate, cargo gate, maintenance gate) of all parking spaces.

[0230] Runway and taxiway network information: the starting point coordinates of the runway, the direction identifier; and the topology of the entire taxiway network, i.e. the taxiway system represented in a graph structure, in which the nodes represent intersections or key points, and the edges represent passable taxiway segments.

[0231] Running constraint rule base: preset airport operation hard constraints, providing a rule base for the subsequent action masking mechanism, mainly including:

[0232] Matching rules for aircraft and parking spaces: defining which aircraft of which type can be parked in which type of parking space.

[0233] Parking space occupancy time conflict rules: based on the flight plan, determine whether the parking space has been occupied at a specific time period.

[0234] Adjacent parking space safety interval rules: define how long the adjacent parking spaces need to be separated for safety reasons after a specific aircraft is parked.

[0235] Flight plan data can be dynamic data within the current decision cycle obtained by accessing the real-time business system (such as the flight information management system) of the target airport, including: the planned arrival time (ETA), the planned departure time (ETD), the aircraft type code, the number of passengers, the use of the runway, and other information for each arriving flight. Flight plan data needs to be verified and cleaned to ensure uniform time format and legal aircraft type code.

[0236] In the embodiment of the application, the jointly scheduled network model (including the policy network, the value network, the parking space conflict evaluation network, and the taxiway path conflict evaluation network) trained offline is loaded into the computing unit of the online scheduling system. In the application stage, only the policy network is used for forward inference, and the value network and the conflict evaluation network do not participate in real-time decision-making. The network serves as a complete and executable software module, and its characteristics include:

[0237] Plug and play: the network has completed sufficient training and convergence on sample airport data, and does not require additional online learning at the target airport, ensuring the stability and real-time response capability of the system.

[0238] Model solidification: the network parameters are locked after deployment, and only forward propagation inference is performed in the actual scheduling process, without parameter updating.

[0239] Input adaptation: ensures that the input interface of the loaded network matches the data format (such as feature dimension, encoding method) of the target airport.

[0240] This step achieves the universality and portability of the intelligent scheduling model by decoupling the training environment and the application environment of the model and clearly defining the required input information.

[0241] S20, dividing the inbound flight sequence into continuous decision-making time windows based on the scheduled arrival time in the flight schedule data.

[0242] This step is a key preprocessing link for the method to realize online and dynamic scheduling, and its role is to convert the continuous flight flow into discrete and batchable decision-making units.

[0243] The application adopts a fixed-length window (for example, 10 minutes or 15 minutes) to divide the inbound flight sequence along the time axis. The start time of the window is dynamically determined according to the current time of the system or the scheduled arrival time of the first flight to be allocated. Based on the scheduled arrival time of the flight, all inbound flights expected to arrive within the time window are collected as the set of flights to be allocated in the current window.

[0244] The division process fully considers the dynamic disturbance in real operation. If a flight is delayed or advanced in real time, the system will automatically classify it into the correct corresponding decision-making time window according to its updated expected arrival time for unified decision-making. This mechanism ensures that the scheduling can actively adapt to the dynamic changes of the flight schedule, thereby enhancing the robustness and practicality of the system in a real disturbance environment.

[0245] S30, for the multiple flights to be allocated in the current decision-making time window, modeling each flight as an agent and generating a state observation vector for each agent.

[0246] This step is the core initialization link of the method, and its purpose is to convert the actual scheduling problem into a decision-making unit and environment that can be processed by the multi-agent system. It includes:

[0247] S31, agent modeling:

[0248] Each flight to be allocated in the current decision-making time window is instantiated as an independent agent.

[0249] S32, generating a state observation vector:

[0250] A comprehensive state observation vector is generated for each flight agent. This vector is a digital representation of the current decision-making environment, which at least integrates the following two levels of information:

[0251] Self-state characteristics: including at least one of the following characteristics of the flight: scheduled arrival time, scheduled departure time, aircraft type, passenger quantity, etc.

[0252] Global parking space state characteristics: covering the real-time occupancy state, geographic coordinates, capacity specifications and type classification of all parking spaces.

[0253] S30 further comprises: constructing a feasible and compliant action space for each agent based on preset airport operation constraints.

[0254] The space is a discrete set containing all available stand numbers. By introducing an action mask mechanism, at least one of the following constraints is used to filter and mask infeasible stand options in real time:

[0255] Matching rules between aircraft type and stand;

[0256] Stand occupancy time conflict rules;

[0257] Adjacent stand safety interval rules.

[0258] This mechanism ensures that the agent only makes decisions in the feasible solution space that meets all hard constraints by generating a binary mask vector at the decision source, which sets the probability of illegal actions to zero.

[0259] S40, input each state observation vector into the policy network of the loaded joint scheduling network model, output the stand allocation action of each agent, and form a joint decision vector.

[0260] This step is the core of intelligent decision-making of the method, and the preliminary scheduling scheme is generated through parallel forward inference of the trained policy network. It includes:

[0261] S41, parallel inference process:

[0262] The state observation vector generated for each flight agent in S30 is input into the policy network of the loaded joint scheduling network model in parallel and independently.

[0263] For each agent, the network outputs a probability distribution over its actionable space (determined by the action mask), representing its selection preference for each available stand.

[0264] S42, action selection and joint decision vector formation:

[0265] Typically, the present application adopts a greedy strategy to directly select the stand with the highest probability as the allocation action for each agent.

[0266] The final stand allocation decisions of all flight agents are collected to form a joint decision vector.

[0267] S50, conflict detection and analysis of the joint decision vector to generate a conflict-free joint scheduling result.

[0268] This step is a key safeguard link to ensure the feasibility and safety of the scheduling scheme, including:

[0269] S51, conflict detection:

[0270] Scan the joint decision vector output by S40 to detect whether multiple agents are assigned to the same physical stand.

[0271] S52, conflict resolution:

[0272] If there is no conflict, the joint decision vector is the final scheduling result.

[0273] If a conflict is detected, a fast conflict resolver is started:

[0274] Priority sorting: For all conflicting flights competing for the same stand, the system automatically sorts them according to predefined priority rules (e.g. the earlier the planned arrival time, the higher the priority).

[0275] Resource confirmation and reallocation: Confirm the allocation of the conflict stand for the highest priority flight. The remaining conflict flights are marked as unassigned and based on the updated global stand state, let their strategy network immediately re-infer once to generate new allocation actions.

[0276] This process can be done in one or a few quick iterations until the conflict is resolved or there is no available compatible stand.

[0277] S53, result output:

[0278] Output the conflict-free joint scheduling result of the current decision time window. The result is a deterministic and executable scheduling instruction set that explicitly specifies the stand for each successfully assigned flight.

[0279] According to the airport apron plan of an international airport, the airport point-line map model with two runways is created combined with the actual environment. According to the number of flights in the off-season and peak season of the real airport, the improved algorithm model is trained and tested respectively. The number of arrival flights in the peak season is set to 1000.

[0280] The bridge docking rate of the first round of training of the present application is 35.67%, and the bridge docking rate after training for 600 rounds is 88.72%, as shown in Figure 2 The smooth upward trend of the curve shows that the algorithm learning is stable, and there is no performance collapse or severe shock. The final success rate of nearly 90% is a very excellent practical level considering the airport resource constraints and complex operating environment, which meets the actual bridge docking rate value.

[0281] As shown in Figure 3As shown in the glide time result graph, the algorithm achieved significant results in efficiency improvement, starting from the initial total glide time of 5800-5900 minutes, the system gradually reduced this indicator to 5400-5450 minutes through continuous learning, achieving a significant improvement of about 8-9%. The continuous downward trend shown by the curve indicates that the algorithm not only learned the basic gate assignment, but more importantly, mastered how to choose a gate with shorter glide distance and better path while meeting the constraints. The time savings of 400-500 minutes in actual airport operations means a significant reduction in fuel costs and carbon emissions.

[0282] The reward curve graph is shown in Figure 4 , which verifies the effectiveness of the entire training process and the rationality of the reward function design. The total reward experienced a continuous and stable upward process and finally reached a convergence value. The smooth upward feature of the curve indicates that the learning process of the algorithm is stable, and there is no problem of reward shock or learning stagnation. This confirms the trend of continuous improvement, and the significant growth of the reward reflects the comprehensive improvement of the system in multiple dimensions such as assignment success rate, gate selection quality, time efficiency, and conflict avoidance, proving the effectiveness of the MAPPO-Lagrangian framework in complex optimization problems.

[0283] Figure 5 The biaxial graph in clearly shows the dynamic relationship between assignment success rate and glide time optimization and the evolution of learning strategies. In the early stage of training, the algorithm prioritizes learning how to improve the assignment success rate to ensure that more flights can obtain a gate. As the success rate stabilizes, the system begins to focus on efficiency optimization, continuously reducing the total glide time. This reflects the intelligent strategy of the present application in multi-objective optimization, which first ensures feasibility and then pursues optimality, ultimately achieving the dual goals of high success rate and high efficiency.

[0284] The experiment sets the number of inbound flights during the off-season to 500. The bridge approach rate and glide time of the present application vary with the round as shown in Figure 6 . The bridge approach rate of the algorithm in the first round of training is 39.32%, and after 600 rounds of training, the bridge approach rate is 98.42%, showing a typical S-shaped growth pattern of reinforcement learning. The final result of the bridge approach rate reached 98.42%, proving that the algorithm can effectively handle multiple constraints such as aircraft type matching, time conflicts, and capacity limitations. The glide time shows the significant results of the algorithm in efficiency improvement, continuously decreasing from the initial total glide time of 5100 minutes to the final level of 4600-4700 minutes. The red curve shows a clear optimization process, and the overall time savings of 400-500 minutes correspond to an efficiency improvement of 8-10%.

[0285] With the training going on, the success rate steadily rises and the taxiing time continuously decreases, which reflects the intelligent learning strategy of the algorithm: on the basis of ensuring high success rate, further optimize the allocation quality, select the shorter taxiing distance and more efficient parking position. The volatility of taxiing time is larger than that in peak season, the reason may be that the flights are more intensive in a short time, not randomly distributed, the number of flights landing in the early morning is less, affecting the degree of resource competition, leading to more complex taxiing time calculation. This volatility reflects the continuous learning and adaptability of the algorithm to some extent, which is a normal phenomenon in the reinforcement learning process, the key is to ensure that the final convergence performance meets the actual application requirements.

[0286] The embodiment of the application further provides an electronic device, comprising: at least one processor; and a memory connected with the at least one processor in communication; wherein the memory stores instructions executable by the at least one processor, and the instructions are configured to execute the method described in the embodiment of the application.

[0287] The embodiment of the application further provides a computer readable storage medium, which stores computer executable instructions, and the computer executable instructions are used for executing the method described in the embodiment of the application.

[0288] It should be understood that the steps shown above can be reordered, added or deleted. For example, the steps described in the application can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in the application can be achieved, which is not limited herein.

[0289] The above specific embodiments do not constitute a limitation on the scope of protection of the application. Those skilled in the art should understand that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modifications, equivalent replacements and improvements made within the spirit and principles of the application should be included in the scope of protection of the application.

Claims

1. A method for constructing a joint scheduling network model using multi-agent reinforcement learning, characterized in that, The method includes the following steps: S100, Construct a joint scheduling network model and initialize model parameters; the joint scheduling network model includes: a policy network, a value network, a parking position conflict assessment network, and a taxiing path conflict assessment network. S200 constructs a training sample set based on historical flight data and pre-calculates the baseline taxiing time based on airport layout information; S300, based on the training sample set and pre-calculated baseline coasting time, runs the joint scheduling network model in the simulation environment, collects empirical data and stores it in the empirical replay buffer; S400, after a training round, sample batch data from the experience replay buffer, calculate the hybrid dominance function, and update the parameters of the joint scheduling network model accordingly. The hybrid dominance function is constructed to simultaneously optimize immediate rewards, parking position conflict costs, and taxiing path conflict costs. The S400 specifically includes: After a training round ends, a batch of experience trajectory fragments are sampled from the experience replay buffer; Based on sampled empirical trajectory segments, the value advantage function, parking position conflict state advantage function, and taxiing path conflict state advantage function are calculated using the generalized advantage estimation method. The value advantage function is calculated based on the predicted state value and immediate reward; the parking position conflict state advantage function is calculated based on the predicted parking position conflict cost and immediate parking position conflict cost; and the taxiing path conflict state advantage function is calculated based on the predicted taxiing path conflict cost and immediate taxiing path conflict cost. Based on the calculated dominance functions, a hybrid dominance function is constructed by combining dynamically adjusted Lagrange multipliers. Based on the hybrid dominance function, the parameters of the policy network, value network, parking position conflict assessment network, taxiing path conflict assessment network, and the Lagrange multiplier are updated respectively.

2. The method according to claim 1, characterized in that, The training sample set contains training samples constructed based on flight data within multiple consecutive decision time windows; S300 specifically includes: for each decision time window, performing the following steps: Instantiate each inbound flight to be assigned within the current decision-making time window as an intelligent agent; Generate a state observation vector for each agent; Each state observation vector is input into the policy network, and the stop position allocation action of each agent is output to form a joint decision vector. Conflict detection is performed on the joint decision vector. When a conflict is detected, a priority and logical time step reassignment mechanism is initiated, and a conflict-free scheduling result is generated through multiple rounds of iteration. Execute conflict-free scheduling results and obtain the instant reward, instant stop position conflict cost, and instant taxi path conflict cost for each agent; The joint state before the action is executed is input into the value network, the parking position conflict assessment network, and the taxiing path conflict assessment network, respectively, to obtain the state value prediction value, parking position conflict cost prediction value, and taxiing path conflict cost prediction value corresponding to the current decision time window. For each agent within the current decision-making time window, an experience tuple is generated and stored in the experience replay buffer. The experience tuple includes: state observation vector, action, state value prediction, parking position conflict cost prediction, taxiing path conflict cost prediction, immediate reward, immediate parking position conflict cost, and immediate taxiing path conflict cost.

3. The method according to claim 2, characterized in that, Each agent's state observation vector incorporates at least its own state characteristics and global stop position state characteristics.

4. The method according to claim 2, characterized in that, The instantaneous stop position conflict cost increases non-linearly with the number of conflicts that occur in the agent.

5. The method according to claim 4, characterized in that, The instantaneous parking space conflict cost satisfies the following condition: Cost g =cost base +k×(n conflict ) α Among them, Cost g Cost of immediate parking space conflict base As the cost of a baseline parking position conflict, n conflict Let k be the cumulative number of conflicts, k be the penalty coefficient, and α be the penalty increase exponent.

6. The method according to claim 1, characterized in that, The hybrid advantage function is specifically: AM = AV - λ1 × Ag - λ2 × At, where AM is the hybrid advantage function, AV is the value advantage function, Ag is the parking position conflict state advantage function, At is the taxiing path conflict state advantage function, and λ1 and λ2 are Lagrange multipliers.

7. The method according to claim 2, characterized in that, The joint state is generated using a fixed-dimensional encoding mechanism.

8. A joint scheduling method for multi-agent reinforcement learning, characterized in that, Includes the following steps: Obtain airport infrastructure information and flight schedule data of the target airport, and load the joint scheduling network model obtained by the multi-agent reinforcement learning joint scheduling network model construction method according to any one of claims 1 to 7; Based on the planned arrival times in the flight schedule data, the inbound flight sequence is divided into continuous decision time windows; For multiple flights to be assigned within the current decision-making time window, each flight is modeled as an agent, and a state observation vector is generated for each agent; Each state observation vector is input into the policy network of the joint scheduling network model, and the stop position allocation action of each agent is output to form a joint decision vector. Conflict detection is performed on the joint decision vector, and agents with detected conflicts are reassigned to generate a conflict-free joint scheduling result for the current decision window.

Images