A method and system for dynamic scheduling decision of aircraft cooperative recovery based on parallel reinforcement learning

Abstract

The application discloses a kind of based on parallel reinforcement learning's aircraft cooperative recovery dynamic scheduling decision method and system, it is related to the intersection field of air traffic control and reinforcement learning, this method includes creating multiple parallel environment instances, the initialization attribute of each aircraft is configured in each environment instance, generates fleet state vector, executes recovery action to the aircraft selected by agent in the decision phase of each environment instance, calculates multi-objective reward, updates global timestamp by this recovery time, synchronizes this fleet state vector, repeats the above steps until all recovery;Select the action sequence corresponding to the maximum total cumulative as the target dynamic scheduling decision sequence, according to the multi-objective reward mechanism of fuel reward, priority reward, safety reward compound, the problems that fuel consumption randomness is not considered carefully, priority task guarantee is insufficient, safety risk cannot be effectively avoided caused by static scheduling are solved, and the low training efficiency problem of single environment reinforcement learning is solved.

Description

Technical Field

[0001] This application relates to the interdisciplinary field of air traffic control and reinforcement learning, and in particular to a dynamic scheduling decision-making method and system for aircraft cooperative recovery based on parallel reinforcement learning. Background Technology

[0002] Traditional aircraft recovery scheduling relies on fixed interval rules or single agent decision-making, which has three shortcomings: First, it does not consider emergency situations caused by the randomness of fuel consumption. Dynamic changes in fuel status can cause contingency plans to fail, and low-fuel aircraft are prone to triggering a chain of safety risks when they go around. Second, it does not adequately guarantee priority tasks. Third, single-environment training is inefficient. Single-environment reinforcement learning training is inefficient and cannot cover scenarios with combinations of random variables such as wake intervals and recovery success rates.

[0003] Related technologies employ heuristic rules to adjust the recovery sequence, but lack dynamic adaptability; or they use fixed-interval queuing strategies, failing to consider real-time state changes caused by fuel consumption rates. Others rely on single-agent reinforcement learning for scheduling optimization, but training is slow and fails to model policy generalization across parallel environments. Still others suffer from slow convergence during single-environment training, and the go-around strategy simply resets the position without establishing a state transition mechanism. Especially in scenarios involving multiple objective trade-offs such as fluctuating recovery success probabilities, wake turbulence constraints, and fuel safety, these methods struggle to achieve real-time optimal decision-making.

[0004] This invention addresses the aforementioned problems of dynamic response lag and multi-objective balance through a parallel environment interaction framework and a composite reward mechanism. Summary of the Invention

[0005] The purpose of this application is to provide a dynamic scheduling decision-making method and system for aircraft cooperative recovery based on parallel reinforcement learning. It considers the dynamic changes in fuel quantity, task priority, and safety costs. Through a multi-objective reward mechanism that combines fuel rewards, priority rewards, and safety rewards, it effectively ensures aircraft fuel safety, dynamically schedules according to demand, and trains in a parallel environment to obtain the optimal target dynamic scheduling decision sequence, thereby improving training efficiency and effectively covering multi-objective trade-off scenarios. It is suitable for real-time optimization scheduling of aircraft recovery on offshore platforms or multi-aircraft cooperative recovery in high-density airport environments, as well as scenarios requiring high real-time dynamic scheduling such as airport emergency landings.

[0006] To achieve the above objectives, this application provides the following solution:

[0007] Firstly, this application provides a dynamic scheduling decision-making method for aircraft cooperative recovery based on parallel reinforcement learning, including:

[0008] Multiple parallel environment instances are created, and the initialization attributes of each aircraft are configured in each environment instance. A fleet state vector consisting of a normalized fuel state vector and an aircraft recovery state vector is generated. The attributes include at least fuel quantity and normalized priority.

[0009] Perform the following steps during the decision-making phase for each environment instance:

[0010] Perform the recovery action on the aircraft selected by the agent and update the aircraft recovery state vector.

[0011] The multi-objective reward is calculated based on the fuel reward calculated from the fuel quantity, the priority reward calculated from the normalized priority, the safety reward obtained from the safety cost based on the fuel quantity, and the weight coefficient of each reward.

[0012] Update the fuel quantity of all aircraft and recalculate the normalized fuel state vector.

[0013] The global timestamp is updated based on the time of this recycling, and the cluster state vector is synchronized.

[0014] The agent selects the next aircraft to be recovered and proceeds to the next decision-making stage, repeating the above steps until all aircraft are recovered.

[0015] Calculate the total cumulative reward of the multi-objective decision trajectory for each environmental instance, and select the action sequence corresponding to the maximum total cumulative reward as the target dynamic scheduling decision sequence.

[0016] Secondly, this application provides a dynamic scheduling decision-making device for aircraft cooperative recovery based on parallel reinforcement learning, including: an initialization module, an action execution module, and a target sequence determination module.

[0017] The initialization module is used to create multiple parallel environment instances, configure the initialization attributes of each aircraft in each environment instance, and generate a group state vector consisting of a normalized fuel state vector and an aircraft recovery state vector. The attributes include at least fuel quantity and normalized priority.

[0018] The action execution module is used to perform the following steps during the decision-making phase of each environment instance: Perform a recovery action on the aircraft selected by the agent and update the current aircraft recovery state vector; calculate the multi-objective reward based on the fuel reward calculated from the fuel quantity, the priority reward calculated from the normalized priority, the safety reward obtained from the safety cost based on the fuel quantity, and the weight coefficients of each reward; update the fuel quantity of all aircraft and recalculate the current normalized fuel state vector; update the global timestamp based on the current recovery time and synchronize the current fleet state vector; the agent selects the next aircraft to be recovered and enters the next decision-making phase, repeating the above steps until all aircraft are recovered.

[0019] The target sequence determination module is used to calculate the total cumulative reward of the multi-objective decision trajectory for each environmental instance, and select the action sequence corresponding to the maximum total cumulative reward as the target dynamic scheduling decision sequence.

[0020] Thirdly, this application provides a computer device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the aircraft cooperative recovery dynamic scheduling decision method based on parallel reinforcement learning as described above.

[0021] Fourthly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the aircraft cooperative recovery dynamic scheduling decision method based on parallel reinforcement learning as described above.

[0022] Fifthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the aircraft cooperative recovery dynamic scheduling decision method based on parallel reinforcement learning as described above.

[0023] According to the specific embodiments provided in this application, the following technical effects are disclosed:

[0024] This application provides a dynamic scheduling decision-making method and system for aircraft cooperative recovery based on parallel reinforcement learning. By creating multiple parallel environment instances and configuring the initial attributes of each aircraft in each instance, a fleet state vector is generated, solving the problem of low training efficiency in single-environment reinforcement learning. It also more comprehensively covers various random variable combinations and improves the generalization ability of the trained strategy. By executing recovery actions on the aircraft selected by the agent, a multi-objective reward mechanism combining fuel rewards, priority rewards, and safety rewards is applied, simultaneously considering multiple objectives such as fuel consumption, priority task assurance, and safety risks. This solves the problems of insufficient consideration of fuel consumption randomness, inadequate priority task assurance, and ineffective avoidance of safety risks caused by static scheduling, making the decision more comprehensive and optimized. After each decision, the aircraft recovery state vector, fuel quantity, normalized fuel state vector, and global timestamp are updated, and the fleet state vector is synchronized. This mechanism ensures the continuity and real-time nature of the decision-making process and avoids erroneous decisions caused by inconsistent states. Selecting the action sequence corresponding to the maximum total accumulation as the target dynamic scheduling decision sequence can optimize the scheduling strategy from a global perspective, avoiding the problem of local optima, ensuring the generalization of the strategy in different environments, and improving the robustness of the strategy. Attached Figure Description

[0025] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0026] Figure 1 This is an application environment diagram of a dynamic scheduling decision-making method for aircraft cooperative recovery based on parallel reinforcement learning in one embodiment of this application;

[0027] Figure 2 A flowchart illustrating a dynamic scheduling decision-making method for aircraft cooperative recovery based on parallel reinforcement learning, provided as an embodiment of this application;

[0028] Figure 3 A schematic diagram of the functional modules of an aircraft cooperative recovery dynamic scheduling decision-making device based on parallel reinforcement learning, provided in an embodiment of this application;

[0029] Figure 4 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0030] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0031] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0032] The aircraft cooperative recovery dynamic scheduling decision-making method based on parallel reinforcement learning provided in this application embodiment can be applied to, for example... Figure 1The application environment shown is illustrated. Terminal 101 communicates with server 102 via a network. A data storage system can store the data that server 102 needs to process. The data storage system can be set up independently, integrated into server 102, or placed in the cloud or on another server. Terminal 101 can configure the initialization attributes of each aircraft in each environment instance and generate a fleet state vector consisting of a normalized fuel state vector and an aircraft recovery state vector, which is then sent to server 102. Server 102 receives the initialization attributes of each aircraft in each environment instance and generates a fleet state vector consisting of a normalized fuel state vector and an aircraft recovery state vector. During the decision-making phase of each environment instance, server 102 executes the following steps: performing a recovery action on the aircraft selected by the agent and updating the current aircraft recovery state vector; calculating multi-objective rewards based on fuel quantity, priority rewards calculated by normalization, safety rewards obtained based on fuel quantity safety costs, and the weight coefficients of each reward; updating the fuel quantity of all aircraft and recalculating the current normalized fuel state vector; updating the global timestamp based on the current recovery time and synchronizing the current fleet state vector; the agent selects the next aircraft to be recovered and enters the next decision-making phase, repeating the above steps until all aircraft are recovered; calculating the total cumulative multi-objective rewards of the decision trajectories of each environment instance, and selecting the action sequence corresponding to the maximum total cumulative reward as the target dynamic scheduling decision sequence. Server 102 can feed back the obtained target dynamic scheduling decision sequence to terminal 101. Furthermore, in some embodiments, the aircraft cooperative recovery dynamic scheduling decision method based on parallel reinforcement learning can also be implemented separately by server 102 or terminal 101. For example, terminal 101 can directly configure the initial attributes of each aircraft in each environment instance to be processed, and generate a group state vector composed of a normalized fuel state vector and an aircraft recovery state vector, and perform aircraft cooperative recovery dynamic scheduling decision processing based on parallel reinforcement learning. Alternatively, server 102 can obtain the initial attributes of each aircraft configured in each environment instance from the data storage system, generate a group state vector composed of a normalized fuel state vector and an aircraft recovery state vector, and perform aircraft cooperative recovery dynamic scheduling decision processing based on parallel reinforcement learning.

[0033] The terminal 101 can be, but is not limited to, various desktop computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, and smart in-vehicle devices. Portable wearable devices can include smartwatches, smart bracelets, and head-mounted devices. The server 102 can be implemented using a standalone server or a server cluster composed of multiple servers, or it can be a cloud server.

[0034] In one exemplary embodiment, such as Figure 2As shown, a dynamic scheduling decision-making method for aircraft cooperative recovery based on parallel reinforcement learning is provided. This method is executed by computer equipment, specifically by a terminal or server alone, or by both a terminal and a server. In this embodiment, the method is applied to... Figure 1 Taking server 102 as an example, the explanation includes the following steps 201 to 203. Wherein:

[0035] Step 201: Create multiple parallel environment instances, configure the initialization attributes of each aircraft in each environment instance, and generate a fleet state vector consisting of a normalized fuel state vector and an aircraft recovery state vector. The attributes include at least fuel quantity and normalized priority.

[0036] Step 202: Perform the following steps during the decision-making phase for each environment instance:

[0037] Perform the recovery action on the aircraft selected by the agent and update the aircraft recovery state vector.

[0038] A multi-objective reward is calculated based on the fuel reward calculated from fuel quantity, the priority reward calculated from normalized priority, the safety reward obtained from the safety cost based on fuel quantity, and the weight coefficients of each reward.

[0039] Update the fuel levels of all aircraft and recalculate the normalized fuel state vector.

[0040] The global timestamp is updated based on the current recycling time, and the cluster state vector is synchronized.

[0041] The agent selects the next aircraft to be recovered and proceeds to the next decision-making stage, repeating the above steps until all aircraft are recovered.

[0042] Step 203: Calculate the total cumulative reward of the multi-objective decision trajectory for each environmental instance, and select the action sequence corresponding to the maximum total cumulative reward as the target dynamic scheduling decision sequence.

[0043] By implementing steps 201 to 203 above, multiple parallel environment instances are created, and the initialization attributes of each aircraft are configured in each environment instance to generate a fleet state vector. This solves the problem of low training efficiency in single-environment reinforcement learning, more comprehensively covers various random variable combination scenarios, and improves the generalization ability of the trained policy. By executing recovery actions on the aircraft selected by the agent, a multi-objective reward mechanism combining fuel rewards, priority rewards, and safety rewards is used, simultaneously considering multiple objectives such as fuel consumption, priority task guarantee, and safety risks. This solves the problems of insufficient consideration of fuel consumption randomness, inadequate priority task guarantee, and ineffective avoidance of safety risks caused by static scheduling, making the decision more comprehensive and optimized. After each decision, the aircraft recovery state vector, fuel quantity, normalized fuel state vector, and global timestamp are updated, and the fleet state vector is synchronized. This mechanism ensures the continuity and real-time nature of the decision-making process and avoids erroneous decisions caused by inconsistent states. Selecting the action sequence corresponding to the maximum total accumulation as the target dynamic scheduling decision sequence can optimize the scheduling policy from a global perspective, avoid the problem of local optima, ensure the generalization of the policy in different environments, and improve the robustness of the policy.

[0044] To better define the aircraft recovery decision-making framework, a decision-making phase sequence is defined before step 201. Each decision-making stage This corresponds to the moment when the next aircraft to be recovered is selected; in each environment instance, decisions are made sequentially in this order until all aircraft have been recovered.

[0045] Specifically, in step 201, multiple parallel environment instances are created. Initialization attributes for each aircraft are configured in each environment instance, and a fleet state vector consisting of a normalized fuel state vector and an aircraft recovery state vector is generated. The attributes include at least fuel quantity and normalized priority, specifically including:

[0046] create A single, independently running environment instance, satisfying Examples for each environment Configure the initialization attributes of each aircraft and generate a group state vector consisting of a normalized fuel state vector and an aircraft recovery state vector.

[0047] The cluster state vector is ,in, For the normalized fuel state vector, For the decision-making phase of environment instance e Aircraft number i Normalized fuel consumption , Let this be the aircraft recovery state vector. For the decision-making phase of environment instance e Aircraft number i The aircraft recovery state vector, (Indicates initial items to be recycled). i Number the aircraft For the assembly of airplanes.

[0048] The attribute includes fuel quantity. Normalization of task priorities Fuel consumption rate At this stage of decision-making. .

[0049] in, Indicates the interval A random function that is uniformly distributed on the upper bound, and this function is... These are mutually independent uniform distribution functions. Optionally, a minimum fuel quantity is set. Maximum fuel capacity Fuel consumption rate .

[0050] Task priority is a pre-defined relative importance index based on aircraft type and flight mission nature. Its value is obtained through linear normalization, and the calculation formula is as follows:

[0051] .

[0052] in, The original priority score. and These are the minimum and maximum original priority scores in the cluster.

[0053] In this application, task priority The local evaluation feedback calculation priority reward directly affects subsequent steps. The reinforcement learning agent will dynamically select actions in the subsequent decision-making stage based on global state information (i.e., the swarm state vector) and global reward signal (i.e., multi-objective reward), that is, determine the aircraft number of the next aircraft to be recovered, thereby learning to generate the final execution sequence, which is a potential improvement or optimization of the initial task priority sequence.

[0054] Optionally, a recycling success rate threshold and an initial timestamp can also be set for each environment instance. Recycling success rate threshold , Let represent a random function uniformly distributed on the interval [0.8, 0.95], and let the threshold define the minimum probability of success required for each recovery attempt; this function is related to... The functions are mutually independent and uniformly distributed; initial timestamps .

[0055] In another exemplary embodiment of this application, in order to effectively execute the dynamic scheduling decision for aircraft cooperative recovery, the recovery action performed on the aircraft selected by the agent in step 202 is replaced by the following steps 301-302:

[0056] Step 301: The normal action selection strategy of the agent is as follows: In each decision-making stage, a single action is dynamically output based on the swarm state vector, normalized priority, and multi-objective reward. Regarding actions Mapped aircraft number i Execute the recycling action; among which, the action From discrete action space The selected integer corresponds directly to the aircraft number of the aircraft to be recovered. i .

[0057] This application is based on the decision-making stage of environmental example e. The cluster state vector provided in this context The probability distribution of actions is calculated through a reinforcement learning policy network, and the output actions are sampled. That is, select the aircraft number i To enable the execution of the recycling action.

[0058] Step 302: The emergency recovery action selection strategy of the intelligent agent is as follows: when the fuel level of the aircraft to be recovered is lower than a set value, the current action or the next candidate action is forced to directly lock the aircraft number of the aircraft to be recovered that is lower than the set value. i As the highest priority, it is forced to be recycled first, and the corresponding status is updated. The emergency recycling action selection strategy has a higher priority than the normal action selection strategy.

[0059] Optional, when satisfy When this occurs, an emergency recovery action is triggered, and a strategy is selected.

[0060] The recycling action is performed in this application. Corresponding aircraft number i It must be currently not recycled, i.e., the restriction condition must be met. Otherwise, the condition would have already been excluded when the agent selects an action.

[0061] Normalization of task priorities As merely a feature input or local reward factor in the learning process, the reinforcement learning agent will base its decisions at each stage on global state information (i.e., the swarm state vector). The system dynamically outputs an action (i.e., selecting the aircraft number of the next aircraft to be recovered) based on the global reward signal of the training objective and the training objective (i.e., multi-objective reward). iThis will eventually form a sequence of actions that may differ from or even be better than the initial priority.

[0062] The normal action selection strategy of the agent in this application dynamically outputs actions based on the aircraft group state vector, normalized priority, and multi-objective rewards. This dynamic decision-making method does not rely on fixed rules, but optimizes scheduling efficiency through dynamic decision-making and multi-objective reward mechanisms, making the optimal choice based on the real-time state and improving overall scheduling efficiency. The emergency recovery action selection strategy of the agent has a higher priority than the normal action selection strategy, ensuring a rapid response in emergency situations, prioritizing the handling of low-fuel aircraft, and effectively avoiding the safety risks caused by low-fuel aircraft. This mechanism, which combines the normal action selection strategy and the emergency recovery action selection strategy, not only ensures dynamic optimization under normal conditions, but also enables rapid safety decisions in emergency situations, allowing the scheduling system to achieve more efficient, safer, and more reliable aircraft recovery scheduling in complex dynamic environments.

[0063] In another exemplary embodiment of this application, in order to ensure the consistency of the decision-making process and improve training efficiency, the update of the current aircraft recovery state vector in step 202 is replaced by the following steps 401-402:

[0064] Step 401: When the probability of successful recovery of the aircraft is greater than the set recovery success rate threshold, the aircraft recovery state vector is assigned a value of 1, marking it as successfully recovered.

[0065] Specifically, in the decision-making phase of environment instance e Aircraft number i Success rate of recovery Then execute .

[0066] Success rate of recovery The calculation formula is:

[0067] .

[0068] in, The sigmoid function is used to smooth probability mappings. This is the fuel emergency warning threshold. Optionally, kg. The probability of successful recovery is characterized by the aircraft number. i The remaining fuel of the aircraft The lower the value, the closer it is to the alarm threshold. Its success rate of recovery The lower the value, the better.

[0069] In addition, the recovery time for this operation is assigned as the previous recovery time plus the successful recovery time and the wake interval time, i.e. ,in To successfully recover time, This is the wake interval time. Optionally, it is the successful recovery time. Seconds. Specifically, the formula for calculating the wake interval is:

[0070] .

[0071] in, For safe distance, i For the recovery of this aircraft, For the previous recovered aircraft, Let this be the velocity vector of the recovered aircraft. This represents the velocity vector of the previously recovered aircraft; the value is the instantaneous velocity when both aircraft begin the recovery decision phase (i.e., when the selected aircraft is chosen for the action). Optional, .

[0072] Step 402: When the probability of successful aircraft recovery is less than or equal to the set recovery success rate threshold, execute the go-around strategy and update the aircraft recovery state vector to the state value after executing the go-around strategy.

[0073] Specifically, in the decision-making phase of environment instance e Aircraft number i Success rate of recovery If so, the go-around strategy will be implemented. Update the aircraft recovery state vector to the state value after executing the go-around strategy. .

[0074] In addition, the recovery time for this mission is set to the previous recovery time plus the go-around time. Optionally, Second.

[0075] This application sets a recovery success rate threshold to clearly define the criteria for judging whether an aircraft has been successfully recovered, enabling rapid determination of aircraft recovery success, thereby saving time and resources and improving overall scheduling efficiency. By accurately updating the aircraft's recovery status, it avoids erroneous decisions caused by inaccurate status information, improving scheduling reliability. Timely marking of recovered aircraft allows the system to allocate resources more rationally, focusing on unrecovered aircraft and improving resource utilization efficiency. Through a go-around strategy, aircraft can quickly adjust their status when recovery fails, avoiding safety risks caused by prolonged unsafe states. The go-around strategy allows the system to flexibly adjust the aircraft recovery order and strategy based on real-time conditions, improving scheduling flexibility. Dynamically updating status information enables the agent to make more reasonable decisions based on the latest status, further optimizing the scheduling strategy.

[0076] In another exemplary embodiment of this application, in order to flexibly execute the go-around strategy when recovery fails, the step 402 of executing the go-around strategy, which updates the aircraft recovery state vector to the state value after executing the go-around strategy, is replaced by the following steps 501 to 503:

[0077] Step 501: When the aircraft is in the first lap go-around hold state, execute the PS strategy:

[0078] After one lap of go-around waiting, the aircraft's state changes from "first lap go-around waiting" to "single lap go-around waiting," which is the go-around strategy function. .

[0079] Step 502: When the aircraft is in a single-lap go-around holding state, execute the SS strategy:

[0080] After one lap of go-around waiting, the aircraft's status changed from "single lap go-around waiting" to "first lap go-around waiting," meaning... We are preparing to try again later.

[0081] Step 503: When the number of consecutive go-arounds exceeds the go-around limit, execute the termination strategy:

[0082] The aircraft changes from any state that has reached the maximum number of go-around attempts to "go-around terminated pending analysis," meaning... ,in This represents any state input that indicates the maximum number of go-around attempts has been reached. At this point, the aircraft will not be selected in subsequent decisions, meaning that the aircraft number of that aircraft will be excluded from the action space.

[0083] Optionally, the maximum number of flights that can be retaken. It can be done 3 times.

[0084] Among them, the go-around strategy (function) Defined as a state transition function, whose input is the current aircraft recovery state vector. The output is the new state value after the go-around strategy is executed.

[0085] The above assignment operation in this application only changes the aircraft number. i The current aircraft recovery status. Go-around strategy (function) The mapping output depends on the current aircraft recovery status. And the strategy selection (PS strategy, SS strategy, termination strategy), its possible output values ​​are These correspond to "single lap go-around waiting", "go-around termination pending analysis", and "first lap go-around waiting", respectively. However, the values ​​assigned during this step are used as the aircraft number. iThe new value of the current aircraft in the state vector will not automatically become 1 (indicating successful recovery). The successful recovery flag is only assigned in the successful recovery branch above. Whether the aircraft is ultimately successfully recovered depends on whether it can be retried via a go-around.

[0086] This application employs a go-around strategy (function) and clear strategy selection (PS strategy, SS strategy, termination strategy), ensuring that each strategy has clear inputs and outputs. This guarantees that the aircraft's state changes during the go-around process are predictable and controllable, improving the transparency and manageability of the entire go-around process. Multiple go-around attempts provide the aircraft with more opportunities, especially when encountering temporary problems (such as weather changes, runway occupancy, etc.), allowing for flexible strategy adjustments. When the number of consecutive go-arounds exceeds the limit, a termination strategy is executed to prevent the aircraft from endlessly attempting go-arounds, saving time and resources, avoiding flight safety risks that may result from multiple go-arounds, and ensuring flight safety. Excluding terminated aircraft from the action space allows the scheduling system to handle other operable aircraft more efficiently, improving overall scheduling efficiency.

[0087] In another exemplary embodiment of this application, in order to comprehensively consider multiple objectives such as fuel consumption, priority task guarantee, and safety risks, and to solve the problems of insufficient consideration of fuel consumption randomness, inadequate priority task guarantee, and inability to effectively avoid safety risks caused by static scheduling, making the decision more comprehensive and optimized, the fuel reward calculated based on fuel quantity, the priority reward calculated based on normalized priority, the safety reward obtained based on fuel quantity safety cost, and the weight coefficient of each reward in step 202 to calculate the multi-objective reward are replaced by the following steps 601 to 605:

[0088] Step 601: Calculate the fuel bonus based on the fuel levels of all aircraft to be recovered before the recovery operation; the formula for calculating the fuel bonus is:

[0089] .

[0090] in, r fuel For fuel rewards, Assemble the planes. i Number the aircraft f max For maximum fuel consumption, For the decision-making phase of environment instance e Aircraft number i Fuel quantity.

[0091] The fuel reward in step 601 is based on the decision-making phase in the current environment instance e. The reward system allocates fuel based on the aircraft's actual fuel consumption, rather than on a fixed amount, making the rewards fairer and more reasonable. This optimizes resource allocation, effectively avoids emergencies caused by the randomness of fuel consumption, and enables dynamic scheduling based on changes in fuel status.

[0092] Step 602: Calculate the priority reward based on the normalized priority. The formula for calculating the priority reward is as follows:

[0093] .

[0094] in, r priority As a priority reward, To normalize priority, For indicator functions, For the aircraft number in environment instance e i The aircraft recovery state vector, This indicates that the item was not recycled.

[0095] The priority reward in step 602 is for the decision-making stage. Successfully executed the transfer of high-priority aircraft (larger ones) (as an action) Select the object (i.e., aircraft number) i The operation of the aircraft (provided that the aircraft number is selected) is required. i After the aircraft executes the command, its recovery state vector does not change to 1 (indicating successful recovery). In the go-around scenario, the aircraft's recovery state vector... The priority reward is established, but because the recovery was not successful, this priority reward will not provide positive weight incentives. In the successful recovery branch, after the aircraft's recovery state vector becomes 1, this priority reward will be 0 for the aircraft in the subsequent decision-making stage. That is, there is no reward after successful recovery, thus ensuring that the priority reward mechanism can effectively encourage the selection of high-priority aircraft for operation, while avoiding unnecessary rewards for aircraft that have been successfully recovered.

[0096] Step 603: Calculate the safety cost based on the fuel levels of all aircraft to be recovered before the recovery operation. The formula for calculating the safety cost is as follows:

[0097] .

[0098] in, For the decision-making phase of environment instance e Security costs in the process; This is the penalty coefficient; f min Minimum fuel consumption. Optionally, a penalty coefficient. .

[0099] The safety cost calculation in step 603 is for environment instance e, at the current decision-making stage. The middle has not yet been recycled (i.e. The cost of insufficient fuel risk faced by all aircraft. Although the normalized task prioritization or the actions selected by the reinforcement learning agent at this point determine the order in which aircraft will be recovered, this safety cost... The evaluation targets the overall current fuel safety status of all aircraft awaiting recovery.

[0100] Step 604: Calculate the security reward based on the security cost. The formula for calculating the security reward is as follows:

[0101] .

[0102] in, r safety As a safety reward.

[0103] The safety cost in step 603 is only used to calculate the safety reward in step 604, including the fuel safety risk penalty for all aircraft to be recovered. It is fed back to the agent as a training signal for learning and does not affect the current decision-making stage. The sequence of actions to be executed (this sequence is implicitly determined by the reinforcement learning policy network when outputting action probabilities based on the current state).

[0104] Step 605: Calculate the multi-objective reward based on the fuel reward, priority reward, safety reward, and their weighting coefficients. The formula for calculating the multi-objective reward is as follows:

[0105] .

[0106] in, For the decision-making phase of environment instance e Multi-objective rewards in Assign fuel reward weight; Priority reward weight; As a safety reward weight. Optionally, .

[0107] This application employs a composite multi-objective reward mechanism that comprehensively considers fuel rewards, priority rewards, and safety rewards. By taking into account both fuel efficiency and priority, the system can allocate limited resources more rationally, reduce unnecessary resource waste, and improve system sustainability. By giving appropriate weight to safety rewards, the system can prioritize safety factors, thereby improving the overall safety of the system and reducing the probability of accidents. This comprehensive optimization approach makes the system more stable and efficient in actual operation, reducing performance bottlenecks caused by single-objective optimization.

[0108] Specifically, step 202, which involves updating the fuel quantity of all aircraft and recalculating the normalized fuel state vector, includes:

[0109] Update the fuel levels for all aircraft to the previous fuel levels for all aircraft, minus the product of the fuel consumption rate and the time interval. ,in For the previous decision-making stage Termination (or this decision-making stage) From the beginning to this decision-making stage The actual elapsed time interval (in seconds). This time interval includes the recovery execution time of the previous aircraft (successful or failed) and the corresponding waiting or scheduling time increment.

[0110] The fuel quantities of all aircraft were normalized, and the normalized fuel state vector was calculated. The normalization formula is:

[0111] .

[0112] in, For the decision-making phase of environment instance e Aircraft number i Normalized fuel consumption For the decision-making phase of environment instance e Aircraft number i Fuel quantity.

[0113] Update the fuel status of all aircraft based on the above information, regardless of their aircraft recovery state vector. This includes aircraft successfully recovered in previous decision-making phases. The aim is to maintain the integrity and timeliness of fuel information in the fleet state vector. Even after an aircraft has been recovered, its fuel consumption calculation continues in subsequent decision-making cycles until the overall system terminates, and participates in the global safety cost calculation. The max function in the safety cost calculation formula ensures that recovered aircraft do not contribute negatively. This process involves normalizing the fuel state vector. The update provides the latest fuel values.

[0114] Specifically, the global timestamp is updated based on the current recycling time, and the cluster state vector is synchronized, including:

[0115] The current recovery time will be updated to the previous recovery time and the previous decision-making stage. Termination (or this decision-making stage) From the beginning to this decision-making stage The sum of the actual time intervals that have elapsed so far, i.e. .

[0116] Based on the normalized fuel state vector and aircraft recovery state vector, synchronize the current fleet state vector, i.e. .

[0117] The synchronization of the aforementioned aircraft group state vector is based on the calculated results of the normalized fuel state vector and the modification of the aircraft recovery state vector of the aircraft selected by the agent (if any), updating the aircraft group state vector of the entire environment instance. The updated aircraft group state vector... It is a subsequent decision (the next decision-making stage after re-entry). (and the basic input for multi-objective reward calculation).

[0118] At the beginning of each decision stage, the time at the end of the previous decision stage is used as the time at the beginning of the next decision stage, and the cluster state vector at the end of the previous decision stage is used as the cluster state vector at the beginning of the next decision stage, thus realizing the inheritance of time and cluster state vector.

[0119] The above state inheritance occurs during the decision-making stage. The purpose of initial execution is to use the cluster state vector (or time) at the end of the previous decision-making phase as the starting point for the cluster state vector (or time) of the next decision-making phase. This provides foundational data for subsequent security assessments, decision execution, and state updates, ensuring that the latest state information is available before initiating a new decision. Each environment instance e operates independently, and their state inheritance operation logic is the same, but the specific cluster state vector values ​​differ due to the independent operating history and random initialization of each environment instance.

[0120] Specifically, repeat the above steps until all items are recycled. This includes repeating the above steps if... ( If the indicator function is used, then the training process for that environment instance e is terminated. This step's decision depends solely on the recovered state vector, and the updated cluster state vector, obtained during the repetition of the above steps in each decision phase, is the core state input for the next decision.

[0121] In another exemplary embodiment of this application, in order to obtain the optimal target dynamic scheduling decision sequence, step 203 above is replaced by steps 701 to 702:

[0122] Step 701: After all environment instances e have completed training and reached the termination state, calculate the total cumulative multi-objective reward of each environment instance's decision trajectory. The calculation formula is as follows:

[0123] .

[0124] in, The total cumulative multi-objective reward under environment instance e; The total number of decision stages when environment instance e terminates; For the decision-making stage; For the decision-making phase of environment instance e Multi-objective rewards in the game.

[0125] Step 702: Among the decision trajectories corresponding to all environment instances e, select the one with the largest total cumulative value. The action sequence corresponding to the decision trajectory is used as the target dynamic scheduling decision sequence obtained based on the current parallel reinforcement learning training.

[0126] This application, by comparing the total cumulative rewards of multiple environment instances, can select the optimal decision path from a global perspective, avoiding the limitations of local optima and realizing the exploration of global optima. Parallel training across multiple environment instances allows for the simultaneous exploration of various possible decision paths, accelerating the training process, significantly reducing training time, and improving training efficiency, especially in complex tasks where the optimal solution can be found more quickly. Furthermore, by comprehensively considering rewards from multiple objectives such as fuel efficiency, priority, and safety, the merits of each decision path can be fully evaluated. This ensures that the final selected decision sequence performs well across multiple key indicators, improving the overall performance of the system. Finally, it enhances the system's adaptability to the environment and strengthens its robustness.

[0127] The following example illustrates this application using a specific dynamic scheduling decision-making process for coordinated aircraft recovery.

[0128] Step 1: Create multiple parallel environment instances. Configure the initialization attributes of five aircraft (aircraft numbers: 1-5) in one of the environment instances, as shown in Table 1, and generate a group state vector consisting of a normalized fuel state vector and an aircraft recovery state vector.

[0129] Table 1 Initial Attributes of the Five Aircraft

[0130]

[0131] The cluster state vector is:

[0132] .

[0133] The initial timestamp is: .

[0134] In addition, a minimum fuel quantity is set. Maximum fuel capacity Fuel consumption rate Recovery success rate threshold: .

[0135] Step 2, (1) In this environmental example, proceed to the decision-making stage. The agent selects actions This means selecting aircraft number 1 to perform the recovery operation. At this point, the timestamp is updated: (Initial decision has no time interval), cluster state vector inheritance: It remains unchanged.

[0136] Calculate the probability of successfully recovering aircraft number 1:

[0137] .

[0138] because The aircraft recovery state vector is assigned a value of 1 to mark it as successfully recovered, i.e., the aircraft recovery state vector is updated: The timestamp is now updated to: (The first aircraft had no wake gap.)

[0139] Fuel bonus calculated based on fuel consumption:

[0140] .

[0141] Calculate priority rewards based on normalized priority:

[0142] (That is, there is no reward after successful recycling).

[0143] Calculate the safety cost based on the fuel levels of all aircraft to be recovered before the recovery operation is performed:

[0144] Because the fuel capacity of aircraft number 3 is This incurs costs; .

[0145] Calculate the security reward based on the security cost:

[0146] .

[0147] Multi-objective rewards are calculated based on fuel rewards, priority rewards, safety rewards, and their weighting coefficients:

[0148] .

[0149] Updated fuel levels for all aircraft:

[0150] The updated fuel capacity for aircraft number 1 is 1800-120. 0.15 = 1782 kg.

[0151] The updated fuel capacity of aircraft number 2 is: 1500-18=1482kg.

[0152] The updated fuel capacity of aircraft number 3 is: 600 - 18 = 582 kg.

[0153] The updated fuel capacity of aircraft number 4 is: 1200-18=1182kg.

[0154] The updated fuel capacity of aircraft number 5 is: 900-18=882kg.

[0155] , , , , .

[0156] The normalized fuel state vector for this operation is calculated as follows: .

[0157] The global timestamp is updated based on the time of this recycling: ( (Depending on the time taken by the previous action); synchronize the current cluster state vector: .

[0158] (2) Entering the decision-making stage The agent selects actions This means that aircraft number 3, which has a low priority but an urgent fuel need, is selected to perform the recovery operation. At this time, the timestamp is: Cluster state vector inheritance: .

[0159] Calculate the probability of successfully recovering aircraft number 3:

[0160] .

[0161] because This triggers the go-around strategy. When the aircraft is in a single-lap go-around waiting state, the SS strategy is executed: after one lap of go-around waiting, the aircraft's state changes from "single-lap go-around waiting" to "first-lap go-around waiting," i.e., the aircraft recovery state vector is: The timestamp is now updated to: .

[0162] Fuel bonus calculated based on fuel consumption:

[0163] .

[0164] Calculate priority rewards based on normalized priority:

[0165] .

[0166] Calculate the safety cost based on the fuel levels of all aircraft to be recovered before the recovery operation is performed: There are no costs involved. Therefore, the security reward is also zero.

[0167] Multi-objective rewards are calculated based on fuel rewards, priority rewards, safety rewards, and their weighting coefficients:

[0168] .

[0169] Updated fuel levels for all aircraft:

[0170] The updated fuel capacity for aircraft number 1 is 1782-180. 0.15 = 1755 kg.

[0171] The updated fuel capacity of aircraft number 2 is: 1482 - 27 = 1455 kg.

[0172] The updated fuel capacity of aircraft number 3 is: 582 - 27 = 555 kg.

[0173] The updated fuel capacity of aircraft number 4 is: 1182 - 27 = 1155 kg.

[0174] The updated fuel capacity of aircraft number 5 is: 900-18=882kg.

[0175] The normalized fuel state vector for this operation is calculated as follows: .

[0176] The global timestamp is updated based on the time of this recycling: Synchronize the cluster state vector for this operation: .

[0177] (3) Entering the decision-making stage Fuel levels were detected in aircraft number 3. Forced locking action Repeat the above steps.

[0178] (4) Entering the decision-making stage The aircraft with serial number 3 attempted to be recovered again. What is the probability of success? If the value is still below the threshold, the termination strategy is triggered, i.e., the aircraft recovery state vector: And remove aircraft number 3 from the waiting queue.

[0179] (5) Entering the decision-making stage Continue to dispatch the remaining aircraft until all are recovered.

[0180] Step 3: Calculate the total cumulative multi-objective reward for the decision trajectory of each environmental instance. Select the action sequence corresponding to the maximum total cumulative value (e.g.) () as the optimal target dynamic scheduling decision sequence.

[0181] Based on the same inventive concept, this application also provides a parallel reinforcement learning-based aircraft cooperative recovery dynamic scheduling decision-making device for implementing the above-mentioned parallel reinforcement learning-based aircraft cooperative recovery dynamic scheduling decision-making method. The solution provided by this device is similar to the solution described in the above method. Therefore, the specific limitations of one or more embodiments of the parallel reinforcement learning-based aircraft cooperative recovery dynamic scheduling decision-making device provided below can be found in the limitations of the parallel reinforcement learning-based aircraft cooperative recovery dynamic scheduling decision-making method described above, and will not be repeated here.

[0182] In one exemplary embodiment, such as Figure 3 As shown, a dynamic scheduling decision-making device for aircraft cooperative recovery based on parallel reinforcement learning is provided, including: initialization module 1, action execution module 2, and target sequence determination module 3.

[0183] Initialization module 1 is used to create multiple parallel environment instances, configure the initialization attributes of each aircraft in each environment instance, and generate a group state vector consisting of a normalized fuel state vector and an aircraft recovery state vector. The attributes include at least fuel quantity and normalized priority.

[0184] Action execution module 2 is used to perform the following steps during the decision-making phase of each environment instance: perform a recovery action on the aircraft selected by the agent and update the aircraft recovery state vector; calculate the multi-objective reward based on the fuel reward calculated from the fuel quantity, the priority reward calculated from the normalized priority, the safety reward obtained from the safety cost based on the fuel quantity, and the weight coefficient of each reward; update the fuel quantity of all aircraft and recalculate the normalized fuel state vector; update the global timestamp based on the recovery time and synchronize the current fleet state vector; the agent selects the next aircraft to be recovered and enters the next decision-making phase, repeating the above steps until all aircraft are recovered.

[0185] Action execution module 2 is responsible for time sequence advancement, state maintenance (including inheritance and update), safety assessment, decision execution and result processing, and is also responsible for calculating multi-objective rewards based on fuel rewards, priority rewards, safety rewards and their weight coefficients.

[0186] The target sequence determination module 3 is used to calculate the total cumulative reward of the multi-objective decision trajectory for each environmental instance, and select the action sequence corresponding to the maximum total cumulative reward as the target dynamic scheduling decision sequence.

[0187] In one exemplary embodiment, a computer device is provided, which may be a server or a terminal, and its internal structure diagram may be as follows. Figure 4As shown, the computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database is used for processing data for dynamic scheduling decisions in aircraft cooperative recovery based on parallel reinforcement learning. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network connection. When the computer program is executed by the processor, it implements a dynamic scheduling decision-making method for aircraft cooperative recovery based on parallel reinforcement learning.

[0188] Those skilled in the art will understand that Figure 4 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0189] In one exemplary embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0190] In one exemplary embodiment, a computer-readable storage medium is provided storing a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0191] In one exemplary embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0192] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0193] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM).

[0194] The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0195] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0196] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A parallel reinforcement learning-based dynamic scheduling decision method for aircraft cooperative recovery, characterized in that, The method comprises the following steps of: creating a plurality of parallel environment instances, configuring initialization attributes of each aircraft in each environment instance, and generating a fleet state vector composed of a normalized fuel state vector and an aircraft recovery state vector, the attributes at least including fuel quantity, normalized priority, fuel consumption rate; in the decision-making stage of each environment instance, the following steps are performed: The aircraft selected by the intelligent agent performs a recovery action, and the aircraft recovery state vector of this time is updated; the aircraft recovery state vector of this time is updated, specifically including: when the recovery success probability of the aircraft of this time is greater than the set recovery success rate threshold, the aircraft recovery state vector is assigned a value of 1, indicating that the recovery is successful; when the recovery success probability of the aircraft of this time is less than or equal to the set recovery success rate threshold, a reflight strategy is executed, and the aircraft recovery state vector is updated to the state value after the reflight strategy is executed; the reflight strategy is executed, and the aircraft recovery state vector is updated to the state value after the reflight strategy is executed, specifically including: when the aircraft is in the first circle reflight waiting state, the PS strategy is executed: the state of the aircraft changes from 'first circle reflight waiting' to'single circle reflight waiting' after one circle of reflight waiting, that is, the reflight strategy function ; when the aircraft is in the single circle reflight waiting state, the SS strategy is executed: the state of the aircraft changes from'single circle reflight waiting' to 'first circle reflight waiting' after one circle of reflight waiting, that is , and the subsequent reattempt is prepared; when the number of consecutive reflight times of the aircraft exceeds the upper limit of reflight, the termination strategy is executed: the aircraft changes from any state reaching the upper limit of reflight times to'reflight termination waiting for analysis', that is , wherein represents any state input reaching the upper limit of reflight times, at this time the aircraft will not be selected in subsequent decision-making, that is, the action space will exclude the aircraft number of the aircraft. calculating a multi-objective reward according to a fuel reward calculated based on fuel quantity, a priority reward calculated based on normalized priority, a safety reward obtained based on safety cost of fuel quantity, and a weight coefficient of each reward; updating the fuel quantity of all aircrafts in the current flight, and recalculating the normalized fuel state vector of the current flight, including: updating the fuel quantity of all aircrafts in the current flight to the fuel quantity of all aircrafts in the previous flight minus the product of the fuel consumption rate and the time interval, i.e. wherein is the time interval actually elapsed from the end of the previous decision stage to the beginning of the current decision stage ; normalizing the fuel quantity of all aircrafts in the current flight to calculate the normalized fuel state vector of the current flight , the normalization formula being: ; wherein, is the normalized fuel quantity of aircraft number i in the decision stage of the environmental instance e, is the fuel quantity of aircraft number i in the decision stage of the environmental instance e, is the minimum fuel quantity, is the maximum fuel quantity; updating a global timestamp by a current recovery time, and synchronizing a current fleet state vector; the agent selects a next aircraft to be recovered to enter a next decision-making stage to repeat the above steps until all recovery is completed; calculating a total accumulation of multi-objective rewards of decision-making trajectories of each environment instance, and selecting an action sequence corresponding to a maximum total accumulation as a target dynamic scheduling decision sequence.

2. The parallel reinforcement learning based dynamic scheduling decision method for aircraft cooperative recovery according to claim 1, characterized in that, performing a recovery action on the aircraft selected by the agent, specifically comprising: The normal action selection strategy of an agent is as follows: at each decision stage, dynamically output an action based on the swarm state vector, normalized priority, and multi-objective reward. Regarding actions Mapped aircraft number i Execute the recycling action; among which, the action From discrete action space The selected integer corresponds directly to the aircraft number of the aircraft to be recovered. i ; The emergency recovery action selection strategy of the intelligent agent is: when the fuel quantity of the aircraft to be recovered is lower than a set value, the current action or the next action to be selected is forced to directly lock the aircraft number of the aircraft to be recovered which is lower than the set value i The emergency recovery action selection strategy has a higher priority than the normal action selection strategy, and is forced to have the highest priority to be recovered.

3. The parallel reinforcement learning based dynamic scheduling decision method for aircraft cooperative recovery according to any one of claims 1-2, characterized in that, calculating a multi-objective reward according to a fuel reward calculated based on fuel quantity, a priority reward calculated based on normalized priority, a safety reward obtained based on safety cost of fuel quantity, and a weight coefficient of each reward, specifically comprising: calculating a fuel reward according to fuel quantity of all aircrafts to be recovered before the recovery action is performed; the calculation formula of the fuel reward is: ； wherein, r fuel is the fuel reward, is the set of aircraft, i is the aircraft number, f max is the maximum fuel amount, is the fuel amount for the aircraft number in the decision phase of the environmental instance e. i is the fuel amount for the aircraft number calculating a priority reward according to normalized priority; the calculation formula of the priority reward is: ； wherein, r Priority is a priority reward, is a normalized priority, is an indicator function, is the aircraft number in the environment instance e i is the aircraft recovery state vector, denotes not recovered; calculating a safety cost based on fuel quantity of all aircrafts to be recovered before the recovery action is performed; the calculation formula of the safety cost is: ； wherein, is the security cost in the decision phase of the environmental instance e; is the penalty coefficient; f min is the minimum fuel quantity;​ calculating a safety reward according to the safety cost; the calculation formula of the safety reward is: ； wherein, r safety for safe rewards; calculating a multi-objective reward according to the fuel reward, the priority reward, the safety reward, and the weight coefficient thereof; the calculation formula of the multi-objective reward is: ； in, For the decision-making phase of environment instance e Multi-objective rewards in Assign fuel reward weight; Priority reward weight; Weighting for security rewards.

4. The parallel reinforcement learning based dynamic scheduling decision method for aircraft cooperative recovery according to claim 1, characterized in that, calculating a total accumulation of multi-objective rewards of decision-making trajectories of each environment instance, and selecting an action sequence corresponding to a maximum total accumulation as a target dynamic scheduling decision sequence, specifically comprising: after all environment instances e complete training and reach a termination state, calculating a total accumulation of multi-objective rewards of decision-making trajectories experienced by each environment instance, and the calculation formula is: ； wherein, is the total cumulative of multi-objective rewards under environment instance e; is the total number of decision stages at the termination of environment instance e; is the decision stage; is the multi-objective reward in decision stage of environment instance e. In all the decision trajectories corresponding to the environment instances e, select the maximum total cumulative corresponding to the action sequence as the target dynamic scheduling decision sequence obtained based on the current parallel reinforcement learning training.

5. An aircraft cooperative recovery dynamic scheduling decision device based on parallel reinforcement learning, characterized in that, The method comprises the following steps of: an initialization module for creating a plurality of parallel environment instances, configuring initialization attributes of each aircraft in each environment instance, and generating a fleet state vector composed of a normalized fuel state vector and an aircraft recovery state vector, the attributes at least including fuel quantity, normalized priority, fuel consumption rate; The action execution module is used to perform the following steps during the decision-making phase of each environment instance: Perform a recovery action on the aircraft selected by the agent and update the aircraft recovery state vector; Updating the aircraft recovery state vector specifically includes: when the probability of successful recovery is greater than a set recovery success rate threshold, assigning a value of 1 to the aircraft recovery state vector, marking it as successfully recovered; when the probability of successful recovery is less than or equal to the set recovery success rate threshold, executing a go-around strategy and updating the aircraft recovery state vector to the state value after executing the go-around strategy; Executing the go-around strategy and updating the aircraft recovery state vector to the state value after executing the go-around strategy specifically includes: when the aircraft is in the first lap go-around waiting state, executing the PS strategy: after one lap of go-around waiting, the aircraft's state changes from "first lap go-around waiting" to "single lap go-around waiting," i.e., the go-around strategy function. When the aircraft is in a single-lap go-around hold state, the SS strategy is executed: after the aircraft completes one lap of go-around hold, the state changes from "single-lap go-around hold" to "first-lap go-around hold," that is... Prepare for further attempts; when the number of consecutive go-arounds exceeds the go-around limit, execute the termination strategy: the aircraft changes from any state that has reached the go-around limit to "go-around termination pending analysis", that is... ,in This represents any state input indicating that the maximum number of go-around attempts has been reached. At this point, the aircraft cannot be selected in subsequent decisions; that is, the aircraft number will be excluded from the action space. The multi-objective reward is calculated based on the fuel reward calculated from fuel quantity, the priority reward calculated from normalized priority, the safety reward obtained from the safety cost based on fuel quantity, and the weighting coefficients of each reward. The fuel quantity of all aircraft in this current operation is updated, and the normalized fuel state vector for this current operation is recalculated. Specifically, the fuel quantity of all aircraft in this current operation is updated to the previous fuel quantity of all aircraft minus the product of the fuel consumption rate and the time interval. ,in For the previous decision-making stage Termination up to this decision-making stage The actual elapsed time interval; normalize the fuel quantity of all aircraft in this operation, and calculate the normalized fuel state vector for this operation. The normalization formula is: ;in, For the decision-making phase of environment instance e The normalized fuel quantity of aircraft number i in the data. For the decision-making phase of environment instance e The fuel quantity of aircraft number i in the diagram. Minimum fuel quantity, The maximum fuel quantity; the global time stamp is updated by the current recovery time, and the current fleet state vector is synchronized; the agent selects the next aircraft to be recovered to enter the next decision stage and repeats the above steps until all the aircrafts are recovered. a target sequence determination module for calculating a total accumulation of multi-objective rewards of decision-making trajectories of each environment instance, and selecting an action sequence corresponding to a maximum total accumulation as a target dynamic scheduling decision sequence.

6. A computer device comprising: A memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that the processor executes the computer program to implement the aircraft cooperative recovery dynamic scheduling decision method based on parallel reinforcement learning in any one of claims 1-4.

7. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the aircraft cooperative recovery dynamic scheduling decision method based on parallel reinforcement learning in any one of claims 1-4.

8. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to implement the parallel reinforcement learning-based aircraft cooperative recovery dynamic scheduling decision method in any one of claims 1-4.

Images