Aircraft sortie scheduling strategy generation methods, devices, equipment, media and products

Abstract

This application discloses a method, apparatus, device, medium, and product for generating aircraft sortie scheduling strategies, relating to the fields of intelligent decision-making and automation technology. The method includes: initializing an intelligent agent; within any scheduling cycle, performing sequential interactions between the agent and the scheduling environment at each decision step based on a dataset, and storing the determined trajectory data in an experience replay memory; after the sequential interactions of one or more scheduling cycles are completed, calculating the strategy loss function using a near-end strategy optimization algorithm based on the corresponding trajectory data stored in the experience replay memory, and updating the parameters of the hierarchical strategy network using gradient descent; performing instance solving on a preset verification dataset based on the updated hierarchical strategy network at preset verification cycle intervals to determine performance indicators; and determining scheduling information based on the scheduling strategy network and the dataset. This application can improve the efficiency and quality of scheduling strategy generation.

Description

Technical Field

[0001] This application relates to the field of intelligent decision-making and automation technology, and in particular to a method, apparatus, equipment, medium and product for generating aircraft sortie scheduling strategies. Background Technology

[0002] Aircraft sortie support scheduling is a crucial link in ensuring sortie efficiency. Its goal is to efficiently plan the execution sequence and resource allocation of all support procedures while meeting strict process timing constraints, resource availability constraints, and conflict constraints related to parallel support of multiple aircraft types, in order to minimize core metrics such as total make-span. This problem is essentially a highly complex flexible job shop scheduling problem, a typical NP-hard combinatorial optimization problem. As the scale and complexity of support tasks increase, the solution space grows exponentially, placing extremely high demands on the efficiency and quality of the scheduling methods.

[0003] Traditional solutions mainly include exact algorithms, heuristic algorithms, and metaheuristic algorithms. Exact algorithms, such as mixed integer programming, theoretically guarantee finding the optimal solution, but their computational time is enormous, making it difficult to solve medium-sized or larger problems within an acceptable timeframe, thus failing to meet the need for rapid response. Heuristic and metaheuristic algorithms, such as genetic algorithms, simulated annealing, and tabu search, quickly find high-quality approximate optimal solutions by designing specific search rules. However, these methods often rely on a large number of manually designed rules and parameter adjustments, resulting in weak generalization ability and a tendency to get trapped in local optima, making them difficult to adapt to dynamically changing deployment environments and support requirements.

[0004] In recent years, artificial intelligence technologies, represented by deep reinforcement learning, have opened up new paths for solving such problems. By modeling the scheduling problem as a Markov decision process, reinforcement learning agents can autonomously learn scheduling strategies through trial and error with the environment. However, existing reinforcement learning-based scheduling methods still face challenges: on the one hand, directly using the joint action of "which process to choose" and "which resource to allocate" as the decision output leads to an extremely large action space, making it difficult for the agent to effectively explore and learn; on the other hand, traditional feature engineering methods often fail to fully capture and utilize the inherent, complex graph structure constraints in scheduling problems, limiting the decision-making performance of the policy network. Therefore, how to design a reinforcement learning scheduling method that can effectively handle the huge decision space and deeply understand the inherent structural constraints of the problem is a pressing problem to be solved in the current technological field. Summary of the Invention

[0005] The purpose of this application is to provide a method, apparatus, equipment, medium, and product for generating aircraft sortie scheduling strategies, which can improve the efficiency and quality of scheduling strategy generation.

[0006] To achieve the above objectives, this application provides the following solution:

[0007] Firstly, this application provides a method for generating aircraft sortie scheduling strategies, including:

[0008] Initialize the intelligent agent; the intelligent agent includes: a hierarchical policy network and an experience replay memory;

[0009] Within any scheduling cycle, based on the dataset, the agent and the scheduling environment perform sequential interactions at each decision step, and the determined trajectory data is stored in the experience replay memory; the dataset includes: state data;

[0010] After the sequential interaction of one or more scheduling cycles is completed, the corresponding trajectory data stored in the memory is replayed based on the experience. The near-end policy optimization algorithm is used to calculate the policy loss function, and the gradient descent method is used to update the parameters of the hierarchical policy network to obtain the updated hierarchical policy network.

[0011] Using a preset verification period as the interval, the preset verification dataset is solved for instances based on the updated hierarchical policy network to determine performance indicators; the performance indicators include: average maximum completion time.

[0012] Based on the scheduling policy network, scheduling information is determined according to the dataset; the scheduling information includes: scheduling procedures and maintenance resources; the scheduling policy network is determined based on the optimal parameters and the updated hierarchical policy network; the optimal parameters are determined by comparing the performance indicators with the historical best performance indicators.

[0013] In one embodiment, the hierarchical strategy network includes a process selection strategy network and a resource allocation strategy network; the process selection strategy network includes a graph convolutional network encoder and a multilayer perceptron decoder; the resource allocation strategy network includes a multilayer perceptron.

[0014] In one implementation, within any scheduling cycle, based on the dataset, the agent and the scheduling environment perform sequential interactions at each decision step, and the determined trajectory data is stored in the experience replay memory, specifically including:

[0015] Based on the process node feature matrix and scheduling graph adjacency matrix contained in the process selection strategy network received state data, information aggregation and feature extraction are performed on each process node through a graph convolutional network encoder to obtain the hidden state of the process node and the hidden state of the global graph.

[0016] The hidden state of the global graph, the context hidden state output by the resource allocation network of the previous decision step, and the candidate process hidden state extracted from the hidden state of the process node according to the current available process index are concatenated and fused, and then input into the decoder of the multilayer perceptron. The Softmax function is used to generate the probability distribution of process selection, and then the process action is sampled.

[0017] Based on the resource allocation strategy network, the processing time vector related to the process and the current completion time vector of the resource are received from the state data. Combined with the hidden state of the global graph output by the process selection network and the hidden state of the node corresponding to the process, the probability distribution of resource allocation is generated by the multilayer perceptron and the Softmax function. Then, the resource allocation action is sampled and the context hidden state of the current decision step is output.

[0018] Trajectory data is determined based on process actions, resource allocation actions, and status data, and the trajectory data is stored in the experience playback memory.

[0019] In one embodiment, a graph convolutional network encoder is used to aggregate information and extract features for each process node. The mathematical expression for the corresponding information transmission mechanism is as follows:

[0020] ;

[0021] in, For the first The process node at the _ Hidden representation of layers; To modify the activation function of the linear unit; For layer normalization operation; For the first Multilayer perceptron; For the first The set of neighboring nodes of each process node; For the first The process node at the _ Hidden representation of layers; For the first The process node at the _ Hidden representation of layers.

[0022] In one embodiment, the aircraft sortie scheduling strategy generation method further includes:

[0023] Periodic performance monitoring is performed during the scheduling period; wherein, performance monitoring logs are determined based on the performance indicators and the policy loss function to monitor performance convergence and loss change trends in real time.

[0024] In one embodiment, the expression for the policy loss function is:

[0025] ;

[0026] in, Let the policy loss function be used. Loss for proxy strategies with pruning objectives; It is a value function loss; This is the policy entropy regularization term; and These are all weighting coefficients for the loss term; This is status data; Expected based on experience; These are the parameters for the hierarchical strategy network; For the reason The resulting hierarchical scheduling strategy.

[0027] Secondly, this application provides an aircraft sortie scheduling strategy generation device, comprising:

[0028] An initialization module is used to initialize the agent; the agent includes: a hierarchical policy network and an experience replay memory;

[0029] An interaction module is used to perform sequential interaction between the agent and the scheduling environment at each decision step based on the dataset within any scheduling cycle, and to store the determined trajectory data into the experience playback memory; the dataset includes: state data;

[0030] The update module is used to calculate the policy loss function based on the trajectory data stored in the corresponding memory according to the experience replay memory after the sequential interaction of one or more scheduling cycles is completed, and to update the parameters of the hierarchical policy network using the near-end policy optimization algorithm and the gradient descent method to obtain the updated hierarchical policy network.

[0031] The performance verification module is used to perform instance solving on the preset verification dataset based on the updated hierarchical policy network at preset verification intervals to determine performance indicators; the performance indicators include: average maximum completion time.

[0032] The scheduling module is used to determine scheduling information based on a scheduling policy network and a dataset. The scheduling information includes scheduling procedures and maintenance resources. The scheduling policy network is determined based on optimal parameters and an updated hierarchical policy network. The optimal parameters are determined by comparing the performance indicators with historical best performance indicators.

[0033] Thirdly, this application provides a computer device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the aircraft sortie scheduling strategy generation method described above.

[0034] Fourthly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the aircraft sortie scheduling strategy generation method described above.

[0035] Fifthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the aircraft sortie scheduling strategy generation method described above.

[0036] According to the specific embodiments provided in this application, the following technical effects are disclosed:

[0037] This application provides a method, apparatus, device, medium, and product for generating aircraft sortie scheduling strategies. It initializes an intelligent agent; within any scheduling cycle, based on a dataset, the agent and the scheduling environment perform sequential interactions at each decision step, and the determined trajectory data is stored in an experience replay memory; after the sequential interactions of one or more scheduling cycles are completed, based on the corresponding trajectory data stored in the experience replay memory, a proximal policy optimization algorithm is used to calculate the policy loss function, and gradient descent is used to update the parameters of the hierarchical policy network; this process learns and updates parameters on a large amount of interactive trajectory data, significantly reducing the decision space at each step. Furthermore, with a preset verification cycle as the interval, the updated hierarchical policy network is used to solve for instances on a preset verification dataset to determine performance indicators; based on the scheduling policy network, scheduling information is determined according to the dataset to ensure that the final result is the best-performing scheduling policy network on the preset verification dataset. This effectively learns high-performance aircraft sortie scheduling schemes, improving the efficiency and quality of scheduling strategy generation. Attached Figure Description

[0038] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0039] Figure 1 A flowchart for generating aircraft sortie scheduling strategies;

[0040] Figure 2 A structural diagram of an aircraft sortie scheduling strategy generation device;

[0041] Figure 3 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0042] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0043] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, this application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0044] In one exemplary embodiment, such as Figure 1 As shown, a method for generating aircraft sortie scheduling strategies is provided, including:

[0045] Step 100: Initialize the agent. The agent includes a hierarchical policy network and experience replay memory.

[0046] In one embodiment, the hierarchical strategy network includes a process selection strategy network and a resource allocation strategy network; the process selection strategy network includes a graph convolutional network encoder and a multilayer perceptron decoder; the resource allocation strategy network includes a multilayer perceptron.

[0047] Step 200: Within any scheduling cycle, based on the dataset, the agent and the scheduling environment perform sequential interactions at each decision step, and the determined trajectory data is stored in the experience replay memory. The dataset includes: state data.

[0048] In one embodiment, within any scheduling cycle, based on the dataset, the agent and the scheduling environment perform sequential interactions at each decision step, and the determined trajectory data is stored in the experience replay memory, specifically including:

[0049] Based on the process selection strategy network receiving state data containing the process node feature matrix and scheduling graph adjacency matrix, a graph convolutional network encoder is used to perform information aggregation and feature extraction on each process node to obtain the hidden state of the process node and the hidden state of the global graph.

[0050] In this process, a graph convolutional network encoder is used to aggregate information and extract features for each process node. The mathematical expression for the corresponding information transmission mechanism is as follows:

[0051] .

[0052] in, For the first The process node at the _ Hidden representation of layers; To modify the activation function of the linear unit; For layer normalization operation; For the first Multilayer perceptron; For the first The set of neighboring nodes of each process node; For the first The process node at the _ Hidden representation of layers; For the first The process node at the _ Hidden representation of layers.

[0053] The hidden state of the global graph, the context hidden state output by the resource allocation network of the previous decision step, and the candidate process hidden state extracted from the hidden state of the process node according to the current available process index are concatenated and fused, and then input into the decoder of the multilayer perceptron. The Softmax function is used to generate the probability distribution of process selection, and then the process action is sampled.

[0054] Based on the resource allocation strategy network, the processing time vector related to the process and the current completion time vector of the resource are received from the state data. Combined with the global graph hidden state output by the process selection network and the node hidden state corresponding to the process, the probability distribution of resource allocation is generated by multilayer perceptron and Softmax function. Then, the resource allocation action is sampled and the context hidden state of the current decision step is output.

[0055] Trajectory data is determined based on process actions, resource allocation actions, and status data, and the trajectory data is stored in the experience playback memory.

[0056] Step 300: After the sequential interaction of one or more scheduling cycles is completed, the corresponding trajectory data stored in the memory is replayed based on the experience. The near-end policy optimization algorithm is used to calculate the policy loss function, and the gradient descent method is used to update the parameters of the hierarchical policy network to obtain the updated hierarchical policy network.

[0057] In one embodiment, the policy loss function is expressed as follows:

[0058] .

[0059] in, Let the policy loss function be used. Loss for proxy strategies with pruning objectives; It is a value function loss; This is the policy entropy regularization term; and These are all weighting coefficients for the loss term; This is status data; Let be the empirical expectation, representing the empirical expectation calculated for a batch of time-series samples; These are the parameters for the hierarchical strategy network; For the reason The resulting hierarchical scheduling strategy.

[0060] Step 400: Using a preset verification period as the interval, perform instance solving on the preset verification dataset based on the updated hierarchical policy network to determine performance metrics. Performance metrics include: average maximum completion time.

[0061] Step 500: Determine scheduling information based on the dataset using the scheduling policy network. The scheduling information includes scheduling procedures and maintenance resources; the scheduling policy network is determined based on optimal parameters and the updated hierarchical policy network; the optimal parameters are determined by comparing performance metrics with historical best performance metrics.

[0062] As an optional implementation method, the aircraft sortie scheduling strategy generation method further includes:

[0063] Periodic performance monitoring is performed within the scheduling cycle; performance monitoring logs are determined based on performance metrics and policy loss functions to monitor performance convergence and loss change trends in real time.

[0064] This application mentions a method using deep reinforcement learning and graph neural network techniques to solve a class of combinatorial optimization problems with complex constraints, which is particularly suitable for aircraft sortie support scheduling, flexible job shop scheduling (FJSP), and other industrial production and logistics scheduling scenarios that require sequential resource allocation and task ordering.

[0065] This application relates to the interdisciplinary fields of artificial intelligence and operations research, aiming to address the difficulties in policy solving caused by the large decision space and complex constraints in complex scheduling problems such as aircraft dispatch support. By constructing a hierarchical reinforcement learning agent comprising a top-level process selection network and a bottom-level resource allocation network, complex joint decisions are decomposed into sequential and more manageable sub-decisions. The core of this approach lies in utilizing graph convolutional networks to deeply mine the inherent graph structure information of process constraints and resource occupancy constraints between processes in the scheduling problem, generating state feature representations rich in topological relationships. Based on this, the attention mechanism in the hierarchical policy network dynamically focuses on the state information most relevant to the current decision. The top-level network is responsible for selecting the optimal process to be scheduled, while the bottom-level network matches the most suitable maintenance resources for that process. The entire method is trained end-to-end using a proximal policy optimization algorithm, enabling the agent to autonomously learn and iteratively optimize scheduling strategies through continuous interaction with the scheduling environment. This application significantly improves the efficiency of policy learning and the quality of the final scheduling scheme through structured hierarchical decision-making and graph information mining, generating high-performance aircraft dispatch scheduling schemes.

[0066] In practical applications, the specific implementation steps of the method mentioned in this application include:

[0067] S1. System Initialization and Data Loading: Initialize a reinforcement learning agent and load the dataset. The agent includes a hierarchical policy network for sequential decision-making and experience replay memory. The hierarchical policy network consists of a top-level process selection policy network. and a low-level resource allocation strategy network Composition; Process selection strategy network It includes a graph convolutional network encoder and a multilayer perceptron decoder, and a resource allocation policy network. It includes a multilayer perceptron.

[0068] S2. Interactive Training Iteration: During a preset training cycle Within this process, the following steps S2.1 to S2.4 are executed repeatedly.

[0069] S2.1. Environmental Interaction and Data Acquisition: Within a scheduling cycle, the agent and the aircraft sortie scheduling environment interact at each decision step. To conduct sequential interactions, the process first selects a strategy network. Based on the current state, i.e., state data Output the processes to be scheduled Subsequently, the resource allocation strategy network based on and Output the machine resources allocated to this process. Record the decision-making process Complete action ,award and the logarithmic probability of actions Until the scheduling is completed, a complete trajectory is formed. The trajectory data is then stored in the experience playback memory. .

[0070] S2.2. Policy Network Update: After one or more scheduling cycles have completed, use experience to replay memory. The data in the dataset is used to calculate the policy loss function using the Proximal Policy Optimization (PPO) algorithm. The process selection strategy network is updated using gradient descent. and resource allocation strategy network The parameters, i.e., the parameters of the hierarchical policy network. .in, Select the parameters of the strategy network for the process; The parameters for the resource allocation strategy network.

[0071] S2.3. Model Performance Verification: Based on a preset verification cycle. For intervals, where The value range is [10, 200] training epochs, using the currently updated policy network on the validation dataset. Solve using examples to calculate the average maximum completion time. As a performance indicator.

[0072] S2.4. Optimal Model Saving: Comparison Compared with historical best performance metrics ,like Then the current policy network parameters, i.e., the parameters of the hierarchical policy network, will be... Save as optimal model parameters .

[0073] Specifically, step S2.1, which involves the agent interacting with the environment to select an action, includes:

[0074] S2.1.1. Process Feature Extraction: Process Selection Strategy Network take over process node feature matrix And the adjacency matrix of the scheduling graph , through a containing The graph convolutional network encoder of the layer performs information aggregation and feature extraction for each process node to obtain the hidden state of the process node. and the hidden state of the global graph .in The value range is [2, 5].

[0075] S2.1.2. Process Selection Decision: Hidden states of the global graph Previous decision step The context hidden state of the resource allocation network output and through from The hidden state of candidate processes is extracted based on the current available process index. Perform splicing and fusion, input into a containing In the multilayer perceptron decoder, a score is calculated for each optional step, and a probability distribution of step selection is generated using a softmax function. Then, the process actions are obtained through sampling. The value range is [2, 4]. For the specific selected process steps.

[0076] S2.1.3. Resource Allocation Decision: Resource Allocation Strategy Network Received status data and Related processing time vector and the current completion time vector of resources And in conjunction with the process, select the hidden state of the global graph output by the network. and with The corresponding node hidden state , through a containing The multilayer perceptron calculates a score for each available resource and generates a probability distribution for resource allocation using a softmax function. This allows for the sampling of resource allocation actions and the output of new contextual hidden states. .in, The value range is [2, 4]. For the specific resource action selected.

[0077] In step S2.1.1, the convolutional network encoder is used in each layer. The information transmission mechanism is as follows:

[0078] .

[0079] In the process selection decision in step S2.1.2, the multilayer perceptron decoder uses an additive attention mechanism before calculating the score. The calculation method is as follows:

[0080] .

[0081] in, For the first Attention score for each candidate process; For the query vector, the hidden states of the global graph are used. The context-hidden state of the previous decision step It is pieced together; For the first The hidden state of each candidate process is the hidden state of the candidate process. Part of; and All are learnable weight matrices. These are learnable weight vectors. This is a vector transpose operation.

[0082] The policy loss function of the PPO algorithm in step S2.2 Defined as:

[0083] .

[0084] in, The loss of the agent policy with pruning objective is expressed as follows: ,and Importance sampling ratio, This is the clipping factor, with a value range of [0.1, 0.3]. It is a value function loss; It is the policy entropy regularization term; and These are all weighting coefficients for the loss term, with values ​​ranging from [0.01, 1.0]. The advantage function estimate, calculated using the generalized advantage estimation (GAE) method, is used to evaluate the dominance in state data. Next action Advantages compared to average movements. This is the old strategy used for collecting samples; This is a clipping function that restricts the value of the first parameter to a closed interval defined by the last two parameters.

[0085] In steps S2.1 and S2.3, the status data of the aircraft sortie dispatching environment. It is a composite entity, consisting of: a graph adjacency matrix representing inter-process constraints and machine occupancy constraints. A matrix containing the eigenvectors of each process node. The feature vector includes at least the lower bound of the estimated completion time of the process and a completion status marker; and a list of candidate actions indicating the set of currently schedulable processes. A binary mask matrix that restricts process selection. A vector containing the current completion times of each maintenance resource. A matrix containing processing time information for all processes across all available machine resources. And a basis and The generated binary mask matrix used to restrict resource allocation choices .

[0086] The method further includes a periodic performance monitoring step:

[0087] After each training iteration, the average maximum completion time obtained for the current training instance is used. Policy loss function and value function loss The data is written to a logging system implemented using TensorBoard, which is used to monitor the performance convergence and loss change trends during the training process in real time, so as to adjust the hyperparameters and diagnose training problems.

[0088] The main objective of this application is to overcome the shortcomings of existing technologies in handling aircraft sortie scheduling problems, such as large decision space, difficulty in effective learning, and insufficient utilization of problem structure information. This application provides a method for generating aircraft sortie scheduling strategies based on hierarchical graph attention networks. This method effectively decomposes the complex scheduling decision problem through an innovative network structure and decision-making process, and deeply utilizes the modeling capabilities of graph neural networks to understand problem constraints, thereby achieving efficient and high-quality scheduling strategy learning.

[0089] The core of this application's technical solution lies in constructing a hierarchical reinforcement learning decision-making framework. This framework decomposes the originally complex one-step decision into two hierarchical, interconnected sequential sub-decisions. First, the system initializes a reinforcement learning agent containing a hierarchical policy network, which consists of a top-level process selection policy network and a bottom-level resource allocation policy network. During training, the agent interacts with a simulated aircraft sortie scheduling environment. At each decision moment, the top-level process selection policy network is activated first. It uses a graph convolutional network encoder to process the graph structure representation of the current scheduling state, which includes the process dependencies and resource occupancy relationships between all processes. The graph convolutional network can aggregate information along the edges of the graph, thereby generating a deep feature representation for each process node that includes its neighborhood structure information. Subsequently, an attention-based decoder integrates the features of the global graph, the contextual information of historical decisions, and the features of all currently executable processes to intelligently evaluate and select the process that should be scheduled most at the moment.

[0090] Once the top-level network determines the scheduling tasks, the bottom-level resource allocation strategy network takes over the decision-making power. It receives information about the selected tasks, combines the processing time of the task across different maintenance resources, the current load of each resource, and the contextual features transmitted from the upper-level network to evaluate the merits of allocating each available resource to the task, and ultimately makes a resource allocation decision. This hierarchical decision-making process not only significantly reduces the decision space at each step but also makes the decision logic clearer. The entire agent is trained using an advanced proximal policy optimization algorithm, learning and updating parameters on a large amount of interaction trajectory data. Furthermore, this method includes periodic model validation and an optimal model saving mechanism to ensure that the final scheduling strategy model performs best on the validation set. This strategy generation method, which hierarchically manages decision-making, utilizes graph networks for deep feature extraction, and combines attention mechanisms for intelligent focusing, can effectively learn high-performance aircraft dispatch scheduling schemes.

[0091] In this embodiment, the execution flow of the entire method follows a standard reinforcement learning training paradigm. First, system initialization is performed, during which a reinforcement learning agent is created. Its core is a parameterized hierarchical policy network, which includes a top-level network for selecting operations and a bottom-level network for allocating resources, both composed of deep neural networks. Simultaneously, the system loads pre-prepared aircraft sortie scheduling problem instances as training and validation datasets. These instances define the support task flow for multiple aircraft, the available resources for each operation, and their processing times.

[0092] After training begins, the system enters a macroscopic iterative loop. In each iteration, the agent interacts with a scheduling problem instance extracted from the training dataset, constituting a complete scheduling cycle. At the beginning of a scheduling cycle, the environment is reset to its initial state, i.e., all processes have not started. Subsequently, at each decision time step, the agent begins to make sequential decisions.

[0093] The first stage of decision-making is handled by the top-level process selection strategy network. This network receives the complete state of the current scheduling environment, which is carefully designed as a combination of multiple components, including a graph adjacency matrix describing the sequence and resource occupancy relationships of all processes, a node feature matrix recording the dynamic characteristics of each process, and a series of masks indicating which processes and resources are currently available. The graph convolutional network encoder, acting as the front end of the process selection network, processes this graph structure input. Through multi-layer information propagation and aggregation, it computes a high-dimensional hidden state representation for each process node and aggregates them into a global graph representation representing the current state of the entire scheduling graph. Next, the decoder uses an additive attention mechanism, taking the global graph representation and the contextual information from the previous decision step as queries, to focus on the hidden states of all currently executable candidate processes. It evaluates the scheduling priority of each candidate process by calculating attention weights and finally outputs a probability distribution. The agent then samples and selects a process to be scheduled based on this distribution.

[0094] The second phase of decision-making then begins, executed by the underlying resource allocation strategy network. At this stage, the input focuses on the process selected in the previous step. The network receives the processing time of this process across all available machine resources, the current estimated completion time of each machine resource, and relevant graph feature information from the top-level network. Using a multilayer perceptron, the network scores the expected effect of allocating each available resource to this process, generating a probability distribution for resource selection. The agent samples this distribution to determine the final machine resources allocated to the process. This complete (process, resource) pair constitutes a complete action. Simultaneously, the resource allocation network generates a new context state, which is passed to the process selection network in the next decision step.

[0095] After an action is executed, the scheduling environment updates its internal state, calculates the start and end times of the scheduled process, and returns a scalar reward value based on a preset reward function. This reward value is typically related to the scheduling objective (such as the reduction in total completion time). All information regarding the state, actions, and rewards throughout the decision-making process is fully recorded and stored in the experience replay memory.

[0096] Once data collection for one or more scheduling cycles is complete, the policy network update phase begins. The system employs the Proximal Policy Optimization (PPO) algorithm, sampling data from the experience replay memory and calculating a composite loss function that integrates policy boosting, value function fitting, and exploration incentives. This algorithm evaluates the merits of each action by calculating the advantage function and uses importance sampling and pruning techniques to ensure the stability of policy updates. Then, all learnable parameters of the hierarchical policy network are updated synchronously using backpropagation and gradient descent algorithms.

[0097] To ensure the effectiveness of training and the generalization ability of the model (scheduling policy network), the system periodically pauses training and performs performance tests on independent validation datasets using the currently learned policy network, calculating key performance indicators such as average maximum completion time. If the performance of the current model surpasses all historically saved best models, the system saves the network parameters of the current model as the new optimal model. The entire training process continues for tens or even hundreds of thousands of iterations until the model performance converges. Furthermore, throughout the training process, key performance indicators and loss function values ​​are recorded and visualized in real time, facilitating monitoring, diagnosis, and adjustments by researchers. Through these detailed implementation steps, this application ultimately generates an intelligent policy model (i.e., a scheduling policy network) capable of quickly solving aircraft sorting problems and providing high-quality scheduling solutions.

[0098] In one exemplary embodiment, such as Figure 2 As shown, an aircraft sortie scheduling strategy generation device is provided, comprising:

[0099] The initialization module is used to initialize the agent. The agent includes a hierarchical policy network and experience replay memory.

[0100] The interaction module is used to sequentially interact with the agent and the scheduling environment at each decision step based on the dataset within any scheduling cycle, and to store the determined trajectory data in the experience playback memory. The dataset includes: state data.

[0101] The update module is used to calculate the policy loss function based on the trajectory data stored in memory after the sequential interaction of one or more scheduling cycles is completed, and to update the parameters of the hierarchical policy network using the near-end policy optimization algorithm and the gradient descent method, so as to obtain the updated hierarchical policy network.

[0102] The performance verification module is used to solve for instances of the preset verification dataset based on the updated hierarchical policy network at preset verification intervals to determine performance metrics, including: average maximum completion time.

[0103] The scheduling module is used to determine scheduling information based on the scheduling policy network and the dataset. The scheduling information includes scheduling procedures and maintenance resources. The scheduling policy network is determined based on the optimal parameters and the updated hierarchical policy network. The optimal parameters are determined by comparing the performance indicators with the historical best performance indicators.

[0104] In one exemplary embodiment, a computer device is provided, which may be a server or a terminal, and its internal structure diagram may be as follows. Figure 3 As shown, this computer device includes a processor, memory, input / output interfaces (I / O), and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores data used to generate aircraft sortie scheduling strategies. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communication with external terminals via a network connection. When the computer program is executed by the processor, it implements the aircraft sortie scheduling strategy generation method.

[0105] Those skilled in the art will understand that Figure 3 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0106] In one exemplary embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0107] In one exemplary embodiment, a computer-readable storage medium is provided storing a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0108] In one exemplary embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0109] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0110] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM).

[0111] The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0112] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0113] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method for generating aircraft sortie scheduling strategies, characterized in that, include: Initialize the intelligent agent; The agent includes: a hierarchical policy network and an experience replay memory; Within any scheduling cycle, based on the dataset, the agent and the scheduling environment perform sequential interactions at each decision step, and the determined trajectory data is stored in the experience replay memory; the dataset includes: state data; After the sequential interaction of one or more scheduling cycles is completed, the corresponding trajectory data stored in the memory is replayed based on the experience. The near-end policy optimization algorithm is used to calculate the policy loss function, and the gradient descent method is used to update the parameters of the hierarchical policy network to obtain the updated hierarchical policy network. Using a preset verification period as the interval, the preset verification dataset is solved for instances based on the updated hierarchical policy network to determine performance indicators; the performance indicators include: average maximum completion time. Based on the scheduling policy network, scheduling information is determined according to the dataset; the scheduling information includes: scheduling procedures and maintenance resources; the scheduling policy network is determined based on the optimal parameters and the updated hierarchical policy network; the optimal parameters are determined by comparing the performance indicators with the historical best performance indicators.

2. The aircraft sortie scheduling strategy generation method according to claim 1, characterized in that, The hierarchical strategy network includes a process selection strategy network and a resource allocation strategy network; the process selection strategy network includes a graph convolutional network encoder and a multilayer perceptron decoder; the resource allocation strategy network includes a multilayer perceptron.

3. The aircraft sortie scheduling strategy generation method according to claim 2, characterized in that, Within any scheduling cycle, based on the dataset, the agent and the scheduling environment perform sequential interactions at each decision step, and the determined trajectory data is stored in the experience replay memory, specifically including: Based on the process node feature matrix and scheduling graph adjacency matrix contained in the process selection strategy network received state data, information aggregation and feature extraction are performed on each process node through a graph convolutional network encoder to obtain the hidden state of the process node and the hidden state of the global graph. The hidden state of the global graph, the context hidden state output by the resource allocation network of the previous decision step, and the candidate process hidden state extracted from the hidden state of the process node according to the current available process index are concatenated and fused, and then input into the decoder of the multilayer perceptron. The Softmax function is used to generate the probability distribution of process selection, and then the process action is sampled. Based on the resource allocation strategy network, the processing time vector related to the process and the current completion time vector of the resource are received from the state data. Combined with the hidden state of the global graph output by the process selection network and the hidden state of the node corresponding to the process, the probability distribution of resource allocation is generated by the multilayer perceptron and the Softmax function. Then, the resource allocation action is sampled and the context hidden state of the current decision step is output. Trajectory data is determined based on process actions, resource allocation actions, and status data, and the trajectory data is stored in the experience playback memory.

4. The aircraft sortie scheduling strategy generation method according to claim 3, characterized in that, The graph convolutional network encoder performs information aggregation and feature extraction for each process node. The mathematical expression for the corresponding information transmission mechanism is as follows: ； in, For the first The process node at the _ Hidden representation of layers; To modify the activation function of the linear unit; For layer normalization operation; For the first Multilayer perceptron; For the first The set of neighboring nodes of each process node; For the first The process node at the _ Hidden representation of layers; For the first The process node at the _ Hidden representation of layers.

5. The aircraft sortie scheduling strategy generation method according to claim 1, characterized in that, The method for generating aircraft sortie scheduling strategies also includes: Periodic performance monitoring is performed during the scheduling period; wherein, performance monitoring logs are determined based on the performance indicators and the policy loss function to monitor performance convergence and loss change trends in real time.

6. The aircraft sortie scheduling strategy generation method according to claim 1, characterized in that, The expression for the policy loss function is: ； in, Let the policy loss function be used. Loss for proxy strategies with pruning objectives; It is a value function loss; This is the policy entropy regularization term; and These are all weighting coefficients for the loss term; This is status data; Expected based on experience; These are the parameters for the hierarchical strategy network; For the reason The resulting hierarchical scheduling strategy.

7. An aircraft sortie scheduling strategy generation device, characterized in that, include: An initialization module is used to initialize the agent; the agent includes: a hierarchical policy network and an experience replay memory; An interaction module is used to perform sequential interaction between the agent and the scheduling environment at each decision step based on the dataset within any scheduling cycle, and to store the determined trajectory data into the experience playback memory; the dataset includes: state data; The update module is used to calculate the policy loss function based on the trajectory data stored in the corresponding memory according to the experience replay memory after the sequential interaction of one or more scheduling cycles is completed, and to update the parameters of the hierarchical policy network using the near-end policy optimization algorithm and the gradient descent method to obtain the updated hierarchical policy network. The performance verification module is used to perform instance solving on the preset verification dataset based on the updated hierarchical policy network at preset verification intervals to determine performance indicators; the performance indicators include: average maximum completion time. The scheduling module is used to determine scheduling information based on a scheduling policy network and a dataset. The scheduling information includes scheduling procedures and maintenance resources. The scheduling policy network is determined based on optimal parameters and an updated hierarchical policy network. The optimal parameters are determined by comparing the performance indicators with historical best performance indicators.

8. A computer device, comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor executes the computer program to implement the aircraft sortie scheduling strategy generation method according to any one of claims 1-6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the aircraft sortie scheduling strategy generation method as described in any one of claims 1-6.

10. A computer program product, comprising a computer program, characterized in that, When executed by a processor, the computer program implements the aircraft sortie scheduling strategy generation method as described in any one of claims 1-6.

Images