Multi-target aircraft dynamic scheduling method based on entropy weight method and deep reinforcement learning

Abstract

The invention relates to a multi-target aircraft dynamic scheduling method based on an entropy weight method and deep reinforcement learning, and the method comprises the steps: obtaining a to-be-maintained aircraft set, building a heterogeneous graph model based on a maintenance project set and a maintenance aircraft position set of each aircraft, carrying out the extraction of graph features through an encoder, carrying out the feature aggregation of the extracted nodes, and obtaining a multi-target aircraft dynamic scheduling model; aggregated feature vectors are obtained; obtaining a multi-target aircraft dynamic scheduling strategy based on the SAC network obtained by reinforcement learning training; a multi-target aircraft dynamic scheduling strategy is obtained by combining the SAC network obtained by reinforcement learning training, and in the training method of the SAC network, based on an entropy weight method, an aircraft maintenance aircraft position scheduling problem is converted into a Markov decision process, so that a multi-target aircraft maintenance aircraft position scheduling strategy can be obtained based on the obtained dynamic weight. While the flexibility of the SAC algorithm is brought into full play, it is guaranteed that multi-target balance and optimal action selection are balanced, dynamic scheduling is reasonably dealt with, and efficient coordination of multi-target optimization and real-time response of dynamic events under complex constraints are achieved.

Description

Technical Field

[0001] The present invention relates to the technical field of multi-objective aircraft scheduling, and in particular to a multi-objective aircraft dynamic scheduling method based on entropy weight method and deep reinforcement learning. Background Art

[0002] With the continued growth of global air traffic, aircraft maintenance scheduling, a core component of ensuring flight safety and operational efficiency, is no longer simply a single-objective scheduling optimization problem. Instead, it faces the dual challenges of multi-objective optimization and dynamic event response. Modern aircraft maintenance scheduling must simultaneously meet multiple objectives, such as minimizing maintenance delays, balancing the load on aircraft stand resources, and minimizing scheduling difficulty. Furthermore, it must respond in real time to dynamic events such as aircraft stand failures and the insertion of emergency maintenance tasks. Optimized maintenance scheduling strategies can significantly reduce flight delays and minimize the waste of maintenance resources, playing a significant role in improving the economic benefits and service quality of airlines.

[0003] Existing aircraft scheduling methods are mainly divided into three categories: the first is the precise algorithm based on mathematical programming, which solves the optimal solution by establishing a mixed integer programming model, but faces the problem of computational explosion in large-scale problems. The second is the heuristic algorithm (such as genetic algorithm, simulated annealing) which optimizes the target through local search strategies, but it is easy to fall into local optimality and lacks theoretical convergence guarantee. The third is the rule-based method that relies on expert experience to formulate scheduling strategies, but it is difficult to adapt to complex constraints and dynamic environmental changes. In recent years, deep learning-based methods have shown significant advantages in the field of scheduling. However, existing research has mostly focused on single-objective optimization and relied on fixed-weight linear combinations to deal with multi-objective problems, making it difficult to dynamically adapt to changes in target priorities. In addition, when dynamic events are triggered, existing rescheduling strategies may cause response delays due to global recalculation, or cause conflicts and imbalances between targets due to local adjustments. Summary of the Invention

[0004] The technical problem to be solved by this application is to provide a multi-objective aircraft dynamic scheduling method based on entropy weight method and deep reinforcement learning, which has the characteristics of being able to achieve efficient coordination of multi-objective optimization under complex constraints and real-time response to dynamic events.

[0005] In one embodiment, a multi-objective aircraft dynamic scheduling method based on entropy weight method and deep reinforcement learning is provided, including: Obtain a set of aircraft to be repaired, and build a heterogeneous graph model based on the set of maintenance items and maintenance bays for each aircraft; Based on the heterogeneous graph model, an encoder is used to extract graph features, and the extracted nodes are subjected to feature aggregation to obtain an aggregated feature vector; The SAC network trained by reinforcement learning obtains a multi-objective aircraft dynamic scheduling strategy; The training method of the SAC network includes: Based on the entropy weight method, the aircraft maintenance bay scheduling problem is transformed into a Markov decision process to implement MDP modeling; Based on multiple preset scheduling objectives, a SAC network architecture of a multi-critic Q network is constructed in combination with the Markov decision process; the multi-critic Q network corresponds one-to-one to the multiple scheduling objectives; Initialize the actor network and multi-critic Q network of the SAC network, and initialize the experience revisit pool and historical data buffer of the MDP; Perform environmental interactions at multiple time steps to collect multiple sample data and multiple historical data. Each sample parameter includes all state vectors in the input environment state, the obtained random actions, and the reward value and joint reward value corresponding to each scheduling target. Each historical data includes the reward value in the sample data obtained at the corresponding time step and the dynamic weight of each scheduling target calculated based on the reward value. Batch sampling the plurality of sample parameters and dynamic weights, inputting the sample parameters into the actor network and the multi-critic Q network, and inputting the sampled dynamic weights into the entropy parameter network; Based on the sampled data, a loss function is constructed, and iterative training is performed to update the entropy parameter network, actor network, and multiple critic Q networks.

[0006] The beneficial effects of the present invention are: The SAC network obtained by reinforcement learning training is combined to obtain a multi-objective aircraft dynamic scheduling strategy. In the training method of the SAC network, based on the entropy weight method, the aircraft maintenance stand scheduling problem is converted into a Markov decision process to realize MDP modeling. Therefore, based on the obtained dynamic weights, the flexibility of the SAC algorithm can be fully utilized while ensuring the balance of multiple objectives and the optimal action selection, and reasonably responding to dynamic scheduling, realizing efficient coordination of multi-objective optimization under complex constraints and real-time response to dynamic events. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Figure 1 This is a flowchart of a multi-objective aircraft dynamic scheduling method according to an embodiment of the present application; Figure 2 This is a flow chart of a training method for a SAC network according to an embodiment of the present application; Figure 3 This is a schematic diagram of the SAC network architecture and training process of an embodiment of the present application; Figure 4 This is a schematic diagram of the structure and iterative update of a multi-critic Q network in one embodiment of the present application; Figure 5 It is a schematic diagram of the iterative update process of the entropy parameter network according to an embodiment of the present application. DETAILED DESCRIPTION

[0008] The present invention will be further described in detail below by means of specific embodiments in conjunction with the accompanying drawings. Similar elements in different embodiments are numbered with associated similar elements. In the following embodiments, many detailed descriptions are provided to enable the present application to be better understood. However, those skilled in the art will readily appreciate that some of the features may be omitted in different circumstances, or may be replaced by other elements, materials, or methods. In some cases, some operations related to the present application are not shown or described in the specification. This is to avoid the core portion of the present application being overwhelmed by excessive descriptions, and for those skilled in the art, it is not necessary to describe these related operations in detail. They will fully understand the related operations based on the description in the specification and the general technical knowledge in the art.

[0009] In addition, the features, operations, or characteristics described in the specification may be combined in any appropriate manner to form various embodiments. Furthermore, the steps or actions in the method description may be reordered or adjusted in a manner readily apparent to those skilled in the art. Therefore, the various sequences in the specification and drawings are provided solely for the purpose of clearly describing a particular embodiment and are not intended to be mandatory, unless otherwise specified.

[0010] Component numbers used herein, such as "first" and "second," are used solely to distinguish the components being described and do not convey any sequential or technical meaning. References to "connection" and "coupling" herein, unless otherwise specified, include both direct and indirect connections (couplings).

[0011] To facilitate the description of the inventive concept of this application, the target optimization technology is briefly described below.

[0012] The SAC (Soft Actor-Critic) algorithm is a reinforcement learning-based policy optimization method. Its core lies in combining a maximum entropy framework with a dual-critic network. By maximizing the joint objective of expected cumulative reward and policy entropy, it enhances policy diversity while ensuring decision stability, demonstrating excellent performance in areas such as aircraft maintenance resource scheduling. However, the SAC algorithm only uses greedy exploration to ensure policy diversity and avoid convergence to local optima. Therefore, it cannot fully consider the selection of appropriate actions in multi-objective optimization. Therefore, how to fully utilize the flexibility of the SAC algorithm while ensuring multi-objective balance and optimal action selection, and properly addressing dynamic scheduling, has become a major challenge in improving this method.

[0013] In view of this, the present application proposes a multi-objective aircraft dynamic scheduling method based on the entropy weight method and deep reinforcement learning. In this method, the SAC network obtained by reinforcement learning training is combined to obtain a multi-objective aircraft dynamic scheduling strategy. In the training method of the SAC network, based on the entropy weight method, the aircraft maintenance stand scheduling problem is converted into a Markov decision process to realize MDP modeling. Therefore, based on the obtained dynamic weights, the flexibility of the SAC algorithm can be fully utilized while ensuring the balance of multiple objectives and the optimal action selection, and reasonably responding to dynamic scheduling, so as to achieve efficient coordination of multi-objective optimization under complex constraints and real-time response to dynamic events.

[0014] Please refer to Figure 1 In an embodiment of the present application, a multi-objective aircraft dynamic scheduling method based on entropy weight method and deep reinforcement learning is provided, comprising: Step S10: Obtain a set of aircraft to be maintained, including a set of maintenance items and a set of maintenance bays for each aircraft, and construct a heterogeneous graph model.

[0015] In one embodiment, the set of aircraft to be repaired can be represented as , where A represents the set of aircraft to be repaired, i represents the index of the aircraft to be repaired, represents the i-th aircraft to be repaired, I represents the total number of aircraft to be repaired, 1≤i≤I, then the maintenance item set for any aircraft can be expressed as ,in, represents the set of maintenance items for the i-th aircraft to be maintained, including completed maintenance items and unfinished maintenance items, j represents the index of the maintenance item, represents the total number of maintenance items for the i-th aircraft to be maintained, 1≤j≤ The maintenance stand set of a maintenance airport can be expressed as , where B represents the set of maintenance bays, represents the kth maintenance bay, k represents the index of the maintenance bay, K represents the total number of maintenance bays, and 1≤k≤K.

[0016] Then step S10 may include: adopting heterogeneous graph modeling, the maintenance project node is , , O represents the maintenance project node set; the maintenance station node is , , B represents the set of maintenance positions, k represents the index of the maintenance position; undirected arc Indicates the matching relationship between OB nodes. Assigned to a maintenance stand Afterwards, undirected arc is retained; the connecting arc is , Represents a set of connected arcs; connected arcs Used to describe the processing order and priority constraints of adjacent maintenance items.

[0017] In one embodiment, the scheduling objectives include minimizing delay time, balancing aircraft loads, and minimizing scheduling distances. The original feature vector of the maintenance project node includes scheduling status, maintenance duration, number of selectable maintenance aircraft positions, expected start time, number of remaining maintenance items, and deadline. The original feature vector of the maintenance aircraft position node includes idle time, number of repairable items, utilization rate, aircraft position load, and tail dock aircraft position coordinates.

[0018] In step S20 , based on the heterogeneous graph model, an encoder is used to extract graph features, and the extracted nodes are subjected to feature aggregation to obtain an aggregated feature vector.

[0019] In one embodiment, step S20 may include: Step S201 : performing original feature linear transformation on original feature vectors of the maintenance project node and the maintenance machine position node to map the original features to a higher-dimensional feature space, thereby obtaining two corresponding high-dimensional feature vectors.

[0020] In one embodiment, the high-dimensional feature vector of any maintenance project node and the high-dimensional feature vector of any maintenance position node can be expressed as: in, A high-dimensional feature vector representing a maintenance project node, A high-dimensional feature vector representing a maintenance bay node; The weight matrix representing the maintenance project node is obtained through learning; The weight matrix representing the maintenance station node is obtained through learning; represents the original feature vector of the maintenance project node, Represents the original feature vector of the maintenance bay node.

[0021] Those skilled in the art will appreciate that, during the model training process, the above two weight matrices are learnable parameters that can be continuously updated through the back-propagation algorithm and the optimizer (e.g., stochastic gradient descent) to minimize the loss function of the model and thus be learned.

[0022] In step S202 , the two high-dimensional feature vectors are respectively normalized and then residually connected to obtain a first maintenance item feature vector and a first maintenance position feature vector.

[0023] Normalization can bring different features into the same scale and improve stability.

[0024] In one embodiment, in step S202, any first maintenance item feature vector and first maintenance position feature vector are obtained, which can be expressed as: in, Represents the normalized result of the high-dimensional feature vector of the maintenance project node, Represents the normalized result of the high-dimensional feature vector of the maintenance station node; Represents the average value of the high-dimensional feature vector of each maintenance project node, Represents the average value of the high-dimensional feature vector of each maintenance station node; Represents the standard deviation of the high-dimensional feature vector of each maintenance project node, Represents the standard deviation of the high-dimensional feature vector of each maintenance station node; represents the first maintenance item feature vector, Represents the second maintenance item feature vector.

[0025] In an embodiment of the present application, the high-dimensional feature vectors of the maintenance project nodes and the maintenance machine position nodes are standardized and then connected using residual connections. This can retain the original feature information, avoid the loss of important information during the feature processing process, and help the model learn richer feature representations.

[0026] In step S203, the first maintenance project feature vector and the first maintenance position feature vector are mapped to the query, key, and value representation spaces respectively, and the cross-attention weights of the maintenance project node and the maintenance position node are calculated respectively using the cross-attention mechanism.

[0027] Query (query vector), Key (key vector) and Value (value vector) representation space are the core concepts in the attention mechanism. Query is a vector used to find or focus on specific information in the data. It can be imagined as a question, and the model looks for relevant answers in the data based on this question. Key corresponds to each element in the data, and its function is to match the Query to determine the relevance of each element to the Query. Value also corresponds to each element in the data and contains the actual information of the element. After determining the relevance of each element to the Query, these relevances are used to perform a weighted summation on the Value to obtain the output. These vectors are used to calculate the attention weights in the subsequent cross-attention mechanism to capture the dependencies between nodes. Then in step S203, it can be expressed as: in, 、 and Represent the query vector, key vector and value vector of the maintenance project node respectively, 、 and Represent the query vector, key vector and value vector of the maintenance bay respectively; 、 and They represent the query weight matrix, key weight matrix and value weight matrix of the maintenance project node respectively; 、 and They represent the query weight matrix, key weight matrix, and value weight matrix of the maintenance station node respectively; softmax represents normalization processing, T represents transposition, represents the cross attention weight of the maintenance project node, Represents the cross attention weight of the maintenance station node.

[0028] In step S204, the node function is updated using the value vector representation and cross-attention weight of the maintenance project node and the maintenance position node, and a graph convolution operation is performed followed by dimensionality reduction and simplification. After the dimensionality reduction and simplification, a second maintenance project feature vector and a second maintenance position feature vector are obtained.

[0029] In one embodiment, in step S204, the second maintenance item feature vector of any maintenance item node and the second maintenance position feature vector of any maintenance position node are obtained, which can be expressed as: in, and They represent the maintenance project node function representation and the maintenance position node function representation respectively. and They represent the graph convolution operation results of the maintenance project node and the maintenance position node respectively; ReLU represents the activation function, and Represent the degree matrices of maintenance project nodes and maintenance bay nodes respectively; and Represent the adjacency matrix of maintenance project node and maintenance station node respectively, Represents the weight matrix of graph convolution; and represent the second maintenance item feature vector and the second maintenance item feature vector respectively, and They represent the deviations of maintenance project nodes and maintenance stand nodes respectively; and Represent respectively.

[0030] The node function combines the maintenance project node and the corresponding maintenance machine node information through attention-based feature aggregation. Updating the node function allows the node to integrate information from other related nodes and enhance the understanding of the relationship between nodes. The degree matrix is ​​a diagonal matrix used to record the degree of each node. The elements on its diagonal represent the degree of the node, that is, the number of edges connected to the node. The adjacency matrix is ​​used to describe the connection relationship between nodes in the graph. The weight matrix of the graph convolution is the learnable parameter matrix of the graph convolution layer. Its function is similar to the weight matrix in traditional neural networks. It is used to perform linear transformations on the input features. By continuously adjusting the value of the weight matrix of the graph convolution, the model can learn the complex relationships between the nodes in the graph and mine the global structural information of the graph.

[0031] The high-dimensional vectors obtained through graph convolution contain redundant information, increasing computational complexity. Therefore, in one embodiment, an MLP (Multi-Layer Perceptron) is introduced for dimensionality reduction and simplification to reduce computational complexity while retaining key information. In one embodiment, the deviations between maintenance project nodes and maintenance station nodes are randomly sampled from a uniform or normal distribution as initial deviations.

[0032] In step S205, the second maintenance item feature vector and the second maintenance position feature vector are first pooled, and then the cross-attention mechanism is applied to the pooling result to splice it with the second maintenance item feature vector, the second maintenance position feature vector and the corresponding high-dimensional feature vector to obtain an aggregated feature vector.

[0033] In one embodiment, step S205 can be expressed as: in, and They represent the results of pooling the second maintenance item feature vector and the second maintenance position feature vector, Represents the aggregated feature vector.

[0034] For the simplified results after dimensionality reduction, pooling can preserve the main features while reducing the data size, lowering the computational effort, and preventing overfitting to a certain extent. The aggregated feature vector integrates feature information from different stages and levels.

[0035] In this application, through the diffusion process of the above-mentioned encoder, the long-distance dependencies between nodes can be captured, and the local information of the nodes can be gradually diffused to the entire graph, so as to better extract the global structural features of the heterogeneous graph model.

[0036] Step S30: obtaining a multi-objective aircraft dynamic scheduling strategy based on the SAC network trained by reinforcement learning.

[0037] In one embodiment, please refer to Figure 2 and Figure 3 , the training methods of SAC network include: In step S100 , based on the entropy weight method, the aircraft maintenance bay scheduling problem is transformed into a Markov decision process to implement MDP modeling.

[0038] MDP modeling can clearly define the key elements of the problem for each SAC network, allowing the SAC network to accurately understand the environment, know what actions can be taken in different states, and what results the actions will bring, laying the foundation for subsequent learning and decision-making. At the same time, the theoretical framework of MDP provides a certain guarantee for the convergence of the SAC network.

[0039] In one embodiment, step S100 may include designing Markov decision process elements for the aircraft maintenance bay scheduling problem and setting multiple scheduling objectives. The elements include a state space, an action space, state transitions, and a reward space. The reward space includes reward values ​​for each scheduling objective and a joint reward. The joint reward is derived based on the reward values ​​and dynamic weights of each scheduling objective. The dynamic weights are calculated using an entropy weighting method.

[0040] In one embodiment, the multiple scheduling objectives include minimizing maintenance delay time, balancing maintenance stand load, and minimizing the distance traveled by scheduled aircraft. The state space can be expressed as: in, Represents the set of all state vectors and key data that the SAC agent (SAC network) may perceive in the environment; Represents the state vector set of the aircraft. The state vector of each aircraft includes the scheduling status, maintenance duration, number of available maintenance slots, expected start time, number of remaining maintenance items, maintenance deadline, and aircraft location coordinates; The state vector set representing all maintenance bays, where each bay's state vector includes whether it is idle or occupied, the number of repairable items, utilization rate, bay load, and bay location coordinates; Indicates the current time step; Represents the historical reward data stored in the historical data buffer. Each historical reward data includes the rewards corresponding to the three scheduling targets at that time step and their corresponding three dynamic weights.

[0041] The action space can be expressed as: in, represents the set of all actions that the SAC agent can perform in any state; a represents an action, which is a paired action for selecting a maintenance stand m for an aircraft to perform maintenance; M represents the set of maintenance stands, .

[0042] State transitions can be expressed as: in, Represents the system time at time step t, and the initial state is ; Indicates the next time step. Each time a scheduled maintenance item is paired with a maintenance station, the next time step is entered. , until the scheduling is completed; Bonus spaces include: in, Represents a collection of rewards; 、 and They represent the reward values ​​for minimizing maintenance delay time, balancing maintenance stand load, and minimizing the distance traveled by scheduled aircraft. The smaller the delay time, the smaller the penalty; the smaller the variance of maintenance stand load, the smaller the penalty; the smaller the sum of the distances of aircraft scheduling in all time steps, the smaller the penalty. Indicates joint reward; 、 and Respectively represent the preset first corresponding coefficient, second corresponding coefficient, and third corresponding coefficient, which are used to standardize and compare the three scheduling target reward values; i represents the index of the aircraft to be maintained, 1≤i≤n, and n represents the number of aircraft to be maintained; represents the maintenance completion time of the i-th aircraft, represents the maintenance deadline of the i-th aircraft; L represents the maintenance stand load, represents the variance of maintenance bay load; represents the sum of the dispatch distances of all aircraft at time step t, represents the sum of the dispatch distances of all aircraft in the previous time step of time step t, Indicates the total number of time steps, 1≤t≤ ; represents the scheduling distance of aircraft i to be repaired at time step t; 、 and They represent the dynamic weights of minimizing maintenance delay time, balancing maintenance stand load, and minimizing the movement distance of scheduled aircraft, respectively, and are calculated based on the entropy weight method at each time step.

[0043] The Entropy Weight Method is an objective weighting method widely used in comprehensive multi-indicator evaluations. First, based on historical data, an implementation example uses N historical scheduling plans and the three scheduling objectives mentioned above as an example to create an N×3 historical data matrix: Where X represents the historical data matrix, Indicates the mth scheduling target value under the nth scheduling scheme, 1≤n≤N, m=1, 2, 3. For standardization, we have: in, represents the normalized result of the mth scheduling target value under the nth scheduling scheme, represents the mth scheduling target value under the sth scheduling scheme. According to the constraints, the scheme with the maximum or minimum value under the same scheduling target is determined, and then the proportion of each value is calculated based on the standardized value, then: in, It represents the proportion of the mth scheduling target value under the nth scheduling scheme. Then the entropy value of any scheduling target value can be obtained, including: in, Represents the entropy value of the mth scheduling target value. Then the difference coefficient of any scheduling target can be calculated , which can be expressed as: The dynamic weight of any scheduling target can be calculated , which can be expressed as: Then the dynamic weights of the first scheduling target, the second scheduling target and the third scheduling target can be obtained respectively. 、 and .

[0044] Entropy is a measure of uncertainty. The smaller the information entropy, the greater the amount of information contained in the data, the more important this indicator is in the comprehensive evaluation, and the greater its weight; conversely, the larger the information entropy, the greater the uncertainty of the data, the smaller the amount of information contained, the lower the importance of this indicator, and the smaller its weight. In the present application, the dynamic weight of each scheduling target calculated by the entropy weight method is used as the dynamic weight of each scheduling target, so that the multi-scheduling target optimization scheduling takes into account the historical weights, and a more accurate and reasonable multi-scheduling target optimization scheduling scheme can be obtained.

[0045] The samples in the experience replay pool and the dynamic weights of each scheduling target calculated by the entropy weight method will be used in subsequent training.

[0046] Step S200: Based on the preset multiple scheduling objectives, a SAC network architecture of a multi-critic Q network is constructed in combination with a Markov decision process, wherein the multi-critic Q network corresponds to the multiple scheduling objectives in a one-to-one manner.

[0047] In the embodiment of the present application, the SAC network includes an actor network, an entropy parameter network and a multi-critic Q network. In combination with the above embodiment, when there are three scheduling targets, three critic Q networks are included, and the three critic Q networks correspond to the three scheduling targets one by one. Figure 3 and Figure 4In the specific embodiment of the present application, the multi-critic Q network includes a first critic Q network, a second critic Q network, and a third critic Q network. The first critic Q network corresponds to the first scheduling target (u=1, u is the index of the scheduling target), including the Q1 network and the first target network, the second critic Q network corresponds to the second scheduling target (u=2), including the Q2 network and the second target network, and the third critic Q network corresponds to the third scheduling target (u=3), including the Q3 network and the third target network. Step S300: Initialize the actor network and multi-critic Q network of the SAC network, and initialize the experience revisit pool and historical data buffer of the MDP.

[0048] In the specific embodiment of the present application, random network parameters are used 、 and Initialize three critic Q networks and use the network parameters 、 and Initialize the three target networks corresponding to the three critic Q networks, and = , = , = . Using random network parameters Initialize the actor network. Initialize the experience replay pool, set the historical data buffer size to the same as the number of samples in the experience replay pool, and initialize the historical buffer. Initially, the corresponding target weights are all set to 1 / 3, and the original entropy parameter is initialized to 0.01. The dynamic entropy parameter and the joint entropy parameter are initialized according to the initial weights and the original entropy parameter according to the preset update formula. The entropy parameter is used to encourage the agent to explore the environment and avoid prematurely falling into a local optimum. Initialize the soft update coefficient, which makes the target network parameter update smoother and reduces fluctuations and instability during training. Initialize the discount factor, which is used to weigh the importance of current rewards and future rewards.

[0049] Step S400 involves performing multiple time-step interactions with the environment to collect multiple sample parameters and multiple historical data. Each sample parameter includes all state vectors in the input environment state, the obtained random actions, and the reward and joint reward values ​​corresponding to each scheduling objective. Each historical data includes the reward value of the sample parameter obtained at the corresponding time step and the dynamic weight of each scheduling objective calculated based on the reward value.

[0050] After initialization, the actor network first interacts with the environment. For any time step, based on the current time step environment state (including the feature vector of aircraft characteristics and maintenance stand characteristics), the actor network takes a random action. The random action is to arbitrarily select a maintenance stand from the set of actions for the aircraft to be maintained. The initial random action selection probability is the untrained strategy. The reward value and joint reward value corresponding to each scheduling target are obtained. The reward value only considers the current time step, that is, the last time step is equal to the current time step. =t. Then perform dynamic entropy weight calculation and use the entropy weight method to calculate the dynamic weight of each adjustment target in this time step 、 and .

[0051] The sample parameters are stored in the experience replay pool, and the reward values ​​for all corresponding targets in the sample parameters and the dynamic weights of each scheduling target calculated based on the reward values ​​are stored in the historical data buffer. The next state is then entered, and the next state is still derived from the environment. After multiple interactions, the experience replay pool contains multiple sample parameters, each of which contains all state vectors in the environment state input after the interaction at that time step, the obtained random actions, and the reward values ​​and joint reward values ​​corresponding to each scheduling target. The historical data in the historical data buffer includes the three reward values ​​corresponding to each sample parameter and the dynamic weights of each target.

[0052] Step S500 , batch sampling is performed on a plurality of sample parameters and dynamic weights, and the sampled sample parameters are input into the actor network and the multi-critic Q network, and the sampled dynamic weights are input into the entropy parameter network.

[0053] Based on multiple sampling, the sample parameters can be input into the actor network and the multi-critic Q network, and the sampled dynamic weights can be input into the entropy parameter network and the multi-critic network for iterative training. Those skilled in the art will appreciate that the sample parameters collected in each sampling correspond to the dynamic weights in the history buffer.

[0054] Step S600: Based on the sampled data, a loss function is constructed, and iterative training is performed to update the entropy parameter network, the actor network, and multiple critic Q networks.

[0055] In one embodiment, step S600 includes: Step S6001: Obtain the action selection probability of the current time step and the action selection probability of the next time step based on the actor network.

[0056] Those skilled in the art will appreciate that based on the sample parameters obtained by the actor network, the action selection probability of the current time step and the action selection probability of the next time step can be obtained through the actor network, which will not be further elaborated here.

[0057] Step S6002: Based on each critic Q network, the corresponding action-value function of the current time step and the action-value function of the next time step are obtained.

[0058] Those skilled in the art will appreciate that based on the sample parameters obtained by the critic Q network, the action-value function of the current time step and the action-value function of the next time step can be obtained through the critic Q network, which will not be elaborated here.

[0059] Step S6003: Based on the entropy parameter network, obtain the joint entropy parameter and the dynamic entropy parameter of each scheduling target.

[0060] Those skilled in the art will appreciate that based on the dynamic weights obtained by the entropy parameter network, the joint entropy parameter and the dynamic entropy parameter of each scheduling target can be obtained through the entropy parameter network, which will not be elaborated here.

[0061] Step S6004: Based on the action-value function of the current time step, the action-value function of the next time step, and the action selection probability, a loss function of each critic Q network is constructed for back-propagation iterative training, and the parameters in the critic Q network are updated.

[0062] In one embodiment, step S6004 includes: For any critic Q network, based on its dynamic entropy parameter, reward value, action value function of the current time step and action value function of the next time step, the current time step target value of the target network of any critic Q network is calculated, which can be expressed as: in, represents the current time step target value of the u-th scheduling target, represents the reward value of the u-th scheduling target, represents the preset discount factor, Represents the next time step of the u-th scheduling target network The action value function is used to evaluate the action in a specific state Next action long-term benefits, represents the dynamic entropy parameter corresponding to the u-th scheduling target, Indicates the probability of action selection in the next time step, in a specific state Next action The probability of action selection.

[0063] Please refer to Figure 4 , then the first target value of the first target network corresponding to the first scheduling target can be obtained , the second target value of the second target network corresponding to the second scheduling target , the third target value of the third target network corresponding to the third scheduling target .

[0064] In one embodiment, =0.95.

[0065] The dynamic entropy parameter corresponding to the scheduling target can make the strategy more flexible in exploring the corresponding target direction and will not converge to the local optimum too early.

[0066] The loss function of the critic Q network is constructed based on the calculated current time step target value and the current time step action value function, which can be expressed as: in, represents the critic Q-network loss function for the u-th scheduling objective, represents the mathematical expectation, represents the current time step of the u-th scheduling target critic Q network The action value function is used to evaluate the action in a specific state Next action Please refer to Figure 4 , we can get the loss function of the first critic Q network corresponding to the first scheduling target , the loss function of the second critic Q network corresponding to the second scheduling target , the loss function of the third critic Q network corresponding to the third scheduling target .

[0067] In this way, the critic Q network loss function of the u-th scheduling target can be back-propagated to the critic Q network of the u-th scheduling target for iterative training, and the random network parameters ( 、 and ) to update.

[0068] Furthermore, based on the updated random network parameters, the target network parameters are updated ( 、 and ),include: in, represents the updated target network parameters of the u-th scheduling target critic Q network at the current time step, represents the updated random network parameters of the u-th scheduling target critic Q network at the current time step, represents the target network parameters of the u-th scheduling target critic Q network after the previous time step update, Indicates the preset weight value.

[0069] Step S6005: Based on the joint entropy parameter, the joint action value function and the action selection probability of the current time step, the loss function of the actor network is constructed for back-propagation iterative training, and the parameters in the actor network are updated.

[0070] In a specific embodiment, step S6005 includes: Based on the action selection probability and dynamic weight of the current time step, the joint action value parameters are obtained, including: in, represents the joint action-value function, represents the dynamic weight of the current time step of the u-th scheduling target; Based on the joint entropy parameter, the joint action value function and the action selection probability at the current time step, the loss function of the actor network is constructed, including: in, represents the loss function of the actor-family network, represents the joint entropy parameter, Indicates the probability of action selection at the current time step, in a specific state Next action The probability of action selection; Back-propagation iterative training is performed based on the loss function of the actor network, and the random network parameters in the actor network are updated.

[0071] Step S6006: Based on the action selection probability of the current time step, the original entropy parameter updated in the previous time step, and the entropy target value of the current time step, a loss function of the entropy parameter network is constructed for back propagation, and the parameters in the entropy parameter network are updated.

[0072] In one embodiment, step S6006 includes: The loss function of the entropy parameter network is constructed based on the action selection probability of the current time step, the original entropy parameter updated in the previous time step, and the entropy target value of the current time step, which can be expressed as: in, represents the loss function of the entropy parameter network, Represents the original entropy parameter after the previous time step update. For the first time step, it is the preset original entropy parameter; Represents the entropy target value of the current time step, based on the number of actions in the discrete action space of the current time step get.

[0073] Please refer to the process of updating the parameters in the entropy parameter network Figure 5 , the loss function of the entropy parameter network is minimized to update the original entropy parameters of the current time step. Based on the updated original entropy parameters of the current time step, the dynamic entropy parameters and the joint entropy parameters are updated, including: in, represents the dynamic entropy parameter of the u-th scheduling target after the current time step update, Represents the preset weight coefficient, Represents the updated joint entropy parameter at the current time step.

[0074] Based on the above process of updating the parameters in the entropy parameter network, it can be seen that if the entropy value Below entropy target , then the entropy parameters are increased, encouraging strategy exploration. Above entropy target , then the entropy parameters are reduced, which encourages the improvement of action value, that is, stabilizing actions and reducing exploration.

[0075] Step S6007: Based on the joint action value function, the joint entropy parameter and the action selection probability of the current time step, a joint state value function is constructed, and a joint state value threshold is set to determine whether the current iteration has reached convergence.

[0076] In one embodiment, a joint state value function is constructed based on the joint action value function, the joint entropy parameter, and the action selection probability at the current time step, which can be expressed as: in, Represents the joint state value of the current time step. The joint state value is evaluated by the user at a specific state The expectation of the future cumulative reward under the current measurement is followed. Based on the preset joint state value threshold, if the joint state value calculated based on the joint action value function obtained at the current time step, the joint entropy parameter, and the action selection probability at the current time step reaches the preset joint state value threshold, it means that the entire SAC network has converged and the iterative training process can be terminated.

[0077] Based on the above embodiment, the SAC network obtained by reinforcement learning training is combined to obtain a multi-objective aircraft dynamic scheduling strategy. In the training method of the SAC network, based on the entropy weight method, the aircraft maintenance stand scheduling problem is converted into a Markov decision process to implement MDP modeling. Therefore, based on the obtained dynamic weights, the flexibility of the SAC algorithm can be fully utilized while ensuring the balance of multiple objectives and the optimal action selection, and reasonably responding to dynamic scheduling, thereby achieving efficient coordination of multi-objective optimization under complex constraints and real-time response to dynamic events.

[0078] In one embodiment of the present application, a computer-readable storage medium is provided. A program is stored on the storage medium. The stored program includes a method that can be loaded by a processor and process any of the above embodiments.

[0079] Those skilled in the art will appreciate that all or part of the functions of the various methods in the above embodiments can be implemented by hardware or by computer program. When all or part of the functions in the above embodiments are implemented by computer program, the program can be stored in a computer-readable storage medium, and the storage medium can include: read-only memory, random access memory, disk, optical disk, hard disk, etc., and the program is executed by a computer to implement the above functions. For example, the program is stored in the memory of the device, and when the program in the memory is executed by the processor, all or part of the above functions can be implemented. In addition, when all or part of the functions in the above embodiments are implemented by computer program, the program can also be stored in a storage medium such as a server, another computer, disk, optical disk, flash disk or mobile hard disk, and saved in the memory of the local device by downloading or copying, or the system of the local device is updated. When the program in the memory is executed by the processor, all or part of the functions in the above embodiments can be implemented.

[0080] The above examples are used to illustrate the present invention, which are only used to help understand the present invention and are not intended to limit the present invention. Those skilled in the art can make several simple deductions, modifications or substitutions based on the concept of the present invention.

Claims

1. A multi-objective aircraft dynamic scheduling method based on entropy weight method and deep reinforcement learning, characterized in that: include: Obtain a set of aircraft to be repaired, and build a heterogeneous graph model based on the set of maintenance items and maintenance bays for each aircraft; Based on the heterogeneous graph model, an encoder is used to extract graph features, and the extracted nodes are subjected to feature aggregation to obtain an aggregated feature vector; The SAC network trained by reinforcement learning obtains a multi-objective aircraft dynamic scheduling strategy; The training method of the SAC network includes: Based on the entropy weight method, the aircraft maintenance bay scheduling problem is transformed into a Markov decision process to implement MDP modeling; Based on multiple preset scheduling objectives, a SAC network architecture of a multi-critic Q network is constructed in combination with the Markov decision process; the multi-critic Q network corresponds one-to-one to the multiple scheduling objectives; Initialize the actor network and multi-critic Q network of the SAC network, and initialize the experience revisit pool and historical data buffer of the MDP; Perform environmental interactions at multiple time steps to collect multiple sample data and multiple historical data. Each sample parameter includes all state vectors in the input environment state, the obtained random actions, and the reward value and joint reward value corresponding to each scheduling target. Each historical data includes the reward value in the sample data obtained at the corresponding time step and the dynamic weight of each scheduling target calculated based on the reward value. Batch sampling the plurality of sample parameters and dynamic weights, inputting the sample parameters into the actor network and the multi-critic Q network, and inputting the sampled dynamic weights into the entropy parameter network; Based on the sampled data, a loss function is constructed, and iterative training is performed to update the entropy parameter network, actor network, and multiple critic Q networks.

2. The multi-objective aircraft dynamic scheduling method according to claim 1, characterized in that: The process of obtaining a set of aircraft to be repaired, a set of repair items for each aircraft to be repaired, and a set of repair bays, and constructing a heterogeneous graph model includes: Using heterogeneous graph modeling, the maintenance project node is , , O represents the set of maintenance project nodes, i represents the index of the aircraft to be maintained, and j represents the index of the maintenance project; the maintenance stand node is , , B represents the set of maintenance stands, k represents the index of the maintenance stand; the undirected arc is , represents a set of undirected arcs, Indicates the matching relationship between OB nodes. Assigned to a maintenance stand Afterwards, undirected arc is retained; the connecting arc is , Represents a set of connected arcs; connected arcs It is used to describe the processing sequence and priority constraints of adjacent maintenance items. The original feature vector of the maintenance item node includes the scheduling status, maintenance duration, number of available maintenance bays, expected start time, number of remaining maintenance items, and deadline. The original feature vector of the maintenance bay node includes the idle time, number of repairable items, utilization rate, bay load, and tail dock bay position coordinates.

3. The multi-objective aircraft dynamic scheduling method according to claim 1, characterized in that: The method of extracting graph features using an encoder based on the heterogeneous graph model and aggregating features of the extracted nodes to obtain an aggregated feature vector includes: Performing original feature linear transformation on the original feature vectors of the maintenance project node and the maintenance position node to map the original features to a higher-dimensional feature space to obtain two corresponding high-dimensional feature vectors; The two high-dimensional feature vectors are respectively normalized and then residually connected to obtain a first maintenance item feature vector and a first maintenance position feature vector; Mapping the first maintenance item feature vector and the first maintenance position feature vector into query, key, and value representation spaces, respectively, and calculating the cross-attention weights of the maintenance item node and the maintenance position node respectively using a cross-attention mechanism; The node function is updated using the value vector representation and cross-attention weight of the maintenance project node and the maintenance position node, and a graph convolution operation is performed followed by dimensionality reduction and simplification to obtain the second maintenance project feature vector and the second maintenance position feature vector; The second maintenance item feature vector and the second maintenance position feature vector are first pooled respectively, and then the cross-attention mechanism is applied to the pooling results to splice them with the second maintenance item feature vector, the second maintenance position feature vector and the corresponding high-dimensional feature vector to obtain the aggregated feature vector.

4. The multi-objective aircraft dynamic scheduling method according to claim 1, characterized in that: The entropy weight method is used to convert the aircraft maintenance stand scheduling problem into a Markov decision process to implement MDP modeling, including: designing Markov decision process elements for the aircraft maintenance stand scheduling problem and setting multiple scheduling goals; the elements include state space, action space, state transition and reward space, the reward space includes reward values ​​and joint rewards for each scheduling goal, the joint reward is obtained based on the reward values ​​and dynamic weights of each scheduling goal, and the dynamic weights are calculated based on the entropy weight method.

5. The multi-objective aircraft dynamic scheduling method according to claim 4, characterized in that: The multiple scheduling objectives include minimizing maintenance delay time, balancing maintenance stand load, and minimizing scheduling aircraft movement distance; the state space includes: in, Represents the set of all state vectors and key data that the SAC agent may perceive in the environment; Represents the state vector set of the aircraft. The state vector of each aircraft includes the scheduling status, maintenance duration, number of available maintenance slots, expected start time, number of remaining maintenance items, maintenance deadline, and aircraft location coordinates; The state vector set representing all maintenance bays, where each bay's state vector includes whether it is idle or occupied, the number of repairable items, utilization rate, bay load, and bay location coordinates; Indicates the current time step; Represents the historical reward data stored in the historical data buffer. Each historical reward data includes the rewards corresponding to the three scheduling targets at that time step and their corresponding three dynamic weights; The action space consists of: in, represents the set of all actions that the SAC agent can perform in any state; represents an action, which is a pairing action for an aircraft to select maintenance stand m for maintenance; M represents the set of maintenance stands, ; State transfers include: in, Represents the system time at time step t, and the initial state is ; Indicates the next time step. Each time a scheduled maintenance item is paired with a maintenance station, the next time step is entered. , until the scheduling is completed; Bonus spaces include: in, Represents a collection of rewards; 、 and They represent the reward values ​​for minimizing maintenance delay time, balancing maintenance stand load, and minimizing the distance traveled by scheduled aircraft. The smaller the delay time, the smaller the penalty; the smaller the variance of maintenance stand load, the smaller the penalty; the smaller the sum of the distances of aircraft scheduling in all time steps, the smaller the penalty. Indicates joint reward; 、 and Respectively represent the preset first corresponding coefficient, second corresponding coefficient, and third corresponding coefficient, which are used to standardize and compare the three scheduling target reward values; i represents the index of the aircraft to be maintained, 1≤i≤n, and n represents the number of aircraft to be maintained; represents the maintenance completion time of the i-th aircraft, represents the maintenance deadline of the i-th aircraft; L represents the maintenance stand load, represents the variance of maintenance bay load; represents the sum of the dispatch distances of all aircraft at time step t, represents the sum of the dispatch distances of all aircraft in the previous time step of time step t, Indicates the total number of time steps, 1≤t≤ ; represents the scheduling distance of aircraft i to be repaired at time step t; 、 and They represent the dynamic weights of minimizing maintenance delay time, balancing maintenance stand load, and minimizing the movement distance of scheduled aircraft, respectively, and are calculated based on the entropy weight method at each time step.

6. The multi-objective aircraft dynamic scheduling method according to claim 1, characterized in that: The aforementioned environmental interaction at multiple time steps to realize data collection of multiple sample data and historical data includes: For any time step, the environment state is input into the actor network to obtain random actions, and the reward value and joint reward value corresponding to each scheduling target are obtained, and the dynamic weight of each scheduling target is calculated based on the entropy weight method combined with the reward value; the sample parameters are stored in the experience revisit pool, and the reward value in the sample parameters and the dynamic weight of each scheduling target are stored in the historical data buffer; in this way, after the input of the environment state for multiple time steps, an experience revisit pool storing multiple sample parameters and a historical data buffer with corresponding dynamic weights are obtained.

7. The multi-objective aircraft dynamic scheduling method according to claim 1, characterized in that: The above method is based on sampling data, constructing a loss function, performing iterative training, and updating the entropy parameter network, actor network, and multiple critic Q networks, including: Obtain the action selection probability of the current time step and the action selection probability of the next time step based on the actor network; Based on each critic Q network, obtain the corresponding action-value function of the current time step and the action-value function of the next time step; Based on the entropy parameter network, the joint entropy parameter and the dynamic entropy parameter of each scheduling target are obtained; Based on the action value function of the current time step, the action value function of the next time step and the action selection probability, the loss function of each critic Q network is constructed for back-propagation iterative training, and the parameters in the critic Q network are updated; Based on the joint entropy parameter, the joint action value function and the action selection probability of the current time step, the loss function of the actor network is constructed for back-propagation iterative training, and the parameters in the actor network are updated; Based on the action selection probability of the current time step, the original entropy parameter updated in the previous time step, and the entropy target value of the current time step, the loss function of the entropy parameter network is constructed for backpropagation, and the parameters in the entropy parameter network are updated; Based on the joint action value function, the joint entropy parameter and the action selection probability of the current time step, a joint state value function is constructed, and the joint state value function is set to determine whether the current iteration has reached convergence.

8. The multi-objective aircraft dynamic scheduling method according to claim 7, characterized in that: Based on the action value function of the current time step, the action value function of the next time step, and the action selection probability, the loss function of each critic Q network is constructed for back-propagation iterative training, and the parameters in the critic Q network are updated, including: For any critic Q network, based on its dynamic entropy parameter, reward value, action value function of the current time step and action value function of the next time step, the current time step target value of the target network of any critic Q network is calculated, including: in, represents the current time step target value of the u-th scheduling target, represents the reward value of the u-th scheduling target, represents the preset discount factor, Represents the next time step of the u-th scheduling target network The action value function is used to evaluate the action in a specific state Next action long-term benefits, represents the dynamic entropy parameter corresponding to the u-th scheduling target, Indicates the probability of action selection in the next time step, in a specific state Next action The probability of action selection; The loss function of the critic Q network is constructed based on the calculated current time step target value and the current time step action value function, including: in, represents the critic Q-network loss function for the u-th scheduling objective, represents the mathematical expectation, represents the current time step of the u-th scheduling target critic Q network The action value function is used to evaluate the action in a specific state Next action long-term benefits; For any critic Q network, perform back-propagation iterative training based on its loss function and update the random network parameters in any critic Q network; Based on the updated random network parameters, update the target network parameters, including: in, represents the updated target network parameters of the u-th scheduling target critic Q network at the current time step, represents the updated random network parameters of the u-th scheduling target critic Q network at the current time step, represents the target network parameters of the u-th scheduling target critic Q network after the previous time step update, Indicates the preset weight value.

9. The multi-objective aircraft dynamic scheduling method according to claim 8, characterized in that: The loss function of the actor network is constructed based on the joint entropy parameter, the joint action value function and the action selection probability of the current time step to perform back propagation iterative training and update the parameters in the actor network, including: Based on the action selection probability and dynamic weight of the current time step, the joint dynamic value parameters are obtained, including: in, represents the joint action-value function, represents the dynamic weight of the current time step of the u-th scheduling target; The loss function of the actor network is constructed based on the joint entropy parameter, the joint action value function and the action selection probability of the current time step, including: in, represents the loss function of the actor-family network, represents the joint entropy parameter, Indicates the probability of action selection at the current time step, in a specific state Next action The probability of action selection; Back-propagation iterative training is performed based on the loss function of the actor network, and the random network parameters in the actor network are updated.

10. The multi-objective aircraft dynamic scheduling method according to claim 9, characterized in that: The loss function of the entropy parameter network is constructed based on the action selection probability of the current time step, the original entropy parameter updated in the previous time step, and the entropy target value of the current time step for back propagation, and the parameters in the entropy parameter network are updated, including: The loss function of the entropy parameter network is constructed based on the action selection probability of the current time step, the original entropy parameter updated in the previous time step, and the entropy target value of the current time step, including: in, represents the loss function of the entropy parameter network, Represents the original entropy parameter after the previous time step update. For the first time step, it is the preset original entropy parameter; Represents the entropy target value of the current time step, based on the number of actions in the discrete action space of the current time step get; Minimize the loss function of the entropy parameter network to update the original entropy parameters of the current time step; Update the dynamic entropy parameters and joint entropy parameters based on the original entropy parameters updated at the current time step, including: in, represents the dynamic entropy parameter of the u-th scheduling target after the current time step update, Represents the preset weight coefficient, Represents the updated joint entropy parameter at the current time step.

Images