Multi-constraint supply chain scheduling method based on reinforcement learning

Abstract

This invention discloses a multi-constraint supply chain scheduling method based on reinforcement learning, relating to the fields of supply chain management and artificial intelligence. The method includes: constructing a hierarchical heterogeneous graph for supply chain scheduling based on multi-source heterogeneous data; inputting the hierarchical heterogeneous graph into an encoder based on a relational graph attention network for graph embedding, generating node embedding vectors and global context embedding vectors; constructing a hierarchical reinforcement learning scheduling decision framework including a meta-controller and multiple parallel-executing low-level scheduling agents; the meta-controller generating a sub-objective vector for the current stage based on the global context embedding vector and a multi-constraint objective function; the low-level scheduling agents performing joint action selection based on the node embedding vector and the sub-objective vector, outputting process allocation actions and logistics path selection actions, thereby generating scheduling decision instructions; and updating the model parameters of the scheduling decision framework based on scheduling execution feedback data.

Description

Technical Field

[0001] This invention relates to the fields of supply chain management and artificial intelligence technology, specifically a multi-constraint supply chain scheduling method based on reinforcement learning. Background Technology

[0002] Supply chain scheduling needs to handle numerous aspects simultaneously, including order scheduling, equipment allocation, material distribution, and personnel arrangement, and is subject to various constraints such as delivery time, process sequence, resource capacity, and energy consumption limitations. Faced with these dynamically changing and highly coupled constraints, existing technologies mostly employ centralized scheduling methods based on preset rules or operations research models. Rule-based methods struggle to balance global efficiency with multi-objective equilibrium, and their scheduling performance degrades significantly when production environments or constraints fluctuate. Operations research model-based methods typically solve process allocation and logistics route planning as two independent problems, severing the strong coupling between production and logistics, resulting in solutions that often deviate from the optimal scheduling objective. Some solutions introduce conventional reinforcement learning for single-layer decision-making, encoding the entire supply chain state as a single vector and outputting an integrated action. However, this approach suffers from the curse of dimensionality as the state space dimension increases and has limited capacity to handle multiple constrained objectives, making it difficult to effectively decompose optimization objectives of different granularities. Meanwhile, existing technologies lack sufficient structured utilization of heterogeneous nodes such as orders, equipment, and inventory in the supply chain and the complex dependencies between them. They often use flat graph representations or fully connected networks to characterize states, failing to accurately capture the differentiated interaction patterns between nodes due to their different types. This results in insufficient perception of key local information by the scheduling model. How to effectively decompose multi-constraint optimization objectives into hierarchically executable and coordinated sub-objectives, and on this basis, jointly optimize process allocation and logistics paths, and how to perform high-fidelity, heterogeneous relationship-aware structural representation of dynamic supply chain networks containing multiple node types and multiple connection relationships, have become urgent problems to be solved. Summary of the Invention

[0003] This invention provides a multi-constraint supply chain scheduling method based on reinforcement learning, which aims to effectively integrate the joint decision-making of process allocation and logistics path, achieve collaborative optimization of multi-constraint objectives under a hierarchical framework, and improve the ability to represent complex relationships in heterogeneous supply chain networks and the adaptive adjustment capability of scheduling strategies.

[0004] To achieve the above objectives, the present invention provides the following technical solution: The present invention provides a multi-constraint supply chain scheduling method based on reinforcement learning, comprising the following steps: A hierarchical heterogeneous graph for supply chain scheduling is constructed based on multi-source heterogeneous data. Preferably, order data, inventory data, equipment status data, and worker skill data are acquired. Each order in the order data is treated as an order node, and each piece of equipment in the equipment status data is treated as a resource node. Edges are then constructed based on the bill of materials relationships in the order data and the storage location relationships in the inventory data to generate the hierarchical heterogeneous graph. This hierarchical heterogeneous graph can depict the complex topological relationships between orders, resources, materials, and inventory in the supply chain from multiple dimensions, providing a structured global view for subsequent graph representation learning and scheduling decisions.

[0005] The hierarchical heterogeneous graph is input into an encoder based on a relational graph attention network for graph embedding, generating node embedding vectors and global context embedding vectors. As a preferred embodiment of the invention, the nodes in the hierarchical heterogeneous graph undergo type encoding and temporal encoding, where type encoding can employ one-hot encoding and temporal encoding can employ sinusoidal positional encoding. The encoder based on the relational graph attention network performs heterogeneous attention aggregation on the node's neighbors, updates the node features, and inputs the updated node features into an average pooling layer to generate the global context embedding vector. Through heterogeneous attention aggregation, this encoder can effectively capture the differentiated importance between different types of nodes and relational edges, enabling the generated embedding vector to retain the semantic information of local nodes while extracting a global representation of the entire supply chain state.

[0006] A hierarchical reinforcement learning-based scheduling decision framework is constructed, comprising a meta-controller and multiple parallel low-level scheduling agents. This framework decomposes the complex multi-constraint scheduling problem into two levels: high-level sub-objective planning and low-level joint action execution, reducing the difficulty of solving single-level decisions and improving the scalability of the policy.

[0007] The meta-controller generates a sub-objective vector for the current stage based on the global context embedding vector and the multi-constraint objective function. Preferably, the meta-controller normalizes the delay penalty coefficient, energy consumption constraint coefficient, and resource utilization coefficient in the multi-constraint objective function based on the global context embedding vector of the Long Short-Term Memory network processing time sequence, generates a weight vector, and then performs a dot product operation between the hidden state of the Long Short-Term Memory network and the weight vector to output the sub-objective vector. This process enables the meta-controller to dynamically balance multiple conflicting scheduling objectives, adaptively adjust the optimization focus at different production stages, and achieve macro-level control over the global scheduling rhythm.

[0008] The low-level scheduling agent performs joint action selection based on the node embedding vector and the sub-target vector. This joint action selection includes process allocation actions and logistics path selection actions. In a preferred embodiment of the invention, each low-level scheduling agent processes process node features and path node features separately based on a dual-attention mechanism. It then fuses the sub-target vector and process node features through cross-attention to generate a process allocation preference score. Finally, it processes historical path node features through a gated loop unit to generate a path selection preference score. By using the sub-target issued by the meta-controller as guidance, the low-level scheduling agent jointly optimizes the two strongly correlated actions of process allocation and logistics path selection, thus avoiding equipment waiting or logistics congestion caused by a disconnect between process scheduling and material delivery.

[0009] In the aforementioned joint action selection process, joint sampling is performed based on the process allocation preference score and the path selection preference score to output the process allocation action and the logistics path selection action. Specifically, a dynamic constraint matrix is ​​constructed based on resource capacity constraints, process sequence constraints, and delivery time constraints. The process allocation preference score and the path selection preference score are multiplied element-wise with the dynamic constraint matrix to generate a masked preference score. The masked preference score is then normalized to generate a joint action probability distribution. Sampling is performed based on the joint action probability distribution to output the process allocation action and the logistics path selection action. By introducing a dynamic constraint matrix to perform element-wise masking of hard constraints during the action sampling stage, this invention, while exploring effective scheduling strategies, fundamentally eliminates infeasible scheduling schemes that violate machine capacity, process sequence, or delivery time constraints, significantly improving the safety and effectiveness of decision-making.

[0010] Scheduling decision instructions are generated based on the process allocation actions and the logistics path selection actions. Preferably, the process allocation actions are mapped to machine start / stop instructions and personnel allocation instructions, and the logistics path selection actions are mapped to automated guided vehicle (AGV) path planning instructions and warehouse outbound instructions. The machine start / stop instructions, personnel allocation instructions, path planning instructions, and warehouse outbound instructions are integrated to generate a discrete-time step scheduling decision sequence. This invention directly transforms the abstract action output of reinforcement learning into low-level control instructions executable by production line equipment, personnel, and logistics systems, achieving seamless integration from optimization decision-making to physical execution.

[0011] The model parameters of the scheduling decision framework are updated based on the scheduling execution feedback data. Preferably, the actual completion time, actual energy consumption, and actual resource utilization rate are collected from the scheduling execution feedback data, the deviation between the actual completion time and the planned completion time is calculated, and a sparse reward signal is constructed. The meta-controller and the low-level scheduling agent perform distributed gradient calculation based on the sparse reward signal and the sub-objective vector, and then update the network parameters of the meta-controller and the low-level scheduling agent based on the asynchronous dominant actor-commentator algorithm. During the distributed gradient calculation process, the meta-controller calculates the sub-objective achievement degree based on the sparse reward signal and generates an internal reward feedback to the low-level scheduling agent. The low-level scheduling agent calculates its own advantage function based on the internal reward and the global reward, and summarizes the advantage functions of each low-level scheduling agent based on the gradient synchronization mechanism to update the shared commentator network parameters. By combining sparse rewards with sub-goal-oriented internal rewards, this invention provides a clear credit allocation signal for hierarchical agents, effectively alleviating the difficulty of policy learning under global sparse rewards, enabling the model to continuously evolve from actual execution results and constantly improve the collaborative scheduling performance under multiple constraints.

[0012] The technical effects and advantages provided by the present invention in the above technical solution are as follows: By constructing a hierarchical reinforcement learning scheduling decision framework comprising a meta-controller and multiple parallel low-level scheduling agents, the meta-controller generates a sub-objective vector for the current stage based on a global context embedding vector and a multi-constraint objective function. The low-level scheduling agents then select specialized joint actions based on this sub-objective vector. This hierarchical decision-making mechanism decomposes the originally highly coupled multi-constraint optimization objective into two levels: a top-level stage-specific guiding signal and a bottom-level specific execution action. This allows the meta-controller to dynamically adjust the preference weights for on-time delivery, energy consumption, and resource utilization at different production stages from a global perspective. The low-level agents, upon receiving a clear sub-objective vector, focus on generating process allocation and logistics path selection actions adapted to the guiding signal within their local decision space. Consequently, production task scheduling and material handling instructions can be collaboratively generated under a unified optimization direction, avoiding objective conflicts and optimization gaps caused by solving the two types of actions separately. When facing dynamically changing production environments, the entire scheduling strategy can be flexibly adapted by adjusting the generation method of the sub-objective vector, without redesigning the underlying rules. This architecture significantly improves the overall coordination of scheduling schemes under multi-constraint conditions and its adaptability to complex dynamic environments.

[0013] An encoder based on a relational graph attention network is used to perform heterogeneous attention aggregation on the neighboring nodes of a hierarchical heterogeneous graph constructed from multi-source heterogeneous data. During node embedding, attention weights under different relationships are calculated and features are aggregated for different node types (e.g., order nodes, equipment nodes) and their connections. This accurately captures specific interaction patterns arising from differences in business relationships between different types of nodes. Based on this, a global context embedding vector containing global structural information is generated through average pooling. This graph embedding approach organizes the dispersed multi-source heterogeneous data in the supply chain into a structured graph model and deeply mines the heterogeneous dependencies between nodes through a relation-aware attention mechanism. This enables the scheduling model to obtain a more expressive and refined state representation of complex network structures. When allocating processes and selecting logistics routes, the low-level scheduling agent can more accurately identify key bottleneck processes, suitable processing equipment, and efficient delivery routes based on this high-quality node embedding vector, thereby improving the targeting and execution efficiency of scheduling decisions. Attached Figure Description

[0014] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this invention. For those skilled in the art, other drawings can be obtained based on these drawings.

[0015] Figure 1 This is a flowchart of a multi-constraint supply chain scheduling method based on reinforcement learning; Figure 2 It is a flowchart of supply chain scheduling hierarchical heterogeneous graph node encoding and relationship graph attention aggregation; Figure 3 This is a flowchart of the hierarchical scheduling decision-making process based on the asynchronous advantage actor-critic algorithm; Figure 4 It is a curve showing the variation of the probability distribution of joint actions under different temperature parameters. Detailed Implementation

[0016] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0017] See Figure 1This invention provides a multi-constraint supply chain scheduling method based on reinforcement learning, comprising the following steps: constructing a hierarchical heterogeneous graph for supply chain scheduling based on multi-source heterogeneous data; inputting the hierarchical heterogeneous graph into an encoder based on a relational graph attention network for graph embedding to generate node embedding vectors and global context embedding vectors; constructing a scheduling decision framework based on hierarchical reinforcement learning, which includes a meta-controller and multiple parallel low-level scheduling agents; the meta-controller generating a sub-objective vector for the current stage based on the global context embedding vector and the multi-constraint objective function; the low-level scheduling agents performing joint action selection based on the node embedding vector and the sub-objective vector, the joint action selection including process allocation action and logistics path selection action; generating scheduling decision instructions based on process allocation action and logistics path selection action; and updating the model parameters of the scheduling decision framework based on scheduling execution feedback data. Example 1

[0018] In specific implementation, please refer to Figure 2 After constructing a hierarchical heterogeneous graph for supply chain scheduling based on multi-source heterogeneous data, the nodes in the graph are coded by type and by time sequence. Type coding uses one-hot encoding, pre-defining a set of node types, including order nodes, resource nodes, etc., and assigning a different integer identifier to each node type. The dimension of the one-hot encoding vector equals the total number of node types, and only the index position corresponding to the integer identifier in the vector is set to 1, while the other positions are set to 0. For example, if the total number of node types is C, for a node with type identifier k, the k-th dimension of its type encoding vector is 1, and the other dimensions are 0. Time sequence encoding uses sinusoidal position encoding, where the dimension of the sinusoidal position encoding vector is the same as or independently set as the dimension of the type-encoded vector. The encoding dimension is denoted as […]. For any node in the hierarchical heterogeneous graph, obtain the time step position pos of that node, where pos is a non-negative integer. Then, the 2i-th and 2i+1-th dimensions of the sinusoidal position encoding vector are calculated according to the following formulas: ; ; in, This represents the value of the 2i-th dimension of the encoded vector at time step position pos. This represents the (2i+1)th dimension of the encoded vector at time step position pos, where i ranges from 0 to... Integer index, The vector dimension for sinusoidal position encoding. This indicates a base of 10000 and an exponent of 10000. The exponentiation operation is used. The type encoding vector is concatenated or added to the sinusoidal position encoding vector to serve as the initial feature representation of the hierarchical heterogeneous graph node.

[0019] An encoder based on a relational graph attention network receives the initial feature representations of nodes in a hierarchical heterogeneous graph, performs heterogeneous attention aggregation on the node's neighbors, and updates the node features. The hierarchical heterogeneous graph contains multiple relation types, denoted as R, where each relation type r corresponds to a connection pattern between nodes. For a target node v in the hierarchical heterogeneous graph, its neighbor node set is divided into different subsets according to relation type. The set of neighbor nodes of target node v under relation type r is denoted as... The encoder based on a graph attention network sets an independent attention weight matrix for each relation type r. and attention vector Calculate the attention coefficient of neighbor node u to target node v under different relation types. The calculation method is as follows: ; in, This represents the current feature vector of the target node v. This represents the current feature vector of neighbor node u. This is the learnable weight matrix corresponding to relation type r. This is the learnable attention vector corresponding to relation type r. express transpose, This represents a vector concatenation operation, with LeakyReLU as the activation function. It calculates the recurrence relation (R) for all neighboring nodes u under relation type r. Perform softmax normalization to obtain normalized attention coefficients. : ; Calculate the weighted sum of the features of neighbors under relation type r, and use it as the aggregate feature corresponding to relation type r. : ; in, The activation function is non-linear. A multi-head attention mechanism is used, with K attention heads, each independently executing the above process to obtain K aggregated features for relation type r. These K aggregated features are then concatenated or averaged to obtain the final aggregated features for relation type r. The final aggregated features corresponding to all relation types are summed or averaged to obtain the updated node features for the target node v. All nodes in the hierarchical heterogeneous graph are traversed to complete the node feature update.

[0020] The updated node features are input into an average pooling layer, which calculates the element-wise average of the updated node features for all nodes in the hierarchical heterogeneous graph, generating a global context embedding vector. This global context embedding vector serves as the overall representation of the hierarchical heterogeneous graph and is used by the subsequent meta-controller to generate sub-target vectors. Example 2

[0021] In its implementation, the hierarchical reinforcement learning-based scheduling decision framework includes a meta-controller and multiple parallel low-level scheduling agents. The meta-controller receives a global context embedding vector generated by an encoder based on a relational graph attention network. The meta-controller processes the temporal global context embedding vector using a Long Short-Term Memory (LSTM) network. The LTM network consists of an input gate, a forget gate, an output gate, and cell states. At each scheduling decision stage, the meta-controller receives the global context embedding vector output by the encoder of the current stage and uses it as input to the LTM network. The LTM network's computation process is as follows: the forget gate calculates the forgotten activation value based on the previous stage's hidden state and the current stage's global context embedding vector; the input gate calculates the input activation value and candidate cell states based on the previous stage's hidden state and the current stage's global context embedding vector; the cell states are updated by partially forgetting the previous stage's cell states through the forget gate and partially accepting the candidate cell states through the input gate; and the output gate calculates the current stage's hidden state based on the updated cell states and the previous stage's hidden states.

[0022] The multi-constraint objective function includes a delay penalty coefficient, an energy consumption constraint coefficient, and a resource utilization coefficient. The delay penalty coefficient reflects the penalty weight for delayed order delivery; it is obtained by setting the order priority in the production plan and the delay penalty rules stipulated in the contract, and its value is a real number greater than or equal to 0. The energy consumption constraint coefficient reflects the intensity of the restriction on energy consumption; it is obtained by setting the workshop energy consumption limit standard and real-time electricity price signal, and its value is a real number greater than or equal to 0. The resource utilization coefficient reflects the required weight for the efficiency of equipment and human resources utilization; it is obtained by setting the equipment depreciation cost and management cost allocation scheme, and its value is a real number greater than or equal to 0. The delay penalty coefficient, energy consumption constraint coefficient, and resource utilization coefficient are normalized using L2 normalization. The sum of the squares and square roots of the three coefficients is calculated, and each coefficient is divided by its square root, so that the L2 norm of the weight vector composed of the three normalized coefficients is 1, thus generating the weight vector.

[0023] The current hidden state of the Long Short-Term Memory (LSTM) network is multiplied by its weight vector using a dot product operation. The dot product formula is as follows: ; in, Let represent the element at position t in the sub-objective vector of the current stage t, where D represents the total dimension of the hidden state vector and weight vector, and d is the dimension index. This represents the d-th dimension of the hidden state vector of the Long Short-Term Memory network at the current stage t. This represents the d-th dimension of the weight vector. The total number of dimensions of the hidden layer state vector is the same as the total number of dimensions of the weight vector, both pre-defined by the network structure. The sub-target vector for the current stage is obtained by summing the products of each dimension component of the hidden layer state vector and the corresponding dimension component of the weight vector. The sub-target vector serves as the execution guidance signal for the lower-level scheduling agents and is transmitted to each lower-level scheduling agent. Example 3

[0024] In practical implementation, each low-level scheduling agent processes process node features and path node features respectively based on a dual-attention mechanism. The dual-attention mechanism comprises two parallel self-attention network branches: the first branch receives the process node feature sequence, and the second branch receives the path node feature sequence. The process node feature sequence consists of node features belonging to the process type from the node embedding vector output by the encoder, while the path node feature sequence consists of node features belonging to the logistics path type from the node embedding vector output by the encoder. The first branch maps the process node feature sequence to a query matrix, key matrix, and value matrix through a linear transformation. It uses scaled dot product attention to calculate the mutual influence weights between process nodes. After weighted summation of the value matrix, it performs residual connection and layer normalization to output the self-attention enhanced features of the process nodes. The second branch uses the same structure to process the path node feature sequence, outputting the self-attention enhanced features of the path nodes.

[0025] A process allocation preference score is generated by fusing sub-target vectors and process node features through cross-attention. The cross-attention module takes the sub-target vector output from the meta-controller as the query input and the self-attention enhancement features of the process nodes output from the first self-attention network branch as the key and value inputs. The sub-target vector undergoes a linear transformation to obtain the query matrix, and the self-attention enhancement features of the process nodes undergo a linear transformation to obtain the key and value matrices. The dot product of the query matrix and the transpose of the key matrix is ​​calculated, scaled by the square root of the column dimension of the key matrix, and the attention weights are obtained by applying the softmax function. The attention weights are multiplied by the value matrix to obtain the process feature representation that fuses the sub-target information. This process feature representation is then input into a scoring network, which consists of a two-layer fully connected network. The scoring network outputs the process allocation preference score, which is a non-negative real vector equal to the number of process nodes, with each element corresponding to the degree of allocation preference for a particular process node.

[0026] Historical path node features are processed by a gated recurrent unit (GRU) to generate a path selection preference score. The GRU includes a reset gate and an update gate. At each decision stage, the low-level scheduling agent takes the self-attention enhancement features of the path nodes output from the second self-attention network branch as input, and simultaneously receives the hidden state from the previous stage. The reset gate calculates a reset activation value based on the self-attention enhancement features of the current input path node and the hidden state from the previous stage. The update gate calculates an update activation value based on the self-attention enhancement features of the current input path node and the hidden state from the previous stage. The reset gate controls the degree of retention of the hidden state from the previous stage, and the candidate hidden states are generated by combining them with the current input. The update gate then performs a weighted combination of the hidden state from the previous stage and the candidate hidden states to output the hidden state of the current stage. The hidden state of the current stage is input into another two-layer fully connected network, which outputs a path selection preference score. The path selection preference score is a non-negative real-valued vector equal to the number of path nodes, with each element corresponding to the degree of preference for a particular logistics path.

[0027] A dynamic constraint matrix is ​​constructed based on resource capacity constraints, process sequence constraints, and delivery time constraints. Resource capacity constraints represent the maximum number of operations each machine can handle within a unit of time. Process sequence constraints represent the prerequisite relationships that must be satisfied between operations. Delivery time constraints represent the latest completion time limit for the order. The number of rows in the dynamic constraint matrix equals the number of operation nodes, and the number of columns equals the number of logistics path nodes. Each element in the matrix corresponds to a combination of an operation allocation action and a logistics path selection action. For the element in the u-th row and v-th column of the matrix, if the machine allocated to the u-th operation node and the selection of the v-th logistics path simultaneously satisfy the resource capacity constraint, process sequence constraint, and delivery time constraint, then the matrix element is set to 1; otherwise, it is set to 0. Resource capacity constraints are determined by checking whether the remaining machine capacity during this time period is greater than or equal to 1; if not, it is set to 0. Process sequence constraints are determined by checking whether all prerequisite operations for the u-th operation have been completed; if not, it is set to 0. Delivery time constraints are determined by calculating whether the sum of the current time and the transportation time of the v-th path is less than the upper limit of the order delivery time window; if it exceeds, it is set to 0.

[0028] The process allocation preference score and path selection preference score are expanded into a preference score matrix of the same size as the dynamic constraint matrix through an outer product. The preference score matrix is ​​then multiplied element-wise with the dynamic constraint matrix to generate the masked preference score matrix. Elements corresponding to inactive actions in the masked preference score matrix are set to zero. The masked preference score matrix is ​​then expanded row-wise into a vector of length M, where M is the product of the total number of process nodes and the total number of logistics path nodes. The masked preference score vector is then normalized using a softmax function with a temperature parameter to generate a joint action probability distribution. The calculation formula is as follows: ; in, Let represent the probability of the m-th joint action. This represents the m-th element in the masked preference score vector. Let M represent the nth element in the masked preference score vector, and M represent the total number of joint actions, i.e., the number of all possible combinations of process allocation actions and logistics path selection actions. M = number of process nodes × number of logistics path nodes. Temperature parameters The value range is positive real numbers, temperature parameter By manually setting the temperature parameter to a value less than 1, the probability distribution of joint actions tends to be sharper. When the temperature parameter is set to 1, the probability distribution of joint actions is the original softmax output, which can be adjusted according to the determinism of the scheduling environment, usually selected between 0.5 and 2, and set to 1 in implementation. Sampling is performed based on the probability distribution of joint actions to generate a one-hot encoded sample of length M. The joint actions corresponding to the positions with a value of 1 in the one-hot encoded sample are the output process allocation actions and logistics path selection actions.

[0029] See Figure 4 The figure illustrates the changing trend of the joint action probability distribution under different temperature parameter τ values ​​during the joint action selection process of the low-level scheduling agent described in Example 3. The horizontal axis represents the joint action index, arranged in descending order of joint action probability, ranging from 0 to 500. The vertical axis represents the joint action probability, using a logarithmic scale, with probability values ​​varying from the order of 1e-10 to 1e-1, facilitating the observation of the tail features of the probability distribution.

[0030] The curves correspond to three temperature parameter settings: τ = 0.5, 1.0, and 2.0. When τ is 0.5, the probability distribution of joint actions exhibits a highly sharp characteristic, with a significantly high probability concentration in the first 50 joint actions. Subsequent probability values ​​rapidly decrease from approximately 1e-1 to below 1e-7, indicating that the low temperature parameter makes the model more inclined to select a few high-preference actions, resulting in a sparse yet concentrated distribution of joint action probabilities. When τ is 1.0, the probability distribution is relatively flat, with the first 100 joint actions still having relatively high probabilities. The probability values ​​decrease slowly within the range of 1e-1 to 1e-5, indicating that the model balances exploration and exploitation in action selection, resulting in a relatively uniform distribution of joint action probabilities. When τ is 2.0, the probability distribution is the flattest, with generally uniform joint action probabilities, and the tail probability remains above 1e-3, indicating that the high temperature parameter makes the model more inclined to action exploration, resulting in a more dispersed selection of joint actions. Example 4

[0031] In specific implementation, please refer to Figure 3The low-level scheduling agent performs joint action selection based on node embedding vectors and sub-target vectors, outputting process allocation actions and logistics path selection actions. Process allocation actions are represented as discrete indices, each uniquely corresponding to a combination of a machine and a process node. These actions are mapped to machine start / stop instructions and personnel allocation instructions, a process implemented through a predefined process-machine-person association table. This table stores the set of machines that can execute each process and the worker skill types required for each machine. When a process allocation action specifies a machine to execute a process, it generates a machine start instruction and a corresponding worker allocation instruction. If the specified machine is currently stopped, the machine start / stop instruction is a start instruction; if the machine is already running, it remains running. The personnel allocation instruction specifies the worker number and task content.

[0032] The logistics route selection action is also represented in the form of a discrete index, which uniquely corresponds to a logistics route node. This action is mapped to route planning instructions and warehouse outbound instructions for automated guided vehicles (AGVs). The route planning instruction includes the sequence of route nodes, route length, and estimated travel time, generated by querying logistics network topology data, which stores the connectivity relationships and distance matrices between nodes. The warehouse outbound instruction includes the outbound material code, outbound quantity, and target workstation, determined based on order data and the target node in the route planning instruction.

[0033] The system integrates machine start / stop commands, personnel allocation commands, path planning commands, and warehouse outbound commands to generate a discrete-time-step scheduling decision sequence. This sequence is spaced at fixed time steps, with each step recording the machine actions, personnel actions, logistics actions, and warehouse actions to be executed within that step. The time step is set based on the production cycle time, for example, 1 minute. Within each time step, machine start / stop commands are parsed into machine control signals, personnel allocation commands are decomposed into worker movement paths and operation periods, path planning commands are compiled into the guidance paths of automated guided vehicles (AGVs), and warehouse outbound commands are converted into outbound orders recognizable by the warehouse management system. The final output is a unified-format scheduling decision sequence frame, which the execution system executes frame by frame.

[0034] After the scheduling decision sequence is executed by the Manufacturing Execution System (MAS) and the Warehouse Management System (WMS), the actual completion time, actual energy consumption, and actual resource utilization rate are collected from the scheduling execution feedback data. Actual completion time is the actual duration from the start of order execution to the completion of the last process, collected through the production reporting records of the MAS. Actual energy consumption is the total power consumption of all machines and automated guided vehicles during the scheduling execution period, collected in real time by equipment energy consumption monitoring sensors and integrated over time. Actual resource utilization rate is defined as the ratio of the busy time of all machines and workers to the total available time during the scheduling execution period, collected through equipment status logs and the personnel attendance system.

[0035] Calculate the deviation between the actual completion time and the planned completion time to construct a sparse reward signal. The planned completion time is the theoretical completion time estimated before scheduling decisions are made based on the standard process route and standard transportation time. The deviation is constructed as follows: ; in, Indicates sparse reward signals, Indicates the actual completion time. Indicates the planned completion time. Tolerance coefficient, tolerance coefficient The value range is a real number between [0, 0.2]. The tolerance coefficient is set manually, based on the allowable fluctuation range of delivery time in production management. In implementation, it will... Setting it to 0.1 means that the actual completion time is allowed to exceed the planned completion time by up to 10%, which is still considered as achieving the goal.

[0036] The meta-controller and the low-level scheduling agent perform distributed gradient calculation based on the sparse reward signal and the sub-target vector. Both the meta-controller and the low-level scheduling agent are independent neural network modules. The meta-controller includes a long short-term memory network and an output layer, while the low-level scheduling agent includes a dual attention network, a gated recurrent unit, and a scoring network. The sparse reward signal is distributed as a global reward to all agents, while the sub-target vector output by the meta-controller serves as the internal target benchmark for the low-level scheduling agent. The distributed gradient calculation process is as follows: the meta-controller calculates the sub-target achievement degree, which is equal to the moving average of the sparse reward signal; the low-level scheduling agent calculates its own action value estimate and advantage function estimate based on the sparse reward signal and the sub-target achievement degree. The action value estimate is output through a critic network, which is a three-layer fully connected network. The input is a concatenation of the node embedding vector, the sub-target vector, and the global context embedding vector, and the output is a scalar value.

[0037] The asynchronous advantage actor-critic algorithm updates the network parameters of the meta-controller and low-level scheduling agents. The asynchronous advantage actor-critic algorithm maintains a globally shared set of neural network parameters, including the network parameters of the meta-controller and all low-level scheduling agents. Multiple parallel worker threads run the scheduling environment independently, each holding a local copy of the network parameters. After each scheduling round, the worker thread calculates its local gradient using locally acquired sparse reward signals and sub-objective vectors, employing a policy gradient method and a value function loss. The policy gradient is obtained by multiplying the log probability of the joint action probability distribution output by the actor network by the advantage function, which is estimated using temporal difference error. The worker thread asynchronously updates the calculated local gradient to the globally shared network parameters, then pulls the latest parameters from the globally shared network parameters to overwrite its local copy. This process is repeated until the network parameters of the meta-controller and low-level scheduling agents are updated. Example 5

[0038] In practice, the meta-controller calculates the sub-objective achievement degree based on the sparse reward signal. The sub-objective achievement degree reflects the degree of matching between the sub-objective vector output by the meta-controller and the actual execution result. The meta-controller maintains a sub-objective achievement degree estimation network, which consists of two fully connected layers. The input is the concatenation of the sub-objective vector output by the meta-controller at the current stage and the global context embedding vector; the output is the scalar achievement degree prediction value. The sparse reward signal is obtained from the scheduling execution feedback data. The meta-controller compares the sparse reward signal with the sub-objective achievement degree prediction value, calculates the mean squared error loss, and updates the parameters of the sub-objective achievement degree estimation network through backpropagation. The updated sub-objective achievement degree estimation network re-evaluates the sub-objective vectors and corresponding global context embedding vectors from each historical stage, generating an internal reward that is fed back to the lower-level scheduling agent. The value of the internal reward is equal to the difference between the scalar achievement degree prediction value output by the sub-objective achievement degree estimation network and the baseline value. The baseline value is the exponential moving average of the achievement degree prediction values ​​output by the sub-objective achievement degree estimation network within the most recent time window, with a decay factor set to 0.99.

[0039] After receiving the internal reward, the low-level scheduling agent calculates its own advantage function based on the internal and global rewards. The global reward is the sparse reward signal transmitted by the meta-controller, and the internal reward is the reward signal related to the achievement of sub-goals fed back by the meta-controller. The calculation of the advantage function is performed using the generalized advantage estimation method. Over a time series of length S, the low-level scheduling agent collects the state value estimate, internal reward, and global reward for each time step. The state value estimate is output by a shared critic network. The core architecture of the critic network is a four-layer fully connected neural network, containing an input layer, two hidden layers, and an output layer. The input layer receives input data that is a concatenation of the node embedding vector, sub-goal vector, and global context embedding vector at the current time step. The first hidden layer contains 256 neurons and uses the ReLU activation function, and the second hidden layer contains 128 neurons and uses the ReLU activation function. The output layer outputs a scalar, which represents the expected cumulative reward for performing the current joint action in the current state. The formula for calculating the generalized advantage estimation is: ; in, This represents the dominance function value at time step b. For time step index, The total length of the time series. This is the offset index starting from time step b. Discount factor The value range is a real number between (0, 1), and is set in the implementation. A value of 0.99 indicates that future rewards are factored into the current advantage estimate with an exponential decay of 0.99. The decay parameter is the generalized advantage estimate. The value range is a real number between (0, 1], and is set in the implementation. It is 0.95, used to balance the bias and variance; Indicates at time step The internal reward received by the lower-level scheduling agent; Indicates at time step Global reward at that time; This indicates that the critics' network is in a state of... Estimation of the output state value; This indicates that the critics' network is in a state of... State value estimation of the output; state From time step The node embedding vector, sub-target vector, and global context embedding vector are jointly defined.

[0040] The gradient synchronization mechanism aggregates the advantage functions of each low-level scheduling agent to update the shared commentator network parameters. This mechanism executes during the global network update phase of the asynchronous advantage actor-commentator algorithm. After each worker thread calculates its advantage function in its local environment, it does not directly use the advantage function to update its local commentator network parameters. Instead, it packages the advantage function and its corresponding state input data into a gradient request message and places it in the global gradient queue. The aggregation operation is performed by the global parameter server, which extracts the advantage function data submitted by each low-level scheduling agent from the gradient queue and concatenates the advantage function data into a batch. The loss function of the commentator network is defined as the mean squared error of the advantage function, i.e., the squared difference between the estimated state value output by the commentator network and the target value, where the target value is the sum of the advantage function and the estimated state value. The global parameter server calculates the average commenter network loss for all samples within a batch, calculates the gradient of the commenter network parameters based on this average, and updates the globally shared commenter network parameters using the Adam optimizer. The Adam optimizer's learning rate is set to 0.0001, the first moment decay rate is set to 0.9, and the second moment decay rate is set to 0.999. Each worker thread periodically pulls the updated shared commenter network parameters from the global parameter server to replace its local commenter network parameters.

[0041] Before generating the hierarchical heterogeneous diagram, order data, inventory data, equipment status data, and worker skill data are obtained. Order data is extracted from the order table of the Enterprise Resource Planning (ERP) system, and each order record includes an order number, bill of materials, required quantity, and delivery date. Inventory data is extracted from the inventory table of the Warehouse Management System, and each inventory record includes a material code, storage location number, and available quantity. Equipment status data is extracted from the equipment table of the Manufacturing Execution System (MES), and each equipment status record includes an equipment number, equipment type, current status, and processing capacity. Worker skill data is extracted from the worker table of the Human Resources Management System, and each worker skill record includes a worker number and a list of skill types.

[0042] Each order in the order data is treated as an order node, with attributes including order number, required quantity, and delivery date. Each piece of equipment in the equipment status data is treated as a resource node, with attributes including equipment number, equipment type, and processing capacity. Edges are constructed based on the bill of materials (BOM) relationships in the order data and the storage location relationships in the inventory data. The BOM relationships form demand edges between order nodes and resource nodes. Specifically, the BOM of each order node is traversed, recording the types and quantities of materials required to produce the order. For each material, the storage location in the inventory data is queried, and a demand edge is established from the order node to the resource node of the corresponding material's storage location. The edge attribute of the demand edge includes the required quantity. Storage location relationships form spatial edges between resource nodes. Specifically, the physical adjacency relationships and distances between all storage locations are obtained from the inventory data. For two physically adjacent storage locations, a spatial edge is established between the corresponding resource nodes. The edge attribute of the spatial edge includes the distance value between the storage locations. All order nodes, resource nodes, demand edges, and spatial edges together constitute a hierarchical heterogeneous graph for supply chain scheduling.

[0043] When performing type encoding on nodes in a hierarchical heterogeneous graph, the node types include two categories: order nodes and resource nodes. One-hot encoding is used for type encoding, and the type encoding vector for order nodes is... The type encoding vector of the resource node is The timing coding employs sinusoidal position coding, with a vector dimension of 16. For each time step... The node, the first of the sinusoidal position-encoded vectors peacekeeping The dimension is calculated according to the sinusoidal position coding formula. It takes a value between 0 and 7. The value represents the discrete-time step number at which the node is created or updated in the hierarchical heterogeneous graph. The type encoding vector and the sinusoidal position encoding vector are concatenated dimensionally to form an initial feature vector for the node with a total dimension of 18, which is used by the encoder based on the relational graph attention network.

[0044] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. A multi-constraint supply chain scheduling method based on reinforcement learning, characterized in that, Includes the following steps: A hierarchical heterogeneous graph for supply chain scheduling is constructed based on multi-source heterogeneous data; The hierarchical heterogeneous graph is input into an encoder based on a relational graph attention network for graph embedding, generating node embedding vectors and global context embedding vectors; A scheduling decision framework based on hierarchical reinforcement learning is constructed, which includes a meta-controller and multiple low-level scheduling agents that execute in parallel. The meta-controller generates the sub-objective vector for the current stage based on the global context embedding vector and the multi-constraint objective function; The low-level scheduling agent performs joint action selection based on the node embedding vector and the sub-target vector. The joint action selection includes process allocation action and logistics path selection action. Generate scheduling decision instructions based on the process allocation actions and the logistics route selection actions; The model parameters of the scheduling decision framework are updated based on the scheduling execution feedback data.

2. The multi-constraint supply chain scheduling method based on reinforcement learning according to claim 1, characterized in that, The step of inputting the hierarchical heterogeneous graph into an encoder based on a relational graph attention network for graph embedding, generating node embedding vectors and global context embedding vectors, specifically involves: The nodes in the hierarchical heterogeneous graph are type-encoded and time-encoded. The encoder based on the relational graph attention network performs heterogeneous attention aggregation on the node's neighbors and updates the node features; The updated node features are input into the average pooling layer to generate the global context embedding vector.

3. The multi-constraint supply chain scheduling method based on reinforcement learning according to claim 1, characterized in that, The meta-controller generates the sub-objective vector for the current stage based on the global context embedding vector and the multi-constraint objective function, specifically as follows: The meta-controller is based on the global context embedding vector of the processing time sequence of the Long Short-Term Memory network; The delay penalty coefficient, energy consumption constraint coefficient, and resource utilization coefficient in the multi-constraint objective function are normalized to generate a weight vector. The hidden state of the Long Short-Term Memory network is multiplied by the weight vector to output the sub-target vector.

4. The multi-constraint supply chain scheduling method based on reinforcement learning according to claim 1, characterized in that, The low-level scheduling agent performs joint action selection based on the node embedding vector and the sub-target vector. This joint action selection includes process allocation actions and logistics path selection actions, specifically: Each low-level scheduling agent processes process node features and path node features respectively based on a dual attention mechanism; By fusing the sub-target vectors and process node features through cross-attention, a process allocation preference score is generated. Historical path node features are processed by a gated recurrent unit to generate a path selection preference score. Based on the process allocation preference score and the path selection preference score, joint sampling is performed to output the process allocation action and the logistics path selection action.

5. The multi-constraint supply chain scheduling method based on reinforcement learning according to claim 4, characterized in that, The step of jointly sampling based on the process allocation preference score and the path selection preference score to output the process allocation action and the logistics path selection action is as follows: A dynamic constraint matrix is ​​constructed based on resource capacity constraints, process sequence constraints, and delivery time constraints. The process allocation preference score and the path selection preference score are multiplied element-wise with the dynamic constraint matrix to generate the masked preference score. The masked preference scores are normalized to generate a joint action probability distribution; Sampling is performed based on the probability distribution of the joint actions, and the process allocation actions and the logistics path selection actions are output.

6. The multi-constraint supply chain scheduling method based on reinforcement learning according to claim 1, characterized in that, The step of generating scheduling decision instructions based on the process allocation actions and the logistics route selection actions specifically includes: The process allocation actions are mapped to machine start / stop commands and personnel allocation commands; The logistics route selection action is mapped to the route planning instructions and warehouse outbound instructions of the automated guided vehicle. The machine start / stop instructions, personnel allocation instructions, path planning instructions, and warehouse outbound instructions are integrated to generate a discrete-time step scheduling decision sequence.

7. The multi-constraint supply chain scheduling method based on reinforcement learning according to claim 1, characterized in that, The step of updating the model parameters of the scheduling decision framework based on the scheduling execution feedback data specifically involves: Collect and schedule execution feedback data including actual completion time, actual energy consumption, and actual resource utilization. Calculate the deviation between the actual completion time and the planned completion time, and construct a sparse reward signal; The meta-controller and the low-level scheduling agent perform distributed gradient calculation based on the sparse reward signal and the sub-target vector; The network parameters of the meta-controller and the low-level scheduling agent are updated based on the asynchronous advantage actor-critic algorithm.

8. The multi-constraint supply chain scheduling method based on reinforcement learning according to claim 7, characterized in that, The meta-controller and the low-level scheduling agent perform distributed gradient calculation based on the sparse reward signal and the sub-target vector, specifically as follows: The meta-controller calculates the sub-goal achievement degree based on the sparse reward signal and generates an internal reward to feed back to the low-level scheduling agent; The low-level scheduling agent calculates its respective advantage function based on the internal reward and the global reward; The advantage functions of each low-level scheduling agent are aggregated based on the gradient synchronization mechanism, and the shared commentator network parameters are updated.

9. The multi-constraint supply chain scheduling method based on reinforcement learning according to claim 1, characterized in that, The hierarchical heterogeneous graph for supply chain scheduling constructed based on multi-source heterogeneous data is as follows: Obtain order data, inventory data, equipment status data, and worker skill data; Each order in the order data is taken as an order node, and each device in the device status data is taken as a resource node; Based on the material list relationships in the order data and the storage location relationships in the inventory data, the connection edges are constructed to generate the hierarchical heterogeneous graph.

10. The multi-constraint supply chain scheduling method based on reinforcement learning according to claim 2, characterized in that, The type encoding uses one-hot encoding, and the timing encoding uses sinusoidal position encoding.

Images