Supply chain order dynamic scheduling optimization method based on reinforcement learning
By using an improved GhostNet network and a natural policy gradient training mechanism, the problems of high network structure complexity and insufficient fusion of order preference features in existing scheduling methods are solved, achieving efficient and flexible supply chain order scheduling optimization and improving scheduling efficiency and stability.
Patent Information
- Application Number
- CN202511139671.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-08-14
- Publication Date
- 2025-11-25
AI Technical Summary
Existing reinforcement learning-based supply chain order scheduling methods suffer from high network structure complexity, low inference efficiency, lack of fine-grained modeling capabilities for multi-dimensional states and action preferences, difficulty in achieving rapid decision-making on resource-constrained industrial edge devices, and lack of dynamic policy expression mechanisms that fuse order preference features, affecting scheduling performance and adaptability.
An improved GhostNet network and a natural policy gradient training mechanism are adopted to construct a lightweight scheduling policy generation process. Multi-scale features are extracted through lightweight channel reconstruction structure and state attention-guided units. Combined with order preference labels and feedback information, dynamic path weighting and policy fusion are performed to achieve efficient and accurate scheduling decisions.
It improves the efficiency of scheduling strategy generation, enhances the structural sensitivity and adaptability of strategy updates, realizes dynamic fusion strategy generation driven by order preferences, improves scheduling accuracy and flexibility, and adapts to the real-time scheduling needs in complex scenarios.
Smart Images
Figure CN121010164A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of supply chain scheduling technology, and in particular to a method for optimizing dynamic scheduling of supply chain orders based on reinforcement learning. Background Technology
[0002] In supply chain management, dynamic order scheduling optimization is a core component, directly impacting resource allocation efficiency and the real-time performance of transportation. Traditional order scheduling methods often rely on heuristic rules or static model-based algorithms, making them ill-suited to dynamic order demands, complex delivery route combinations, and multi-objective constraints in real-world scenarios. In recent years, reinforcement learning methods have been gradually introduced into the field of scheduling optimization. Through state-action mapping mechanisms, they enable adaptive policy updates, providing a learning-capable optimization framework for scheduling decisions. However, existing reinforcement learning-based scheduling methods mostly employ general neural networks as the policy modeling structure. These networks suffer from high complexity, low inference efficiency, and a lack of fine-grained modeling capabilities for the relationship between multi-dimensional states and action preferences, hindering rapid decision-making on resource-constrained industrial edge devices.
[0003] Existing natural policy gradient methods offer advantages in improving convergence speed and stability. However, when faced with network architectures featuring lightweight structures and channel branching characteristics, they fail to perceive the varying contributions of different channel paths to the policy output, resulting in a lack of focus in policy updates and impacting overall scheduling performance. Furthermore, existing scheduling schemes generally lack dynamic policy expression mechanisms that integrate with order preference features, ignoring the weight differences in order semantics during action generation and limiting the adaptability of scheduling models to multi-objective preference tasks. Therefore, there is an urgent need to construct a supply chain order scheduling optimization method that integrates lightweight neural structures, efficient policy estimation mechanisms, and multi-objective preference modeling capabilities to meet the requirements of real-time scheduling and feedback-driven updates in complex scenarios.
[0004] Therefore, how to provide a reinforcement learning-based method for dynamic scheduling optimization of supply chain orders is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention
[0005] One objective of this invention is to propose a dynamic scheduling optimization method for supply chain orders based on reinforcement learning. This invention employs an improved GhostNet network and a natural policy gradient training mechanism to construct a scheduling policy generation process that integrates order preference and state information, which has the advantages of high scheduling efficiency, strong policy adaptability, and high structure update accuracy.
[0006] A supply chain order dynamic scheduling optimization method based on reinforcement learning according to an embodiment of the present invention includes the following steps:
[0007] Step 1: Collect supply chain data, preprocess to construct a set of state input vectors, extract order target constraint information, and preprocess to construct a set of order preference labels;
[0008] Step 2: Concatenate and encode the set of state input vectors and the set of order preference labels to generate a scheduling input tensor, and input it into the improved GhostNet with a lightweight channel reconstruction structure and a state attention-guided unit to extract the set of candidate action feature maps;
[0009] Step 3: In the action intent compression mechanism, channel sparsification and high-response path aggregation are performed on the candidate action feature map set to generate an action intent vector set;
[0010] Step 4: Input the action intent vector set and the order preference label set into the policy fusion gating structure, and generate a policy fusion weight set based on the target scheduling preference information;
[0011] Step 5: Perform dynamic path weighting on the action intent vector set based on the policy fusion weight set, and output the fused action vector set;
[0012] Step 6: Map the fused action vector set to an order scheduling action set, which includes warehouse node instructions, transportation route instructions, and scheduling time interval control instructions;
[0013] Step 7: Collect execution feedback information corresponding to the order scheduling action set, construct a feedback vector set, input the feedback vector set and the fused action vector set into the improved natural policy gradient training unit, and update the parameters of the improved GhostNet using a structure-sensitive gradient estimation method.
[0014] Optionally, step one includes: normalizing and filling missing values in the order status data, inventory status data and transportation resource status data in the supply chain data to construct a set of status input vectors with a unified scale and complete attributes; performing feature encoding and tag embedding operations on the delivery time limit, budget cost threshold and order priority level in the order target constraint information to construct a corresponding set of order preference tags.
[0015] Optionally, the improved GhostNet network in step two specifically includes:
[0016] The scheduling input tensor is input into the backbone feature extraction structure, which then performs standard one-dimensional convolution, batch normalization, and ReLU activation operations in sequence to generate a set of basic feature maps.
[0017] In each set of basic feature maps, a 3×1 depthwise separable convolution is performed to generate a set of ghost feature maps, and an initial set of feature maps is constructed through feature concatenation operations.
[0018] The initial feature map set is input into the lightweight channel reconstruction structure, which performs the following operations:
[0019] 1. Calculate the average response value for all feature channels along the channel dimension to construct the channel score vector;
[0020] 2. Set a sparsity threshold and sort the scoring vectors in descending order, retaining the top k% of channels and setting the remaining channels to zero;
[0021] 3. Perform 1×1 convolution and group normalization on the retained channels to generate an enhanced response channel set;
[0022] Fourth, the enhanced response channel set replaces the original channel to form a reconstructed channel feature map set;
[0023] The reconstructed channel feature map set is input into the state attention-guided unit, which performs the following operations:
[0024] 1. Expand the set of state input vectors into a state feature matrix according to the field dimension;
[0025] 2. Calculate the channel-dimensional dot product similarity between the state feature matrix and the reconstructed channel feature map set to generate the channel attention score matrix;
[0026] 3. Normalize the channel attention score matrix along the channel dimension to generate the channel weight vector;
[0027] Fourth, perform a point-by-point weighting operation on the reconstructed channel feature map set to obtain a candidate action feature map set.
[0028] Optionally, the action intent compression mechanism in step three includes:
[0029] Expand the candidate action feature map set by channel dimension to construct the action channel feature matrix;
[0030] For each channel in the action channel feature matrix, calculate the average activation value and standard deviation under the current state input, and construct the channel response score vector;
[0031] Based on the channel responsivity score vector, a sparsity retention ratio is set, and the top k% of channels with the highest score values are selected to construct a set of sparse channel feature maps.
[0032] The sparse channel feature map set is divided into several spatial sub-regions, and a global average pooling operation is performed on each sub-region to generate a set of local path expression vectors.
[0033] The set of local path representation vectors is subjected to max pooling and linear fusion operations in the channel dimension to generate aggregated path representation vectors.
[0034] The aggregated path representation vector and the sparse channel feature map set are weighted and superimposed point by point according to the channel dimension to generate the action intent vector set.
[0035] Optionally, the generation of the policy fusion weight set in step four includes:
[0036] The set of action intent vectors is divided into several continuous sub-vectors according to the channel dimension.
[0037] The order preference label set is normalized to construct a preference weight vector with the same dimension as the sub-vector;
[0038] After performing element-wise multiplication on each subvector and the preference weight vector, the sum is obtained to get the fusion weight value corresponding to each subvector;
[0039] Normalize all fusion weight values to form a policy fusion weight set.
[0040] Optionally, the process of generating the fused action vector set in step five includes: dividing the action intent vector set into several strategy sub-path structure units in the channel dimension; establishing a one-to-one mapping relationship between each fusion weight in the strategy fusion weight set and the corresponding strategy sub-path; and performing element-wise multiplication on each group of mapped strategy sub-path structure units to achieve dynamic weighting; and performing weighted aggregation on all weighted strategy sub-path structure units in the channel dimension to generate the fused action vector set.
[0041] Optionally, the improved natural policy gradient training unit consists of a policy distribution generation structure, a compatibility function construction structure, a Fisher information approximation calculation structure, and a policy gradient correction structure.
[0042] Among them, the strategy distribution generation structure is used to receive the fused action vector set and the order preference label set to generate the current strategy probability distribution function;
[0043] The compatibility function construction structure is used to build a policy-feedback compatibility mapping relationship based on the set of feedback vectors, and jointly expresses the expected action payoff with the current policy probability distribution function;
[0044] The Fisher information approximation structure is used to construct a local approximate representation of the Fisher information matrix based on the current policy probability distribution, and to measure the sensitivity to the policy gradient direction.
[0045] The policy gradient correction structure is used to apply the Fisher information matrix inverse mapping transformation to the gradient estimation results output by the policy distribution generation structure, generating the natural policy gradient estimation vector in the structure-sensitive direction.
[0046] Optionally, the Fisher information matrix inverse mapping transformation includes the following steps:
[0047] Under the logarithmic form of the current policy probability distribution function, the output values of each policy sub-path channel in the fused action vector set are expanded with respect to the network trainable parameters to construct a set of local gradient response vectors;
[0048] The channel-level Fisher information approximation matrix is constructed based on the set of local gradient response vectors, and the numerical estimation of the channel-level Fisher information approximation matrix is performed by the approximate second-order outer product method.
[0049] Without directly inverting the entire matrix, the conjugate gradient method is used to solve the Fisher information matrix by performing a low-rank inverse mapping, thus obtaining the low-rank inverse Fisher matrix.
[0050] The natural policy gradient estimate is left-multiplied by the low-rank inverse Fisher matrix to form a structure-sensitive orientation-adjusted natural policy gradient estimate vector, which drives the directional update of the improved GhostNet network parameters.
[0051] The beneficial effects of this invention are:
[0052] (1) Improve the efficiency of scheduling strategy generation: This invention utilizes the GhostNet network with a lightweight channel reconstruction structure and state attention-oriented unit to extract multi-scale feature information while maintaining inference efficiency, adapting to complex order states and multi-objective scheduling requirements, thereby improving the generation speed and real-time response capability of scheduling strategies.
[0053] (2) Enhance the structural sensitivity of policy updates: This invention introduces an improved natural policy gradient training unit and constructs a path-level gradient correction mechanism based on the Fisher information matrix to achieve targeted optimization of the key channel structure in the fused action policy, avoid the convergence instability problem caused by policy generalization updates, and improve training stability and scheduling accuracy.
[0054] (3) Realize dynamic fusion strategy generation driven by order preference: This invention constructs a strategy fusion gating structure and a dynamic path weighting mechanism, and combines order preference tags to weight the action intention vector set to accurately adjust the response of the scheduling strategy to multi-objective constraints, thereby improving the adaptability and flexibility of the strategy in multi-type order scenarios. Attached Figure Description
[0055] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:
[0056] Figure 1This is an overall flowchart of a reinforcement learning-based supply chain order dynamic scheduling optimization method proposed in this invention.
[0057] Figure 2 This is a diagram of the improved GhostNet architecture for a reinforcement learning-based dynamic scheduling optimization method for supply chain orders proposed in this invention.
[0058] Figure 3 This is a diagram of the improved natural policy gradient training unit structure for a reinforcement learning-based supply chain order dynamic scheduling optimization method proposed in this invention. Detailed Implementation
[0059] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.
[0060] refer to Figure 1-3 A reinforcement learning-based method for dynamic scheduling optimization of supply chain orders includes the following steps:
[0061] Step 1: Collect supply chain data, preprocess to construct a set of state input vectors, extract order target constraint information, and preprocess to construct a set of order preference labels;
[0062] Step 2: Concatenate and encode the set of state input vectors and the set of order preference labels to generate a scheduling input tensor, and input it into the improved GhostNet with a lightweight channel reconstruction structure and a state attention-guided unit to extract the set of candidate action feature maps;
[0063] Step 3: In the action intent compression mechanism, channel sparsification and high-response path aggregation are performed on the candidate action feature map set to generate an action intent vector set;
[0064] Step 4: Input the action intent vector set and the order preference label set into the policy fusion gating structure, and generate a policy fusion weight set based on the target scheduling preference information;
[0065] Step 5: Perform dynamic path weighting on the action intent vector set based on the policy fusion weight set, and output the fused action vector set;
[0066] Step 6: Map the fused action vector set to an order scheduling action set, which includes warehouse node instructions, transportation route instructions, and scheduling time interval control instructions;
[0067] Step 7: Collect execution feedback information corresponding to the order scheduling action set, construct a feedback vector set, input the feedback vector set and the fused action vector set into the improved natural policy gradient training unit, and update the parameters of the improved GhostNet using a structure-sensitive gradient estimation method.
[0068] This invention addresses the supply chain order scheduling problem under multiple constraints by constructing a closed-loop process from data acquisition, feature encoding, action generation to reinforcement learning adaptive optimization. It integrates an improved GhostNet and a natural policy gradient mechanism to effectively solve the problems of insufficient handling of high-dimensional state features and lack of dynamic adaptability in policy generation in traditional scheduling systems. This enables the generation of high-response path scheduling optimization strategies for multi-objective order features, improving scheduling accuracy and feedback response adaptability.
[0069] In this embodiment, step one includes: normalizing and filling missing values in the order status data, inventory status data and transportation resource status data in the supply chain data to construct a set of status input vectors with uniform scale and complete attributes; performing feature encoding and tag embedding operations on the delivery time limit, budget cost threshold and order priority level in the order target constraint information to construct a corresponding set of order preference tags.
[0070] In the process of constructing state input vectors and order preference labels, this invention introduces normalization processing, missing value imputation, and multi-dimensional label embedding operations to ensure the consistency and completeness of input features. Addressing the problems of simple order information structure and weak constraint expression in existing methods, this invention improves the order scheduling system's ability to identify multiple types of order demands and enhances its scalability, laying the foundation for diverse scheduling strategy modeling.
[0071] In this embodiment, the improved GhostNet network in step two specifically includes:
[0072] The scheduling input tensor is input into the backbone feature extraction structure, which then performs standard one-dimensional convolution, batch normalization, and ReLU activation operations in sequence to generate a set of basic feature maps.
[0073] In each set of basic feature maps, a 3×1 depthwise separable convolution is performed to generate a set of ghost feature maps, and an initial set of feature maps is constructed through feature concatenation operations.
[0074] The initial feature map set is input into the lightweight channel reconstruction structure, which performs the following operations:
[0075] 1. Calculate the average response value for all feature channels along the channel dimension to construct the channel score vector;
[0076] 2. Set a sparsity threshold and sort the scoring vectors in descending order, retaining the top k% of channels and setting the remaining channels to zero;
[0077] 3. Perform 1×1 convolution and group normalization on the retained channels to generate an enhanced response channel set;
[0078] Fourth, the enhanced response channel set replaces the original channel to form a reconstructed channel feature map set;
[0079] The reconstructed channel feature map set is input into the state attention-guided unit, which performs the following operations:
[0080] 1. Expand the set of state input vectors into a state feature matrix according to the field dimension;
[0081] 2. Calculate the channel-dimensional dot product similarity between the state feature matrix and the reconstructed channel feature map set to generate the channel attention score matrix;
[0082] 3. Normalize the channel attention score matrix along the channel dimension to generate the channel weight vector;
[0083] Fourth, perform a point-by-point weighting operation on the reconstructed channel feature map set to obtain a candidate action feature map set.
[0084] This invention introduces a lightweight channel reconstruction structure and a state attention-oriented unit into the improved GhostNet structure. It innovatively links order preference information and state features through a channel attention mechanism to make up for the shortcomings of traditional network structures in terms of poor adaptability and low response efficiency in channel-dimensional scheduling decision feature extraction. It achieves a dual improvement in the effective utilization of resources and model inference efficiency, and has high adaptability and deployment feasibility.
[0085] In this embodiment, the action intent compression mechanism in step three includes:
[0086] Expand the candidate action feature map set by channel dimension to construct the action channel feature matrix;
[0087] For each channel in the action channel feature matrix, calculate the average activation value and standard deviation under the current state input, and construct the channel response score vector;
[0088] Based on the channel responsivity score vector, a sparsity retention ratio is set, and the top k% of channels with the highest score values are selected to construct a set of sparse channel feature maps.
[0089] The sparse channel feature map set is divided into several spatial sub-regions, and a global average pooling operation is performed on each sub-region to generate a set of local path expression vectors.
[0090] The local path expression vector set is subjected to max pooling and linear fusion operations in the channel dimension to generate an aggregated path expression vector. Specifically, this involves: first, performing element-wise maximum value extraction on each local path expression vector in the channel dimension to obtain the channel maximum response vector, forming a preliminary response feature; then, constructing a channel weight coefficient vector, which is statistically obtained based on the response frequency distribution of each channel in all local paths, and is used to reflect the significance of each channel in multiple regions; next, performing an element-wise linear weighted fusion operation on the preliminary response feature and the channel weight coefficient vector to obtain a weighted response path expression; finally, performing a unified splicing and normalization operation on all weighted response path expressions in the spatial dimension to generate the final aggregated path expression vector, which is used as a modulation factor in the channel weighted superposition operation.
[0091] The aggregated path representation vector and the sparse channel feature map set are weighted and superimposed point-by-point along the channel dimension to generate an action intent vector set. The point-by-point weighted superposition operation specifically includes the following steps: First, the aggregated path representation vector is copied and expanded along the channel dimension to maintain consistency with the sparse channel feature map set in terms of spatial size and number of channels; then, the expanded aggregated path representation vector and the sparse channel feature map set are multiplied point-by-point at the corresponding channel positions, and each channel value at each spatial position is multiplied by the aggregated path weight value corresponding to the channel to obtain a channel-modulated weighted feature map set; then, all weighted channel feature maps are summed along the channel dimension to integrate them into a single tensor representation, forming an intermediate representation of the fused multi-path feature response; finally, batch normalization and nonlinear activation operations are performed on the fused representation to output the final action intent vector set, which serves as the feature basis for the policy generation stage.
[0092] The action intent compression mechanism proposed in this invention improves the representation quality and generalization ability of scheduling action features through channel sparsity, spatial location path representation, and multi-scale aggregation operations. Compared with conventional channel attention mechanisms, this mechanism takes into account both local response intensity and channel selection sparsity, and can more accurately retain key decision paths, compress redundant information, and improve the efficiency of scheduling strategy generation. It is an important innovation in the compressed representation of scheduling decision features in this invention.
[0093] In this embodiment, the generation of the strategy fusion weight set in step four includes:
[0094] The set of action intent vectors is divided into several continuous sub-vectors according to the channel dimension.
[0095] The order preference label set is normalized to construct a preference weight vector with the same dimension as the sub-vector;
[0096] After performing element-wise multiplication on each subvector and the preference weight vector, the sum is obtained to get the fusion weight value corresponding to each subvector;
[0097] Normalize all fusion weight values to form a policy fusion weight set.
[0098] The strategy fusion weight generation method proposed in this invention establishes a dynamic association mechanism between scheduling strategy and order objective by performing fine-grained matching calculation between action intention sub-vector and order preference weight vector. This solves the problem of lack of preference response adjustment capability in existing strategies, realizes adaptive adjustment of strategy structure based on label driving, and improves the consistency of scheduling response in multi-objective optimization scenarios.
[0099] In this embodiment, the process of generating the fused action vector set in step five includes: dividing the action intent vector set into several strategy sub-path structure units in the channel dimension; establishing a one-to-one mapping relationship between each fusion weight in the strategy fusion weight set and the corresponding strategy sub-path; and performing element-wise multiplication operation on each group of mapped strategy sub-path structure units to achieve dynamic weighting; and performing weighted aggregation operation on all weighted strategy sub-path structure units in the channel dimension to generate the fused action vector set.
[0100] The dynamic path weighting operation proposed in this invention integrates weights and structured action paths through a fusion strategy, enabling fine adjustment of the channel dimension path structure during the scheduling strategy generation process. This design breaks through the technical limitations of uncontrollable channels and poor response consistency in traditional reinforcement learning action representations, enabling the fused action vectors to have adjustable path structure response capabilities, thereby enhancing the flexibility and interpretability of scheduling decisions.
[0101] In this embodiment, the improved natural policy gradient training unit consists of a policy distribution generation structure, a compatibility function construction structure, a Fisher information approximation calculation structure, and a policy gradient correction structure.
[0102] The strategy distribution generation structure receives the fused action vector set and the order preference label set, and generates the current strategy probability distribution function. The strategy distribution generation structure receives the fused action vector set and the order preference label set, and first concatenates them along the feature dimension to construct a joint input vector z = [a; l], where a ∈ R. d Represents the fusion action vector, l∈R q The order preference label is represented by z, and then the joint input vector z is fed into the probability mapping function f, which consists of two fully connected layers. θ (·), thus obtaining the original action score vector s=f θ (z)∈R K Where K is the dimension of the action space;
[0103] When generating the strategy probability distribution, a structure-sensitive temperature coefficient τ∈R is introduced. + The action scores are scaled and the policy distribution π is generated using a softened normalization function. θ :
[0104]
[0105] Among them, s k This is the score for the k-th action;
[0106] The temperature coefficient τ is not a fixed value; it is dynamically calculated from the channel response distribution of the fused action vector and is defined as follows:
[0107] τ = α·std(a) + β;
[0108] Where std(·) represents the standard deviation operation, and α and β are trainable parameters;
[0109] The generation method ensures that the generated policy distribution not only reflects the action scores themselves, but also adapts to the structural complexity of the current fused features, enhancing the distribution's sensitivity to scheduling decision-making scenarios. The final output π θ (a k |z) constitutes the current policy probability distribution function, which serves as the basis for subsequent policy gradient calculation and Fisher information estimation.
[0110] The compatibility function construction structure is used to build a policy-feedback compatibility mapping relationship based on the set of feedback vectors, and jointly express the expected action reward with the current policy probability distribution function. Specifically, the compatibility function construction structure receives a fused action vector and a set of feedback vectors. First, it performs a nonlinear mapping on the feedback vectors to generate a feedback response representation. Then, it performs a Hadamard product operation on the channel dimension of the feedback response representation and the corresponding fused action vector to construct a policy-feedback compatibility vector, which is used to capture the fine-grained coupling relationship between the action channel response and the feedback signal. Finally, by constructing a compatibility function, it weights and combines each compatibility vector with the current policy probability distribution function to generate an expression of the expected action reward that reflects the policy execution effect, providing a structure-sensitive gradient evaluation basis for natural policy gradient calculation.
[0111] The Fisher information approximation structure is used to construct a local approximate representation of the Fisher information matrix based on the current policy probability distribution, and to measure the sensitivity to the policy gradient direction.
[0112] The policy gradient correction structure is used to apply the Fisher information matrix inverse mapping transformation to the gradient estimation results output by the policy distribution generation structure, generating the natural policy gradient estimation vector in the structure-sensitive direction.
[0113] The improved natural policy gradient training unit constructed in this invention combines four structures: policy distribution modeling, compatibility mapping, Fisher information matrix estimation, and natural gradient correction. This improves the directionality and stability of policy gradient updates, solves the problems of unstable training and severe gradient oscillation in the high-dimensional action space of the supply chain in traditional reinforcement learning methods, and provides an efficient and reliable optimization path for the implementation of reinforcement learning scheduling algorithms in complex industrial environments.
[0114] In this embodiment, the Fisher information matrix inverse mapping transformation includes the following steps:
[0115] Under the logarithmic form of the current policy probability distribution function, the output values of each policy sub-path channel in the fused action vector set are expanded with respect to the network trainable parameters to construct a set of local gradient response vectors;
[0116] The channel-level Fisher information approximation matrix is constructed based on the set of local gradient response vectors, and the numerical estimation of the channel-level Fisher information approximation matrix is performed by the approximate second-order outer product method.
[0117] Without directly inverting the entire matrix, the conjugate gradient method is used to solve the Fisher information matrix by performing a low-rank inverse mapping, thus obtaining the low-rank inverse Fisher matrix.
[0118] The natural policy gradient estimate is left-multiplied by the low-rank inverse Fisher matrix to form a structure-sensitive orientation-adjusted natural policy gradient estimate vector, which drives the directional update of the improved GhostNet network parameters.
[0119] This invention introduces channel-level gradient response estimation and low-rank approximation operation of conjugate gradient method in the process of solving the Fisher information matrix inverse mapping, avoiding the direct inversion operation of high-dimensional matrix, effectively reducing algorithm complexity, improving training stability and convergence efficiency. This method takes into account the requirements of industrial-grade systems for computing resources and speed while ensuring the accuracy of natural gradient direction, and is a technical breakthrough in the structure-sensitive update method of reinforcement learning model.
[0120] Example 1:
[0121] To verify the feasibility of this invention in practice, it was applied to a supply chain scheduling task in a manufacturing industry. The order types involved included urgent orders, large-volume standard orders, and multi-destination small-volume orders. Warehouse resources were redundant in multiple regions, and transportation resources had dynamic fluctuation characteristics. The scheduling strategy needed to take into account three scheduling objectives: timeliness, cost control, and resource utilization. In traditional rule-based scheduling systems, problems such as uneven resource utilization, frequent instruction changes, and delayed scheduling response often occur because the real-time preferences of orders and scheduling feedback cannot be fully considered.
[0122] In real-world scenarios, the scheduling system collects historical supply chain execution data to construct a unified set of state input vectors containing attributes such as order status, inventory status, and transportation status. Simultaneously, it performs label encoding on order targets to generate a set of order preference labels. The system uses the state input vectors and order preference labels as basic inputs and feeds them into an improved GhostNet structure. This structure extracts responsive features from scheduling target constraints through lightweight channel reconstruction and a state attention-oriented mechanism, outputting a set of candidate action feature maps.
[0123] Building upon this foundation, the system utilizes an action intent compression mechanism to perform channel-dimensional sparsity processing and spatial path aggregation on candidate action feature maps, constructing action intent vectors. Combined with order preference information, it generates strategy fusion weights and further generates fused action vectors through a dynamic path weighting method. These fused action vectors are mapped to actual scheduling action instructions, including specifying warehouse nodes, selecting transportation routes, and allocating time intervals. The system collects scheduling feedback information and continuously updates the GhostNet network weights by constructing feedback vectors and natural policy gradient estimation structures, thereby achieving dynamically adaptive optimized scheduling decisions.
[0124] The method described in this invention was subjected to multiple rounds of scheduling execution experiments with a comparison system. Scheduling instruction response latency, resource utilization, and order fulfillment rate were measured. Experimental data showed that after adopting the method of this invention, the average scheduling response time decreased from 3.7 seconds to 2.4 seconds, the average warehouse resource utilization rate increased from 64.1% to 82.7%, the transportation resource utilization rate increased by approximately 15.9%, and the on-time order delivery rate increased to 92.3%. Especially in order scheduling scenarios facing tight delivery deadlines and frequent resource competition conflicts, the integrated strategy generation mechanism constructed in this invention can improve decision-making flexibility and strategy targeting, reduce the number of scheduling failure rollbacks, and reduce the time consumed by system replanning.
[0125] The stability of scheduling behavior has also been significantly enhanced. According to statistics, the consistency of scheduling actions in three consecutive rounds of scheduling operations has increased from 68.2% to 87.4%, which shows that the method of the present invention has better strategy continuity and control robustness under complex dynamic constraints. It is particularly suitable for supply chain operation scenarios with multiple constraints and frequent changes in feedback.
[0126] Table 1: Comparison of Supply Chain Scheduling Indicators Based on Improved Reinforcement Learning Methods
[0127] Indicator Name Results of the original scheduling method Results of the method of the present invention Increase Average scheduling response time (seconds) 3.7 2.4 35.1％ Warehouse resource utilization rate (%) 64.1 82.7 29.1％ Transportation resource utilization rate (%) 58.3 74.2 15.9％ On-time delivery rate (%) 76.5 92.3 20.6％ Consistency of scheduling actions (%) 68.2 87.4 28.2％ Feedback processing time (seconds) 1.5 0.9 40.0％
[0128] The table above compares the performance of traditional scheduling methods with the improved reinforcement learning method proposed in this invention in a real supply chain order scheduling scenario based on six key performance indicators. The results fully demonstrate the significant advantages of the proposed method in terms of scheduling efficiency, resource utilization, and strategy stability.
[0129] Regarding scheduling response speed, the method of this invention reduces the average scheduling response time from 3.7 seconds to 2.4 seconds, an improvement of 35.1%. This indicates that the method of this invention has faster processing and decision-making capabilities when facing complex order scheduling requests. In terms of warehouse resource utilization, the method of this invention increases to 82.7%, an improvement of 29.1% compared to the original method's 64.1%. This shows that the method can effectively integrate the storage capacity of warehouse nodes and optimize inventory turnover. In terms of transportation resource utilization, it increases from 58.3% to 74.2%, an improvement of 15.9%. This means that the optimized scheduling action can more rationally allocate transportation routes and times, improving the operational efficiency of the transportation process.
[0130] In terms of on-time order delivery rate, it increased from 76.5% to 92.3%, a 20.6% improvement, reducing delivery delays caused by unreasonable scheduling strategies. Meanwhile, the invention demonstrated excellent performance in scheduling consistency, increasing from 68.2% to 87.4%, a 28.2% increase. This reflects a more stable and reliable learned strategy, which is beneficial for subsequent deployment and large-scale application. Feedback processing time was also reduced from 1.5 seconds to 0.9 seconds, a 40% reduction, enhancing the real-time response capability of the scheduling model and helping to quickly perceive and adapt to dynamic changes in the supply chain.
[0131] In summary, this invention optimizes the state perception, action generation, and policy learning stages in the order scheduling process by introducing an improved GhostNet network structure and a natural policy gradient training mechanism. This effectively improves the overall operational efficiency and intelligence level of the supply chain system, demonstrating outstanding practical value and promising prospects for wider application.
[0132] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.
Claims
1. A supply chain order dynamic scheduling optimization method based on reinforcement learning, characterized in that, Includes the following steps: Step 1: Collect supply chain data, preprocess to construct a set of state input vectors, extract order target constraint information, and preprocess to construct a set of order preference labels; Step 2: Concatenate and encode the set of state input vectors and the set of order preference labels to generate a scheduling input tensor, and input it into the improved GhostNet with a lightweight channel reconstruction structure and a state attention-guided unit to extract the set of candidate action feature maps; Step 3: In the action intent compression mechanism, channel sparsification and high-response path aggregation are performed on the candidate action feature map set to generate an action intent vector set; Step 4: Input the action intent vector set and the order preference label set into the policy fusion gating structure, and generate a policy fusion weight set based on the target scheduling preference information; Step 5: Perform dynamic path weighting on the action intent vector set based on the policy fusion weight set, and output the fused action vector set; Step 6: Map the fused action vector set to an order scheduling action set, which includes warehouse node instructions, transportation route instructions, and scheduling time interval control instructions; Step 7: Collect execution feedback information corresponding to the order scheduling action set, construct a feedback vector set, input the feedback vector set and the fused action vector set into the improved natural policy gradient training unit, and update the parameters of the improved GhostNet using a structure-sensitive gradient estimation method.
2. The supply chain order dynamic scheduling optimization method based on reinforcement learning according to claim 1, characterized in that, Step one includes: normalizing and filling missing values in the order status data, inventory status data and transportation resource status data in the supply chain data to construct a set of status input vectors with uniform scale and complete attributes; performing feature encoding and tag embedding operations on the delivery time limit, budget cost threshold and order priority level in the order target constraint information to construct a corresponding set of order preference tags.
3. The supply chain order dynamic scheduling optimization method based on reinforcement learning according to claim 2, characterized in that, The improved GhostNet network in step two specifically includes: The scheduling input tensor is input into the backbone feature extraction structure, which then performs standard one-dimensional convolution, batch normalization, and ReLU activation operations in sequence to generate a set of basic feature maps. In each set of basic feature maps, a 3×1 depthwise separable convolution is performed to generate a set of ghost feature maps, and an initial set of feature maps is constructed through feature concatenation operations. The initial feature map set is input into the lightweight channel reconstruction structure, which performs the following operations:
1. Calculate the average response value for all feature channels along the channel dimension to construct the channel score vector; 2. Set a sparsity threshold and sort the scoring vectors in descending order, retaining the top k% of channels and setting the remaining channels to zero; 3. Perform 1×1 convolution and group normalization on the retained channels to generate an enhanced response channel set; Fourth, the enhanced response channel set replaces the original channel to form a reconstructed channel feature map set; The reconstructed channel feature map set is input into the state attention-guided unit, which performs the following operations:
1. Expand the set of state input vectors into a state feature matrix according to the field dimension; 2. Calculate the channel-dimensional dot product similarity between the state feature matrix and the reconstructed channel feature map set to generate the channel attention score matrix; 3. Normalize the channel attention score matrix along the channel dimension to generate the channel weight vector; Fourth, perform a point-by-point weighting operation on the reconstructed channel feature map set to obtain a candidate action feature map set.
4. The supply chain order dynamic scheduling optimization method based on reinforcement learning according to claim 3, characterized in that, The action intent compression mechanism in step three includes: Expand the candidate action feature map set by channel dimension to construct the action channel feature matrix; For each channel in the action channel feature matrix, calculate the average activation value and standard deviation under the current state input, and construct the channel response score vector; Based on the channel responsivity score vector, a sparsity retention ratio is set, and the top k% of channels with the highest score values are selected to construct a set of sparse channel feature maps. The sparse channel feature map set is divided into several spatial sub-regions, and a global average pooling operation is performed on each sub-region to generate a set of local path expression vectors. The set of local path representation vectors is subjected to max pooling and linear fusion operations in the channel dimension to generate aggregated path representation vectors. The aggregated path representation vector and the sparse channel feature map set are weighted and superimposed point by point according to the channel dimension to generate the action intent vector set.
5. The supply chain order dynamic scheduling optimization method based on reinforcement learning according to claim 4, characterized in that, The generation of the strategy fusion weight set in step four includes: The set of action intent vectors is divided into several continuous sub-vectors according to the channel dimension. The order preference label set is normalized to construct a preference weight vector with the same dimension as the sub-vector; After performing element-wise multiplication on each subvector and the preference weight vector, the sum is obtained to get the fusion weight value corresponding to each subvector; Normalize all fusion weight values to form a policy fusion weight set.
6. The supply chain order dynamic scheduling optimization method based on reinforcement learning according to claim 5, characterized in that, The process of generating the fused action vector set in step five includes: dividing the action intent vector set into several strategy sub-path structure units in the channel dimension; establishing a one-to-one mapping relationship between each fusion weight in the strategy fusion weight set and the corresponding strategy sub-path; and performing element-wise multiplication on each group of mapped strategy sub-path structure units to achieve dynamic weighting; and performing weighted aggregation on all weighted strategy sub-path structure units in the channel dimension to generate the fused action vector set.
7. The supply chain order dynamic scheduling optimization method based on reinforcement learning according to claim 6, characterized in that, The improved natural policy gradient training unit consists of a policy distribution generation structure, a compatibility function construction structure, a Fisher information approximation calculation structure, and a policy gradient correction structure. Among them, the strategy distribution generation structure is used to receive the fused action vector set and the order preference label set to generate the current strategy probability distribution function; The compatibility function construction structure is used to build a policy-feedback compatibility mapping relationship based on the set of feedback vectors, and jointly expresses the expected action payoff with the current policy probability distribution function; The Fisher information approximation structure is used to construct a local approximate representation of the Fisher information matrix based on the current policy probability distribution, and to measure the sensitivity to the policy gradient direction. The policy gradient correction structure is used to apply the Fisher information matrix inverse mapping transformation to the gradient estimation results output by the policy distribution generation structure, generating the natural policy gradient estimation vector in the structure-sensitive direction.
8. The supply chain order dynamic scheduling optimization method based on reinforcement learning according to claim 7, characterized in that, The Fisher information matrix inverse mapping transformation includes the following steps: Under the logarithmic form of the current policy probability distribution function, the output values of each policy sub-path channel in the fused action vector set are expanded with respect to the network trainable parameters to construct a set of local gradient response vectors; The channel-level Fisher information approximation matrix is constructed based on the set of local gradient response vectors, and the numerical estimation of the channel-level Fisher information approximation matrix is performed by the approximate second-order outer product method. Without directly inverting the entire matrix, the conjugate gradient method is used to solve the Fisher information matrix by performing a low-rank inverse mapping, thus obtaining the low-rank inverse Fisher matrix. The natural policy gradient estimate is left-multiplied by the low-rank inverse Fisher matrix to form a structure-sensitive orientation-adjusted natural policy gradient estimate vector, which drives the directional update of the improved GhostNet network parameters.