A supply chain scheduling optimization system based on reinforcement learning
By using a reinforcement learning-based supply chain scheduling optimization system, the system identifies overlapping order segments and rearranges the arrival sequence, thus resolving inventory management deviations caused by overlapping orders. This enables more accurate replenishment decisions and inventory control, thereby improving the overall scheduling efficiency of the supply chain.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- YANGO UNIV
- Filing Date
- 2026-03-31
- Publication Date
- 2026-07-21
Smart Images
Figure CN121981339B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of intelligent supply chain scheduling technology, and more specifically, to a supply chain scheduling optimization system based on reinforcement learning. Background Technology
[0002] In the field of supply chain management, existing supply chain scheduling methods typically predict inventory consumption based on the order of order placement and formulate corresponding replenishment instructions. However, in actual supply chain operations, factors such as logistics delays and fluctuations in order execution efficiency often cause orders to arrive at supply chain nodes in a different order than the order of order placement, resulting in order overlap. This makes replenishment strategies based on the order of order placement unable to accurately reflect the true changes in inventory.
[0003] Existing supply chain scheduling methods fail to effectively handle inventory management deviations caused by cross-arrivals of orders, resulting in a mismatch between replenishment decisions and actual inventory changes, which reduces the accuracy of overall supply chain inventory control and resource scheduling. Summary of the Invention
[0004] To overcome the aforementioned deficiencies of the prior art, embodiments of the present invention provide a supply chain scheduling optimization system based on reinforcement learning to solve the problems mentioned in the background art.
[0005] To achieve the above objectives, the present invention provides the following technical solution:
[0006] A supply chain scheduling optimization system based on reinforcement learning includes:
[0007] Time-series data acquisition module: Acquires inventory data, replenishment instruction data, order placement time data, and order arrival time data at supply chain nodes to form a time-series dataset of orders in transit;
[0008] Cross-identification module: Calculates the order sequence inversion relationship based on order placement time data and order arrival time data, identifies order cross-sections, and generates order cross-identification data;
[0009] The order reordering module rearranges the arrival order of the time-series dataset of in-transit orders based on the order cross-identification data, and constructs the actual arrival sequence data of in-transit orders;
[0010] Path correction module: Based on the actual arrival sequence data of orders in transit and the inventory data of supply chain nodes, calculate the inventory evolution path under the condition of order crossover and generate state transition correction data;
[0011] Attribution Correction Module: Based on state transition correction data, replenishment instruction data and corresponding arrival feedback data are re-associated to construct revenue attribution correction data;
[0012] Policy Update Module: Inputs state transition correction data and revenue attribution correction data into the reinforcement learning scheduling model to update the policy and outputs a supply chain scheduling optimization policy.
[0013] In a preferred embodiment, inventory data, replenishment instruction data, order placement time data, and order arrival time data of supply chain nodes are acquired to form a time-series dataset of in-transit orders. Specifically:
[0014] Inventory data at each node of the supply chain is continuously collected in chronological order and time-aligned to form a time-series sequence of inventory changes;
[0015] Based on the time series of inventory changes, replenishment order data within the corresponding time range is matched to construct the correspondence data between inventory changes and replenishment orders;
[0016] Time-index the replenishment order data to extract the order placement time data;
[0017] Parse the order execution records corresponding to the replenishment instruction data and extract the order arrival time data;
[0018] The replenishment instruction data is correlated and organized based on the order placement time data and the order arrival time data to form a time-series dataset of orders in transit.
[0019] In a preferred embodiment, the order sequence inversion relationship is calculated based on order placement time data and order arrival time data, order overlap segments are identified, and order overlap identification data is generated, specifically as follows:
[0020] Based on the time-series dataset of orders in transit, the correspondence between order placement time data and order arrival time data is extracted. Order sequences are constructed according to the order placement time order and the corresponding order arrival time positions are recorded synchronously.
[0021] The system compares the order placement time sequence with the order arrival time sequence of adjacent orders in the order sequence to identify order pairs whose order placement time sequence and order arrival time sequence are inconsistent.
[0022] The continuous distribution intervals of order pairs in the time series dataset of orders in transit are merged to form a set of order cross segments;
[0023] Based on the set of order cross segments, the corresponding orders are marked to generate order cross-identification data that corresponds one-to-one with the time series dataset of orders in transit.
[0024] In a preferred embodiment, the arrival order of the in-transit order time series dataset is rearranged based on the order cross-identification data to construct the actual arrival sequence data of the in-transit orders, specifically as follows:
[0025] Based on the order cross-identification data, extract the marked order cross-segments in the time series dataset of in-transit orders;
[0026] Orders within overlapping order segments are reordered based on their arrival time data to construct the order arrival order within the segment;
[0027] Replace the original order placement time order with the new order arrival order order within the order cross-section;
[0028] Consistency verification is performed between the replaced overall order sequence and the original order placement time data to form the actual arrival sequence data of orders in transit.
[0029] In a preferred embodiment, based on the actual arrival sequence data of orders in transit and the inventory data of supply chain nodes, the inventory evolution path under the condition of order crossover is calculated, and state transition correction data is generated, specifically as follows:
[0030] Based on the actual arrival sequence data of orders in transit, the inventory update relationship between the order arrival time and the inventory data of the supply chain nodes is determined, forming a time-by-time inventory status change path;
[0031] Based on the set of order cross segments, extract the time-by-time inventory status change path within the order cross segments, and distinguish the individual impact of order arrivals on inventory changes within the order cross segments;
[0032] By associating and combining the inventory change results after separate differentiation within the order cross-segment with the inventory status change path, the transfer path of inventory status under the order cross-condition is obtained, forming status transfer correction data.
[0033] In a preferred embodiment, replenishment instruction data and corresponding arrival feedback data are re-associated based on state transition correction data to construct revenue attribution correction data, specifically as follows:
[0034] Based on the state transition correction data, extract the transition path of inventory status under the condition of order crossover; determine the order arrival time corresponding to the replenishment instruction data according to the transition path of inventory status;
[0035] The order arrival time is rematched with the replenishment instruction data and a corresponding relationship is established, forming a new association between the replenishment instruction data and the corresponding arrival feedback data under the condition of order overlap;
[0036] Based on the new relationships, the true impact of replenishment order data on inventory changes is determined, resulting in revenue attribution correction data.
[0037] In a preferred embodiment, state transition correction data and revenue attribution correction data are input into the reinforcement learning scheduling model for policy updates, and a supply chain scheduling optimization policy is output, specifically as follows:
[0038] Input data for constructing supply chain scheduling status is based on state transition correction data and revenue attribution correction data;
[0039] The state transition relationships and revenue distribution in the reinforcement learning scheduling model are corrected and updated using the input data of the supply chain scheduling status.
[0040] The updated reinforcement learning scheduling model is used to calculate the replenishment action sequence under the supply chain scheduling state;
[0041] The replenishment sequence is sorted according to its correspondence with the inventory data of the supply chain nodes to form a supply chain scheduling optimization strategy.
[0042] The technical effects and advantages of the supply chain scheduling optimization system based on reinforcement learning in this invention are as follows:
[0043] By acquiring inventory data, replenishment instruction data, order placement time data, and order arrival time data from supply chain nodes to form an in-transit order time-series dataset, the basic data related to replenishment scheduling during supply chain operations has a unified time-series organization format. By identifying the order sequence reversal relationship and generating order cross-identification data, the order cross-segment can be accurately located. By rearranging the arrival sequence of the in-transit order time-series dataset, the actual arrival sequence data of in-transit orders that better reflects the actual business process can be reconstructed. By combining the inventory data of supply chain nodes to calculate the inventory evolution path under order cross-conditions and forming state transition correction data, the inventory state change process can be more realistically represented. By re-linking replenishment instruction data with corresponding arrival feedback data to form revenue attribution correction data, the accuracy of revenue attribution for replenishment actions can be improved. By inputting the state transition correction data and revenue attribution correction data into the reinforcement learning scheduling model for policy updates, the consistency between the supply chain scheduling optimization strategy and the actual inventory evolution process can be improved. Attached Figure Description
[0044] Figure 1 This is a schematic diagram of the structure of a supply chain scheduling optimization system based on reinforcement learning according to the present invention. Detailed Implementation
[0045] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.
[0046] Example 1
[0047] Figure 1This invention presents a supply chain scheduling optimization system based on reinforcement learning, comprising:
[0048] Time-series data acquisition module: Acquires inventory data, replenishment instruction data, order placement time data, and order arrival time data at supply chain nodes to form a time-series dataset of orders in transit;
[0049] Cross-identification module: Calculates the order sequence inversion relationship based on order placement time data and order arrival time data, identifies order cross-sections, and generates order cross-identification data;
[0050] The order reordering module rearranges the arrival order of the time-series dataset of in-transit orders based on the order cross-identification data, and constructs the actual arrival sequence data of in-transit orders;
[0051] Path correction module: Based on the actual arrival sequence data of orders in transit and the inventory data of supply chain nodes, calculate the inventory evolution path under the condition of order crossover and generate state transition correction data;
[0052] Attribution Correction Module: Based on state transition correction data, replenishment instruction data and corresponding arrival feedback data are re-associated to construct revenue attribution correction data;
[0053] Policy Update Module: Inputs state transition correction data and revenue attribution correction data into the reinforcement learning scheduling model to update the policy and outputs a supply chain scheduling optimization policy.
[0054] Acquire inventory data, replenishment instruction data, order placement time data, and order arrival time data at supply chain nodes to form a time-series dataset of orders in transit, including:
[0055] Inventory data at each node of the supply chain is continuously collected in chronological order and time-aligned to form a time-series sequence of inventory changes;
[0056] During the collection of inventory data at each supply chain node, a uniform sampling frequency is set, for example, once every 5 minutes. Inventory quantity data at different locations of each supply chain node are collected simultaneously and synchronously. The collected data is in the form of numerical inventory quantity data, with the unit being pieces. After obtaining the inventory quantity data of each supply chain node, the collection time point of each inventory data is marked in the form of a timestamp, and all the inventory quantity data of the supply chain node are sorted according to the marked timestamp. If the inventory quantity data of the supply chain node is missing at a specific time, a linear interpolation method is used to fill in the missing data to maintain the continuity of the inventory change time series. The inventory quantity data of the supply chain node after collection, marking and interpolation are arranged in chronological order to form an inventory change time series.
[0057] Based on the time series of inventory changes, replenishment order data within the corresponding time range is matched to construct the correspondence data between inventory changes and replenishment orders;
[0058] The replenishment instruction data includes, but is not limited to, the time the instruction was issued, the quantity of the items specified for replenishment, and the supply chain node identifier. The time of instruction issuance is marked with a timestamp accurate to the minute. Based on the time range of inventory quantity changes in the inventory change time series, replenishment instruction data within the same time period of the inventory change time series is extracted. By comparing the inventory quantity change trend in the inventory change time series with the quantity of the items specified for replenishment in the extracted replenishment instruction data, the inventory quantity change trend and the corresponding replenishment instruction are matched one by one. For example, the matching method can be set such that if the inventory quantity change trend and the item quantity change in the replenishment instruction data both show an increasing trend within the same time window, a correlation is established, and the correlation is stored in a structured data table to form the correspondence data between inventory changes and replenishment instructions.
[0059] Time-index the replenishment order data to extract the order placement time data;
[0060] For each replenishment instruction data that matches the inventory change time series, the timestamp of the instruction issuance time recorded in the replenishment instruction data is used as the specific time point of order placement, and the order placement time data corresponding to each replenishment instruction data is recorded. When multiple replenishment instruction data appear at the same time, they are indexed according to the actual sending order recorded in the replenishment instruction data, ensuring a one-to-one correspondence between each replenishment instruction data and its corresponding order placement time data. Through indexing processing, the order placement time data represented by timestamps is obtained, forming a structured dataset containing replenishment instruction data and its corresponding order placement time data.
[0061] Parse the order execution records corresponding to the replenishment instruction data and extract the order arrival time data;
[0062] Order execution records include, but are not limited to, logistics and transportation records, warehousing and inbound records, and records of the flow of goods between supply chain nodes. When parsing order execution records, the time in the order execution record that represents the actual completion and arrival of the order at the target supply chain node is defined. For example, when the order execution record shows a warehousing and inbound completion status and has an inbound timestamp, the warehousing and inbound completion time is used as the order arrival time data. When a logistics and transportation record exists, the time of the last confirmed arrival node in the logistics and transportation record is used as the order arrival time data. The order execution records corresponding to each replenishment instruction data are parsed in turn to form order arrival time data with timestamps accurate to the minute. At the same time, the one-to-one correspondence between order arrival time data and replenishment instruction data is ensured, and a structured dataset containing replenishment instruction data and order arrival time data is constructed.
[0063] Based on the order placement time data and order arrival time data, the replenishment instruction data is correlated and organized to form a time-series dataset of in-transit orders;
[0064] Based on order placement time data, all replenishment instruction data are sorted according to the order placement sequence. Based on order arrival time data, the order arrival time data corresponding to each replenishment instruction is associated with the order placement time data. Using an association and organization method, such as setting the order placement time data as the index and the order arrival time data as the value, each replenishment instruction data is reorganized to obtain the complete order lifecycle for each replenishment instruction, including both order placement and order arrival time nodes. Then, based on the inventory quantity change trend in the inventory change time series, the inventory change impact path of each replenishment instruction data within the order lifecycle is supplemented, improving the dynamic process data of the replenishment instruction data within the order lifecycle. The replenishment instruction data, after sorting, association, and supplementation of inventory change paths, is then continuously arranged and structured in chronological order to form an in-transit order time series dataset. The in-transit order time series dataset includes, but is not limited to, inventory change time series, order placement time data, and order arrival time data.
[0065] Based on order placement time data and order arrival time data, the order sequence inversion relationship is calculated, overlapping order segments are identified, and order overlap identification data is generated, including:
[0066] Based on the time-series dataset of orders in transit, the correspondence between order placement time data and order arrival time data is extracted. Order sequences are constructed according to the order placement time order and the corresponding order arrival time positions are recorded synchronously.
[0067] Order placement time and arrival time data are extracted from the in-transit order time series dataset. Each order placement time corresponds to a unique replenishment instruction and arrival time, ensuring a one-to-one correspondence between order placement and arrival. Order placement and arrival times are recorded with timestamps accurate to the minute, and both are recorded with the same precision. During order sequence construction, the order placement times are arranged chronologically to form an ordered sequence. Simultaneously, each order arrival time is linked to its corresponding order placement time, creating an order sequence based on the order placement times. For example, if multiple orders are placed simultaneously, the order sequence is determined by the actual sending order recorded in the replenishment instruction data.
[0068] The system compares the order placement time sequence with the order arrival time sequence of adjacent orders in the order sequence to identify order pairs whose order placement time sequence and order arrival time sequence are inconsistent.
[0069] From the constructed order sequence, extract the order placement time data and corresponding order arrival time data of any two adjacent orders; define the order placement time order, that is, determine the order based on the arrangement of the order placement time data in the order sequence; then define the order arrival time order, that is, compare the order arrival time data of two adjacent orders to determine the actual arrival order; if the order placement time order and the order arrival time order of two adjacent orders are the same, that is, the order placed first also arrives first, then it is considered that there is no inconsistency; if there is a situation where the order placed first arrives later and the order placed later arrives first, it is determined that the order pair has an inconsistent order order; complete the check of all adjacent order pairs in the order sequence through a comparison operation, and record all order pairs whose order placement time order and order arrival time order are inconsistent.
[0070] The continuous distribution intervals of order pairs in the time series dataset of orders in transit are merged to form a set of order cross segments;
[0071] Based on all identified order pairs where the order placement time sequence does not match the order arrival time sequence, the position of each order pair in the order sequence is determined from the in-transit order time series dataset, and the start and end positions of each order pair are determined. For multiple continuously distributed or partially overlapping order pairs, their position ranges are merged to form continuous order cross segments. The principle of segment merging is set, for example, that if at least one order belongs to the cross position range of multiple order pairs, the position ranges of the order pairs are merged into a single segment to avoid overlap or redundancy between segments. Finally, through continuous merging, one or more order cross segments are obtained, and each order cross segment is identified by the start and end order placement time data to form a set of order cross segments.
[0072] Based on the set of cross-segments of orders, the corresponding orders are marked to generate cross-identification data of orders that correspond one-to-one with the time-series dataset of orders in transit.
[0073] Extract the start and end positions of each order cross segment from the constructed set of order cross segments to define the coverage area of the order cross segments; based on the start and end positions of the order cross segments, identify all orders in the in-transit order time series dataset that belong to the range of each order cross segment; mark each order that belongs to the range of the order cross segment, for example, by adding a cross identifier to each order in the original order sequence; the cross identifier can be represented by Boolean or numeric type, for example, a value of 1 indicates that it belongs to an order cross segment, and a value of 0 indicates that it does not belong to any order cross segment; after the marking process, obtain order cross identifier data that corresponds one-to-one with each order in the in-transit order time series dataset; the order cross identifier data identifies the order cross situation and maintains a correspondence with the original order sequence data.
[0074] Based on the order cross-identification data, the arrival order of the time-series dataset of orders in transit is rearranged to construct the actual arrival sequence data of orders in transit, including:
[0075] Based on the order cross-identification data, extract the marked order cross-segments in the time series dataset of in-transit orders;
[0076] Based on the cross-identifiers contained in the order cross-identification data, it is determined whether each order in the in-transit order time series dataset belongs to an order cross-segment. For example, a cross-identifier value of 1 indicates that the order is within a cross-segment, and a value of 0 indicates that the order is not within any cross-segment. After defining the cross-identifiers, all order data with a cross-identifier of 1 are extracted from the in-transit order time series dataset and arranged in chronological order according to the order placement time data to form an ordered sequence of order cross-segments. If there are multiple non-contiguous order cross-segments in the in-transit order time series dataset, the order data in each order cross-segment is extracted independently to ensure that the boundaries of each order cross-segment are clear. Through the extraction process, order data within all order cross-segments are obtained, and the one-to-one correspondence between each order cross-segment and the order cross-identification data is maintained to form an order cross-segment data set.
[0077] Orders within overlapping order segments are reordered based on their arrival time data to construct the order arrival order within the segment;
[0078] Taking the order data within each order cross-segment as the object, the order arrival time data corresponding to each order in each order cross-segment is extracted from the order cross-segment data set. The order arrival time data is in timestamps accurate to the minute. Based on the order arrival time data, the orders within each order cross-segment are sorted according to their time sequence, that is, orders with earlier arrival time data are placed in the first position of the sort, and orders with later arrival time data are placed in the last position of the sort. For example, when multiple orders have the same order arrival time data, the relative order between the orders is determined by the order of the order placement time data. The sorting operation completes the reconstruction of the order arrival order within each order cross-segment, thereby obtaining the order arrival order corresponding to the order arrival time data within each order cross-segment.
[0079] Replace the original order placement time order with the new order arrival order order within the order cross-section;
[0080] After constructing the order arrival order within each order cross-segment, the position range of all orders within each order cross-segment in the original order sequence of the in-transit order time series dataset is determined. Based on the reordered order arrival order, the positions of the corresponding orders in the original order sequence are replaced sequentially according to their order arrival order. The replacement method is that all orders within the order cross-segment in the original order sequence are overwritten according to the reordered order arrival order, that is, the original order placement time sequence data at the corresponding position is replaced with the new order arrival order. During the replacement process, it is ensured that only segments with order cross-segments are adjusted to form a new order sequence containing the reordered order arrival order, thus completing the reconstruction of the order cross-segments in the in-transit order time series dataset to reflect the actual order arrival status.
[0081] Consistency verification is performed between the replaced overall order sequence and the original order placement time data to form the actual arrival sequence data of orders in transit;
[0082] Using the order sequence after the replacement operation as the overall order order order sequence, the data of each order in the overall order order sequence is compared and verified with the original order placement time data. The verification method is set as follows: for each order data in the overall order order sequence, determine its position in the overall order order sequence and compare it with its position in the original order placement time data sequence. By comparing the differences in the order positions in the two sequences, it is confirmed whether the replacement operation in the order intersection segment is consistent with the original order placement time data. If a difference is found between the order position in the overall order order sequence and the position in the original order placement time data sequence, it is determined whether the difference is caused by reordering in the order intersection segment. Through consistency verification, the correlation between the overall order order sequence after replacement and the original order placement time data is guaranteed, preventing data inconsistency caused by incorrect adjustment of position data outside the order intersection segment. After completing the consistency verification, the overall order sequence that has been verified and passed the consistency verification is defined as the actual arrival sequence data of orders in transit.
[0083] Based on real arrival sequence data of orders in transit combined with inventory data of supply chain nodes, the inventory evolution path under order crossover conditions is calculated, and state transition correction data is generated, including:
[0084] Based on the actual arrival sequence data of orders in transit, the inventory update relationship between the order arrival time and the inventory data of the supply chain nodes is determined, forming a time-by-time inventory status change path;
[0085] The real-time arrival sequence data for orders in transit includes, but is not limited to, the order arrival time data for each order and the corresponding replenishment item quantity data. Order arrival time data is represented by a timestamp accurate to the minute, and replenishment item quantity data is represented by a numerical data item count. When determining inventory update relationships, the order arrival time data for each order in the real-time arrival sequence data is used as a benchmark, and the corresponding replenishment item quantity data is associated with the inventory data at each supply chain node. For example, when the order arrival time data for a certain order matches the timestamp of the supply chain node inventory data, the replenishment item quantity data for that order is added to the inventory data at the corresponding time. If the order arrival time data and inventory data... When the sampling timestamps are not completely consistent, an interpolation method is used to determine the inventory data at the order arrival time data point, and then the updated inventory is calculated. The above interpolation method can be set to use linear interpolation, that is, to perform linear calculations between the inventory data before and after the order arrival time data to determine the initial inventory value at the time of order arrival. Through calculation and interpolation, the inventory data corresponding to each order is updated, forming a time-by-time inventory status change path. The time-by-time inventory status change path is represented by the update of inventory as the order actually arrives, and is marked with timestamps accurate to the minute, thereby reflecting the time-by-time changes in the inventory data of the supply chain nodes as the actual arrival sequence data of orders in transit changes.
[0086] Based on the set of order cross segments, extract the time-by-time inventory status change path within the order cross segments, and distinguish the individual impact of order arrivals on inventory changes within the order cross segments;
[0087] Based on the set of order cross-segments, inventory data within the time range corresponding to the set of order cross-segments is extracted from the time-by-time inventory status change path. The extraction method involves extracting inventory status data within the range of the arrival time data of the starting order and the arrival time data of the ending order corresponding to the order cross-segment in the time-by-time inventory status change path to obtain inventory status change data within the order cross-segment. The extracted inventory status change data within the order cross-segment is then processed separately for each order, i.e., the individual impact of each order on the inventory data is calculated. The processing method involves sequentially extracting the order arrival time data and the replenishment item quantity data of each order, and independently calculating the individual contribution of the order arrival to the inventory data change. When calculating the inventory change contributed by each order individually, an inventory difference method is used, for example, based on the inventory at the time of order arrival, the inventory after the arrival of the previous order is subtracted to determine the amount of inventory status change generated by the arrival of a single order. Through calculation, the quantitative calculation of the individual inventory status change of each order arrival within the order cross-segment is completed, and the inventory status change result contributed by each order individually is recorded, thereby realizing the differentiation and quantification of the impact of each order arrival on inventory change within the order cross-segment.
[0088] By associating and combining the inventory change results that are separately distinguished within the order cross-segment with the inventory status change path, the transfer path of inventory status under the order cross-condition is obtained, forming status transfer correction data;
[0089] Based on the inventory status change results contributed individually by each order within the order crossover segment, the relationship between the individual contribution results and the overall time-by-time inventory status change path is defined, and an association combination method is set. Specifically, the association combination method involves sequentially superimposing the inventory status change results contributed individually by each order within the order crossover segment onto the inventory corresponding to the starting time of the order crossover segment, thereby forming the inventory status transfer path under order crossover conditions. For example, when three orders arrive sequentially within the order crossover segment, the inventory at the starting time of the order crossover segment is used as the initial state. The inventory change contributed individually by each order is sequentially superimposed, and the inventory at each step is calculated and recorded sequentially, thus obtaining the step-by-step inventory status change path within the order crossover segment. The timestamp and inventory of each inventory status change are recorded to ensure the traceability of the inventory status transfer path. After completing the calculation of the inventory status transfer path within all order crossover segments, the obtained inventory status transfer paths under order crossover conditions are uniformly defined as status transfer correction data to represent the transfer process of supply chain node inventory data according to the actual order arrival sequence under order crossover conditions.
[0090] Based on the state transition correction data, replenishment instruction data and corresponding delivery feedback data are re-associated to construct revenue attribution correction data, including:
[0091] Based on the state transition correction data, extract the transition path of inventory state under the condition of order crossover;
[0092] The state transition correction data records the time-by-time changes in inventory data at supply chain nodes under order crossover conditions, following the actual order arrival sequence. The representation of the state transition correction data includes, but is not limited to, timestamps in minutes identifying each inventory change node and its corresponding inventory. The state transition correction data is parsed to extract all data nodes containing timestamps and inventory changes, forming the inventory state transition path under order crossover conditions. The inventory state transition path is a continuous, time-series-identified sequence of inventory state changes, with each sequence node including inventory and its corresponding timestamp.
[0093] Determine the order arrival time corresponding to the replenishment instruction data based on the transfer path of inventory status;
[0094] Based on the inventory status transfer path under cross-order conditions, this method compares and maps each inventory change node recorded in the inventory status transfer path with replenishment instruction data. The replenishment instruction data includes, but is not limited to, the instruction issuance time, the quantity of items to be replenished, and the supply chain node identifier; each replenishment instruction records the replenishment requirement. To determine the correspondence, the inventory increase value for each inventory status change node is first calculated, and then compared with the quantity of items to be replenished specified in each replenishment instruction data to establish a one-to-one correspondence between the inventory increase value and the replenishment instruction data. For example, when the inventory status changes... If the inventory increase value of a certain inventory change node in the movement path is the same as the replenishment quantity of a certain replenishment instruction in the replenishment instruction data, then the timestamp corresponding to the inventory change node is considered to be the actual arrival time of the order corresponding to the replenishment instruction data. If the inventory increase value cannot be completely matched with a single instruction in the replenishment instruction data, a combination matching method is used, that is, multiple replenishment instruction data are combined so that the combined replenishment quantity matches the inventory increase value, thereby determining the arrival time of the orders corresponding to multiple replenishment instruction data. Through one-to-one comparison and combination matching, the relationship between replenishment instruction data and the arrival time of the corresponding order is obtained.
[0095] The order arrival time is rematched with the replenishment instruction data and a corresponding relationship is established, forming a new association between the replenishment instruction data and the corresponding arrival feedback data under the condition of order overlap;
[0096] Based on the determined order arrival time corresponding to the replenishment instruction data, the order arrival time is added to the corresponding replenishment instruction data in timestamp format to form a new association between the replenishment instruction data and the corresponding arrival feedback data. The new association is a structured data table, including but not limited to order placement time data, order arrival time data, replenishment item quantity data, and corresponding supply chain node identifiers. Compared with the original replenishment instruction data, the new association adds order arrival time data calibrated through the actual inventory status transfer path under order crossover conditions. During the establishment of the new association, association verification is performed to ensure that the order placement time data in each replenishment instruction data must be earlier than the corresponding order arrival time data. For example, the verification logic is set to require that the order arrival time data must be later than the order placement time data by at least one time unit; otherwise, the data is reviewed and corrected. Through data combination and verification, a new association between replenishment instruction data and corresponding arrival feedback data under order crossover conditions is formed.
[0097] Based on the new relationships, determine the true impact of replenishment order data on inventory changes, and generate revenue attribution correction data;
[0098] Based on the new correlation between replenishment instruction data and corresponding arrival feedback data under order crossover conditions, the true contribution of each replenishment instruction data to inventory status changes is calculated. For each replenishment instruction data, the corresponding inventory change start state and inventory change end state are defined, and the difference between the inventory before and after the change is calculated as the true inventory contribution of the replenishment instruction data. When multiple replenishment instruction data overlap and affect inventory changes within the order crossover segment, a method of sequential calculation and step-by-step differentiation is adopted, that is, the inventory increment after the arrival of each replenishment instruction data is calculated one by one, and the inventory state after each calculation is used as the starting state for the inventory impact calculation of the next replenishment instruction data, so as to quantify the individual contribution of each replenishment instruction data. Through calculation, the numerical value of the true inventory impact of each replenishment instruction data is recorded to form revenue attribution correction data.
[0099] The revenue attribution correction data includes replenishment instruction data identifiers, corresponding order arrival time data, actual inventory change values, and corresponding supply chain node identifiers. The revenue attribution correction data is used to provide feedback information when the reinforcement learning model adjusts its strategy, enabling the reinforcement learning scheduling optimization method to accurately attribute the actual revenue of replenishment actions based on the real impact of inventory.
[0100] The state transition correction data and revenue attribution correction data are input into the reinforcement learning scheduling model for policy updates, outputting a supply chain scheduling optimization policy, including:
[0101] Input data for constructing supply chain scheduling status is based on state transition correction data and revenue attribution correction data;
[0102] Using state transition correction data and revenue attribution correction data as initial input information, the state transition correction data contains the transition path of inventory status under order crossover conditions, while the revenue attribution correction data contains the actual impact of replenishment instruction data on inventory changes. When constructing the input data for the supply chain scheduling status, the time-by-time inventory status change path recorded in the state transition correction data is analyzed to extract the inventory and its corresponding timestamp, and the inventory is normalized, with the normalized inventory value limited to the range of 0 to 1. At the same time, the actual inventory change value of each replenishment instruction recorded in the revenue attribution correction data is analyzed and normalized using the same normalization method. Through the above processing, structured data representing the changes in supply chain inventory status and the actual contribution of replenishment instruction data are obtained, which is the input data for the supply chain scheduling status.
[0103] The state transition relationships and revenue distribution in the reinforcement learning scheduling model are corrected and updated using the input data of the supply chain scheduling status.
[0104] Using the input data of the supply chain scheduling state as the input source, the reinforcement learning scheduling model is a deep reinforcement learning model, including but not limited to a deep reinforcement learning structure that combines Q-learning algorithm and deep neural network; the state space is defined as the normalized value of the inventory of the supply chain nodes and the inventory change trend, the action space is defined as the replenishment action to be performed on a specific supply chain node, and the reward function is defined as the normalized true contribution of the inventory increase after the corresponding replenishment action is performed; the state transition relationship is represented by the probability distribution of the supply chain inventory state transitioning from the current time to the next time.
[0105] The input data of the supply chain scheduling status is fed into the neural network structure of the deep reinforcement learning model. The neural network structure includes, but is not limited to, a multilayer perceptron network. For example, the network structure is set as follows: the number of neurons in the input layer is consistent with the dimension of the input data; the intermediate hidden layers are set to two hidden layers; the number of neurons in each layer is determined by a grid search method; and the number of neurons in the output layer is set to the number of possible replenishment actions in the corresponding action space. Based on the input data of the supply chain scheduling status, the parameters of the state transition probabilities in the deep reinforcement learning model are updated. The update method adopts a gradient descent optimization algorithm, for example, the Adam optimization algorithm, and the learning rate parameter can be set as follows. The loss function is set to 0.001, and the loss function is set to the mean square error between the predicted state value and the actual state value. Simultaneously, the revenue allocation parameter is updated; this parameter is the attribution coefficient of the actual revenue obtained by the action in the reward function. The update method for the revenue allocation parameter is as follows: based on the actual inventory change value in the revenue attribution correction data, the difference in the normalized inventory state value before and after the action execution is calculated to determine the actual revenue corresponding to each action. The parameter coefficients in the model's reward function are then updated based on the actual revenue. For example, the least squares estimation method for the parameters can be used. Through this parameter update process, the state transition relationship and revenue allocation parameters of the reinforcement learning scheduling model are corrected and updated.
[0106] The updated reinforcement learning scheduling model is used to calculate the replenishment action sequence under the supply chain scheduling state;
[0107] The supply chain scheduling state input data is fed into the modified reinforcement learning scheduling model. The expected return for each possible replenishment action is calculated through forward propagation: the Q-value of each replenishment action corresponding to the output layer neurons of the reinforcement learning scheduling model represents the expected long-term return of executing the corresponding replenishment action under a given supply chain scheduling state. The Q-values of all possible replenishment actions are compared, and the replenishment action with the largest Q-value is selected as the optimal replenishment action for the current state. Through iterative calculations at each time step, replenishment actions at multiple consecutive time points are determined sequentially, gradually forming a replenishment action sequence. For example, after each step of the reinforcement learning model selects an action, the corresponding inventory state change result is fed back to the model as the input for the next step. The model then outputs a new Q-value, and so on, gradually generating a complete replenishment action sequence. The replenishment action sequence records the inventory node and replenishment quantity corresponding to each action, maintaining timestamp order, reflecting the optimization process of the reinforcement learning scheduling model and the decision-making logic under the supply chain scheduling state.
[0108] The replenishment action sequence is sorted according to its correspondence with the inventory data of the supply chain nodes to form a supply chain scheduling optimization strategy;
[0109] Each replenishment action includes a supply chain node identifier, replenishment quantity, and execution time. The replenishment action sequence is sorted chronologically by execution time, in ascending order, meaning actions with earlier execution times precede those with later execution times. The sorted replenishment action sequence is then grouped according to node identifiers in the supply chain node inventory data, with each group containing all replenishment actions for the corresponding supply chain node. Within each group, replenishment actions are further sorted according to the inventory change time series data, ensuring that the execution time of replenishment actions within a group is consistent with the inventory data collection time of the supply chain node. For example, for a given supply chain node identifier, the timestamps of all replenishment actions within its corresponding group are matched and sorted with the inventory data time in the inventory change time series, ensuring logical consistency between the correspondence between the replenishment action execution time and the inventory data change time. Through multi-dimensional sorting, a supply chain scheduling optimization strategy corresponding to supply chain nodes and inventory time series data is ultimately formed. The supply chain scheduling optimization strategy includes, but is not limited to, supply chain node identifiers, replenishment action execution times, replenishment quantities, and corresponding inventory states, reflecting the optimal scheduling decision results generated by the reinforcement learning scheduling model.
[0110] The above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more sets of available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. The semiconductor medium can be a solid-state drive.
[0111] Those skilled in the art will recognize that the modules and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.
[0112] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and modules described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.
[0113] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or modules may be electrical, mechanical, or other forms.
[0114] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.
[0115] In addition, the functional modules in the various embodiments of this application can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module.
[0116] If the aforementioned functions are implemented as software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0117] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.
[0118] In conclusion, the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.
Claims
1. A supply chain scheduling optimization system based on reinforcement learning, characterized in that, include: Time-series data acquisition module: Acquires inventory data, replenishment instruction data, order placement time data, and order arrival time data at supply chain nodes to form a time-series dataset of orders in transit; Cross-identification module: Calculates the order sequence inversion relationship based on order placement time data and order arrival time data, identifies order cross-sections, and generates order cross-identification data; The order reordering module rearranges the arrival order of the time-series dataset of in-transit orders based on the order cross-identification data, and constructs the actual arrival sequence data of in-transit orders; Path correction module: Based on the actual arrival sequence data of orders in transit and the inventory data of supply chain nodes, calculate the inventory evolution path under the condition of order crossover and generate state transition correction data; Attribution Correction Module: Based on state transition correction data, replenishment instruction data and corresponding arrival feedback data are re-associated to construct revenue attribution correction data; Revenue attribution correction data includes replenishment instruction data identifiers, corresponding order arrival time data, actual inventory change values, and corresponding supply chain node identifiers. The revenue attribution correction data is used to provide feedback information when the reinforcement learning model adjusts its strategy, enabling the reinforcement learning scheduling optimization method to accurately attribute the actual revenue of replenishment actions based on the real impact of inventory. Policy Update Module: Inputs state transition correction data and revenue attribution correction data into the reinforcement learning scheduling model to update the policy and outputs a supply chain scheduling optimization policy.
2. The supply chain scheduling optimization system based on reinforcement learning according to claim 1, characterized in that, Acquire inventory data, replenishment instruction data, order placement time data, and order arrival time data at supply chain nodes to form a time-series dataset of in-transit orders, specifically: Inventory data at each node of the supply chain is continuously collected in chronological order and time-aligned to form a time-series sequence of inventory changes; Based on the time series of inventory changes, replenishment order data within the corresponding time range is matched to construct the correspondence data between inventory changes and replenishment orders; Time-index the replenishment order data to extract the order placement time data; Parse the order execution records corresponding to the replenishment instruction data and extract the order arrival time data; The replenishment instruction data is correlated and organized based on the order placement time data and the order arrival time data to form a time-series dataset of orders in transit.
3. The supply chain scheduling optimization system based on reinforcement learning according to claim 2, characterized in that, Based on order placement time data and order arrival time data, the order sequence inversion relationship is calculated, overlapping order segments are identified, and order overlap identifier data is generated, specifically as follows: Based on the time-series dataset of orders in transit, the correspondence between order placement time data and order arrival time data is extracted. Order sequences are constructed according to the order placement time order and the corresponding order arrival time positions are recorded synchronously. The system compares the order placement time sequence with the order arrival time sequence of adjacent orders in the order sequence to identify order pairs whose order placement time sequence and order arrival time sequence are inconsistent. The continuous distribution intervals of order pairs in the time series dataset of orders in transit are merged to form a set of order cross segments; Based on the set of order cross segments, the corresponding orders are marked to generate order cross-identification data that corresponds one-to-one with the time series dataset of orders in transit.
4. The supply chain scheduling optimization system based on reinforcement learning according to claim 3, characterized in that, Based on the order cross-identification data, the arrival order of the time-series dataset of orders in transit is rearranged to construct the actual arrival sequence data of orders in transit. Specifically: Based on the order cross-identification data, extract the marked order cross-segments in the time series dataset of in-transit orders; Orders within overlapping order segments are reordered based on their arrival time data to construct the order arrival order within the segment; Replace the original order placement time order with the new order arrival order order within the order cross-section; Consistency verification is performed between the replaced overall order sequence and the original order placement time data to form the actual arrival sequence data of orders in transit.
5. A supply chain scheduling optimization system based on reinforcement learning according to claim 4, characterized in that, Based on the actual arrival sequence data of orders in transit and the inventory data of supply chain nodes, the inventory evolution path under the condition of order crossover is calculated, and state transition correction data is generated, specifically: Based on the actual arrival sequence data of orders in transit, the inventory update relationship between the order arrival time and the inventory data of the supply chain nodes is determined, forming a time-by-time inventory status change path; Based on the set of order cross segments, extract the time-by-time inventory status change path within the order cross segments, and distinguish the individual impact of order arrivals on inventory changes within the order cross segments; By associating and combining the inventory change results after separate differentiation within the order cross-segment with the inventory status change path, the transfer path of inventory status under the order cross-condition is obtained, forming status transfer correction data.
6. The supply chain scheduling optimization system based on reinforcement learning according to claim 5, characterized in that, Based on the state transition correction data, replenishment instruction data and corresponding arrival feedback data are re-associated to construct revenue attribution correction data, specifically as follows: Based on the state transition correction data, extract the transition path of inventory status under the condition of order crossover; determine the order arrival time corresponding to the replenishment instruction data according to the transition path of inventory status; The order arrival time is rematched with the replenishment instruction data and a corresponding relationship is established, forming a new association between the replenishment instruction data and the corresponding arrival feedback data under the condition of order overlap; Based on the new relationships, the true impact of replenishment order data on inventory changes is determined, resulting in revenue attribution correction data.
7. A supply chain scheduling optimization system based on reinforcement learning according to claim 6, characterized in that, The state transition correction data and revenue attribution correction data are input into the reinforcement learning scheduling model for policy updates, outputting a supply chain scheduling optimization policy, specifically: Input data for constructing supply chain scheduling status is based on state transition correction data and revenue attribution correction data; The state transition relationships and revenue distribution in the reinforcement learning scheduling model are corrected and updated using the input data of the supply chain scheduling status. The updated reinforcement learning scheduling model is used to calculate the replenishment action sequence under the supply chain scheduling state; The replenishment sequence is sorted according to its correspondence with the inventory data of the supply chain nodes to form a supply chain scheduling optimization strategy.