A data-driven perishable age-stratified inventory optimization method and system

Abstract

The application discloses a data-driven perishable product age stratified inventory optimization method and system, the method comprises the following steps: a Markov decision process model is constructed with long-term profit maximization and physical expiration quantity minimization as double objectives, and through ideal point pre-training, state dimensionality reduction coding, preference conditioning strategy network, differentiable Chebyshev scalarization module and discount monotonicity hard constraint sampling mechanism based on sequential mask, complete generation and coverage of the non-convex Pareto optimal strategy set are realized. The application can output a physically feasible optimal joint ordering and age differentiated discount scheme under any given operating preference, and provides an end-to-end intelligent control means covering the whole strategy spectrum for a perishable product inventory management system.

Description

Technical Field

[0001] This invention relates to the field of supply chain management system technology, specifically to a data-driven method and system for optimizing perishable goods inventory by age stratification. Background Technology

[0002] Perishable agricultural products with fixed shelf lives, such as fresh foods, fruits and vegetables, milk, and eggs, are characterized by short shelf lives, high demand volatility, and significant inventory losses, posing significant challenges to their supply chain management. In multi-product, multi-channel distribution systems, traditional methods suffer from technical shortcomings, including coarse inventory age modeling, lack of consumer choice behavior modeling, singular optimization objectives, lack of demand-side control mechanisms, and excessively high dimensionality of action space. Dynamically monitoring inventory status across channels and adjusting replenishment strategies accordingly has become crucial for improving the preservation quality of perishable goods and reducing losses. In recent years, with the development of artificial intelligence and data-driven methods, advanced algorithms such as deep learning and reinforcement learning have been introduced into the field of supply chain management, particularly in inventory control and replenishment optimization scenarios. These algorithms aim to solve complex dynamic decision-making problems that static models and methods struggle to address, gradually becoming a hot topic in the industry.

[0003] The existing technologies mainly suffer from the following technical bottlenecks: First, at the objective level, all the above methods combine profit with various economic costs, including expiration costs, into a single scalar reward function for optimization. This single-objective modeling approach mathematically eliminates the non-convexity between profit and physical losses, causing the algorithm to converge only to a point on the Pareto front. It cannot provide the control system with a complete set of optimal strategies covering the entire spectrum from "aggressive profit-making" to "aggressive loss-reduction".

[0004] Second, many inventory control methods rely on forecasting demand or fixed stochastic demand, treating consumer demand as an exogenous given variable. They fail to characterize the discrete choices consumers make when faced with the same product at different age levels and discount prices during inventory transitions. Therefore, the feedback relationship between discounting actions and sales allocation, stockouts, and expiration is not realistically portrayed.

[0005] Third, if existing neural network decision-making methods directly output discount coefficients for each age group, they are prone to generating actions that violate the physical laws of business, such as selling used goods at a higher price than new goods. If this problem is only addressed through penalty terms or ex-post corrections, inconsistencies may arise between the policy probabilities, executed actions, and gradient updates during training.

[0006] Fourth, existing technologies often focus on offline model training and lack a complete engineering closed loop for generating inventory management strategies such as order quantities directly from real-time inventory status. Summary of the Invention

[0007] This invention proposes a data-driven method and system for optimizing the age-stratified inventory of perishable goods to address the problems mentioned in the background.

[0008] To achieve the above objectives, the present invention adopts the following technical solution: A data-driven method for optimizing the age-stratified inventory of perishable goods, comprising the following steps: S1. Construct a digital twin environment for the dynamic system of perishable goods inventory; S2. Construct a preference-conditional dual-objective control network based on the digital twin environment; S3. The reinforcement learning training of the dual-objective control network is carried out using a differentiable Chebyshev scalarization method based on ideal point deviation; S4. Based on the training results, a sequential mask sampling mechanism is used to generate discount actions that satisfy the discount monotonicity constraint.

[0009] Preferably, the digital twin environment cycles according to a decision-making cycle, and in each cycle... Initially, define an age-stratified inventory state vector:

[0010] in, To fix the number of shelf-life cycles, For age Product inventory levels; The state transition process of the digital twin environment includes: based on order quantity Update inventory, simulate stochastic demand and actual sales of products for different age groups based on a multi-logit consumer choice model, perform inventory aging and physical expiration, and calculate cyclical profit. With negative expired amount Two-dimensional instant reward vector ,in, This represents the amount of goods that have expired at the end of this period.

[0011] Preferably, the dual-target control network uses the current inventory status as the basis for its operation. and business preference vector The network takes as input and outputs order placement actions and discount actions for products in different age groups; the network includes: A state encoder is used to map age-stratified inventory status into state feature vectors. Preference embedding layer, used to embed the business preference vector Mapped to a preference feature vector; The Actor policy network is used to generate a discrete probability distribution of order quantity and a discrete probability distribution of discount coefficient for products of each age group based on the state feature vector and the preference feature vector. The Critic value network is used to independently output the cumulative discounted return estimates for each of the two objectives based on the state feature vector and the preference feature vector. and .

[0012] Preferably, the reinforcement learning training process includes: Single-cycle normalized reference points and cumulative ideal reference points for profit and loss reduction targets are determined through endpoint pre-training. ; During the policy sampling phase, a hybrid preference sampling strategy is adopted to cover the entire preference space; At each policy update, the temporal difference error and generalized advantage estimate of the two objectives are calculated separately to obtain a two-dimensional advantage vector:

[0013] Using differentiable Chebyshev scalarization functions Value of the current state With the dominance vector Combining and calculating the advantages of scalarization:

[0014] This drives the optimization and updating of near-end policies in the policy network.

[0015] Preferably, S4 includes: The sequential mask sampling mechanism samples each age group with inventory in ascending order of age. Determine the discount factor; for age groups The upper limit of its optional discount tiers is forcibly limited to those immediately adjacent to the newer age group. The selected discount tiers are masked to zero to eliminate the probability of invalid tiers. After re-normalization, the samples are then sampled to ensure that the generated discount sequence strictly satisfies the given conditions. .

[0016] Preferably, the multi-logit consumer choice model is as follows: For the age group set with inventory Consumers choose to buy age groups The probability of the item is:

[0017] Among them, utility function , As a basic attractiveness parameter for the product, Price sensitivity coefficient The freshness sensitivity coefficient Indicates the number of remaining marketable cycles; Total consumer demand It follows a Gamma-Poisson mixture distribution and is distributed through a multinomial distribution. The options are assigned to different age groups and those that do not require purchase.

[0018] Preferably, the differentiable Chebyshev scalarization function Implemented using a smooth minimum function based on LogSumExp:

[0019] in,

[0020] To smooth out temperature parameters; The hybrid preference sampling strategy includes three stages: The first stage, from parameters Random sampling preference in a two-dimensional Dirichlet distribution; In the second phase, two extreme preference points are forcibly added in each round of data collection. and The corresponding complete trajectory; In the third stage, during the training convergence period, the process switches to traversing and sampling multiple grid points evenly distributed in the preference space to refine the policy quality in the intermediate preference region.

[0021] Preferably, a preference diversity regularization term is also introduced during the training process of S3. This mandatory requirement applies to the same state. Two different preference vectors and The KL divergence between the action probability distributions output by the Actor policy network is proportional to the Euclidean distance between the two preference vectors; the regularization term is used to prevent the conditional policy network from converging on different preference outputs, thereby ensuring the integrity of the Pareto front.

[0022] Preferably, the sequential mask sampling mechanism remains consistent in both the training and inference phases: in the training phase, it is used to calculate the logarithmic probability of joint actions and the policy gradient; in the inference phase, it is used to deterministically decode the output discount coefficient that satisfies the monotonicity constraint.

[0023] A data-driven age-stratified inventory optimization system for perishable goods includes: The state coding module is used to obtain the quantity vectors and demand context features of goods for each age group in the current inventory, which constitute the system state; The dual-objective reward calculation module is used to calculate the profit reward and the overdue loss penalty separately after the discount action is executed, and output the dual-objective reward vector. The preference conditionalization policy network module includes an Actor network and a Critic network. The Actor network receives a concatenated vector of state and preference weights and outputs a probability distribution of discounted actions. The sequence mask module is used to apply a monotonic mask to the action probability distribution based on the remaining shelf life information of products for each age group in the inventory, thereby blocking actions that violate the discount monotonicity constraint. The consumer choice modeling module is used to calculate the probability of consumers choosing products for each age group and the expected sales volume based on the MNL model. The differentiable Chebyshev scalarization module is used to train and update the preference conditional policy network by employing the PPO algorithm combined with the differentiable Chebyshev scalarization loss function. The decision output module is used to output the optimal discount rate vector for products of each age group based on the current state and preference weights, and connect to the online deployment interface.

[0024] As can be seen from the above technical solution, the present invention provides a data-driven method for optimizing the age-stratified inventory of perishable goods. Compared with the prior art, the present invention has the following advantages: 1. This invention addresses the inventory control system for perishable goods with fixed shelf life and age stratification. It constructs a Markov decision process model with two independent optimization objectives and uses the Chebyshev scalarization method to solve the model, providing a complete technical framework. From the system architecture level, it ensures that the non-convex performance of the profit-physical loss trade-off is fully characterized.

[0025] 2. This invention constructs a differentiable Chebyshev scalarization module based on the LogSumExp smooth approximation and embeds it into the computation graph of the value estimation network to achieve real-time scalarization TD error calculation at each time step; it solves the technical limitation of traditional multi-objective reinforcement learning methods that scalarization can only be performed after the end of the round, so that all single-step learning mechanisms of standard proximal policy optimization can be directly applied to the bi-objective control problem of perishable goods inventory.

[0026] 3. This invention employs a sequential masking sampling technique from the newest to the oldest age to enforce a monotonic constraint that the greater the age, the greater the discount, at the probability calculation level of action generation; it also ensures the physical feasibility of the output pricing instruction from the underlying probability mechanism of the algorithm, with a strictly zero probability of violation.

[0027] 4. This invention combines a specific preference sampling strategy with network optimization regularization to ensure that the final generated policy set can completely and uniformly cover the non-convex Pareto front; it synergistically guarantees the integrity and identifiability of the Pareto optimal policy set from two dimensions: data supply and optimization constraints, enabling a single neural network model to internalize all control strategies from aggressive profit to aggressive loss. Attached Figure Description

[0028] Figure 1 This is a flowchart illustrating a data-driven method for optimizing the age-stratified inventory of perishable goods according to the present invention.

[0029] Figure 2 This is a diagram illustrating the evolution of the inventory status of perishable goods according to the present invention.

[0030] Figure 3 This is a schematic diagram of the preference conditionalization strategy network structure of the present invention.

[0031] Figure 4 This is a flowchart of the multi-objective Chebyshev PPO training process of the present invention.

[0032] Figure 5 This is a convergence curve of the supervolume HV index of the present invention.

[0033] Figure 6 This is the V1 / V2 convergence curve diagram of the present invention.

[0034] Figure 7 The diagram shows the convergence curves of the Chebyshev scalarized objective under different preference weights in this invention.

[0035] Figure 8 This is the physical profit-expiration Pareto front diagram of the present invention.

[0036] Figure 9 This is a typical preference radar chart of the present invention.

[0037] Figure 10 This is a trend chart of the average periodic order volume of the present invention.

[0038] Figure 11 This is a comparison chart of typical preference indicators of the present invention.

[0039] Figure 12 This is an age-based discount distribution chart for the present invention.

[0040] Figure 13 This is an initial state diagram of the online reasoning interface of the intelligent decision-making assistant for perishable goods inventory optimization according to the present invention.

[0041] Figure 14 This is a diagram showing the running results of the online reasoning interface of the intelligent decision-making assistant for perishable goods inventory optimization according to the present invention. Detailed Implementation

[0042] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are some embodiments of the present invention, but not all embodiments.

[0043] like Figure 1 As shown in this embodiment, a data-driven age-stratified inventory optimization method for perishable goods includes the following steps: S1. Construct a digital twin environment for the dynamic system of perishable goods inventory; S2. Construct a preference-conditional dual-objective control network based on the digital twin environment; S3. The reinforcement learning training of the dual-objective control network is carried out using a differentiable Chebyshev scalarization method based on ideal point deviation; S4. Based on the training results, a sequential mask sampling mechanism is used to generate discount actions that satisfy the discount monotonicity constraint.

[0044] For ease of explanation, this embodiment sets the physical shelf life of perishable products as a decision cycle, corresponding to five age groups from age 0 to age 4; the basic full price of the product is... The maximum order quantity per cycle is In the project implementation example, the maximum order quantity Base price The initial inventory is Each training or evaluation round contains 200 cycles by default.

[0045] S1: Construct a digital twin environment for the dynamic system of perishable goods inventory; This embodiment constructs a computer simulation environment that can accurately simulate the evolution of the physical state of perishable products during their shelf life. The simulation environment executes cyclically according to a fixed period, consisting of 7 sub-steps: S11: State space definition and age-level hierarchical representation: cycle The initial system state is described by a vector of spot inventory levels for each age group of products:

[0046] in, Indicates age Product inventory levels; This indicates the new products that have arrived this period. This indicates that the product is nearing its expiration date and will expire if it is not sold by the end of the period.

[0047] This embodiment adopts zero lead time for ordering, with goods arriving immediately after the order is placed at the beginning of the order period, and the following always applies before the ordering action is executed:

[0048] This constraint indicates that new orders for this period have not yet been placed by the model, and the model needs to determine the order quantity first. In the engineering implementation, the actual input of the state encoder is... to These inventory layers, which will actually change before the decision is made, will not enter the state encoder input because their age is fixed at 0.

[0049] S12: Ordering Actions and Available Inventory Updates: First, the cycle The discrete values ​​for order quantity are:

[0050] Then, the order is immediately entered into the warehouse and the age is changed. New stock, after ordering, the available inventory will be updated as follows:

[0051] within the normal shelf life The above formula can be written as:

[0052] S13: Modeling Discount Actions and Discrete Choices in MNL Consumers: express Cycle Age Product discount factor, actual selling price:

[0053] Discount tiers from In the selection process, the pricing of perishable goods should meet the requirement that the price of older goods should not be higher than that of newer goods. Therefore, the system requires:

[0054] Its general form is:

[0055] This constraint is natively guaranteed by the action layer sequence mask sampling mechanism, without relying on loss penalties or ex-post corrections.

[0056] MNL Utility and Choice Probability: For the age group set with inventory ,age The perceived certainty utility of a product is defined as:

[0057] in, As a basic attractiveness parameter for the product, Price sensitivity coefficient The freshness sensitivity coefficient This indicates the remaining number of sales cycles. The younger the age, the longer the remaining sales cycle, and the higher the freshness utility.

[0058] If the out-of-stock age group is removed from the selection set and the utility of the external "Do Not Buy" option is set to 0, then consumers will select the age group to buy. The probability of the item is:

[0059] At this point, the probability that the consumer will not buy is:

[0060] To avoid overflow in exponent calculation, the engineering implementation first subtracts the maximum utility from the candidate utility, and then calculates the exponent and normalized probability. The final result is... There are one probability for each age group and one probability for an external option, with the sum of probabilities being strictly 1.

[0061] S14: Mixed stochastic demand and multivariate allocation: The system uses a Gamma-Poisson mixture model to generate the number of overly discrete potential consumers; first, it samples the intensity of potential demand:

[0062] Subsequently To sample the total number of potential consumers for conditional strength:

[0063] Finally, through multivariate distributions... Potential consumers were assigned to different age groups and external options:

[0064] in, The selection probability vector output by the MNL model. For age The theoretical potential demand for the product; the model can simultaneously represent the random fluctuations in demand size, excessive dispersion, and the discrete choice behavior of consumers between different freshness and price.

[0065] S15: Sales Cut-off and Stockout Calculation: In this embodiment, due to the upper limit of spot supply, the age group The actual sales volume is:

[0066] The stock shortage in a single age group is:

[0067] Total stockout during the period is:

[0068] This embodiment assumes a single selection and no substitution for out-of-stock items; that is, after a consumer selects a certain age group through the MNL model, if the inventory for that age group is insufficient, the excess will be treated as out-of-stock or lost sales for that age group and will not be reallocated to other age groups within the same period. This assumption is consistent with the sales cutoff and out-of-stock calculation in the project code.

[0069] S16: Inventory Aging, Expiration, and Status Transition: At this point, the remaining inventory after the sale is:

[0070] At the end of the cycle, all remaining products for the oldest age group expired:

[0071] exist In the embodiments, there are As inventory ages, the initial state for the next cycle is generated:

[0072] The general form is:

[0073] like Figure 2 As shown, by shelf life For example, this demonstrates the entire chain of a single cycle: initial tiered inventory, order receipt and update of available inventory, MNL random demand splitting, actual sales truncation, near-expiry product scrapping, and remaining inventory age carried over to the next period; used to illustrate the physical flow logic of the simulation environment.

[0074] In the engineering implementation, the environment also performs physical verification of actions and state transitions: the order quantity must be an integer and not exceed [a certain value]. The discount index must be within the valid tier; the activity discount mask must be exactly equal to whether the available inventory after ordering is greater than 0; the sales volume must not exceed the available inventory; the next cycle age must be 0; the expired quantity, carried inventory, and profit must not exceed the safety boundaries set in the configuration; these checks are used to ensure that the state transitions during the training process always comply with the physical constraints of the inventory control problem.

[0075] S17: Dual-objective instant profit calculation: In this embodiment, the periodic profit is defined as:

[0076] The first item is sales revenue, the second is ordering costs, the third is holding costs of unexpired inventory that can be carried over to the next period, and the fourth is stockout losses. (The last part, "expiring at the end of the period," appears to be an unrelated fragment and is omitted from the translation.) As expired quantity Exit the system.

[0077] The loss reduction target is represented by a negative expiration amount:

[0078] Therefore, the original two-dimensional reward for a single cycle is:

[0079] Both objectives are maximization objectives, meaning the higher the profit and the greater the negative expiration amount, the better.

[0080] S2: Construct a preference-conditional bi-objective control network based on the digital twin environment; The network adopts an Actor-Critic (policy-value function) architecture, consisting of four core modules: a state encoder, a preference embedding layer, an Actor policy head, and a Critic value head, and is divided into four sub-steps. The network inputs are inventory status and operating preference weights, and the outputs are order quantity, age-specific discount coefficients, and bi-objective value estimates. The single-condition network can generate a continuous family of policies from "profit priority" to "loss reduction priority", including four sub-steps.

[0081] S21: Constructing a state feature encoder: The state encoder input is:

[0082] The general form is In the embodiments Therefore, the input dimension is 4. The state encoder consists of a 128-dimensional hidden layer and a 64-dimensional output layer, with ReLU activation function. This encoder transforms the age inventory distribution into continuous state features. It is used to characterize total inventory, age structure, and near-expiration risk.

[0083] S22: Constructing the preference embedding layer: The preference vector is defined as:

[0084] in:

[0085] The Actor and Critic each have independent preference embedding layers that map two-dimensional preference vectors to 16-dimensional preference features and activate them using ReLU. Independent preference embeddings enable policy outputs and value estimates to learn preference representations suitable for their respective tasks.

[0086] S23: Actor Policy Network: The Actor concatenates its state features and preference features, then inputs them into a two-layer 128-dimensional fully connected network. The output is then divided into two parts: Order Header: Output Logits, corresponding to order quantity to The discrete probability distribution; Discount pricing header: Outputs 10-dimensional logits for each age group, corresponding to... to The 10 discount levels.

[0087] The logarithmic probability of the joint action is:

[0088] in, Indicates age group There is available inventory after ordering; the age group with no inventory is not included in the training as a random discount action, and its discount uses normalized filler values ​​with a joint log probability contribution of 0; see network topology for details. Figure 3 .

[0089] like Figure 3 As shown, the data flow is as follows: the original inventory state is input, the state encoder extracts latent features, the preference vector is independently embedded to obtain preference features, and then concatenated with the state features; the Actor bifurcates to output order quantity and age-specific discount logits, and the Critic bi-headed outputs the value estimates of the two targets; this is used to illustrate the relationship between the network module composition and data flow.

[0090] S24: Critic Value Network The Critic concatenates state features with Critic preference features, then inputs the concatenation into a two-layer 128-dimensional fully connected network, ultimately outputting a two-dimensional value.

[0091] in, Estimate the cumulative return of the normalized profit bonus with discount. Estimate the discounted cumulative return of normalized negative expired rewards; Critic does not directly output scalarized value, but retains the independent value heads of the two objectives to avoid nonlinear scalarization contaminating single-objective value estimation.

[0092] Compared to training a separate model for each preference, this invention enables a single network to cover policies under multiple preferences through preference-conditional input; during deployment, only changes are needed. You can switch your ordering and discount decision preferences.

[0093] S3: The reinforcement learning training of the dual-objective control network is carried out using a differentiable Chebyshev scalarization method based on ideal point deviation; A differentiable Chebyshev scalarization based on ideal point bias is used to achieve dual-objective coupled optimization. The system first constructs a normalized reference benchmark through endpoint pre-training, then calculates the TD error and GAE advantage of the two objectives separately, and finally performs scalarization combination in the advantage layer to maintain the stability of the PPO training framework, including four sub-steps.

[0094] S31: Reference point pre-training and dynamic update, this sub-step includes 3 sub-steps: S311: Single-target endpoint pre-training: The system first performs single-objective PPO pre-training on the two endpoint targets, including the profit-optimal endpoint and the loss-reduction-optimal endpoint. Among them, the profit-optimal endpoint: only based on To reward training a single-objective PPO strategy; Minimize the optimal endpoint: using only To reward training a single-objective PPO policy.

[0095] Both endpoints are pre-trained using the same age-stratified environment and the same sequential masking action mechanism, thus the endpoint policies also satisfy the order cap and discount monotonicity constraints. In endpoint PPO, the Actor outputs order and discount actions, while the Critic outputs a single value. To reduce the impact of excessively large raw profit residuals on the shared encoder during early training, the endpoint Critic in the engineering implementation uses SmoothL1 loss by default to smooth the parameters. .

[0096] S312: Single-period normalized reference point: After the endpoint strategy training is completed, the system independently evaluates the profit endpoint strategy and the loss-reduction endpoint strategy in a deterministic mode, generating a single-period reference point; Let: =Average periodic profit of the profit endpoint strategy; =Average periodic profit of the fee reduction endpoint strategy; =Average negative expiration time for fee reduction endpoint strategies; = Average periodic negative expiration of profit endpoint strategies; The normalized reward for a single period is then:

[0097] In this embodiment, the normalized reward is not pruned by default. That is, when the performance of a certain period or a certain strategy exceeds the endpoint reference strategy, the normalized value is allowed to be greater than 1; when the random demand is extremely unfavorable, the normalized value may also be less than 0. This setting preserves the true information when the strategy exceeds the reference point or encounters extreme demand.

[0098] S313: Cumulative Ideal Reference Point: In a normalized environment where a single-period reference point has been determined, the endpoint policies are loaded separately and the cumulative normalized return is calculated to obtain the cumulative ideal reference point: Profit endpoint strategy:

[0099] Endpoint mitigation strategy:

[0100] The reference point for project output is:

[0101] S32: Three-stage mixed preference sampling strategy: Preference vector satisfy and , representing the degree of emphasis on profit targets and waste avoidance targets, respectively. To ensure a single conditional network can cover all possible policies, the preference space must be fully explored during training, while also ensuring that the boundaries of extreme policies are accurately learned.

[0102] This embodiment employs a three-stage hybrid sampling strategy: Phase 1 (Random Exploration Period): From a two-dimensional Dirichlet distribution Random sampling preference. This distribution has a higher density at the edges of the simplex and tends to generate near-simplexes. or Extreme preferences naturally favor the exploration of radical strategies.

[0103] Phase Two (Endpoint Forced Coverage): Regardless of the random sampling results, two additional extreme preference points are added during each round of data collection. and The corresponding complete trajectory. That is, during each data collection, a predetermined number of preferences are first randomly generated using the Dirichlet distribution, and then two endpoint preferences are added in a fixed manner, resulting in a total of several trajectories. This design ensures from the sampling mechanism that the two endpoint strategies of the Pareto front can be accurately approximated without drifting due to the lack of probability in random sampling. This is the key technical means of this invention to ensure the integrity of the Pareto front.

[0104] Phase 3 (Grid Fine-tuning): When training tends to converge, switch to... The strategy is fine-tuned by traversing multiple grid points evenly distributed across the interval, refining the policy quality in the intermediate preference region, and ensuring that the final frontier has a uniform distribution density.

[0105] This sampling design, which is closely tied to the multi-strategy management needs of perishable goods, enables the trained network model to cover the entire spectrum of control strategies, from "aggressive profit-making" to "aggressive loss reduction".

[0106] S33: Differentiable smoothed Chebyshev scalarization module: Let the cumulative value vector output by Critic be:

[0107] The cumulative ideal reference point is:

[0108] For a given preference Define the deviation of each target from the ideal point:

[0109] Chebyshev scalar value is defined as:

[0110] This formula selects the more severe bottleneck term from two weighted regrets; maximizing This is equivalent to minimizing the regret of the most severe objective under the current preference.

[0111] This invention emphasizes that scalarization must be based on Instead of using directly If the ideal point is not subtracted, because Choosing a smaller operator value might actually increase the weight of a target, causing the optimizer to shift to another target, resulting in a semantic reversal of preferences. By using ideal point bias, increasing... At the target When the ideal point is not reached, the penalty for failure is increased, thereby stabilizing the guiding strategy to prioritize improving the goal.

[0112] To ensure differentiability, the engineering implementation uses a smooth minimum function of the form LogSumExp:

[0113] Ultimately, it can be quantized as follows:

[0114] The actual calculation uses the numerically stable LogSumExp implementation to avoid the situation where... or Larger and When the value is small, the exponential overflow occurs.

[0115] S34: Advantages of Independent GAE + Rear-mounted Scalar Quantization: This invention does not directly apply nonlinear Chebyshev scaling to random TD targets. Instead, it first performs TD error and generalized advantage estimation on two targets respectively, and then performs scaling combination at the advantage layer.

[0116] For each time step The normalized two-dimensional instantaneous reward is known. Using the target network (parameters) Calculate the TD targets for the two targets separately:

[0117] Among the discount factors Then, the single-step TD error for each of the two targets was calculated:

[0118] Then execute GAE respectively:

[0119]

[0120] The two-dimensional dominance vector is obtained: ,in This is an unbiased, low-variance advantage estimate, reflecting the impact of the current action on the first... The degree of improvement of each objective.

[0121] The advantages of scalarization are defined in this invention as follows:

[0122] in The differentiable Chebyshev scalarization function defined in step three (3); this value represents the current action relative to the current state baseline, when If the action improves the bottleneck objective under the current preference, it will be strengthened in the policy update; otherwise, it will be suppressed.

[0123] Then use the standard PPOClip target:

[0124] in, This represents the joint probability ratio of the old and new strategies. The Critic loss still maintains its two-dimensional form:

[0125] Compared with directly scalarized TD targets Unlike other methods, this method uses nonlinear functions. Value estimation acting on determinism and advantage estimation Rather than acting on random TD targets. (It is expected that it needs to pass through a nonlinear function).

[0126] Specifically, let The true value is (Multi-objective advantage function), then:

[0127] exist Perform a first-order Taylor expansion nearby:

[0128] The first-order approximation is: A linear combination of the components, assuming sufficient accuracy in function approximation, allows the TD error and GAE advantage of each target to accurately reflect actual performance differences; therefore, scalarization advantage... It exhibits lower estimation bias compared to directly scaling the TD objective. The residual bias arises from higher-order terms of the Taylor expansion, function approximation error, and the bias-variance tradeoff of the GAE.

[0129] Based on the advantages of scalarization Directly update the policy using the standard PPOClip target:

[0130] because As a scalar, this update rule is completely consistent in form with the single-objective PPO. All engineering mechanisms of other standard PPOs (mini-batch updates, gradient pruning, entropy regularization, etc.) can be directly reused without any adaptation or improvement.

[0131] like Figure 4 As shown, the offline training closed loop consists of: endpoint pre-training, reference point initialization, mixed preference sampling, environmental interaction trajectory acquisition, sub-target TD and GAE calculation, post-Chebyshev scaling advantage, alternating iterative updates of Actor and Critic, periodic grid evaluation and HV convergence verification; used to fully illustrate the entire algorithm optimization process.

[0132] In this embodiment, the system code for implementing Chebyshev scaling and single-step temporal difference learning is as follows: # Calculate the smoothed Chebyshev value based on the cumulative ideal point deviation defsmooth_phi(values,weights,ideal,tau=0.1): ifvalues.shape[-1]!=2orweights.shape[-1]!=2: raiseValueError("Twoobjectivesarerequired.") iftau<=0ortorch.any(weights<0): raiseValueError("Invalidtemperatureorweights.") ifnottorch.allclose(weights.sum(-1),torch.ones_like(weights[...,0]),atol=1e-5): raiseValueError("Weightsmustsumtoone.") #Weighted signed bias: w_i(V_i-Z_i*) deficit=weights.to(values)*(values-ideal.to(values)) #LogSumExp implements a numerically stable smooth minimum value return-tau*torch.logsumexp(-deficit / tau,dim=-1) #Calculate TD error and GAE independently for the two objectives, without pre-scalarization. defvector_gae(rewards,values,next_values,bootstrap,continuation, (gamma=0.99,gae_lambda=0.95): advantages=torch.zeros_like(rewards) running=torch.zeros_like(rewards[0]) bootstrap,continuation=bootstrap[:,None],continuation[:,None] fortinreversed(range(rewards.shape[0])): delta=rewards[t]+gamma*bootstrap[t]*next_values[t]-values[t] running=delta+gamma*gae_lambda*continuation[t]*run advantages[t]=running Return advantages,advantages+values #A_cheb=Phi(V+A)-Phi(V) defchebyshev_advantage(values, advantages, weights, ideal, tau=0.1): baseline=smooth_phi(values,weights,ideal,tau) improved=smooth_phi(values+advantages,weights,ideal,tau) scalar=improved-baseline #Fuck the new actor return ( scalar - scalar . mean ( ) ) / ( scalar . std ( unbiased = False ) + 1e - 8 ) #Actor has a great time, Critic has a great time defppo_losses(new_log_prob,old_log_prob,advantage, predicted_values, value_targets, clip_epsilon=0.2): ratio=torch.exp(new_log_probe-old_log_probe) clipped=torch.clamp(ratio,1.0-clip_epsilon,1.0+clip_epsilon); actor_loss=-torch.min(ratio*advantage,clipped*advantage).mean(); critical_loss=torch.nn.functional.mse_loss( predicted_values, value_targets.detach() ) return actor_loss,critic_loss S4: Based on the training results, a sequential mask sampling mechanism is used to generate discount actions that satisfy the discount monotonicity constraint; This invention directly implements the discount monotonic hard constraint at the action generation probability layer; the discount coefficient levels are arranged in ascending order as follows:

[0133] A larger index indicates a higher price coefficient. Pricing requirements for perishable goods dictate that older goods cannot be priced higher than newer goods; therefore, the following conditions must be met:

[0134] First, determine the available inventory after ordering based on the status and order quantity:

[0135] Then the activity discount mask is defined as:

[0136] Only when At that time, age group Discounts are used as random actions in training. If there is no inventory available for sale in a certain age group, the discount for that age group will not affect sales in the current period. The system uses the previous valid price index as the normalized fill value, and its joint action log probability contribution is 0.

[0137] For age groups with inventory, sort by age from arrive Execute in sequence: (1) To Using the highest price tier index as the initial upper bound, we directly sample from its available tiers or take the discount with the highest probability. (2) To In (1) above, the age group discount index has been selected as the maximum allowed index. All the levels greater than this index are set to floating-point minimum values ​​and then sampled after normalization. (3) If the training data is replayed to calculate the probability ratio of the new and old strategies, the same restricted distribution is reconstructed using the recorded order quantity, discount index and activity mask to ensure that the PPO probability ratio calculation corresponds to the actual execution strategy.

[0138] The logarithmic probability of the joint action is:

[0139] The technical advantages of this mechanism are: the network can still output independent logits for each age group, preserving its expressive power; however, all actually executable discounted actions satisfy monotonic constraints, and the probabilities, entropies, and gradients during training are calculated based on the legitimate distribution after masking. Compared to post-projection or penalty terms, this mechanism guarantees physical feasibility from the bottom layer of action generation.

[0140] In this embodiment, the system code for implementing the sequential mask sampling mechanism is as follows: #Sequence mask ensures that the price coefficient of used goods is no higher than that of new goods. defmonotone_discount_action(order_logits,discount_logits,state, deterministic=False): order_dist=Categorical(logits=order_logits) order=order_logits.argmax()ifdeterministicelseorder_dist.sample() available = state.clone() available[0]=order.to(available.dtype) active=available>0 levels=discount_logits.shape[-1] selected = torch.full( (discount_logits.shape[0],),levels-1, dtype=torch.long,device=discount_logits.device ) previous=levels-1 log_prob=order_dist.log_prob(order) forageinrange(discount_logits.shape[0]): #For age groups without inventory, use standard filler values; the logarithmic probability contribution is zero. ifnotbool(active[age]): selected[age]=previous continue invalid=torch.arange(levels,device=discount_logits.device)>previous masked=discount_logits[age].masked_fill( invalid,torch.finfo(discount_logits.dtype).min ) dist=Categorical(logits=masked) index=masked.argmax()ifdeterministicelsedist.sample() selected[age]=index log_prob=log_prob+dist.log_prob(index) previous = int(index) returnorder,selected,active,log_prob In practical applications, the training process and online deployment are as follows: First, calculate the Critic value loss: The dual-objective Critic employs a joint loss of independent value errors for two objectives:

[0141] In this embodiment, the pre-training for each of the profit and loss endpoints is 200,000 steps. The main training for the dual-objective endpoint is 2,000,000 steps, and the main training parameters are as follows:

[0142] In reference point estimation, the endpoint policy evaluation rounds are 50, the single-cycle reference evaluation length is 200, and the cumulative ideal point reference length is 1000. During multi-objective training, grid evaluation and checkpoint saving are performed every 50,000 steps.

[0143] like Figure 5 As shown, the hypervolume changes of the multi-objective physics results during training are used to measure the quality of Pareto front coverage; the increase in HV indicates that the policy set has expanded the effective trade-off region in the profit-expiration two-dimensional space, and the later stabilization indicates that the multi-objective policy set gradually stabilizes.

[0144] like Figure 6 As shown in the figure, this diagram illustrates the dual-objective value function during the main training process. )and( The changing trends of the two curves; both curves rise rapidly in the early stages of training and then enter a relatively stable range, indicating that the model's value estimation of the profit and loss reduction targets gradually converges, providing a stable foundation for subsequent strategy optimization; it should be noted that... and With training cumulative ideal point Same scale.

[0145] Then, apply preference diversity regularization (to prevent policy collapse): To avoid the conditional policy network outputting overly similar actions for different preferences, this invention introduces a preference diversity regularization term. In engineering implementation, a batch of preferences is compared with its rolling paired preferences, the KL divergence between the order quantity distribution and the activity discount distribution is calculated, and it is made to approximate the linear objective of preference distance.

[0146] The corresponding regular expression can be represented as:

[0147] In the example:

[0148] The total loss consists of Actor loss, The loss consists of a weighted Critic loss, an entropy regularization term, and a preference diversity regularization term. Before each parameter update, a global gradient norm pruning is performed, and the original gradient norm, the pruned gradient norm, and whether the warning threshold is exceeded are recorded to improve training robustness.

[0149] Finally, convergence determination: Fixed grid preference is periodically simulated and evaluated to calculate the original physical space hypervolume HV; if the relative increase of HV is less than the threshold for multiple consecutive rounds and the average policy entropy of the full preference is lower than the threshold, convergence is determined and the optimal weight is saved.

[0150] like Figure 7 As shown in the figure, this graph illustrates how the quantization target varies with the number of training steps for different profit preference weights. Each subgraph corresponds to a profit weight, ranging from (w=0.0) to (w=1.0), with the horizontal axis representing the number of training steps and the vertical axis representing the Chebyshev quantization target value.

[0151] The final evaluation used 30 independent rounds, each with 200 cycles. The profit, waste, and order quantity in the project's final_policy_points.csv file represent the average cycle results. The results analysis report further sums the cycle-by-cycle trajectory within each round and then averages the results over the rounds to obtain the cumulative profit and cumulative overdue amount over 200 cycles. To avoid confusion, the table below simultaneously presents the average cycle results, the cumulative results over 200 cycles, and the average order quantity.

[0152] Table 1 Preferences Data table after final evaluation

[0153] The results above show that as the profit weight increases overall, the strategy generally increases order volume and inventory pressure in order to obtain higher profits; as the loss reduction weight increases, the strategy reduces order volume and inventory backlog, thereby significantly reducing expiration.

[0154] Specifically, compared to the profit-first strategy, the balancing strategy reduces the average profit over 200 cycles from 13670.57 to 12747.09, sacrificing only about 6.76% of the profit; the average expiration over 200 cycles decreases from 158.60 to 28.53, a reduction of about 82.01%. This demonstrates that in this embodiment, the system can identify low-cost loss reduction opportunities, eliminating the need for operators to make an extreme choice between profit and loss reduction.

[0155] Furthermore, the evaluation results indicate that the preference weights and physical outcomes are not a strictly monotonic mechanical correspondence. In this training output, the preferences are dominated by other strategies on the physical profit-expiration plane. The system retains these preference points as a complete preference grid output, while simultaneously labeling Pareto effective strategies through a non-dominated filtering module. This design allows the system to both demonstrate the trajectory of preference changes and provide operators with a set of physically effective strategies.

[0156] like Figure 8 As shown, the average profit and average expiration corresponding to different preference weights are displayed in the real physical indicator space, and non-dominated strategy points are marked; the further to the right, the higher the profit, and the further down, the lower the expiration; the points on the frontier represent that no other strategy can achieve both higher profit and lower expiration at the same time, which is the core result diagram for managers to make trade-off choices.

[0157] As can be seen from the figure, the weights generally rise rapidly in the early stage of training and then gradually enter the stable range. This indicates that the model can learn effective strategies quickly under different business preferences and there is no obvious divergence in the later stage of training.

[0158] Meanwhile, strategies with higher profit weights show a deeper scalar target decrease and a greater improvement in the early stages of training, indicating that profit-oriented strategies face stronger profit-over-profit tradeoff pressure. However, after training, the curves under each weight tend to stabilize, reflecting the adaptability of the preference-conditional PPO network to different preference strategies.

[0159] like Figure 9 As shown in the figure, this diagram illustrates the differences in profit, expiration, and order quantity under two typical preferences: (w=0.5) and (w=1.0). The results indicate that the profit-first strategy achieves higher profits but comes with higher order quantities and expiration risk; the balanced strategy, on the other hand, effectively reduces expiration while maintaining higher profits, demonstrating the trade-off between profit and loss reduction.

[0160] like Figure 10 As shown in the figure, the average periodic order volume changes under different profit preference weights. As the profit weight increases, the overall order volume rises, indicating that the model automatically adjusts the replenishment intensity according to business preferences: the stronger the profit orientation, the more aggressive the strategy; the stronger the loss reduction orientation, the more conservative the strategy.

[0161] like Figure 11 As shown, typical preference strategies are selected, and indicators such as profit, expiration, ordering, and stockout are compared side-by-side. This chart is suitable for explaining to business personnel the behavioral differences caused by different preferences; for example, a loss mitigation strategy is more conservative, while a profit-oriented strategy involves ordering more stock but carries a higher risk of expiration.

[0162] like Figure 12 As shown, the distribution of discount coefficients selected by the model for products in different age groups is displayed. This is used to verify the strategy characteristics of discounts changing with age and to help check whether the order mask constraint ensures that the price of older goods is not higher than that of new goods.

[0163] In this embodiment, the project is deployed online: After the system is connected to the enterprise's WMS inventory system or local inventory decision support software, the following online process will be executed: At the beginning of the cycle, read the real-time inventory of each age and verify whether the length of the input inventory status is equal to the shelf life, whether the inventory is non-negative, and whether the age before the decision is 0. Business personnel can input their operational preferences, such as prioritizing profitability during peak seasons. If the input weights are not equal to 1, the system will automatically normalize them. Automatically match the closest preference-specific checkpoint based on profit weight, or use a user-specified checkpoint; By inputting the state and preferences into the preference-conditionalized Actor-Critic network, we can obtain order quantity logits, discount logits, and two-dimensional value estimates. Decode using a deterministic order mask to output the order quantity with the highest probability and the discount coefficient that satisfies the monotonic constraint; Convert the discount factor into the actual selling price and mark whether there is available inventory for each age group; Optionally, several rounds of rolling simulation can be performed starting from the current state to show operators the expected profit and expected expiration amount of the strategy in future cycles.

[0164] During the deployment phase, the system can receive real-time inventory and preference weights through a command-line inference program or a graphical interface. The online inference interface can display parameter input, order quantity, age-based discount schemes, selling price, inventory bar chart, and optional rolling simulation statistics. Since the online phase only requires one neural network forward propagation and limited discount tier mask filtering, the computational complexity is low, which can meet the real-time requirements of daily inventory replenishment and price adjustment.

[0165] like Figure 13 As shown in the diagram, this figure illustrates the initial state of the system before it executes inference. Users can select completed training experiments, input the inventory quantity for each age group, and set profit and loss weights using sliders or input boxes. At this point, the results area has not yet generated a specific solution and displays "Waiting for inference results." After completing the parameter settings, clicking "Generate Intelligent Decision" will allow the system to call the corresponding model for inference.

[0166] like Figure 14 As shown in the figure, this diagram illustrates the system's operational results after inference. Based on the input inventory status and preference weights, the system automatically generates the current period's order quantity and age-group discount schemes. In the example, the model suggests ordering 10 units, with discounts for age groups 0-4 at 1.00, 1.00, 0.90, 0.80, and 0.80 respectively. The visualization area on the right further displays the order intensity and suggested prices for each age group, facilitating understanding and decision-making for operators.

[0167] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the flow or function according to the embodiments of this application is generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk).

[0168] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes the element.

[0169] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0170] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A data-driven method for optimizing age-stratified inventory of perishable goods, characterized in that, Includes the following steps: S1. Construct a digital twin environment for the dynamic system of perishable goods inventory; S2. Construct a preference-conditional dual-objective control network based on the digital twin environment; S3. The reinforcement learning training of the dual-objective control network is carried out using a differentiable Chebyshev scalarization method based on ideal point deviation; S4. Based on the training results, a sequential mask sampling mechanism is used to generate discount actions that satisfy the discount monotonicity constraint.

2. The data-driven age-stratified inventory optimization method for perishable goods according to claim 1, characterized in that: The digital twin environment of S1 cycles according to a decision-making cycle, in each cycle Initially, define an age-stratified inventory state vector: in, To fix the number of shelf-life cycles, For age Product inventory levels; The environmental state transition process includes: based on order quantity Update inventory, simulate stochastic demand and actual sales of products for different age groups based on a multi-logit consumer choice model, perform inventory aging and physical expiration, and calculate cyclical profit. With negative expired amount Two-dimensional instant reward vector ,in, This represents the amount of goods that have expired at the end of this period.

3. The data-driven age-stratified inventory optimization method for perishable goods according to claim 2, characterized in that: The dual-target control network of S2 uses the current inventory status. and business preference vector The network takes as input and outputs order placement actions and discount actions for products in different age groups; the network includes: A state encoder is used to map age-stratified inventory status into state feature vectors. Preference embedding layer, used to embed the business preference vector Mapped to a preference feature vector; The Actor policy network is used to generate a discrete probability distribution of order quantity and a discrete probability distribution of discount coefficient for products of each age group based on the state feature vector and the preference feature vector. The Critic value network is used to independently output the cumulative discounted return estimates for each of the two objectives based on the state feature vector and the preference feature vector. and .

4. The data-driven age-stratified inventory optimization method for perishable goods according to claim 3, characterized in that: The reinforcement learning training process of S3 includes: Single-cycle normalized reference points and cumulative ideal reference points for profit and loss reduction targets are determined through endpoint pre-training. ; During the policy sampling phase, a hybrid preference sampling strategy is adopted to cover the entire preference space; At each policy update, the temporal difference error and generalized advantage estimate of the two objectives are calculated separately to obtain a two-dimensional advantage vector: Using differentiable Chebyshev scalarization functions Value of the current state With the dominance vector Combining and calculating the advantages of scalarization: This drives the optimization and updating of near-end policies in the policy network.

5. The data-driven age-stratified inventory optimization method for perishable goods according to claim 4, characterized in that: The sequential mask sampling mechanism of S4 operates in ascending order of age, sequentially sampling each age group with available inventory. Determine the discount factor; for age groups The upper limit of its optional discount tiers is forcibly limited to those immediately adjacent to the newer age group. The selected discount tiers are masked to zero to eliminate the probability of invalid tiers. After re-normalization, the samples are then sampled to ensure that the generated discount sequence strictly satisfies the given conditions. .

6. The data-driven age-stratified inventory optimization method for perishable goods according to claim 2, characterized in that: The multi-logit consumer choice model is as follows: For the age group set with inventory Consumers choose to buy age groups The probability of the item is: Among them, utility function , As a basic attractiveness parameter for the product, Price sensitivity coefficient The freshness sensitivity coefficient Indicates the number of remaining marketable cycles; Total consumer demand It follows a Gamma-Poisson mixture distribution and is distributed through a multinomial distribution. The options are assigned to different age groups and those that do not require purchase.

7. The data-driven age-stratified inventory optimization method for perishable goods according to claim 4, characterized in that: The differentiable Chebyshev scalarization function Implemented using a smooth minimum function based on LogSumExp: in, To smooth out temperature parameters; The hybrid preference sampling strategy includes three stages: The first stage, from parameters Random sampling preference in a two-dimensional Dirichlet distribution; In the second phase, two extreme preference points are forcibly added in each round of data collection. and The corresponding complete trajectory; In the third stage, during the training convergence period, the process switches to traversing and sampling multiple grid points evenly distributed in the preference space to refine the policy quality in the intermediate preference region.

8. The data-driven age-stratified inventory optimization method for perishable goods according to claim 7, characterized in that: The training process of S3 also introduces a preference diversity regularization term. The regularization term enforces the following requirement: for the same state Two different preference vectors and The KL divergence between the action probability distributions output by the Actor policy network is proportional to the Euclidean distance between the two preference vectors; the regularization term is used to prevent the conditional policy network from converging on different preference outputs, thereby ensuring the integrity of the Pareto front.

9. The data-driven age-stratified inventory optimization method for perishable goods according to claim 8, characterized in that: The sequential mask sampling mechanism remains consistent in both the training and inference phases: in the training phase, it is used to calculate the logarithmic probability of joint actions and the policy gradient; in the inference phase, it is used to deterministically decode the output discount coefficient that satisfies the monotonicity constraint.

10. A data-driven age-stratified inventory optimization system for perishable goods, employing the data-driven age-stratified inventory optimization method for perishable goods as described in any one of claims 1-9, characterized in that, include: The state coding module is used to obtain the quantity vectors and demand context features of goods for each age group in the current inventory, which constitute the system state; The dual-objective reward calculation module is used to calculate the profit reward and the overdue loss penalty separately after the discount action is executed, and output the dual-objective reward vector. The preference conditionalization policy network module includes an Actor network and a Critic network. The Actor network receives a concatenated vector of state and preference weights and outputs a probability distribution of discounted actions. The sequence mask module is used to apply a monotonic mask to the action probability distribution based on the remaining shelf life information of products for each age group in the inventory, thereby blocking actions that violate the discount monotonicity constraint. The consumer choice modeling module is used to calculate the probability of consumers choosing products for each age group and the expected sales volume based on the MNL model. The differentiable Chebyshev scalarization module is used to train and update the preference conditional policy network by employing the PPO algorithm combined with the differentiable Chebyshev scalarization loss function. The decision output module is used to output the optimal discount rate vector for products of each age group based on the current state and preference weights, and connect to the online deployment interface.

Images