Centralized dynamic decision system for artificial intelligence-based dual-channel supply chain

Abstract

The application discloses a centralized dynamic decision system of a dual-channel supply chain based on artificial intelligence, relates to the technical field of deep reinforcement learning, and has the following core modules: a strategy library module pre-stores multiple strategy models; a parameter setting module manages environment parameters; an agent management module constructs a joint agent fusing a production party and a first sales party based on parameters, and constructs an ordinary agent for each second sales party; a strategy matching and scheduling module matches an optimal strategy for each agent, that is, the agent respectively performs single-time trial operation based on each strategy model, scores according to the result, and selects the highest-scored model as the optimal strategy; and a simulation module makes each agent perform simulation iterative training of resource allocation decision-making according to the optimal strategy. Through the combined architecture of the joint agent and the independent agent, multiple reinforcement learning strategies are fused, the system can dynamically adapt to the optimal learning paradigm in the heterogeneous role collaborative task, and the automation degree of strategy configuration and the deployment efficiency are significantly improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of deep reinforcement learning, and particularly relates to a centralized dynamic decision system for a dual-channel supply chain based on artificial intelligence. BACKGROUND

[0002] With the wide application of digital technology in the retail and manufacturing fields, the supply chain structure of enterprises presents a trend of multi-channel, multi-role and multi-task coordination. In particular, in the "dual-channel" environment, the system needs to handle data input and business response from two channels of self-operated platform and ordinary retailers. This structure increases the dimension of the state space in the supply chain, makes the control logic of the roles heterogeneous, and increases the complexity of the strategy optimization task, thus requiring an intelligent technology framework with self-adaptive ability to cope with it.

[0003] In order to solve the above problems, in recent years, reinforcement learning (RL) in artificial intelligence has been introduced as a decision optimization method. RL has the ability to learn strategies through interaction in a high-uncertainty environment, and can be used to complete task sequence optimization without precise modeling. The literature "Reinforcement Learning for Supply Chain Optimization" (published in the artificial intelligence conference European Workshop on Reinforcement Learning) proposes a centralized RL method modeled by Markov Decision Process (MDP) for simulating an "one factory + multiple warehouses" inventory control system. The method uses approximate SARSA and REINFORCE two strategies to achieve system response under periodic fluctuating demand scenarios through joint control of factory production and warehouse replenishment. The research results show that compared with traditional heuristic strategies (i.e. (s,Q) rules), the reinforcement learning method has advantages in strategy generalization and inventory control efficiency.

[0004] The core feature of the above method is a single agent structure, and its strategy is based on the calculation of joint actions based on the global state vector, and the strategy expression uses linear or nonlinear feature function approximation. However, this method mainly focuses on fixed structure and single role decision model, and is difficult to cope with the case of heterogeneous roles, differentiated control objectives and unequal strategy complexity in the dual-channel structure in reality. SUMMARY

[0005] In view of the problems in the prior art, the application provides a centralized dynamic decision system of a dual-channel supply chain based on artificial intelligence, which is constructed by combining a joint agent and an independent agent, fuses multiple types of reinforcement learning strategy models, supports the system in dynamically adapting an optimal learning paradigm in a heterogeneous role collaborative task, and thus improves the automation degree and deployment efficiency of overall strategy configuration.

[0006] The technical scheme of the application is implemented as follows:

[0007] The centralized dynamic decision system of the dual-channel supply chain based on artificial intelligence comprises a strategy library module, a parameter setting module, an agent management module, a strategy matching and scheduling module, and a simulation module.

[0008] The strategy library module pre-stores multiple strategy models.

[0009] The parameter setting module is used for managing parameters, including training parameters and environment parameters; the training parameters refer to a preset parameter set for controlling the training process of any one of the strategy models, and specifically include learning rate, discount factor, exploration rate, weight, proportion and other hyperparameters; the environment parameters refer to a parameter set for representing the state of a learning object and the feedback of an environment, and specifically include state space definition, reward function and action space.

[0010] The environment parameters include parameters of multiple objects; the objects include one producer, one first seller and multiple second sellers; the producer produces resources and allocates the resources according to the requests of the first seller and the second sellers.

[0011] The agent management module constructs agents based on the environment parameters, wherein a joint agent is constructed based on the parameters of the producer and the first seller, and an ordinary agent is constructed based on the parameters of each second seller.

[0012] The strategy matching and scheduling module performs strategy matching for each agent; the strategy matching is that one agent performs single training based on each strategy model, generates a score according to the single training result, and selects one strategy model with the highest score as the optimal strategy of the agent.

[0013] The simulation module performs simulation iterative training of resource allocation decision-making of each agent according to the corresponding optimal strategy; in the training process, the agents perform state updating and decision interaction based on shared environment parameters, and update the optimal strategy.

[0014] The double channel corresponds to a first seller and a second seller; the supply chain corresponds to a production party, which allocates resources for the first seller and the second seller respectively.

[0015] By modeling the joint agent and the common agent, the resource allocation behavior of the direct operation platform end and the retail distribution end can be effectively coordinated, the node heterogeneous environment can be adapted, and the inventory turnover structure and the resource allocation mode can be optimized. Specifically, the adaptive matching and centralized scheduling mechanism for the heterogeneous reinforcement learning strategy structure is provided by the strategy matching and scheduling module, which is particularly suitable for business networks such as double-channel supply chains that have node differences and complex control structures. By pre-training each agent, the optimal strategy with the strongest adaptability is obtained, that is, the system supports dynamic adaptation of the optimal learning paradigm in the heterogeneous role collaborative task, realizes multi-agent multi-strategy collaborative operation and iterative training, breaks through the heterogeneous strategy collaboration bottleneck, realizes dynamic performance optimization and global decision convergence, improves the convergence speed, and solves the control complexity explosion and strategy adaptation problem in the multi-strategy coexistence environment.

[0016] The strategy scheduling mechanism of the present application can be integrated with various control system structures without relying on specific business rules or management objectives, and has good engineering integration and cross-scene adaptability.

[0017] As a further optimization of the above scheme, in the strategy matching and scheduling module, the evaluation function is used to calculate the score.

[0018] The evaluation function is:

[0019] ;

[0020] Wherein, represents the score of the kth strategy model; represents the weight, ;

[0021] represents the score response function of the state dimension of the agent;

[0022] represents the score response function of the data size of the agent;

[0023] represents the score response function of the convergence period of the agent;

[0024] represents the score response function of the state error of the agent;

[0025] a score response function representing time complexity of the agent;

[0026] The optimal strategy at the tth iteration is represented as , i.e. ; The kth policy model is represented as

[0027] Further, , The values of , and , i.e. the error and time complexity of the policy run have greater influence on the score. Specifically, , The values of The value of , The values of

[0028] As a further optimization of the above scheme, the score response function of the state dimension of the agent is:

[0029] , ; and

[0030] The first sensitivity coefficient is represented as The dimension of the state input vector of an agent is represented as

[0031] Further, .

[0032] is used to measure the adaptability of a policy under the current state space dimension; is an exponential function. The sensitivity coefficient of the policy to the state dimension is represented as For rule-based policies (such as heuristic policies, Q-Learning), the first sensitivity coefficient is set to a larger value, such as [0.8, 1.0], i.e. the higher the state latitude, the faster the score of such a policy decreases; for deep reinforcement learning policies (such as DQN deep reinforcement learning algorithm, Actor-Critic deep reinforcement learning algorithm), the first sensitivity coefficient is set to a smaller value, such as [0.1, 0.2], indicating that it tolerates high latitude states. The rest of the policies are set in the middle, i.e. [0.2, 0.8], preferably 0.5.

[0033] As a further optimization of the above scheme, the score response function of the data size of the agent is:

[0034] , ; and

[0035] represents the second sensitivity coefficient, represents the number of samples input into an agent.

[0036] Further, .

[0037] for measuring the performance stability of a strategy under the number of available training samples. represents the data efficiency coefficient of the strategy, indicating how much data the strategy needs to achieve effective learning. For rule-based strategies, the second sensitivity coefficient is set to a larger value, such as [0.85, 1.0], indicating that a small amount of data can achieve a high score; for deep reinforcement learning strategies, the second sensitivity coefficient is set to a smaller value, such as [0.05, 0.25], indicating that a large number of samples are needed for convergence, and the score increases slowly. The rest of the strategies are set in the middle, i.e. [0.25, 0.85], preferably 0.5.

[0038] As a further optimization of the above scheme, the scoring response function of the convergence period of the agent is:

[0039] , ;

[0040] represents the third sensitivity coefficient, represents the maximum number of training periods, i.e. the number of iterations, represents the empirical convergence period of the kth strategy model.

[0041] Further, .

[0042] for measuring the convergence efficiency of a strategy under the maximum training period acceptable to the user. When the response score rises rapidly, it indicates that the strategy can achieve the desired effect relatively quickly within the tolerable training time; otherwise, the score is close to 0, indicating that the strategy is not suitable for fast-paced tasks.

[0043] represents the convergence speed sensitivity parameter of the strategy. For rule-based strategies, the third sensitivity coefficient is set to a larger value, such as [0.4, 0.5], a smaller value, such as 200000, indicating that the strategy can quickly converge in the early training period without the need for a large number of training times and sample accumulation, i.e. the strategy is suitable for supply chain agents with tight training time; for deep reinforcement learning strategies, the third sensitivity coefficient is set to a smaller value, such as [0.05, 0.1], is set to a large value, such as 1000000, indicating that the strategy learns slowly in the early stage of training and needs more training times to fully accumulate performance, i.e. the strategy is suitable for a supply chain agent with sufficient training time. In the remaining strategies, is set to a middle value, such as [0.1, 0.4], preferably 0.25, is set to 600000, indicating that the training speed and iteration number required by the strategy are moderate, i.e. the strategy is suitable for a supply chain agent with moderate training time.

[0044] As a further optimization of the above scheme, the scoring response function of the state error of the agent is:

[0045] , ;

[0046] represents the fourth sensitivity coefficient, represents the variance of the predicted return and the true return of the strategy.

[0047]

[0048] is a state nonlinear error response function, which is used to measure the prediction error degree of a strategy in the current state on the long-term return, i.e. the prediction accuracy. represents the sensitivity coefficient of the strategy to the nonlinear error, which measures the tolerance of the strategy to the prediction error of the return in the actual environment; is a logarithmic function. The function uses a nonlinear logarithmic compression function to improve the punishment for abnormal fluctuations and increase the anti-noise ability of the strategy in the high fluctuation state.

[0049] For rule-based strategies, the fourth sensitivity coefficient is set to a small value, such as [0.1, 0.3], indicating that the strategy has a small sensitivity to error fluctuations, and should be used for supply chain scenarios with linear system data relationships and small jumps; for deep reinforcement learning strategies that rely on prediction accuracy, the fourth sensitivity coefficient is set to a large value, such as [0.8, 1.0], i.e. the strategy uses a relatively complex deep neural network, and should be used for supply chain scenarios with nonlinear system data relationships and large jumps. The remaining strategies are set to a middle value, such as [0.3, 0.8], preferably 0.55, indicating that the strategy has a moderate sensitivity to error fluctuations, i.e. the strategy uses a feature value function with a moderate complexity, and should be used for supply chain scenarios with nonlinear system data relationships but small jumps.

[0050] As a further optimization of the above scheme, the scoring response function of the time complexity of the agent is:

[0051] , ;

[0052] denotes the fifth sensitivity coefficient, denotes the average time consumption of a single iteration training, is the training time consumption threshold.

[0053]

[0054] is the unit training time consumption response function, which is used to measure the cost of a strategy in unit training time consumption. denotes the sensitivity coefficient of a strategy to unit training time length, is equal to the total training time divided by , and is the unit training step time consumption. is the hyperbolic tangent function. By modeling with a piecewise function, early time consumption penalty and extreme time consumption strong penalty are taken into account. The combination of tanh function and Gaussian function is used to improve smoothness, and the adaptability of the strategy to inference delay and resource consumption in industrial deployment is emphasized.

[0055] For heuristic strategies, the fifth sensitivity coefficient is set to a large value, such as [0.4, 0.5], and the training time consumption threshold is set to 100 ms, indicating that the strategy is more sensitive to the difference between training time consumption and tolerance threshold. That is, the strategy should have shorter training iteration time consumption and lower delay tolerance because of its simple algorithm; for deep reinforcement learning strategies, the fifth sensitivity coefficient is set to a small value, such as [0.05, 0.1], and the training time consumption threshold is set to 10000 ms, indicating that the strategy is more sluggish to the difference between training time consumption and tolerance threshold. That is, the strategy should have longer training iteration time consumption and higher delay tolerance because of its complex algorithm. The rest of the strategies are set to a medium value, such as [0.1, 0.4], preferably 0.3, and the training time consumption threshold is set to 100 ms.

[0056] As a further optimization of the above scheme, a joint state vector is constructed according to the environmental parameters , that is,

[0057] ;

[0058] wherein, denotes n groups of sequences, corresponding to n said second sellers, ;

[0059] denotes the resource stock state of the production party at period t;

[0060] denotes the resource stock state of the first seller at period t;

[0061] represents the resource inventory state of the i-th second seller at period t;

[0062] represents the resource request amount of the first seller to the producer at period t-1;

[0063] represents the resource request amount of the i-th second seller to the producer at period t-1;

[0064] represents the environmental disturbance variable of the first seller at period t-1;

[0065] represents the environmental disturbance variable of the i-th second seller at period t-1;

[0066] The environmental disturbance variable is an external factor that affects the resource request amount, such as weather factors, seasonal factors, etc., which is used to estimate the system evolution trend.

[0067] represents the resource demand amount of the external environment to the first seller at period t-1;

[0068] represents the resource demand amount of the external environment to the i-th second seller at period t-1;

[0069] The joint agent adopts as input features, and the i-th ordinary agent adopts as input features;

[0070] constructs corresponding actions, i.e. : ;

[0071] represents the resource production amount of the producer at period t; are the action prediction outputs of a strategy at period t according to , and are used for updating .

[0072] wherein the three-dimensional action output of the joint agent is: ; and the two-dimensional action output of the i-th ordinary agent is: ;

[0073] " represents the strategy allocated to the joint agent , and the input state ​The obtained action output represents the calculation results of the upstream resource allocation, the direct operation platform supply distribution and the strategy characteristic parameters, and is used to drive the subsequent state evolution and strategy evaluation. Accordingly, the control intention of the local resource request and the response variable output is expressed for the system environment response execution.

[0074] For any action of any agent, the reward value is calculated according to the reward function, the optimal action is selected according to the reward value, the state is updated according to the optimal action, and the state transition is constructed;

[0075] The state update is updated once and twice;

[0076] The first update is represented as: = ;

[0077] The second update is based on the first update The resources are allocated to the first seller and the second seller, and the number of allocated resources is represented as and ;

[0078] If , then , , the second update is represented as: ;

[0079] If , then the resources are allocated to and according to the preset request priority or the preset allocation ratio, and the second update is represented as: ; the request priority allocation is to allocate resources to the first seller or the second seller first, until .

[0080] The resource demand is updated according to and , that is:

[0081] , ;

[0082] , ;

[0083] wherein represents the reference demand, which can be fitted by historical data; is an environmental disturbance variable ​corresponding coefficients of the environment adjustment parameters, used to describe the weight of the environment adjustment parameters in the simulation demand generation, ; is a Gaussian distribution random term, used to simulate the demand disturbance and volatility, is a disturbance amplitude; d represents the parameter corresponding to the first seller, i.e., the joint agent; i represents the parameter corresponding to the ith second seller, i.e., the ordinary agent; the above demand generation mechanism is used to construct the state evolution in the "strategy-state-return" path.

[0084] The update of the strategy model is represented as: ;

[0085] The update of the strategy model is represented as: ; represents taking the minimum value;

[0086] Finally, , the state transition is represented as: ;

[0087] The state transition is used to update the strategy model.

[0088] All agents use the experience replay mechanism (Replay Buffer) to manage the training samples, and store the state transition as a sample into the local experience pool and maintain and update it.

[0089] As a further optimization of the above scheme, the reward function is represented as:

[0090] ;

[0091] wherein, represents the reward value, used for reward calculation in the reinforcement learning objective function; are respectively preset system feedback weighting coefficients; j is used to represent d or i; represents the unit transfer loss coefficient of the resource; represents the production cost of the resource; , respectively represent the acceptance cost of the resource by the joint agent and the ordinary agent; , respectively represent the holding cost of the resource by the joint agent and the ordinary agent; , respectively represent other associated costs of the resource by the joint agent and the ordinary agent.

[0092] The above-mentioned "production costs" and "holding costs" do not specifically refer to "price" costs. In any field, they can be key physical parameters that affect the final reward value, such as "resource consumption rate," "equipment wear rate," "storage space utilization rate," "material decay rate," and "wastewater treatment volume."

[0093] For example, in the field of "fresh food", the final decision-making goal is to select a solution that can achieve the lowest cargo damage rate in the entire supply chain process, then the reward value is " " represents the estimated level of cargo remaining, then " " indicates the cargo retention rate under the temperature and humidity of the storage environment;" "Indicates the damage rate of goods at corresponding temperature and humidity under the cold chain transportation;" "Indicates the damage rate of goods under the temperature and humidity inside the warehouse;" " represents the random interference coefficient. Various costs can be fitted through historical data.

[0094] As a further optimization of the above scheme, the strategy model includes a rule-based heuristic strategy, which can be expressed as (s,Q), (s,S); tabular Q-Learning; REINFORCE characterized by quadratic basis; REINFORCE characterized by RBF; DQN deep reinforcement learning algorithm; Actor-Critic deep reinforcement learning algorithm.

[0095] Different strategies have different update methods. Heuristic strategies do not need gradient updates and directly output preset actions according to rule conditions; Tabular Q-Learning Incremental update, based on time difference error correction; REINFORCE updates the strategy function by using the parameter vector, relying on the feature function mapping ; The deep reinforcement learning algorithm optimizes neural network parameters, uses mini-batch SGD / Adam, introduces target networks, and uses experience replay to improve convergence and stability.

[0096] Compared with the prior art, the present invention achieves the following beneficial effects:

[0097] (1) The application provides a self-adaptive matching and centralized scheduling mechanism for a heterogeneous reinforcement learning strategy structure through a strategy matching and scheduling module, which is particularly suitable for a business network such as a dual-channel supply chain that has node differences and control structure complexity, and through pre-training of each agent, the most optimal strategy with the strongest adaptability is obtained, that is, the system supports dynamic adaptation of the optimal learning paradigm in the heterogeneous role collaborative task, and then realizes multi-agent multi-strategy collaborative operation and iterative training, breaks through the heterogeneous strategy collaboration bottleneck, realizes dynamic performance optimization and global decision convergence, improves the convergence speed, and solves the control complexity explosion and strategy adaptation problem in the multi-strategy coexistence environment.

[0098] (2) The strategy scheduling mechanism of the application can be integrated with various control system structures without relying on specific business rules or management targets, and has good engineering integration and cross-scene adaptability.

[0099] (3) The selected strategy structure is dynamically distributed to the target agent through the strategy matching and scheduling module, and supports the matching parameter initialization, strategy hot start, model migration and strategy structure replacement, forming a unified management, on-demand calling and continuous evolvable control structure scheduling closed loop. The mechanism effectively realizes the evolution of the strategy structure from "static configuration" to "dynamic scheduling".

[0100] (4) Through the modeling mode of joint agents and ordinary agents, the resource allocation behavior of different ports can be effectively coordinated, the node heterogeneous environment can be adapted, and the inventory turnover structure and resource allocation mode can be optimized. BRIEF DESCRIPTION OF DRAWINGS

[0101] Figure 1 is a module interaction schematic diagram of the centralized dynamic decision system of the dual-channel supply chain based on artificial intelligence provided by the embodiment of the application;

[0102] Figure 2 is a process schematic diagram of strategy matching provided by the embodiment of the application. DETAILED DESCRIPTION

[0103] In order to make the purpose, technical scheme and advantages of the application clearer, the technical scheme in the embodiment of the application will be described clearly and completely in combination with the drawings in the embodiment of the application. Obviously, the described embodiments are only a part of the embodiments of the application, not all the embodiments. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative labor belong to the protection scope of the application.

[0104] As Figure 1 , 2As shown, the embodiment provides a centralized dynamic decision system of a dual-channel supply chain based on artificial intelligence, which comprises a strategy library module, a parameter setting module, an agent management module, a strategy matching and scheduling module, and a simulation module.

[0105] The strategy library module pre-stores a plurality of strategy models; in the embodiment, the strategy models include rule-based heuristic strategies, which can be represented as (s,Q), (s,S); table type Q-Learning; REINFORCE characterized by quadratic basis; REINFORCE characterized by RBF; DQN deep reinforcement learning algorithm; and Actor-Critic deep reinforcement learning algorithm.

[0106] The parameter setting module is used for managing parameters, including training parameters and environment parameters; the training parameters refer to a preset parameter set containing a training process of controlling any one strategy model, and specifically, the training parameters include learning rate, discount factor, exploration rate, and other hyperparameters; and the environment parameters refer to a parameter set representing a state of a learning object and environment feedback.

[0107] The environment parameters include parameters of a plurality of objects; the objects include one producer, one first seller, and a plurality of second sellers; the producer produces resources and allocates the resources according to requests of the first seller and the second sellers.

[0108] The agent management module constructs agents based on the environment parameters, wherein a joint agent is constructed based on parameters of the producer and the first seller, and a common agent is constructed based on parameters of each second seller.

[0109] In the embodiment, a joint state vector is constructed according to the environment parameters, which is represented as , that is,

[0110] ;

[0111] wherein, n groups of sequences corresponding to n second sellers, ;

[0112] represents a resource inventory state of the producer at period t;

[0113] represents a resource inventory state of the first seller at period t;

[0114] represents a resource inventory state of the i-th second seller at period t;

[0115] represents a resource request amount of the first seller to the producer at period t-1;

[0116] denotes the resource request amount of the i-th second seller to the producer at period t-1;

[0117] denotes the environmental disturbance variable of the first seller at period t-1;

[0118] denotes the environmental disturbance variable of the i-th second seller at period t-1; the environmental disturbance variable is an external factor affecting the resource request amount, such as weather factors, seasonal factors, etc., and is used to estimate the system evolution trend.

[0119] denotes the resource demand amount of the external environment to the first seller at period t-1;

[0120] denotes the resource demand amount of the external environment to the i-th second seller at period t-1;

[0121] The joint agent adopts as input features, and the i-th normal agent adopts as input features.

[0122] The corresponding action, i.e. , is constructed as : ;

[0123] denotes the resource production amount of the producer at period t; are the action prediction outputs of a strategy at period t according to , and are used for updating .

[0124] wherein the three-dimensional action output of the joint agent is ; and the two-dimensional action output of the i-th normal agent is .

[0125] “ ” denotes the action output obtained after inputting the state to the strategy allocated to the joint agent, representing the calculation results of the upstream resource allocation, direct operation platform supply allocation, and strategy feature parameters, which are used to drive the subsequent state evolution and strategy evaluation; “ ” is the same, and is used to express the control intention of the related features of the local resource request and response variable output, for system environment response execution.

[0126] The policy matching and scheduling module performs policy matching for each agent respectively; the policy matching is that each agent performs single training based on each policy model, generates a score according to the single training result, and selects a policy model with the highest score as the optimal policy of the agent. Specifically, the score is calculated by using an evaluation function; the evaluation function is:

[0127] ;

[0128] wherein, represents the score of the kth policy model; the optimal policy at the tth iteration is represented as , that is, ; represents the kth policy model. The numerical value of each of the is 0.15, The numerical value of the is 0.1, The numerical value of each of the is 0.3.

[0129] represents the score response function of the state dimension of the agent, and specifically is:

[0130] ;

[0131] represents the first sensitivity coefficient, represents the dimension of the state input vector of an agent. is used to measure the adaptability of a policy under the current state space dimension; is an exponential function. represents the sensitivity coefficient of the policy to the state dimension. For a rule-based policy (such as a heuristic policy, Q-Learning), the first sensitivity coefficient is set to 0.9, that is, the higher the state latitude, the faster the score of the policy decreases; for a deep reinforcement learning policy (such as a DQN deep reinforcement learning algorithm, an Actor-Critic deep reinforcement learning algorithm), the first sensitivity coefficient is set to 0.1, indicating that it tolerates high latitude states. The rest is set to 0.5.

[0132] represents the score response function of the data size of the agent, and specifically is:

[0133] ;

[0134] represents the second sensitivity coefficient, represents the number of samples input into an agent. is used to measure the performance stability of a policy under the number of available training samples. Data efficiency coefficient of the strategy, indicating how much data the strategy needs to achieve effective learning. For rule-based strategies, the second sensitivity coefficient is set to 0.85, indicating that a small amount of data can achieve a high score; for deep reinforcement learning strategies, the second sensitivity coefficient is set to 0.25, indicating that a large number of samples are needed for convergence, and the score increases slowly. The rest of the strategies are set to 0.5.

[0135] Score response function representing the convergence period of the agent, specifically:

[0136] ;

[0137] Third sensitivity coefficient, Maximum number of training periods, i.e. iteration number, Experience convergence period of the kth strategy model. Used to measure the convergence efficiency of a strategy under the maximum training period acceptable to the user. The response score rises rapidly, indicating that the strategy can achieve the desired effect relatively quickly within the tolerable training time; otherwise, the score is close to 0, indicating that the strategy is not suitable for fast-paced tasks.

[0138] Convergence speed sensitivity parameter of the strategy. For rule-based strategies, the third sensitivity coefficient is set to 0.45, Set to 200000, indicating that the strategy can quickly converge at the beginning of training, without the need for a large number of training times and sample accumulation, i.e. the strategy is suitable for supply chain agents with tight training time; for deep reinforcement learning strategies, the third sensitivity coefficient is set to 0.1, Set to 1000000, indicating that the strategy learns slowly at the beginning of training, and needs more training times to accumulate sufficient iterations to reflect performance, i.e. the strategy is suitable for supply chain agents with sufficient training time. In the rest of the strategies, Set to 0.25, Set to 600000, indicating that the training speed and iteration number required by the strategy are moderate, i.e. the strategy is suitable for supply chain agents with moderate training time.

[0139] Score response function representing the state error of the agent, represented as:

[0140] ;

[0141] Fourth sensitivity coefficient, Variance of the predicted return and the true return of the strategy.

[0142] is a state nonlinear error response function, which measures the degree of prediction error of a strategy in the current state on the long-term return, that is, the prediction accuracy. is a sensitivity coefficient of the strategy to the nonlinear error, which measures the tolerance of the strategy to the prediction error of the return in the actual environment; is a logarithmic function. The function uses a nonlinear logarithmic compression function to improve the punishment for abnormal fluctuations and increase the anti-noise ability of the strategy in a high fluctuation state.

[0143] For a rule-based strategy, the fourth sensitivity coefficient is set to 0.2, indicating that the strategy is less sensitive to error fluctuations, and should be used to handle supply chain scenarios with linear system data relationships and small jumps; for deep reinforcement learning strategies that rely on prediction accuracy, the fourth sensitivity coefficient is set to 0.9, that is, the strategy uses a relatively complex deep neural network, and should be used to handle supply chain scenarios with nonlinear system data relationships and large jumps. The remaining strategies are set to 0.55, indicating that the strategy is moderately sensitive to error fluctuations, that is, the strategy uses a feature value function with moderate complexity, and should be used to handle supply chain scenarios with nonlinear system data relationships but small jumps.

[0144] is a scoring response function that represents the time complexity of the agent, specifically:

[0145] , ;

[0146] is the fifth sensitivity coefficient, is the average time consumption of a single iteration of training, is the training time threshold. is a unit training time response function, which is used to measure the cost of a strategy in unit training time. is a sensitivity coefficient of the strategy to the unit training time, is the unit training step time consumption, which is equal to the total training time divided by . is a hyperbolic tangent function. The piecewise function models both early time consumption punishment and extreme time consumption strong punishment, and uses a combination of tanh functions and Gaussian functions to improve smoothness, emphasizing the strategy's ability to adapt to inference delays and resource consumption in industrial deployment.

[0147] For the heuristic strategy, the fifth sensitivity coefficient is set to 0.5, and the training time consumption threshold is set to 100 ms, indicating that the strategy is more sensitive to the difference between the training time consumption and the patience threshold, that is, the strategy should have shorter training iteration time consumption and lower delay tolerance because of the simplicity of the algorithm; for the deep reinforcement learning strategy, the fifth sensitivity coefficient is set to 0.1, and the training time consumption threshold is set to 10000 ms, indicating that the strategy is more sluggish to the difference between the training time consumption and the patience threshold, that is, the strategy should have longer training iteration time consumption and higher delay tolerance because of the complexity of the algorithm. The remaining strategies are set to 0.3, and the training time consumption threshold is set to 100 ms.

[0148] The simulation module is that each agent respectively performs simulation iteration training of resource allocation decision according to the corresponding optimal strategy. During the training process, the agents update the state and interact with the decision based on the shared environment parameters, and update the optimal strategy.

[0149] Specifically, the iteration training includes: for any action of any agent, calculating a reward value according to a reward function, and selecting an optimal action according to the reward value; updating the state according to the optimal action, and constructing a state transition.

[0150] When updating the state, the first salesperson and the second salesperson are updated once and twice. The once update is represented as:

[0151]

[0152] The twice update is according to the once updated to the first salesperson and the second salesperson, and the number of resources allocated is represented as and , respectively.

[0153] If , then , , and the twice update is represented as: .

[0154] If , then resources are allocated to and according to the preset request priority or preset allocation ratio, and the twice update is represented as: ; the request priority allocation is that resources are preferentially allocated to the first salesperson or the second salesperson until .

[0155] The resource demand is updated according to and , that is:

[0156] ​​​​ , ;

[0157] , ;

[0158] wherein, represents the benchmark demand, which can be fitted by historical data; is the corresponding coefficient of the environmental disturbance variable , used to depict the weight of the environmental adjustment parameter in the simulation demand generation, ; is a Gaussian distribution random term, used to simulate the demand disturbance and volatility, is the disturbance amplitude; d represents the parameter corresponding to the first seller, that is, the joint agent; i represents the parameter corresponding to the ith second seller, that is, the ordinary agent; the above demand generation mechanism is used to construct the state evolution in the "strategy-state-return" path.

[0159] The update of ;

[0160] The update of ; represents taking the minimum value;

[0161] Finally, , the state transition is represented as: ;

[0162] The strategy model is updated by using the state transition. The six strategy update methods indicated in the embodiment are not the invention points of the present application, and the prior art can be used to implement them.

[0163] All agents use the experience replay mechanism (Replay Buffer) to manage the training samples, and store the state transition as a sample into the local experience pool and maintain and update it.

[0164] In the embodiment, the reward function is represented as:

[0165] ;

[0166] wherein, represents the reward value, used for reward calculation in the reinforcement learning objective function; are respectively preset system feedback weighting coefficients; j is used to represent d or i; represents the unit transfer loss coefficient of the resource; represents the production cost of the resource; , respectively represent the joint agent and the ordinary agent's holding cost of the resource. , respectively represent the joint agent and the ordinary agent's holding cost of the resource. , respectively represent the joint agent and the ordinary agent's other associated cost of the resource.

[0167] In this embodiment, one action with the maximum reward value is selected to construct the state transition.

[0168] The double channel corresponds to the first seller and the second seller; the supply chain corresponds to the production party, and the production party allocates resources to the first seller and the second seller. In the application environment, the production party is a factory, has the resource production capacity, the first seller is a factory direct store, and the second seller is a retail store.

[0169] Through the modeling mode of the joint agent and the ordinary agent, the resource allocation behavior of the direct operation platform end and the retail distribution end can be effectively coordinated, the node heterogeneous environment can be adapted, and the inventory turnover structure and the resource allocation mode can be optimized. Specifically, the application provides an adaptive matching and centralized scheduling mechanism for a heterogeneous reinforcement learning strategy structure through a strategy matching and scheduling module, is particularly suitable for a business network such as a double-channel supply chain that has node differences and control structure complexity, obtains the optimal strategy with the strongest adaptability by pre-training each agent, that is, supports the system to dynamically adapt the optimal learning paradigm in the heterogeneous role collaborative task, realizes the collaborative operation and iterative training of the multi-agent and the multi-strategy, breaks through the heterogeneous strategy collaboration bottleneck, realizes dynamic performance optimization and global decision convergence, improves the convergence speed, and solves the control complexity explosion and strategy adaptation problem in the multi-strategy coexistence environment.

[0170] The strategy scheduling mechanism of the application can be integrated with various control system structures, does not need to depend on specific business rules or management targets, has good engineering integrability and cross-scene adaptability.

[0171] According to the disclosure and teaching of the above description, those skilled in the art of the application can also make changes and modifications to the above embodiments. Therefore, the application is not limited to the specific embodiments disclosed and described above, and some modifications and changes of the application should fall within the protection scope of the claims of the application. In addition, although some specific terms are used in the specification, these terms are only for convenience of explanation and do not constitute any limitation on the application.

Claims

1. A centralized dynamic decision system for dual-channel supply chain based on artificial intelligence, characterized in that, The method comprises the following steps: A strategy library module is configured to store a plurality of strategy models; A parameter setting module is configured to manage parameters, including training parameters and environment parameters; the training parameters refer to a preset parameter set for controlling the training process of any strategy model; the environment parameters refer to a parameter set representing the state of the learning object and the feedback of the environment; The environment parameters include parameters of a plurality of objects; the objects include a producer, a first seller and a second seller; the producer produces resources and allocates the resources according to the requests of the first seller and the second seller; Constructing a joint state vector representation from environmental parameters That is: ； wherein, denotes n groups of sequences, corresponding to n said second sellers, ; , , denote the resource stock state of the producer, the resource stock state of the first seller, the resource stock state of the i-th second seller, respectively, at period t. 、 、 、 、 、 Respectively represent the resource request amount of the first seller to the producer, the resource request amount of the i-th second seller to the producer, the environmental disturbance variable of the first seller, the environmental disturbance variable of the i-th second seller, the resource demand amount of the external environment on the first seller, and the resource demand amount of the external environment on the i-th second seller in period t-1; The intelligent agent management module is configured to construct intelligent agents based on the environment parameters; wherein a joint intelligent agent is constructed based on the parameters of the production party and the first sales party, and the joint intelligent agent adopts as input features; and an i-th common intelligent agent is constructed based on the parameters of the i-th second sales party, and the i-th common intelligent agent adopts as input features. A strategy matching and scheduling module is configured to perform strategy matching for each agent; the strategy matching is that each agent is trained based on each strategy model, a score is generated according to the single training result, and the strategy model with the highest score is selected as the optimal strategy of the agent; A simulation module, wherein the agents perform simulation iterative training of resource allocation decision-making according to the corresponding optimal strategies; in the training, each agent updates the state and interacts with the decision-making based on the shared environment parameters, and updates the optimal strategy. 2.The centralized dynamic decision system of the dual-channel supply chain based on artificial intelligence according to claim 1, wherein, In the strategy matching and scheduling module, the score is calculated by using an evaluation function; The evaluation function is a weighted sum of a plurality of evaluation indexes, and the sum of the weights corresponding to each evaluation index is 1; The evaluation indexes include state dimension, data size, convergence period, state error and time complexity of the agent; The optimal strategy at the tth iteration is expressed as ,Right now ; represents the k-th strategy model, represents the score of the kth policy model. 3.The centralized dynamic decision system of the dual-channel supply chain based on artificial intelligence according to claim 2, wherein, In the evaluation function, the score of the state dimension is calculated based on the dimension of the state input vector of an agent and a preset first sensitivity coefficient; the value range of the first sensitivity coefficient is [0.1, 1.0]; Based on the score of the state dimension, the adaptability of one strategy model under the current state space dimension is measured.

4. The centralized dynamic decision system for the dual-channel supply chain based on artificial intelligence according to claim 2, wherein, In the evaluation function, the score of the data size is calculated based on the number of samples of an agent and a preset second sensitivity coefficient; the value range of the second sensitivity coefficient is [0.05, 1.0]; Based on the score of the data size, the performance stability of one strategy model under the number of available training samples is measured. 5.The centralized dynamic decision system of the dual-channel supply chain based on artificial intelligence according to claim 2, characterized in that, In the evaluation function, the score of the convergence period is calculated based on the maximum training period number of an agent, the experience convergence period of one strategy model and a preset third sensitivity coefficient; The value range of the third sensitivity coefficient is [0.05, 0.5]; Based on the score of the convergence period, the convergence efficiency of one strategy model is measured.

6. The centralized dynamic decision system for the dual-channel supply chain based on artificial intelligence according to claim 2, characterized in that, In the evaluation function, the score of the state error is calculated based on the variance of the predicted reward and the real reward obtained after the training of an agent, a preset fourth sensitivity coefficient; the value range of the fourth sensitivity coefficient is [0.1, 1.0]; Based on the score of the state error, the prediction error degree of one strategy model is measured. In the evaluation function, the score of the time complexity is calculated based on the variance of the predicted reward and the real reward obtained after the training of an agent, a preset fifth sensitivity coefficient; the value range of the fifth sensitivity coefficient is [0.05, 0.5].

7. The centralized dynamic decision system for the dual-channel supply chain based on artificial intelligence of claim 2, wherein, ​ Based on the time complexity score, measure the time consumption cost of one of the policy models in the training process.

8. The centralized dynamic decision system for the artificial intelligence-based two-channel supply chain according to claim 1, wherein, Construction Corresponding actions, i.e. : ; represents the resource production of the producer at period t; The three-dimensional action output of the joint agent is: The two-dimensional action output of the i-th common agent is: ; For any one of the actions of any one of the agents, calculate a reward value according to a reward function, select an optimal action according to the reward value, update the state according to the optimal action, and construct a state transition; The state is updated at primary and secondary updates. The one update is represented as: = ;​ The secondary update is according to the primary update The first seller and the second seller are allocated resources, and the number of allocated resources is respectively represented as and ; wherein, if then , , the secondary update is expressed as: ; If , then allocate the resource to and , the secondary update is expressed as: ; According to and updating the resource demand, i.e.: , ； , ； wherein, represents a reference demand amount; is an environmental disturbance variable with a corresponding coefficient, ; is a Gaussian distributed random term, is a disturbance amplitude; d represents a parameter corresponding to the first seller; i represents a parameter corresponding to the i-th second seller; The update representation is: ; The update representation is: ; Finally, , the state transition is represented as: ; Update the policy model by using the state transition.

9. The centralized dynamic decision system for the artificial intelligence-based two-channel supply chain according to claim 8, wherein, The reward function is represented as: ； wherein, represents the reward value; respectively represent preset system feedback weighting coefficients; j is used to represent d or i; represents a unit transfer loss coefficient of the resource; represents a production cost of the resource; , respectively represent an acceptance cost of the resource for the joint agent and the ordinary agent; , respectively represent a holding cost of the resource for the joint agent and the ordinary agent; , respectively represent other associated costs of the resource for the joint agent and the ordinary agent.

10. The centralized dynamic decision system for the dual-channel supply chain based on artificial intelligence of claim 1, wherein, The policy model includes a rule-based heuristic policy, a table type Q-Learning, a REINFORCE characterized by a quadratic basis, a REINFORCE characterized by an RBF, a DQN deep reinforcement learning algorithm, and an Actor-Critic deep reinforcement learning algorithm.

Images