Multi-objective optimization method and system for centralized procurement with quantity based on MODDPG

Abstract

The invention discloses a multi-objective optimization method and system for centralized procurement with quantity based on MODDPG. The method comprises the following steps: constructing a collection chain multi-objective optimization model which comprises a supply enterprise, a procurement alliance and a third-party group purchase organization and is based on deep reinforcement learning and a multi-objective optimization algorithm; based on the historical state of the supply enterprise, the purchase alliance or the third-party group purchase organization, control operation is generated through a strategy network, and random variation is carried out on the control operation according to the mutation probability; executing control operation, obtaining rewards and states fed back by the environment, and adding the historical state, the control operation, the rewards and the states into an experience pool; selecting a high-quality population sample from the experience pool by using a multi-objective optimization algorithm; training a strategy network and a value evaluation network of a set sampling chain multi-objective optimization model based on the high-quality population samples; and using the trained model to generate a control operation strategy according to the current states of the supply enterprise, the purchase alliance and the third-party group purchase organization.

Description

Technical Field

[0001] This invention relates to the field of centralized cost optimization technology for centralized volume-based procurement, and more specifically to a multi-objective optimization method and system for centralized volume-based procurement based on MODDPG. Background Technology

[0002] Globally, it's common for governments to promote the establishment of purchasing alliances and group purchasing organizations (GPOs) to consolidate demand through centralized bulk purchasing, achieving economies of scale, reducing procurement costs, and improving supply chain efficiency. Typically, GPOs aggregate the demands of the purchasing alliance, procure from suppliers, negotiate prices, provide high-quality procurement services, and simultaneously ensure suppliers receive stable and substantial orders through policy interventions, attracting numerous suppliers to bid and provide supply. However, the pursuit of profit maximization by individual members within the centralized procurement supply chain leads to decreased overall supply chain profits and increased costs, exacerbating the double marginal effect. To help suppliers increase profits, reduce operating costs for purchasing alliances, and ensure the financial stability of GPOs, multi-objective optimization algorithms can be used to balance the interests of all parties. Traditional optimization algorithms often use linear weighting to transform multi-objective functions into single-objective functions when solving multi-objective optimization problems, leading to human intervention in the final results. Summary of the Invention

[0003] This invention provides a multi-objective optimization method and system for centralized volume-based procurement based on MODDPG, which solves the technical problem that traditional optimization algorithms often use linear weighting to transform multi-objective functions into single-objective functions when solving multi-objective optimization problems, resulting in the final result being subject to human intervention.

[0004] To address the aforementioned technical problems, according to one aspect of the present invention, a multi-objective optimization method for centralized volume-based procurement based on MODDPG is provided, comprising: A multi-objective optimization model for the centralized procurement chain, based on deep reinforcement learning and multi-objective optimization algorithms, is constructed, incorporating supply enterprises, procurement alliances, and third-party group-buying organizations. The comprehensive objective function of the multi-objective optimization model for the centralized procurement chain is: The reward function is , and These represent the profit objective function of the supplier, the procurement fund surplus objective function of the third-party group buying organization, and the cost objective function of the procurement alliance, respectively. Based on the aforementioned multi-objective optimization model for the centralized procurement chain, the target multi-objective optimization model for the centralized procurement chain is obtained by training it with the historical states and control operations of supply enterprises, procurement alliances, and third-party group-buying organizations. Specifically, it includes: Collect historical status and control operations of suppliers, purchasing alliances, and third-party group buying organizations; Based on the historical status of the supply enterprises, procurement alliances and third-party group buying organizations, control operations are generated through a policy network, and the control operations are randomly mutated according to the mutation probability. Execute the mutated / generated control operations and obtain rewards and status feedback from the environment. The historical state, the mutated control operation / generated control operation, the rewards and states fed back by the environment will be included. The transformed tuple samples are added to the experience pool; When the amount of data in the experience pool reaches the threshold, a multi-objective optimization algorithm is used to select parent transformation tuple samples from the experience pool, and the parent transformation tuple samples are processed to generate child transformation tuple samples. Non-dominated sorting and crowding distance calculation are performed on the state data in the population sample combination after merging the parent generation transformed tuple samples and the offspring generation transformed tuple samples. Based on the calculated crowding distance, high-quality population samples are selected from the population sample combination. Based on the high-quality population samples, the strategy loss function and value loss function are calculated, and the network parameters are updated using the gradient descent algorithm. The target network parameters are synchronized through a soft update mechanism, and the training is terminated according to the Pareto convergence condition, thus obtaining the target collection chain multi-objective optimization model. The target-oriented centralized procurement chain multi-objective optimization model is used to generate optimized control operation strategies based on the current status of supply enterprises, procurement alliances and third-party group buying organizations.

[0005] To address the aforementioned technical problems, according to another aspect of the present invention, a centralized volume-based procurement multi-objective optimization system based on MODDPG is provided, comprising: The model building module is used to construct a multi-objective optimization model for the centralized procurement chain, based on deep reinforcement learning and multi-objective optimization algorithms, which includes supply enterprises, procurement alliances, and third-party group-buying organizations. The comprehensive objective function of the centralized procurement chain multi-objective optimization model is: The reward function is , and These represent the profit objective function of the supplier, the procurement fund surplus objective function of the third-party group buying organization, and the cost objective function of the procurement alliance, respectively. The model training module is used to collect historical states and control operations from supply enterprises, procurement alliances, and third-party group-buying organizations. Based on the historical states of these entities, control operations are generated through the policy network of a multi-objective optimization model for the centralized procurement chain, and these operations are randomly mutated according to mutation probabilities. The mutated control operations / generated control operations are executed to obtain rewards and states from environmental feedback. The historical state, the mutated control operation / generated control operation, the rewards and states fed back by the environment will be included. The transformed tuple samples are added to the experience pool. When the amount of data in the experience pool reaches a threshold, a multi-objective optimization algorithm is used to select parent transformed tuple samples from the experience pool and process them to generate child transformed tuple samples. The state data in the population sample combination after merging the parent and child transformed tuple samples are sorted by non-dominated order and crowding distance is calculated. Based on the calculated crowding distance, high-quality population samples are selected from the population sample combination. The strategy loss function and value loss function are calculated based on the high-quality population samples, and the network parameters are updated using the gradient descent algorithm. The target network parameters are synchronized through a soft update mechanism. The training is terminated based on the Pareto convergence condition, resulting in the target collection chain multi-objective optimization model. The multi-objective optimization module is used to generate optimized control operation strategies based on the current status of supply enterprises, procurement alliances and third-party group buying organizations using the multi-objective optimization model of the target procurement chain.

[0006] The beneficial technical effects of this invention are as follows: Compared with the prior art, this invention constructs a multi-objective optimization model for the centralized procurement chain based on deep reinforcement learning and multi-objective optimization algorithms, which includes supply enterprises, procurement alliances, and third-party group-buying organizations. By combining multi-objective optimization and deep reinforcement learning, and through non-dominated sorting, selection, crossover, and mutation operations, the convergence speed of the neural network can be accelerated, ensuring the diversity and convergence of the solution set. It can simultaneously optimize multiple objective functions and avoid getting trapped in local optima, thereby improving the adaptability and robustness of the model. Furthermore, the model uses a scalar reward function, eliminating the need for weight allocation of objectives, thus avoiding human intervention and simplifying the optimization process. It also introduces a Pareto stopping condition, which can accurately solve the Pareto optimal solution of the multi-objective optimization model for the centralized procurement chain, improving the scientific nature and efficiency of decision-making. Attached Figure Description

[0007] Figure 1 This is a flowchart illustrating a multi-objective optimization method for centralized volume-based procurement based on MODDPG according to the present invention.

[0008] Figure 2 This is a schematic diagram of a sub-process of a centralized volume-based procurement multi-objective optimization method based on MODDPG according to the present invention.

[0009] Figure 3 yes Figure 1 The diagram illustrates the interaction between the multi-objective optimization model of the centralized procurement chain constructed by the method shown and the supply enterprise, procurement alliance and third-party group buying organization.

[0010] Figure 4 yes Figure 1 The diagram shows the distribution of the first round of Pareto solutions during the verification of the multi-objective optimization model of the centralized procurement chain constructed by the method shown.

[0011] Figure 5 yes Figure 1 The diagram shows the distribution of the Pareto solution in the last round during the verification of the multi-objective optimization model of the centralized procurement chain constructed by the method shown.

[0012] Figure 6 This is a schematic diagram of the structure of a centralized volume-based procurement multi-objective optimization system based on MODDPG according to the present invention. Detailed Implementation

[0013] To enable those skilled in the art to more clearly understand the purpose, technical solution, and advantages of the present invention, the present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0014] Reference Figure 1 , Figure 1 This is a flowchart illustrating the multi-objective optimization method for centralized volume-based procurement based on MODDPG according to the present invention. In the embodiment shown in the figure, the multi-objective optimization method for centralized volume-based procurement based on MODDPG includes: S101. Construct a multi-objective optimization model for the centralized procurement chain that includes supply enterprises, procurement alliances, and third-party group buying organizations, based on deep reinforcement learning and multi-objective optimization algorithms.

[0015] In this embodiment, the comprehensive objective function of the multi-objective optimization model for the centralized procurement chain can be: The reward function can be , and These represent the profit objective function of the supplier, the procurement fund surplus objective function of the third-party group buying organization, and the cost objective function of the procurement alliance, respectively.

[0016] Understandably, in the centralized procurement chain of this invention, the supplying enterprise is responsible for the supply of products, pursuing profit maximization, and provides information such as product cost, price, and quality. It can adjust the production plan and delivery volume according to the optimization results of the centralized procurement chain multi-objective optimization model. The third-party group buying organization is responsible for organizing and supervising centralized procurement, pursuing the maximization of capital surplus, and provides information such as centralized procurement policies and reimbursement ratios. It can adjust the centralized procurement strategy and fund allocation according to the optimization results of the centralized procurement chain multi-objective optimization model. The procurement alliance is responsible for purchasing and using products, pursuing cost minimization, and provides information such as procurement needs and storage costs. It can adjust the procurement plan and storage strategy according to the optimization results of the centralized procurement chain multi-objective optimization model.

[0017] In this invention, combining multi-objective optimization and deep reinforcement learning, the expression of the multi-objective optimization model for the centralized procurement chain is as follows: (a) (b) (c) (d) (e) (f) (g) (h) (i) (g) in, Indicates the quantity of product demand. Indicates product size, Indicates the price sensitivity coefficient. This indicates the bulk purchase price of the product. This represents the service sensitivity coefficient for each region. This indicates the service level in different regions. This represents the product's quality coefficient. Indicates the level of product defects. This indicates the delivery volume for each region. This represents the distance between the location of the SME (i.e., the enterprise to be delivered) and the supplier, h represents the unit logistics cost, c represents the product raw material procurement cost, and t represents the stockout rate. Indicates product quality cost, Indicates the cost of delivery delay. This represents the unit product storage cost. This indicates the service fees for small and medium-sized enterprises. This indicates the product price before it was included in the centralized procurement. This indicates the total volume of centralized procurement in each region.

[0018] In this invention, formula (a) uses a market demand function to determine the product demand, and market demand depends on the product's selling price, quality level, and service level; that is, the demand for the product decreases as the selling price increases, and increases as the quality and service level improves. Formulas (b), (c), and (d) are the profit objective function of the supplier, the procurement fund surplus objective function of the third-party group buying organization, and the cost objective function of the procurement alliance, respectively. Formula (e) is a comprehensive objective function, operating with the goal of minimizing costs. In this embodiment, the objective functions of the supplier and the third-party group buying organization are... The numbers are adjusted with a negative sign, and a decision is made on the delivery quantity; Formula (f) is the range constraint, indicating that the service level, defect rate, and stockout rate of the procurement alliance are percentages; Formula (g) is the procurement capacity constraint, where the delivery quantity is lower than the total procurement quantity during centralized procurement; Formula (h) is the quantity constraint of procurement demand; Formula (i) is the centralized procurement price constraint, considering that the supplier's profit is used for re-production and R&D, as well as the scale effect of centralized procurement, the centralized procurement price is greater than the procurement price; Formula (g) indicates the actual range of the three objective functions, where the supplier's profit, the third-party group buying organization's cash surplus, and the procurement alliance's cost are all greater than or equal to zero.

[0019] S102. Collect the historical status and historical control operations of the supply enterprises, procurement alliances and third-party group buying organizations, and generate control operations through a policy network based on the historical status of the supply enterprises, procurement alliances and third-party group buying organizations, and randomly mutate the control operations according to the mutation probability.

[0020] In this invention, the control operation includes at least one business decision operation among product pricing strategy, procurement quantity decision, and resource allocation ratio (such as distribution volume allocation, fund allocation, etc.); the status includes multiple of the following: total procurement volume in each region, unit inventory cost, transportation distance, unit transportation cost, price sensitivity coefficient, quality sensitivity coefficient, procurement price, stockout rate, product quality level, service sensitivity coefficient, service level, and raw material cost.

[0021] During training, the multi-objective optimization model of the data collection chain based on deep reinforcement learning and multi-objective optimization algorithms is first initialized, and a set of control operations is introduced. Furthermore, the learning rate of the policy network (actor_lr), the learning rate of the value network (critic_lr), the total number of training epochs (num_eposodes), the number of hidden layers in the policy network (hidden_dim), and the depreciation factor can be preset. GAE parameters Number of iterations (epochs), random action policy parameters The parameters include: experience pool size (buffer_size), batch size (sample size), weight decay parameter (weight_decay), soft update parameter (tau), transition dictionary (transition_dict), and mutation probability. Historical status data, etc. Actions, i.e., control operations, are generated through the policy network in the multi-objective optimization model of the centralized procurement chain. This historical state As the initial state, a mutation operation is introduced, and random mutation is performed according to the mutation probability, that is, the mutation probability determines whether the action will result in a mutation.

[0022] S103. Execute the mutated / generated control operation to obtain the reward and status feedback from the environment. The historical state, the mutated control operation / generated control operation, the rewards and states fed back by the environment will be included. The transformed tuple samples are added to the experience pool.

[0023] In this step, control operations are performed, the comprehensive objective function is calculated, and rewards are provided. and new status Collect historical states, mutated / generated control operations, environmental feedback rewards, and new states to form transformation tuple samples, which are then added to the experience pool.

[0024] S104. When the amount of data in the experience pool reaches the threshold, a multi-objective optimization algorithm is used to select parent transformation tuple samples from the experience pool, and the parent transformation tuple samples are processed to generate child transformation tuple samples.

[0025] In this step, the transformation tuple samples in the experience pool are selected, and high-quality samples and random samples are selected to form the parent generation. The actions of the parent generation are cross-operated to generate the next generation of actions, and interact with the environment to generate the offspring generation.

[0026] Specifically, step S104 includes the following steps S1041-S1043: S1041. Use a multi-objective optimization algorithm to analyze the states in all transition tuple samples of the experience pool. Non-dominated sorting and crowding distance calculation are performed, and all transformed tuple samples in the experience pool are sorted in ascending order according to the crowding distance of each sample to obtain the non-dominated level.

[0027] In this embodiment, the multi-objective optimization algorithm can be the NSGA-II algorithm, which explores the optimal solution set in the search space by simulating natural selection and genetic mechanisms.

[0028] In this step, the NSGA-II algorithm is used to perform fast non-dominated sorting on the transformed tuple samples, resulting in multiple Pareto fronts. The crowding distance between samples within the same Pareto front is calculated, and the samples in the experience pool are sorted according to the crowding distance of each sample to obtain the non-dominated level. .

[0029] S1042. Select samples from the non-dominated layer according to the non-dominated level to obtain high-quality samples, and select random samples from the experience pool based on the random selection method to obtain parent transformation tuple samples composed of the high-quality samples and random samples.

[0030] In this step, the sample size for each non-dominated level is calculated. Check the number of samples in the non-dominated layer. Select non-dominated layer All samples in, if and In the non-dominated layer In this process, the crowding distance of the samples is calculated using the crowding formula. Samples with larger crowding distances are selected and added to the above samples to form a high-quality sample, until... High-quality samples and random samples form a high-quality parent transformation tuple sample.

[0031] S1043. Perform crossover and mutation operations on the control operation data in the parent generation transformed tuple sample to generate new control operations, and interact with the environment to obtain rewards and status feedback from the environment. This generates offspring transformed tuple samples.

[0032] In this step, the uniform block crossover operator and adaptive mutation operator in the NSGA-II algorithm are used to perform crossover and adaptive mutation operations on the control operation data to generate new actions, which interact with the environment to obtain rewards and states from environmental feedback. They then had offspring.

[0033] S105. Perform non-dominated sorting and crowding distance calculation on the state data in the population sample combination after merging the parent generation transformed tuple samples and the offspring generation transformed tuple samples, and select high-quality population samples from the population sample combination according to the calculated crowding distance.

[0034] In this step, parent and offspring generations are merged, and high-quality population samples are selected after non-dominated sorting. Train the neural network.

[0035] S106. Calculate the strategy loss function and value loss function based on the high-quality population samples, update the network parameters using the gradient descent algorithm, synchronize the target network parameters through a soft update mechanism, and determine the termination of training based on the Pareto convergence condition to obtain the target collection chain multi-objective optimization model.

[0036] In this step, the policy network and value evaluation network of the multi-objective optimization model of the collection and acquisition chain are trained based on high-quality population samples. The loss functions of the policy network and value evaluation network in the model are calculated, and the gradient descent algorithm is used to update the parameters of the policy network and value evaluation network. Synchronization is carried out step by step through a soft update mechanism until the Pareto convergence condition is reached.

[0037] Specifically, in this embodiment, the Pareto convergence condition is: if and only if there is no previous control operation. Make Simultaneously established, and satisfying At that time, control operation The Pareto optimal solution is achieved.

[0038] In this embodiment, the internal architecture of the multi-objective optimization model for the centralized procurement chain may include a strategy network, a first value assessment network for predicting the profit target of the supplier, a second value assessment network for predicting the cost target of the procurement alliance, and a third value assessment network for predicting the capital surplus target of the third-party organization. An action exploration module is also connected after the strategy network to randomly mutate the control operation based on the mutation probability. An environment model, an experience pool, a training control module, and an action post-processing module may also be provided. The environment model can be used to execute control operations to obtain rewards and states from environmental feedback. The experience pool can be used to store transition tuple samples consisting of the initial input state, the mutated control operation / generated control operation, and the rewards and states from environmental feedback. The training control module can be used to perform non-dominated sorting and crowding distance calculation on the transition tuple samples in the experience pool based on a multi-objective optimization algorithm, and select high-quality samples to train the strategy network, the first value assessment network, the second value assessment network, and the third value assessment network. The action post-processing module can be used to ensure that the control operations generated by the generated strategy network meet the business decision constraints (e.g., the above formulas (g) and (h)).

[0039] S107. Utilize the target procurement chain multi-objective optimization model to generate optimized control operation strategies based on the current status of supply enterprises, procurement alliances, and third-party group buying organizations.

[0040] In this embodiment, the policy network adopts a deep neural network structure, including hidden layers and fully connected layers. After the model is trained, when real-time state data is input, the real-time state is used as the input vector of the target procurement chain multi-objective optimization model. The target procurement chain multi-objective optimization model performs multi-level nonlinear feature transformation on the input vector to generate control operations. Based on business constraints, the control operations are adjusted for feasibility to generate executable decision instructions. The value evaluation network calculates the expected effect of the decision instructions on each optimization objective to find the Pareto optimal solution. Finally, the model outputs the optimal solution (action vector), which represents the recommended control operation, such as order quantity, purchase price, delivery quantity, etc. It can also output multi-objective expected values, decision credibility, etc.

[0041] Understandably, in order to verify the beneficial effects of this invention, the drugs procured in my country's ninth round of centralized volume-based procurement are used as an example to verify the multi-objective optimization model of the centralized procurement chain constructed in this invention (such as the interaction between the model and the environment). Figure 3 (As shown), with each province representing a medical institution in that region. This validation uses the winning data of argatroban injection in the ninth centralized volume-based procurement as the dataset. Based on the actual situation of my country's pharmaceutical market, the total drug demand Q reported by the region is set as the market size q of that region. , , , , Regarding the service sensitivity coefficient, since patients generally choose to seek medical treatment at medical institutions within their own province, a coefficient is set. , .

[0042] Table 1 shows the total procurement volume Q of argatroban injection in each province, the inventory holding cost H, and the transportation distance D between the supplier and the medical institution n to be delivered. The transportation cost is h = 0.001 yuan / km / piece. Based on the pharmaceutical company's gross profit margin of 45.43% in its financial report one year after the centralized procurement, the following is set... .

[0043] Table 1 Simulation experiments were conducted on a computer using the MATLAB tool, with each sample size... Maximum iteration Gmax = 1000, initial crossover probability mutation probability With actor_lr=3e-4, critic_lr=3e-3, hidden_dim=4, and tau=0.005, the above state data was used for verification. Furthermore, the process was run 200 times and the average result was taken to prevent random factors from affecting the running process. Figure 4 and Figure 5 The diagram shows the distribution of Pareto solutions, with the three axes being Z1, Z2, and Z3, respectively. Figure 4 Initial first round and Figure 5 The distribution diagram of the last round of Pareto solutions shows that... Figure 5 The red dot representing Pareto convergence is located in Figure 4 From the bottom of the initial solution, it can be seen that, compared with the initial solution, the optimal Pareto solution is optimized in all aspects of the multi-objective problem.

[0044] Reference Figure 6 , Figure 6 This is a schematic block diagram of a centralized volume-based procurement multi-objective optimization system based on MODDPG according to the present invention. In the embodiment shown in the figure, the centralized volume-based procurement multi-objective optimization system based on MODDPG includes a model building module 110, a model training module 120, and a multi-objective optimization module 130. The model building module 110 is used to construct a multi-objective optimization model for the centralized procurement chain, which includes supply enterprises, procurement alliances, and third-party group-buying organizations, based on deep reinforcement learning and multi-objective optimization algorithms; wherein, the comprehensive objective function of the centralized procurement chain multi-objective optimization model is: The reward function is , and These represent the profit objective function of the supplier, the procurement fund surplus objective function of the third-party group buying organization, and the cost objective function of the procurement alliance, respectively. The model training module 120 is used to collect historical states and control operations from supply enterprises, procurement alliances, and third-party group-buying organizations; based on the historical states of these entities, control operations are generated through the policy network of a multi-objective optimization model for the centralized procurement chain, and these control operations are randomly mutated according to mutation probabilities; the mutated control operations / generated control operations are executed to obtain rewards and states from environmental feedback. The historical state, the mutated control operation / generated control operation, the rewards and states fed back by the environment will be included. The transformed tuple samples are added to the experience pool. When the amount of data in the experience pool reaches a threshold, a multi-objective optimization algorithm is used to select parent transformed tuple samples from the experience pool and process them to generate child transformed tuple samples. The state data in the population sample combination after merging the parent and child transformed tuple samples are sorted by non-dominated order and crowding distance is calculated. Based on the calculated crowding distance, high-quality population samples are selected from the population sample combination. The strategy loss function and value loss function are calculated based on the high-quality population samples, and the network parameters are updated using the gradient descent algorithm. The target network parameters are synchronized through a soft update mechanism. The training is terminated based on the Pareto convergence condition, resulting in the target collection chain multi-objective optimization model. The multi-objective optimization module 130 is used to generate optimization control operation strategies based on the current status of the supply enterprise, the procurement alliance and the third-party group buying organization using the multi-objective optimization model of the target procurement chain.

[0045] In some embodiments, the expression of the multi-objective optimization model of the centralized procurement chain is as follows: (a) (b) (c) (d) (e) (f) (g) (h) (i) (g) in, Indicates the quantity of product demand. Indicates product size, Indicates the price sensitivity coefficient. This indicates the bulk purchase price of the product. This represents the service sensitivity coefficient for each region. This indicates the service level in different regions. This represents the product's quality coefficient. Indicates the level of product defects. This indicates the delivery volume for each region. The distance between the location of the SME and its suppliers is represented by h, the unit logistics cost is represented by c, the raw material procurement cost is represented by t, and the stockout rate is represented by t. Indicates product quality cost, Indicates the cost of delivery delay. This represents the unit product storage cost. This indicates the service fees for small and medium-sized enterprises. This indicates the product price before it was included in the centralized procurement. This indicates the total volume of centralized procurement in each region.

[0046] In some embodiments, the multi-objective optimization model of the centralized procurement chain includes: The strategy network is used to generate control actions based on the status of suppliers, purchasing alliances and third-party group buying organizations, and also uses an action exploration module to randomly mutate the control actions according to the mutation probability. The first value assessment network is used to predict the profit targets of supplying companies; A second value assessment network is used to predict the cost targets of the procurement alliance; A third-party valuation network is used to predict the funding balance targets of third-party organizations. An environment model is used to execute control operations and obtain rewards and status feedback from the environment. ; The experience pool is used to store historical states, mutated / generated control actions, rewards from environmental feedback, and states. The transformed tuple sample; The training control module includes a multi-objective optimization engine and a training update unit. The multi-objective optimization engine, when the amount of data in the experience pool reaches a threshold, uses a multi-objective optimization algorithm to select parent transformation tuple samples from the experience pool, processes these parent transformation tuple samples to generate child transformation tuple samples, and performs non-dominated sorting and crowding distance calculation on the state data in the combined population samples after merging the parent and child transformation tuple samples. Based on the calculated crowding distance, it selects high-quality population samples from the combined population samples. The training update unit calculates the policy loss function and value loss function based on the high-quality population samples, updates the network parameters using a gradient descent algorithm, synchronizes the policy network parameters and value evaluation network parameters through a soft update mechanism, and terminates training when the Pareto convergence condition is met. The post-action processing module ensures that the generated control operations comply with business decision constraints.

[0047] In some embodiments, the control operation includes at least one business decision operation among product pricing strategy, procurement quantity decision, and resource allocation ratio; the status includes multiple parameters among total procurement volume in each region, unit inventory cost, transportation distance, unit transportation cost, price sensitivity coefficient, quality sensitivity coefficient, procurement price, stockout rate, product quality level, service sensitivity coefficient, service level, and raw material cost.

[0048] In some embodiments, the model training module 120 is further configured to: The NSGA-II algorithm is used to analyze the states in all transition tuple samples in the experience pool. Perform non-dominated sorting and crowding distance calculation, and sort all transformed tuple samples in the experience pool in ascending order according to the crowding distance of each sample to obtain the non-dominated level; Based on the non-dominated level, samples of the non-dominated layer are selected to obtain high-quality samples, and random samples are selected from the experience pool based on the random selection method, thereby obtaining a parent transformation tuple sample composed of the high-quality samples and random samples. The control operation data in the parent transformation tuple samples are subjected to crossover and adaptive mutation operations to generate new control operations, which then interact with the environment to obtain rewards and states from environmental feedback. This generates offspring transformed tuple samples.

[0049] Understandably, the specific limitations regarding the MODDPG-based centralized volume-based procurement multi-objective optimization system can be found in the limitations of the MODDPG-based centralized volume-based procurement multi-objective optimization method described above, and will not be repeated here. Each module in the aforementioned MODDPG-based centralized volume-based procurement multi-objective optimization system can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the computer device's memory as software, so that the processor can call and execute the corresponding operations of each module.

[0050] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention in any way. Those skilled in the art can make various equivalent changes and improvements based on the above embodiments, and all equivalent variations or modifications made within the scope of the claims should fall within the protection scope of the present invention.

Claims

1. A multi-objective optimization method for centralized volume-based procurement based on MODDPG, characterized in that, include: A multi-objective optimization model for the centralized procurement chain, based on deep reinforcement learning and multi-objective optimization algorithms, is constructed, incorporating supply enterprises, procurement alliances, and third-party group-buying organizations. The comprehensive objective function of the multi-objective optimization model for the centralized procurement chain is: The reward function is , , and These represent the profit objective function of the supplier, the procurement fund surplus objective function of the third-party group buying organization, and the cost objective function of the procurement alliance, respectively. Based on the aforementioned multi-objective optimization model for the centralized procurement chain, the target multi-objective optimization model for the centralized procurement chain is obtained by training it with the historical states and control operations of supply enterprises, procurement alliances, and third-party group-buying organizations. Specifically, it includes: Collect historical status and control operations of suppliers, purchasing alliances, and third-party group buying organizations; Based on the historical status of the supply enterprises, procurement alliances and third-party group buying organizations, control operations are generated through a policy network, and the control operations are randomly mutated according to the mutation probability. Execute the mutated / generated control operations and obtain rewards and status feedback from the environment. The historical state, the mutated control operation / generated control operation, the rewards and states fed back by the environment will be included. The transformed tuple samples are added to the experience pool; When the amount of data in the experience pool reaches the threshold, a multi-objective optimization algorithm is used to select parent transformation tuple samples from the experience pool, and the parent transformation tuple samples are processed to generate child transformation tuple samples. Non-dominated sorting and crowding distance calculation are performed on the state data in the population sample combination after merging the parent generation transformed tuple samples and the offspring generation transformed tuple samples. Based on the calculated crowding distance, high-quality population samples are selected from the population sample combination. Based on the high-quality population samples, the strategy loss function and value loss function are calculated, and the network parameters are updated using the gradient descent algorithm. The target network parameters are synchronized through a soft update mechanism, and the training is terminated according to the Pareto convergence condition to obtain the target collection chain multi-objective optimization model. The target-oriented centralized procurement chain multi-objective optimization model is used to generate optimized control operation strategies based on the current status of supply enterprises, procurement alliances and third-party group buying organizations.

2. The multi-objective optimization method for centralized volume-based procurement based on MODDPG as described in claim 1, characterized in that, The expression for the multi-objective optimization model of the centralized procurement chain is as follows: in, Indicates the quantity of product demand. Indicates product size, This represents the price sensitivity coefficient. This indicates the bulk purchase price of the product. This represents the service sensitivity coefficient for each region. This indicates the service level in different regions. This represents the product's quality coefficient. Indicates the level of product defects. This indicates the delivery volume for each region. The distance between the location of the SME and its suppliers is represented by h, the unit logistics cost is represented by c, the raw material procurement cost is represented by t, and the stockout rate is represented by t. Indicates product quality cost, Indicates the cost of delivery delay. This represents the unit product storage cost. This indicates the service fees for small and medium-sized enterprises. This indicates the product price before it was included in the centralized procurement. This indicates the total volume of centralized procurement in each region.

3. The multi-objective optimization method for centralized volume-based procurement based on MODDPG as described in claim 1, characterized in that, The control operations include at least one of the following business decision operations: product pricing strategy, procurement quantity decision, and resource allocation ratio; the status includes multiple of the following: total procurement volume in each region, unit inventory cost, transportation distance, unit transportation cost, price sensitivity coefficient, quality sensitivity coefficient, procurement price, stockout rate, product quality level, service sensitivity coefficient, service level, and raw material cost.

4. The multi-objective optimization method for centralized volume-based procurement based on MODDPG as described in claim 1, characterized in that, The step of selecting parent transformation tuple samples from the experience pool using a multi-objective optimization algorithm and processing the parent transformation tuple samples to generate child transformation tuple samples specifically includes: The state in all transformed tuple samples of the experience pool is evaluated using a multi-objective optimization algorithm. Perform non-dominated sorting and crowding distance calculation, and sort all transformed tuple samples in the experience pool in ascending order according to the crowding distance of each sample to obtain the non-dominated level; Based on the non-dominated level, samples of the non-dominated layer are selected to obtain high-quality samples, and random samples are selected from the experience pool based on the random selection method, thereby obtaining a parent transformation tuple sample composed of the high-quality samples and random samples. The control operation data in the parent transformation tuple sample is crossover and mutation operations are performed to generate new control operations, which are then interacted with the environment to obtain rewards and states from the environment's feedback. This generates offspring transformed tuple samples.

5. The multi-objective optimization method for centralized volume-based procurement based on MODDPG as described in claim 4, characterized in that, The multi-objective optimization algorithm is the NSGA-II algorithm, wherein the step of performing crossover and mutation operations on the control operation data in the parent transformed tuple samples to generate new control operations specifically includes: The NSGA-II algorithm is used to perform crossover and adaptive mutation operations on the control operation data in the parent transformation tuple samples to generate new control operations.

6. The multi-objective optimization method for centralized volume-based procurement based on MODDPG as described in claim 1, characterized in that, The Pareto convergence condition is: if and only if there is no previous control operation. Make Simultaneously established, and satisfying At that time, control operation The Pareto optimal solution is achieved.

7. A centralized volume-based procurement multi-objective optimization system based on MODDPG as described in claim 1, characterized in that, The MODDPG-based centralized volume-based procurement multi-objective optimization system includes: The model building module is used to construct a multi-objective optimization model for the centralized procurement chain, based on deep reinforcement learning and multi-objective optimization algorithms, which includes supply enterprises, procurement alliances, and third-party group-buying organizations. The comprehensive objective function of the centralized procurement chain multi-objective optimization model is: The reward function is , and These represent the profit objective function of the supplier, the procurement fund surplus objective function of the third-party group buying organization, and the cost objective function of the procurement alliance, respectively. The model training module is used to collect historical states and control operations from supply enterprises, procurement alliances, and third-party group-buying organizations. Based on the historical states of these entities, control operations are generated through the policy network of a multi-objective optimization model for the centralized procurement chain, and these operations are randomly mutated according to mutation probabilities. The mutated control operations / generated control operations are executed to obtain rewards and states from environmental feedback. The historical state, the mutated control operation / generated control operation, the rewards and states fed back by the environment will be included. The transformed tuple samples are added to the experience pool. When the amount of data in the experience pool reaches a threshold, a multi-objective optimization algorithm is used to select parent transformed tuple samples from the experience pool and process them to generate child transformed tuple samples. The state data in the population sample combination after merging the parent and child transformed tuple samples are sorted by non-dominated order and crowding distance is calculated. Based on the calculated crowding distance, high-quality population samples are selected from the population sample combination. The strategy loss function and value loss function are calculated based on the high-quality population samples, and the network parameters are updated using the gradient descent algorithm. The target network parameters are synchronized through a soft update mechanism. The training is terminated based on the Pareto convergence condition, resulting in the target collection chain multi-objective optimization model. The multi-objective optimization module is used to generate optimized control operation strategies based on the current status of supply enterprises, procurement alliances and third-party group buying organizations using the multi-objective optimization model of the target procurement chain.

8. The centralized volume-based procurement multi-objective optimization system based on MODDPG as described in claim 7, characterized in that, The expression for the multi-objective optimization model of the centralized procurement chain is as follows: in, Indicates the quantity of product demand. Indicates product size, This represents the price sensitivity coefficient. This indicates the bulk purchase price of the product. This represents the service sensitivity coefficient for each region. This indicates the service level in different regions. This represents the product's quality coefficient. Indicates the level of product defects. This indicates the delivery volume for each region. The distance between the location of the SME and its suppliers is represented by h, the unit logistics cost is represented by c, the raw material procurement cost is represented by t, and the stockout rate is represented by t. Indicates product quality cost, Indicates the cost of delivery delay. This represents the unit product storage cost. This indicates the service fees for small and medium-sized enterprises. This indicates the product price before it was included in the centralized procurement. This indicates the total volume of centralized procurement in each region.

9. The centralized volume-based procurement multi-objective optimization system based on MODDPG as described in claim 7 or 8, characterized in that, The multi-objective optimization model for the centralized procurement chain includes: The strategy network is used to generate control actions based on the status of suppliers, purchasing alliances and third-party group buying organizations, and also uses an action exploration module to randomly mutate the control actions according to the mutation probability. The first value assessment network is used to predict the profit targets of supplying companies; A second value assessment network is used to predict the cost targets of the procurement alliance; A third-party valuation network is used to predict the funding balance targets of third-party organizations. An environment model is used to execute control operations and obtain rewards and status feedback from the environment. ; The experience pool is used to store historical states, mutated / generated control actions, rewards from environmental feedback, and states. The transformed tuple sample; The training control module includes a multi-objective optimization engine and a training update unit. The multi-objective optimization engine, when the amount of data in the experience pool reaches a threshold, uses a multi-objective optimization algorithm to select parent transformation tuple samples from the experience pool, processes these parent transformation tuple samples to generate child transformation tuple samples, and performs non-dominated sorting and crowding distance calculation on the state data in the combined population samples after merging the parent and child transformation tuple samples. Based on the calculated crowding distance, it selects high-quality population samples from the combined population samples. The training update unit calculates the policy loss function and value loss function based on the high-quality population samples, updates the network parameters using a gradient descent algorithm, synchronizes the policy network parameters and value evaluation network parameters through a soft update mechanism, and terminates training when the Pareto convergence condition is met. The post-action processing module ensures that the generated control operations comply with business decision constraints.

10. The centralized volume-based procurement multi-objective optimization system based on MODDPG as described in claim 7, characterized in that, The control operations include at least one of the following business decision operations: product pricing strategy, procurement quantity decision, and resource allocation ratio; the status includes multiple of the following: total procurement volume in each region, unit inventory cost, transportation distance, unit transportation cost, price sensitivity coefficient, quality sensitivity coefficient, procurement price, stockout rate, product quality level, service sensitivity coefficient, service level, and raw material cost.

Images