A battery inventory management optimization method for the shared electric bicycle market

Abstract

The present invention belongs to a dual deep reinforcement learning method, and discloses a battery inventory management optimization method for a shared electric bicycle market, including: constructing a multi-level spare parts inventory model with shared electric bicycles as the research object, the multi-level spare parts inventory model is a shared electric bicycle system with a three-level supply chain, from top to bottom, the first-level, second-level and third-level supply chains represent a central warehouse, several secondary warehouses and several bases, respectively; designing a shared electric bicycle operation simulation platform under the multi-level spare parts inventory model based on a dual deep reinforcement learning algorithm, the simulation platform simulates the actual operation of the shared electric bicycle, and provides corresponding rewards and subsequent states according to its operation behavior and inventory status, and also integrates a dual deep reinforcement learning algorithm to optimize battery inventory management; the present invention is used to solve the multi-dimensional inventory problem of batteries in a shared electric bicycle system and improve the operation efficiency of the shared electric bicycle system.

Description

Technical Field

[0001] The present invention belongs to a dual deep reinforcement learning method, and specifically relates to a battery inventory management optimization method for a shared electric bicycle market. Background Art

[0002] The normal use of shared electric bicycles depends largely on the battery supply, and the reasonable allocation of batteries is an important factor for the normal operation of the system. In order to improve the operating efficiency of shared electric bicycles in the market, it is necessary to have a deep understanding of the battery inventory problem and establish a reliable mathematical optimization model to provide managers with an effective decision-making basis. Research in recent years has mainly focused on solving the problem of supply and demand imbalance. So far, most studies have assumed that the inventory level is known. Since it is a challenging task to formulate an optimal inventory strategy, the existing research on shared electric bicycles is very limited. For example, only the interval availability of a two-level spare parts inventory system has been studied; the two-echelon spare parts inventory path problem based on lateral transfer with random demand has been studied. Since then, the multi-level inventory problem has received widespread attention. For example, researchers have developed a model to effectively study the batch problem of multi-level spare parts inventory. The above studies mainly focus on the inventory problem of simple systems, while the spare parts inventory problem of multi-level systems is still an under-researched area.

[0003] In the operation of shared electric bicycle systems, inventory management and replacement policies play a vital role. Previous studies have only focused on the supply and demand relationship of shared electric bicycle systems. Most studies often do not consider the failure of shared electric bicycles, but shared electric bicycle failures do occur from time to time. Therefore, not considering maintenance strategies may result in considerable cost losses.

[0004] That is to say, the normal operation of shared electric bicycles depends to a large extent on the rationality of battery supply, and the rational allocation of batteries is the key to the normal operation of the system. At present, the battery inventory problem is limited to a single-level network structure, which has deficiencies in accuracy and efficiency. These methods are based on intelligent optimization algorithms to solve such problems. These methods have the following limitations: First, lack of comprehensive consideration: Traditional methods mostly consider the inventory problem of a single-level network, and a single-level network considers fewer random factors and more deterministic factors, which is not suitable for complex and changeable real environments. Second, the original invention considered a relatively single problem, and did not jointly optimize the replacement strategy, charging strategy and inventory strategy of shared electric bicycle batteries. Third, most of the existing technologies solve the inventory model of small spaces, but fail to solve the inventory model problem of large spaces. Summary of the invention

[0005] In view of the shortcomings of the prior art, the present invention provides a battery inventory management optimization method for the shared electric bicycle market, which is used to solve the multi-dimensional inventory problem of batteries in the shared electric bicycle system, especially for the optimization of complex multi-dimensional state space, to improve the operating efficiency of the shared electric bicycle system. Specifically, the following steps are included: first, the three-level inventory problem of shared electric bicycle batteries is analyzed, and a multi-level spare parts inventory model considering the central warehouse, secondary warehouse and base of electric bicycles is established; secondly, a dual deep reinforcement learning method is used to minimize the system cost and improve the inventory management efficiency.

[0006] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0007] A battery inventory management optimization method for a shared electric bicycle market, comprising the following steps:

[0008] Step 1: Construct a multi-level spare parts inventory model with shared electric bicycles as the research object;

[0009] The multi-level spare parts inventory model is a shared electric bicycle system with a three-level supply chain. From top to bottom, the first-level, second-level and third-level supply chains represent a central warehouse, several secondary warehouses and several bases respectively; the central warehouse is used to store spare parts and distribute them to each secondary warehouse after receiving a replenishment notice; the secondary warehouse replaces the faulty battery with a new one according to the inventory policy, and triggers a replenishment notice when the inventory level drops below s; each base is a parallel system consisting of multiple shared electric bicycles;

[0010] The multi-level spare parts inventory model uses the number of charging piles and the order point as decision variables, and the objective function is the cost of the shared electric bicycle system;

[0011] Step 2: Train a multi-level spare parts inventory model based on a dual deep reinforcement learning algorithm to minimize the cost of the shared electric bicycle system;

[0012] A shared electric bicycle operation simulation platform based on a multi-level spare parts inventory model with a dual deep reinforcement learning algorithm is designed. The simulation platform simulates a three-level supply chain system consisting of a primary warehouse, a secondary warehouse, and a base to simulate the actual operation of shared electric bicycles, and provides corresponding rewards and follow-up status based on their operating behavior and inventory status. It also integrates a dual deep reinforcement learning algorithm to optimize battery inventory management.

[0013] Furthermore, the workflow of the shared electric bicycle system of the three-level supply chain is as follows:

[0014] The shared electric bicycles operate normally. Whenever a battery runs out of power, it is transported to the base for charging. After charging, the battery is replaced to keep the shared electric bicycles operating normally. When the battery of a shared electric bicycle fails and cannot be used, a replacement notice will be sent when a certain number is reached, and the secondary warehouse will replenish the stock. When the number of spare parts in the secondary warehouse is lower than s, a replenishment notice will be sent to the central warehouse, and the central warehouse will replenish the stock.

[0015] Furthermore, at each base, the failure process of shared electric bicycles is considered as a function with parameter λ 2 Poisson process, due to the internal degradation of the battery, the life of any electric bicycle is uncertain, it is a random variable T U , and is a parameter λ 1 exponential distribution; when the number of failed batteries reaches l, new batteries will be shipped from the secondary warehouse to the base, where the time from the secondary warehouse to the base plus the replacement time is a variable with parameter α;

[0016] In the primary warehouse and the secondary warehouse, the inventory strategy (s, S) is adopted, where s and S represent the order point from the secondary warehouse and the maximum inventory of the secondary warehouse respectively. When the number of spare parts in the secondary warehouse is less than s 2 When , the ordering behavior is triggered, after which some spare parts will be transported from the central warehouse to the secondary warehouse, and the time is exponentially distributed with the parameter η;

[0017] Among them, λ 1 is the time when the battery is exhausted, λ 2 is the time when the battery fails, α is the time to replace the battery from the secondary warehouse to the base, and η is the transportation time from the primary warehouse to the secondary warehouse.

[0018] Furthermore, the decision variables of the multi-level spare parts inventory model are as follows:

[0019] Number of charging piles m: represents the maximum number of shared electric bicycles that can be charged at each base;

[0020] Order point s: indicates the minimum inventory quantity of the secondary warehouse;

[0021] The objective function is as follows:

[0022] The goal is to minimize the cost of the system and establish the following objective function:

[0023] (1);

[0024] Among them, k 1 represents the number of shared electric bicycles running in the system, k 2 Indicates the number of dead batteries in the system, s 1is the number of spare parts, i.e. batteries, in the secondary warehouse, m is the number of charging piles in the base, n is the initial number of shared electric vehicles in the system, s is the order point of the secondary warehouse, S is the maximum inventory of the secondary warehouse, represents the system cost, The system status is The probability of Indicates that the system status is The expected cost when The system status is The system expected stay time is is the probability of system failure, The system status is The probability of represents the constraints of this formula, represents the state transition probability of the system.

[0025] Furthermore, By solving the following formula:

[0026] (2);

[0027] cost It is expressed as:

[0028] (3);

[0029] in is the action space of the system, For charging action, For ordering behavior, To replace the behavior, The construction cost of a charging station is is the battery charging cost, represents the inventory cost per unit, is the ordering cost, is the replacement cost, represents the failure cost of the system, is the current state of the system, Indicates that the system status is The probability of Indicates the system status by Transfer to The probability of represents the state space of the system, , and is an indicative function, which is specifically expressed as follows:

[0030] (4);

[0031] (5);

[0032] (6).

[0033] Furthermore, the process of optimizing battery inventory management using dual deep reinforcement learning algorithm is as follows:

[0034] First, in the dual deep reinforcement learning algorithm, two neural networks are defined: one is the estimation network and the other is the target network. The estimation network is used to select the best "action" according to the current "environment" state and estimate the Q value after executing the selected action. The Q value refers to the discounted sum of the expected rewards; the target network is used to estimate the target Q value. The structure of the target network is the same as that of the estimation network, but the parameter update frequency is low. The target network is updated only after the estimation network has been updated many times.

[0035] Among them, "environment" refers to the shared electric bicycle system, and the "environment" status includes the real-time operation data and inventory status of shared electric bicycles. The inventory status includes the battery inventory and battery failure rate of warehouses at all levels; "action" includes battery charging and replenishment;

[0036] By comparing the estimated Q value output by the estimation network with the target Q value output by the target network, the loss function is calculated, and the gradient descent algorithm is used to update the parameters of the estimation network;

[0037] Then, the experience replay pool D is introduced to store the experience samples generated when the shared electric bicycles interact with the simulation platform. The experience samples include the "environment" state, action, reward and next state;

[0038] During the training process, small batches of experience samples are randomly sampled from the experience replay pool D to update the parameters of the estimation network.

[0039] Furthermore, in step 2, during the training phase, the environment will input the current state ,in accordance with Greedy strategy to choose actions , the environment performs actions Enter the next state , and generate instant rewards , forming a sequence sample The samples are stored in the experience replay pool D. After a certain number of rounds of calculation, enough samples are generated. Then, small batches of random sampling are performed to train the estimation network. The gradient descent method is used to calculate the difference and update the parameters. The above steps are iterated continuously until the training reaches the end stage.

[0040] Furthermore, the specific detailed process of the training phase is as follows:

[0041] First, the algorithm initializes an experience replay pool D to store the experience data generated when the shared electric bicycle interacts with the shared electric bicycle simulation operation platform; at the beginning of each training cycle, the algorithm resets the environment state and sets the total cost and the end flag; then, the algorithm enters a loop in which the shared electric bicycle decides whether to randomly select an action or use the learned strategy to select an action based on the current exploration rate. After selecting an action, the shared electric bicycle sends the action to the shared electric bicycle operation simulation platform model, executes the action and receives the reward, new state and a flag indicating whether the terminal state has been reached returned by the shared electric bicycle operation simulation platform;

[0042] Next, the shared electric bicycle stores the experience data such as the current state, action, reward, new state and end flag in the experience replay pool D; then, the algorithm randomly extracts a certain number of samples from the experience replay pool D and uses these samples to update the estimation network; this process involves calculating the predicted value, target value and loss function, and using gradient descent to optimize the estimated network parameters; after each update, the algorithm also periodically uses the parameters of the estimated network to update the parameters of the target network.

[0043] Compared with the prior art, the present invention has the following advantages:

[0044] (1) The present invention establishes a three-level inventory optimization model for a shared electric bicycle system, including a central warehouse, a secondary warehouse, and a base. The optimization model makes a trade-off between the construction cost, ordering cost, and replacement cost of charging piles and the failure probability of the entire electric bicycle system.

[0045] (2) The present invention uses a dual deep reinforcement learning method to solve the complex multi-dimensional inventory problem of the shared electric bicycle system.

[0046] (3) The present invention jointly optimizes the replacement strategy, charging strategy and inventory strategy of shared electric bicycle batteries. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0048] Figure 1 This is a structural diagram of the shared electric bicycle system of the three-level supply chain of the present invention;

[0049] Figure 2 Flowchart of the dual depth enhancement algorithm of the present invention. DETAILED DESCRIPTION

[0050] The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments.

[0051] like Figure 1-Figure 2 As shown, this embodiment provides a battery inventory management optimization method for a shared electric bicycle market, comprising the following steps:

[0052] Step 1: Construct a multi-level spare parts inventory model with shared electric bicycles as the research object;

[0053] The multi-level spare parts inventory model is a shared electric bicycle system with a three-level supply chain. From top to bottom, the first-level, second-level and third-level supply chains represent a central warehouse, several secondary warehouses and several bases, respectively; the central warehouse is used to store spare parts and distribute them to each secondary warehouse after receiving a replenishment notice; the secondary warehouse replaces faulty batteries with new batteries based on the inventory policy, and triggers a replenishment notice when the inventory level drops below s; each base is a parallel system composed of multiple shared electric bicycles. The base only charges depleted batteries. The base is considered to be operating normally if and only if at least one shared electric bicycle is working.

[0054] The multi-level spare parts inventory model takes the number of charging piles and the order point as decision variables, and the objective function is the cost of the shared electric bicycle system.

[0055] The workflow of the shared electric bicycle system of the three-level supply chain is as follows:

[0056] The shared electric bicycles operate normally. Whenever a battery runs out of power, it is transported to the base for charging. After charging, the battery is replaced to keep the shared electric bicycles operating normally. When the battery of a shared electric bicycle fails and cannot be used, a replacement notice will be sent when a certain number is reached, and the secondary warehouse will replenish the stock. When the number of spare parts in the secondary warehouse is lower than s, a replenishment notice will be sent to the central warehouse, and the central warehouse will replenish the stock.

[0057] At each base, the failure process of shared electric bicycles is considered as a function with parameter λ 2 Poisson process, due to the internal degradation of the battery, the life of any electric bicycle is uncertain, it is a random variable T U , and is a parameter λ 1 exponential distribution; when the number of failed batteries reaches l, new batteries will be shipped from the secondary warehouse to the base, where the time from the secondary warehouse to the base plus the replacement time is a variable with parameter α;

[0058] In the primary warehouse and the secondary warehouse, the inventory strategy (s, S) is adopted, where s and S represent the order point from the secondary warehouse and the maximum inventory of the secondary warehouse respectively. When the number of spare parts in the secondary warehouse is less than s 2 When , the ordering behavior is triggered, after which some spare parts will be transported from the central warehouse to the secondary warehouse, and the time is exponentially distributed with the parameter η.

[0059] Setting of multi-level spare parts inventory model:

[0060] (1) The batteries in the shared electric bicycle system are identical, and each battery is in a like-new condition at the beginning;

[0061] (2) All batteries at all bases and warehouses are in cold reserve. In other words, batteries that are not in operation will not fail;

[0062] (3) Charging time of a dead battery (T C ) is exponentially distributed β.

[0063] The decision variables of the multi-echelon spare parts inventory model are as follows:

[0064] Number of charging piles m: represents the maximum number of shared electric bicycles that can be charged at each base;

[0065] Order point s: indicates the minimum inventory quantity of the secondary warehouse;

[0066] The objective function is as follows:

[0067] The goal is to minimize the cost of the system and establish the following objective function:

[0068] (1);

[0069] Among them, k 1 represents the number of shared electric bicycles running in the system, k 2 Indicates the number of dead batteries in the system, s 1 is the number of spare parts, i.e. batteries, in the secondary warehouse, m is the number of charging piles in the base, n is the initial number of shared electric vehicles in the system, s is the order point of the secondary warehouse, S is the maximum inventory of the secondary warehouse, represents the system cost, The system status is The probability of Indicates that the system status is The expected cost when The system status is The system expected stay time is is the probability of system failure, The system status is The probability of represents the constraints of this formula, represents the state transition probability of the system.

[0070] As a preferred embodiment, By solving the following formula:

[0071] (2);

[0072] cost It is expressed as:

[0073] (3);

[0074] in is the action space of the system, For charging action, For ordering behavior, To replace the behavior, The construction cost of a charging station is is the battery charging cost, represents the inventory cost per unit, is the ordering cost, is the replacement cost, represents the failure cost of the system, is the current state of the system, Indicates that the system status is The probability of Indicates the system status by Transfer to The probability of represents the state space of the system, , and is an indicative function, which is specifically expressed as follows:

[0075] (4);

[0076] (5);

[0077] (6).

[0078] Step 2: Train a multi-level spare parts inventory model based on a dual deep reinforcement learning algorithm to minimize the cost of the shared electric bicycle system;

[0079] A shared electric bicycle operation simulation platform based on a multi-level spare parts inventory model with a dual deep reinforcement learning algorithm is designed. The simulation platform simulates a three-level supply chain system consisting of a primary warehouse, a secondary warehouse, and a base to simulate the actual operation of shared electric bicycles, and provides corresponding rewards and follow-up status based on their operating behavior and inventory status. It also integrates a dual deep reinforcement learning algorithm to optimize battery inventory management.

[0080] The process of optimizing battery inventory management using dual deep reinforcement learning algorithm is as follows:

[0081] First, in the dual deep reinforcement learning algorithm, two neural networks are defined: one is the estimation network and the other is the target network. The estimation network is used to select the best "action" according to the current "environment" state and estimate the Q value after executing the selected action. The Q value refers to the discounted sum of the expected rewards; the target network is used to estimate the target Q value. The structure of the target network is the same as that of the estimation network, but the parameter update frequency is low. The target network is updated only after the estimation network has been updated many times.

[0082] Among them, "environment" refers to the shared electric bicycle system, and the "environment" status includes the real-time operation data and inventory status of shared electric bicycles. The inventory status includes the battery inventory and battery failure rate of warehouses at all levels; "action" includes battery charging and replenishment;

[0083] By comparing the estimated Q value output by the estimation network with the target Q value output by the target network, the loss function is calculated, and the gradient descent algorithm is used to update the parameters of the estimation network;

[0084] Then, the experience replay pool D is introduced to store the experience samples generated when the shared electric bicycles interact with the simulation platform. The experience samples include the "environment" state, action, reward and next state;

[0085] During the training process, small batches of experience samples are randomly sampled from the experience replay pool D to update the parameters of the estimation network.

[0086] In step 2, during the training phase, the environment will input the current state ,in accordance with Greedy strategy to choose actions , the environment performs actions Enter the next state , and generate instant rewards , forming a sequence sample The samples are stored in the experience replay pool D. After a certain number of rounds of calculation, enough samples are generated. Then, small batches of random sampling are performed to train the estimation network. The gradient descent method is used to calculate the difference and update the parameters. The above steps are iterated continuously until the training reaches the end stage.

[0087] As a preferred implementation, the specific detailed process of the training phase is as follows:

[0088] First, the algorithm initializes an experience replay pool D to store the experience data generated when the shared electric bicycles interact with the shared electric bicycle simulation operation platform; at the beginning of each training cycle, the algorithm resets the environment state and sets the total cost and end flag ; The algorithm then enters a loop in which the shared electric bicycle decides whether to randomly select an action or select an action using the learned strategy based on the current exploration rate. After selecting an action, the shared electric bicycle sends the action to the shared electric bicycle operation simulation platform model, executes the action and receives the reward, new state, and a flag indicating whether the terminal state has been reached returned by the shared electric bicycle operation simulation platform. Next, the shared electric bicycle stores these experience data (including current state, action, reward, new state, and end flag) in the experience replay pool. After that, the algorithm randomly draws a certain number of samples from the experience replay pool D and uses these samples to update its deep neural network (usually an estimation network). This process involves calculating the predicted value, target value, and loss function, and using gradient descent to optimize the network parameters. After each update, the algorithm also periodically updates the parameters of the target network with the parameters of the estimated network to improve the stability and accuracy of learning. The entire training process continues until the preset number of training cycles is reached or other termination conditions are met. Through this process, the dual deep reinforcement learning algorithm enables shared electric bicycles to gradually learn the ability to make optimal decisions in complex environments.

[0089] The detailed process of the training phase is shown in Table 1.

[0090] Table 1. Dual deep reinforcement learning algorithms

[0091]

[0092] in Represents the time for each iteration of the system; The time when the battery runs out of power; The time when the battery fails; Time required to replace batteries from secondary warehouse to base; Battery charging time; It is the transportation time from the primary warehouse to the secondary warehouse; The maximum capacity of the experience replay pool; is the sampling sample size; is the interval step size for replacing the target parameter; is the parameter of the greedy function; is the discount factor; is the reward value of the system at time t; is the future reward value of the system; is the state space of the system at time t.

[0093] Of course, the above description is not a limitation of the present invention, and the present invention is not limited to the above examples. Any changes, modifications, additions or substitutions made by ordinary technicians in this technical field within the essential scope of the present invention should fall within the protection scope of the present invention.

Claims

1. A battery inventory management optimization method for the shared electric bicycle market, characterized in that: The following steps are involved: Step 1: Construct a multi-level spare parts inventory model with shared electric bicycles as the research object; The multi-level spare parts inventory model is a shared electric bicycle system with a three-level supply chain. From top to bottom, the first-level, second-level and third-level supply chains represent a central warehouse, several secondary warehouses and several bases respectively; the central warehouse is used to store spare parts and distribute them to each secondary warehouse after receiving a replenishment notice; the secondary warehouse replaces the faulty batteries with new batteries according to the inventory policy, and triggers a replenishment notice when the inventory level drops below s; each base is a parallel system composed of multiple shared electric bicycles. The base only charges the exhausted batteries. The base is considered to be operating normally if and only if at least one shared electric bicycle is working; The multi-level spare parts inventory model uses the number of charging piles and the order point as decision variables, and the objective function is the cost of the shared electric bicycle system; The decision variables of the multi-echelon spare parts inventory model are as follows: Number of charging piles m: represents the maximum number of shared electric bicycles that can be charged at each base; Order point s: indicates the minimum inventory quantity of the secondary warehouse; The objective function is as follows: The goal is to minimize the cost of the system and establish the following objective function: Among them, k1 represents the number of shared electric bicycles in operation in the system, k2 represents the number of batteries without power in the system, s1 represents the number of spare parts in the secondary warehouse, i.e., the number of batteries, m represents the number of charging piles in the base, n represents the initial number of shared electric vehicles in the system, s represents the order point of the secondary warehouse, S represents the maximum inventory of the secondary warehouse, and C(m, s) represents the system cost. is the probability that the system state is (k1, k2, s1), represents the expected cost when the system state is (k1, k2, s1), is the expected stay time of the system when the system state is (k1, k2, s1), P r is the probability of system failure, is the probability that the system state is at (0, k2, s1), st represents the constraint condition of this formula, and p represents the state transition probability of the system; By solving the following formula: cost It is expressed as: Among them, (a1, a2, a3) is the action space of the system, a1 is the charging action, a2 is the ordering action, a3 is the replacement action, C p is the construction cost of a charging pile, C h is the battery charging cost, C s represents the inventory cost per unit, C o is the ordering cost, C r is the replacement cost, C f represents the failure cost of the system, (k′1, k′2, s′1) is the current state of the system, represents the probability that the system state is (k′1, k′2, s′1), represents the probability that the system state transfers from (k1, k2, s1) to (k′1, k′2, s′1), Ω represents the state space of the system, and is an indicative function, which is specifically expressed as follows: Step 2: Train a multi-level spare parts inventory model based on a dual deep reinforcement learning algorithm to minimize the cost of the shared electric bicycle system; Design a shared electric bicycle operation simulation platform based on a multi-level spare parts inventory model with a dual deep reinforcement learning algorithm. The simulation platform simulates a three-level supply chain system consisting of a primary warehouse, a secondary warehouse, and a base to simulate the actual operation of shared electric bicycles, and provides corresponding rewards and subsequent status based on their operating behavior and inventory status. It also integrates a dual deep reinforcement learning algorithm to optimize battery inventory management. The process of optimizing battery inventory management using dual deep reinforcement learning algorithm is as follows: First, in the dual deep reinforcement learning algorithm, two neural networks are defined: one is the estimation network and the other is the target network. The estimation network is used to select the best "action" according to the current "environment" state and estimate the Q value after executing the selected action. The Q value refers to the discounted sum of the expected rewards; the target network is used to estimate the target Q value. The structure of the target network is the same as that of the estimation network, but the parameter update frequency is low. The target network is updated only after the estimation network has been updated many times. Among them, "environment" refers to the shared electric bicycle system, and the "environment" status includes the real-time operation data and inventory status of shared electric bicycles. The inventory status includes the battery inventory and battery failure rate of warehouses at all levels; "action" includes battery charging and replenishment; By comparing the estimated Q value output by the estimation network with the target Q value output by the target network, the loss function is calculated, and the gradient descent algorithm is used to update the parameters of the estimation network; Then, introduce the experience replay pool Utilizing the Experience Replay Pool To store the experience samples generated when the shared electric bicycles interact with the simulation platform, which include the "environment" state, action, reward and next state; During training, the experience replay pool A small batch of experience samples are randomly sampled in the training phase to update the parameters of the estimated network; during the training phase, the environment inputs the current state z t , select action a according to the ε greedy strategy t , the environment performs action a t Enter the next state z′ t , and generates an immediate reward R t , forming a sequence sample (z t , (a1, a2, a3)(t), R t , z′ t ) is stored in the experience replay pool In the process, enough samples are generated after a certain number of rounds of calculations, and then small batches of random sampling are performed to train the estimation network; the difference is calculated using the gradient descent method, the parameters are updated, and the above steps are continuously iterated until the training reaches the end stage; The specific details of the training phase are as follows: First, the algorithm initializes an experience replay pool Used to store the experience data generated when the shared electric bicycle interacts with the shared electric bicycle simulation operation platform; at the beginning of each training cycle, the algorithm resets the environment state and sets the total cost and the end flag; then, the algorithm enters a loop in which the shared electric bicycle decides whether to randomly select an action or use the learned strategy to select an action based on the current exploration rate. After selecting an action, the shared electric bicycle sends the action to the shared electric bicycle operation simulation platform model, executes the action and receives the reward, new state and a flag indicating whether the terminal state has been reached returned by the shared electric bicycle operation simulation platform; Next, the shared electric bicycle stores the current state, action, reward, new state, and end flag as experience data into the experience replay pool. After that, the algorithm replays the experience from the pool A certain number of samples are randomly selected from the network and used to update the estimated network; this process involves calculating the predicted value, the target value, and the loss function, and using gradient descent to optimize the estimated network parameters; after each update, the algorithm also periodically uses the parameters of the estimated network to update the parameters of the target network.

2. A battery inventory management optimization method for the shared electric bicycle market according to claim 1, characterized in that: The workflow of the shared electric bicycle system of the three-level supply chain is as follows: when the shared electric bicycles operate normally, whenever a battery runs out of power, the battery is transported to the base for charging, and then replaced after charging is complete, so that the shared electric bicycles can operate normally. When the battery of a shared electric bicycle fails and cannot be used any more, a replacement notice will be sent when a certain number of spare parts is reached, and the secondary warehouse will replenish the stock. When the number of spare parts in the secondary warehouse is less than s, a replenishment notice will be sent to the central warehouse, and the central warehouse will replenish the stock.

3. The battery inventory management optimization method for the shared electric bicycle market according to claim 1 is characterized in that: At each base, the failure process of shared electric bicycles is considered as a Poisson process with parameter λ2. Due to the influence of internal degradation of the battery, the life of any electric bicycle is uncertain, which is a random variable T U , and is an exponential distribution with parameter λ1; when the number of failed batteries reaches l, new batteries will be shipped from the secondary warehouse to the base, where the time from the secondary warehouse to the base plus the replacement time is a variable with parameter α; In the primary warehouse and the secondary warehouse, the inventory strategy (s, S) is adopted, where s and S represent the order point from the secondary warehouse and the maximum inventory of the secondary warehouse respectively. When the number of spare parts in the secondary warehouse is less than s2, the ordering behavior is triggered. After that, some spare parts will be transported from the central warehouse to the secondary warehouse, and the time is exponentially distributed with the parameter η; Among them, λ1 is the time when the battery is exhausted, λ2 is the time when the battery fails, α is the time to replace the battery from the secondary warehouse to the base, and η is the transportation time from the primary warehouse to the secondary warehouse.

Images