Dryer group scheduling method based on PPO algorithm
Through the dryer group scheduling method based on the PPO algorithm, the problem of global optimal solution in the dryer group scheduling is solved, and the electricity bill and production efficiency are reduced, and the production needs of the factory are met.
Patent Information
- Application Number
- CN202410948296.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-16
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2044-07-16
AI Technical Summary
The prior art lacks a global optimal solution in dryer group scheduling, making it difficult to effectively reduce electricity bills and improve production efficiency, especially in the coordination and scheduling of multiple dryers.
Using the dryer group scheduling method based on the PPO algorithm, the working state and tank state of the dryer group are modeled into Markov decision-making process, the state space, action space and reward functions are defined, and the PPO algorithm is used to train the strategy network and the value network. To optimize the scheduling of the dryer group.
Based on electricity bill fluctuations and production demands, we have achieved flexible adjustment of the regeneration operation time period, significantly reducing total electricity bill expenditure, improving scheduling efficiency, reducing manual scheduling dependence, improving the degree of system automation, and meeting production needs.
Smart Images

Figure CN118915647A_ABST
Abstract
Description
Technical Field
[0001] The invention relates to the technical field of industrial control and optimization, and in particular to a drying machine group scheduling method based on a PPO algorithm. Background Art
[0002] Drying is one of the indispensable technologies in industrial manufacturing. Its purpose is to reduce the water activity of products to slow down the deterioration of products and maintain quality. In industries such as food production and chemical industry, drying accounts for a large proportion of global energy consumption. According to statistics, in developed countries, 9-25% of the total industrial energy consumption is used for the operation of dryers. Therefore, reducing the production cost of dryers, especially electricity expenses, has always been a hot research direction.
[0003] At present, domestic and foreign scholars have achieved certain research results in saving electricity costs for dryers. For example, in food drying, the use of continuous renewable energy technology is used to reduce the use of heat and electricity generated by fossil energy and its derivatives; the spray dryer uses exhaust heat recovery technology to recover the low to medium temperature waste heat generated, effectively improving the energy efficiency of the spray dryer. The literature studied the heat recovery system of the spray dryer, which directly mixed the exhaust gas with the feed air, reducing the energy consumption of the dryer by 14%. For the drying of textiles, mathematical modeling is used to achieve process design and minimize energy costs. For example, a semi-empirical model for clothing drying is constructed in the literature, covering the final constant speed period and the decreasing speed period.
[0004] However, the above research mainly focuses on the energy-saving optimization of a single dryer, and there is still little research on the scheduling optimization of a group of dryers. Although the optimization method of a single dryer can reduce a certain amount of electricity expenditure, in actual industrial production, dryers are usually used in groups, and the coordination and scheduling issues between multiple dryers need to be considered. In addition, the existing optimization methods are mostly based on heuristic algorithms such as linear programming and simulated annealing. Although they can quickly provide scheduling solutions, it is difficult to guarantee the global optimal solution. Summary of the invention
[0005] In view of the deficiencies in the prior art, the present invention provides a dryer group scheduling method based on the PPO algorithm, which solves the problem of how to reduce electricity expenditure and improve production efficiency by optimizing the scheduling of dryer groups.
[0006] To achieve the above objectives, the present invention is implemented through the following technical solutions: A drying machine group scheduling method based on the PPO algorithm, comprising the following steps:
[0007] S1. Model the working state of the dryer group and the tank state as a Markov decision process;
[0008] S2. Define state space, action space and reward function;
[0009] S3. Use the PPO algorithm to train the policy network and value network to optimize the scheduling of the dryer fleet.
[0010] Preferably, the step S1 specifically includes the following steps:
[0011] S1.1. Construct a tank status table to record whether each tank has completed the regeneration operation and whether it can be dried;
[0012] S1.2. Construct a working status table for the dryer group to record the current working status of each tank.
[0013] Preferably, in step S1.2, the current working state of each tank includes drying operation, regeneration operation and standby state.
[0014] Preferably, the step S2 specifically includes the following steps:
[0015] S2.1. Define the state space, including the tank state table and the dryer group working state table;
[0016] S2.2, define the action space, including the actions that each tank can choose at each time step;
[0017] S2.3. Design reward functions, including action reward function, time reward function and cooperation reward function.
[0018] Preferably, the step S3 specifically includes the following steps:
[0019] S3.1. Initialize the policy network, value network and environment;
[0020] S3.2. Filter the actions that the tank can perform according to the tank status table and the dryer group working status table;
[0021] S3.3, the dryer group executes the action and updates the environment;
[0022] S3.4, determine whether the set working time has been reached, and if so, calculate the advantage function;
[0023] S3.5. Determine whether the number of iterations has been reached. If not, update the strategy network and value network according to the formula;
[0024] S3.6. Determine whether the total number of training times has been reached. If so, select the optimal strategy; otherwise, return to step S3.1.
[0025] Preferably, the reward function in step S2.3 is designed as follows:
[0026] The action reward function is:
[0027]
[0028] Preferably, the formula of the advantage function in step S3.4 is:
[0029] Where γ represents the discount factor, λ represents the advantage function discount factor, γ and λ are used to balance the importance of current rewards and future rewards, t represents the current time step, T represents the final time step, δ t It is expressed as an estimate of the advantage function at time step t as follows:
[0030] δ t =R t +γ * V(s t+1 )-V(s t ), where R t represents the reward obtained at time t, V(s t+1 ) means s t+1 The state value estimation, V(s t ) means s t The estimated state value.
[0031] Preferably, in step S3.5, an objective function is used to update the policy network parameters, wherein the objective function is:
[0032] where π θ represents the decision of the new policy network, represents the decision of the old policy network, π θ and The ratio of is called the new and old strategy ratio.
[0033] The present invention provides a method for scheduling a drying machine group based on a PPO algorithm. It has the following beneficial effects:
[0034] 1. The present invention optimizes the scheduling of the dryer group and arranges the regeneration operation during the off-peak period when the electricity price is lower, thereby significantly reducing the total electricity expenditure. In addition, the PPO algorithm in deep reinforcement learning is adopted, combined with the Markov decision process model, and the intelligent agent is able to autonomously optimize the scheduling strategy and improve the scheduling efficiency through continuous interaction and learning with the environment.
[0035] 2. The present invention flexibly adjusts the time period of the regeneration operation based on the fluctuation of electricity prices and production needs. It is independent of the specific structure of the dryer and the drying object, has a wide range of applicability, and while optimizing the scheduling, ensures that a certain number of tanks are dried at each moment to meet the production needs of the factory without affecting the normal progress of production tasks.
[0036] 3. The present invention reduces the reliance on manual scheduling, improves the automation level of the system, and reduces labor costs and error rates through an intelligent scheduling optimization method. BRIEF DESCRIPTION OF THE DRAWINGS
[0037] Figure 1 is a flow chart of the present invention;
[0038] Figure 2 It is a schematic diagram of the dryer scheduling optimization process of the present invention;
[0039] Figure 3 This is a working state diagram of the drying machine group of the present invention;
[0040] Figure 4 This is a schematic diagram of different numbers of tanks for drying operations of the present invention;
[0041] Figure 5 It is a schematic diagram of different starting moments of the present invention. DETAILED DESCRIPTION
[0042] The technical solution of the present invention will be described clearly and completely below in conjunction with the accompanying drawings of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0043] Example:
[0044] Please see attached Figure 1 -Attached Figure 2 The embodiment of the present invention provides a drying machine group scheduling method based on the PPO algorithm, comprising the following steps:
[0045] S1. Model the working state of the dryer group and the tank state as a Markov decision process;
[0046] S2. Define state space, action space and reward function;
[0047] S3. Use the PPO algorithm to train the policy network and value network to optimize the scheduling of the dryer fleet.
[0048] Specifically, the present invention provides a scheduling method for a dryer group based on a PPO algorithm, which significantly improves the operating efficiency and economic benefits of the dryer group.
[0049] 1. Reduce electricity expenses: By optimizing scheduling, regeneration operations are scheduled during off-peak hours when electricity costs are lower, effectively reducing the total electricity expenses of regeneration operations.
[0050] 2. Improve scheduling efficiency: Using the PPO algorithm in deep reinforcement learning, the intelligent agent continuously interacts and learns with the environment, autonomously optimizes the scheduling strategy, and significantly improves the scheduling efficiency.
[0051] 3. Strong adaptability: The method can flexibly adjust the regeneration operation time period based on the fluctuation of electricity prices and production demand. It is independent of the specific dryer structure and drying object and has wide applicability.
[0052] 4. Meet production needs: Ensure that while optimizing scheduling, a certain number of tanks are dried at each moment to meet the factory's production needs without affecting the normal progress of production tasks.
[0053] 5. Reduce manual intervention: Intelligent scheduling optimization methods reduce dependence on manual scheduling, improve the degree of system automation, and reduce labor costs and error rates.
[0054] The S1 step specifically includes the following steps:
[0055] S1.1. Construct a tank status table to record whether each tank has completed the regeneration operation and whether it can be dried;
[0056] S1.2. Construct a working status table for the dryer group to record the current working status of each tank.
[0057] In step S1.2, the current working status of each tank includes drying operation, regeneration operation and standby state.
[0058] Specifically, step S1 provides a systematic framework to formalize the complex dryer fleet scheduling problem so that the problem can be processed by deep reinforcement learning algorithms.
[0059] Specific solutions:
[0060] S1.1. Construct a tank status table: record whether each tank has completed the regeneration operation and whether the drying operation can be performed.
[0061] The problem of inability to accurately record the status of the tank is solved. Through the status table, the status of each tank is accurately recorded to ensure the accuracy and timeliness of the data.
[0062] Specific steps:
[0063] The state space consists of two key parts: the tank state table and the dryer group working state table. These two tables are used to describe the states existing in the system and play a key role in the tank's perception of environmental information.
[0064] Create a tank status table to record the regeneration status (completed or not completed) and drying status (available or not available) of each tank. As shown in Table 1, the tank status table:
[0065]
[0066] A Boolean value (0 or 1) is used to represent the state of each tank. For example, c,1=0 indicates that tank No. 1 of the i-th dryer can be dried, and c,1=1 indicates that it cannot be dried.
[0067] The tank status table reflects whether the tank can be dried. Specifically, after the tank is dried, it must be regenerated, otherwise it cannot be dried. The tank status table records the status of the tank. If the tank has completed the regeneration operation, its status will be marked as "dryable" and subsequent drying operations can be performed; if the tank has not completed the regeneration operation, its status will be marked as "undryable" and subsequent drying operations cannot be performed. The tank status table is very important. It determines which tank can be dried and always affects the decision-making process of the drying machine group.
[0068] S1.2. Construct a working status table for the dryer group: record the current working status of each tank, including drying operation, regeneration operation and standby status.
[0069] Solve the problem of being unable to monitor the working status of the dryer group in real time. Through the working status table, the working status of each tank is monitored and updated in real time.
[0070] Specific steps:
[0071] In order to comply with the production requirements and physical limitations of the dryer group, the tank needs to obtain the working status information of other tanks during the decision-making process. Table 2 records the working status of the dryer group. By viewing the information in Table 2, the tank can obtain the working status information of other tanks and make better decisions.
[0072] Create a dryer group working status table to record the current working status of each tank, including drying operation, regeneration operation and standby.
[0073] Table 2 Dryer group working status table:
[0074]
[0075] The S2 step specifically includes the following steps:
[0076] S2.1. Define the state space, including the tank state table and the dryer group working state table;
[0077] S2.2, define the action space, including the actions that each tank can choose at each time step;
[0078] S2.3. Design reward functions, including action reward function, time reward function and cooperation reward function.
[0079] Specifically, the present invention systematically optimizes the scheduling of the drying machine group by defining the state space, action space and reward function, effectively improving the operating efficiency and reducing the electricity cost. The specific beneficial effects and functions are as follows:
[0080] 1. Improve decision-making efficiency: By accurately defining the state space and action space, the PPO algorithm can efficiently optimize decisions based on a comprehensive understanding of the system state and optional actions.
[0081] 2. Reduce operating costs: Design a reasonable reward function to guide the tank to perform regeneration operations during periods of lower electricity costs, significantly reducing total electricity costs.
[0082] 3. Ensure production demand: Through the design of cooperative reward function, ensure that the drying machine group meets the production demand of the factory while optimizing the scheduling, without affecting the normal progress of production tasks.
[0083] Solve the following problems faced by the scheduling of dryer groups in the existing technology:
[0084] 1. Incomplete status records: Traditional methods make it difficult to fully record the working status of the dryer fleet, resulting in a lack of accurate data support for scheduling decisions.
[0085] 2. Inflexible action selection: The action space is not clearly defined, resulting in inflexible scheduling strategies and difficulty in coping with dynamically changing environments and demands.
[0086] 3. Unclear optimization goals: The reward function is not designed reasonably and cannot effectively guide the algorithm to optimize in the direction of reducing costs and meeting production needs.
[0087] Specific solutions
[0088] S2.1. Define the state space, including the tank state table and the dryer group working state table:
[0089] Providing comprehensive status information enables the algorithm to accurately grasp the current status of each tank and the entire dryer fleet.
[0090] Specific solutions:
[0091] Tank status table: records whether each tank has completed the regeneration operation and whether it can be dried. This solves the problem that the tank status cannot be accurately recorded.
[0092] Ensure the accuracy and timeliness of data, making the status of each tank clearly visible.
[0093] Dryer group working status table: records the current working status of each tank, including drying operation, regeneration operation and standby state. It solves the problem of being unable to monitor the working status of the dryer group in real time.
[0094] Monitor and update the working status of each tank in real time, providing comprehensive status information.
[0095] S2.2. Define the action space, including the actions that each tank can choose at each time step:
[0096] Providing a variety of decision-making options enables the algorithm to flexibly respond to different situations and optimize scheduling strategies.
[0097] Specific solutions:
[0098] Action definition: including drying operation, regeneration operation and standby operation. It solves the problem of unclear action selection.
[0099] The action space consists of the actions that the tank can choose in each time step. The action space is shown as follows:
[0100] A={0 i , 1 i , 2 i}
[0101] Where 0 i Indicates that the No. 1 tank of the i-th dryer selects the regeneration operation, 1 i Indicates that the No. 2 tank of the i-th dryer selects the regeneration operation, 2 i Indicates that the i-th tank selects standby operation.
[0102] Provide flexible scheduling options so that the algorithm can make reasonable decisions based on current status and demand.
[0103] S2.3. Design reward functions, including action reward function, time reward function and cooperation reward function:
[0104] Through a reasonable reward mechanism, the algorithm is guided to optimize in the direction of reducing electricity costs and meeting production needs.
[0105] Specific solutions:
[0106] Action reward function: Encourage the tank to perform regeneration operations. Solve the problem of unclear algorithm optimization direction.
[0107] The reward function design is as follows:
[0108]
[0109] The algorithm is guided to select actions to perform regeneration operations to improve tank utilization.
[0110] The action reward function for judging whether the tank performs regeneration operation:
[0111]
[0112] Time reward function: guide the tank to perform regeneration operations during off-peak hours to reduce electricity costs.
[0113] Through time rewards, the tank is guided to regenerate during periods of low electricity costs, thus reducing electricity costs. The time reward function is a design for the division of electricity price intervals, and its purpose is to guide the dryer fleet to make decisions in the direction of saving electricity costs. Time rewards will receive corresponding rewards or penalties based on the electricity price interval.
[0114] The time reward function is as follows:
[0115]
[0116] Cooperation reward function: While ensuring production needs, guide the tanks to cooperate to complete the task.
[0117] Through cooperative incentives, we ensure that the factory's production needs are met while optimizing scheduling.
[0118]
[0119] The cooperative reward function is a reward function designed for the production needs of the factory.
[0120] The purpose is to guide the dry tanks to cooperate with each other to meet the production needs of the factory. If the number of tanks in the drying operation at a certain moment is less than the number of tanks set by the factory, the alarm at that moment is 1 and the cooperation reward is -1; if the number of tanks in the drying operation at a certain moment is the number of tanks set by the factory, the alarm at that moment is 0 and the cooperation reward is 1.
[0121] In order to reduce the difference between the time reward function and the cooperation reward function and optimize the algorithm stability, we adopted the method of normalizing the rewards. The normalization formula is as follows:
[0122]
[0123] Among them, R is the reward calculated by the reward function, R mean Represents the mean operation of the reward, Rstd Represents the standard deviation of the reward.
[0124] Summarize:
[0125] By defining the state space, action space and designing a reasonable reward function, the dryer group scheduling method of the present invention solves many problems in the prior art. Specifically, comprehensive recording of state information enables the algorithm to accurately grasp the system state, flexible action selection improves the adaptability of the scheduling strategy, and a reasonable reward mechanism ensures the clarity of the algorithm optimization direction, effectively reducing electricity costs and meeting production needs.
[0126] The S3 step specifically includes the following steps:
[0127] S3.1. Initialize the policy network, value network and environment;
[0128] S3.2. Filter the actions that the tank can perform according to the tank status table and the dryer group working status table;
[0129] S3.3, the dryer group executes the action and updates the environment;
[0130] S3.4, determine whether the set working time has been reached, and if so, calculate the advantage function;
[0131] S3.5. Determine whether the number of iterations has been reached. If not, update the strategy network and value network according to the formula;
[0132] S3.6. Determine whether the total number of training times has been reached. If so, select the optimal strategy; otherwise, return to step S3.1.
[0133] Specifically, the present invention uses the PPO algorithm to train the strategy network and the value network to optimize the scheduling of the dryer group, which significantly improves the operating efficiency and economic benefits of the dryer group. The specific beneficial effects and their functions are as follows:
[0134] 1. Improve optimization efficiency: Through the adaptive learning of intelligent algorithms, the policy network and value network can continuously improve the scheduling strategy, improve optimization efficiency, and reduce manual intervention.
[0135] 2. Reduce electricity costs: The intelligent scheduling strategy can flexibly adjust the regeneration operation time according to the electricity fee period, significantly reducing electricity expenses.
[0136] 3. Meet production needs: Ensure that while optimizing scheduling, the working status of the drying machine group can meet the production needs of the factory and does not affect normal production.
[0137] 4. Reduce manual operations: The automated optimization process reduces manual scheduling operations, improves the automation level of the system, and reduces the possibility of human errors.
[0138] Specific solutions:
[0139] S3.1. Initialize the policy network, value network and environment:
[0140] Provide initial conditions and environment for the algorithm to ensure the effectiveness of the training process.
[0141] Specific solutions:
[0142] Initialize the parameters of the policy network and the value network, set the initial environment state, and ensure a stable foundation when training begins.
[0143] The problem of unreasonable initial training conditions is solved, allowing the algorithm to start learning in a stable environment.
[0144] S3.2. According to the tank status table and the dryer group working status table, select the actions that the tank can perform:
[0145] According to the current status, the executable actions are screened out to ensure the rationality of the scheduling strategy.
[0146] Specific solutions:
[0147] According to the tank state table and the dryer group working state table, determine the actions (drying, regeneration or standby) that each tank can perform at the current time step.
[0148] The problem of ambiguous action selection is solved, making the decision at each time step more accurate and reasonable.
[0149] S3.3, Drying machine group executes action and updates environment:
[0150] Execute the selected action and update the state of the environment to provide a basis for the next decision.
[0151] Specific solutions:
[0152] Execute the filtered actions, update the tank status table and the dryer group working status table to reflect the execution results of the current actions.
[0153] Solved the problem of untimely environment status updates, ensuring that the environment status is always synchronized with actual operations.
[0154] S3.4. Determine whether the set working time has been reached. If the set working time has been reached, calculate the advantage function:
[0155] By calculating the advantage function, the pros and cons of the current strategy can be evaluated, providing a basis for strategy optimization.
[0156] Specific solutions:
[0157] Determine whether the set working time has been reached. If so, calculate the advantage function to evaluate the effect of the current action.
[0158] Advantage function calculation formula:
[0159] Among them, γ represents the discount factor, λ represents the advantage function discount factor, γ and λ are used to balance the importance of current rewards and future rewards, t represents the current time step, T represents the final time step, δ t It is expressed as an estimate of the advantage function at time step t as follows:
[0160] δ t =R t +γ * V(s t+1 )-V(s t ), where R t represents the reward obtained at time t, V(s t+1 ) means s t+1 The state value estimation, V(s t ) means s t The state value estimates are obtained through the value network calculation.
[0161] The problem of inaccurate strategy evaluation is solved, and the effect of the current strategy can be accurately evaluated through the calculation of the advantage function.
[0162] S3.5. Determine whether the number of iterations has been reached. If not, update the strategy network and value network according to the formula:
[0163] Through multiple iterations, the strategy network and value network are continuously optimized to improve the effectiveness of the scheduling strategy.
[0164] Specific solutions:
[0165] Determine whether the set number of iterations has been reached. If not, update the strategy network and value network according to the following formula:
[0166] Use the objective function for policy network update, the formula is as follows:
[0167] where π θ represents the decision of the new policy network, represents the decision of the old policy network, π θ and The ratio of is called the new and old strategy ratio, and ε is the shear rate of the shear function.
[0168] The shear function is to limit the ratio of the new strategy to the old strategy in the range (1-ε,1+ε) neritic, limiting π θ and The gap between them controls the update amplitude of the decision network.
[0169] The value network loss function is:
[0170]
[0171] The value network loss function is used to train and optimize the value network parameters V(s) represents the current parameters The calculated value network's estimate of state s, V target Represents the target value network's estimate of state s as follows:
[0172]
[0173] Among them, t represents the current time step, T represents the final time step, and γ represents the discount factor.
[0174] The problem of slow algorithm convergence was solved. Through multiple iterations and continuous optimization of strategies, the convergence speed and effect were improved.
[0175] S3.6, determine whether the total number of training times has been reached, if so, select the optimal strategy; otherwise, return to step S3.1:
[0176] By setting the total number of training times, we ensure that the algorithm eventually converges to the optimal strategy.
[0177] Specific solutions:
[0178] Determine whether the set total number of training times has been reached. If so, select the optimal strategy; if not, return to step S3.1 and continue training.
[0179] The problem of insufficient algorithm training is solved by setting a reasonable number of training times to ensure that the algorithm can fully learn and optimize.
[0180] Summarize:
[0181] Through the above steps, the dryer group scheduling method of the present invention solves many problems in the prior art. Specifically, the effectiveness of training is ensured through initialization and environment setting; the rationality of scheduling decisions is ensured through state screening and action execution; the continuous optimization of strategies is ensured through advantage functions and network updates; and the sufficient training and final convergence of the algorithm are ensured by setting the number of training times. Ultimately, the effects of improving scheduling efficiency, reducing electricity costs, meeting production needs and reducing manual operations are achieved.
[0182] The reward function in step S2.3 is designed as follows:
[0183] The action reward function is:
[0184]
[0185] The formula of the advantage function in step S3.4 is:
[0186] Where γ represents the discount factor, λ represents the advantage function discount factor, γ and λ are used to balance the importance of current rewards and future rewards, t represents the current time step, T represents the final time step, δ t It is expressed as an estimate of the advantage function at time step t as follows:
[0187] δ t =R t +γ * V(s t+1 )-V(s t ), where R t represents the reward obtained at time t, V(s t+1 ) means s t+1 The state value estimation, V(s t ) means s t The estimated state value.
[0188] In step S3.5, the objective function is used to update the policy network parameters, where the objective function is:
[0189] where π θ represents the decision of the new policy network, represents the decision of the old policy network, π θ and The ratio of is called the new and old strategy ratio.
[0190] Please see attached Figure 3 -Attached Figure 5 To verify the effectiveness of the optimization algorithm of the present invention, the optimization algorithm can more effectively reduce electricity costs by reducing the number of tanks performing drying operations and shortening the allowed time of the dryer group. The following experiment used four tanks out of seven dryers for drying operations at the same time, and after running for 240 hours, it successfully reduced the electricity costs generated by the regeneration operation by about 43.2%. Even if only five tanks are set up for drying operations at the same time, the regeneration electricity costs can be reduced by about 15.8%. The specific experimental contents are as follows:
[0191] Purpose:
[0192] This experiment aims to verify the effectiveness of the dryer group scheduling method based on the PPO algorithm, mainly evaluating its performance in reducing electricity expenditure and meeting production needs.
[0193] Experimental steps:
[0194] Experimental preparation:
[0195] Initialize the policy network, value network, and environment.
[0196] Set up a dryer cluster consisting of seven dryers, each with two tanks, and record their initial conditions.
[0197] Determine the experimental parameters and electricity price range (peak period, flat period, and valley period), as shown in the following table.
[0198]
[0199] Data collection
[0200] Create a tank status table and a dryer group working status table to record the current working status of each tank (drying operation, regeneration operation, standby).
[0201] According to the experimental time period, the state changes of the tank and the electricity consumption at each time step are recorded.
[0202] Experimental procedures
[0203] Follow these steps to conduct the experiment:
[0204] S1: Modeling the working status of the dryer group and the tank status as a Markov decision process
[0205] S1.1: Construct a tank status table to record whether each tank has completed the regeneration operation and whether the drying operation can be performed.
[0206] S1.2: Construct a dryer group working status table to record the current working status of each tank (drying operation, regeneration operation and standby).
[0207] S2: Define state space, action space, and reward function
[0208] S2.1: Define the state space, including the tank state table and the dryer group working state table.
[0209] S2.2: Define the action space, including the actions that each tank can choose at each time step (drying operation, regeneration operation, standby).
[0210] S2.3: Design reward functions, including action reward function, time reward function and cooperation reward function.
[0211] S3: Use the PPO algorithm to train the policy network and value network to optimize the scheduling of the dryer fleet
[0212] S3.1: Initialize the policy network, value network, and environment.
[0213] S3.2: According to the tank status table and the dryer group working status table, filter the actions that the tank can perform.
[0214] S3.3: The dryer cluster executes the action and updates the environment.
[0215] S3.4: Determine whether the set working time has been reached. If the set working time has been reached, calculate the advantage function.
[0216] S3.5: Determine whether the number of iterations has been reached. If not, update the strategy network and value network according to the formula.
[0217] S3.6: Determine whether the total number of training times has been reached. If so, select the optimal strategy; otherwise, return to step S3.1.
[0218] Data analysis
[0219] According to the experimental data, the working status and electricity expenditure of the drying machine group in different time periods are analyzed. The working status of the drying machine group is shown in the attached figure. Figure 3 As shown in the figure, the electricity cost comparison when different numbers of tanks are used for drying operation is as follows: Figure 4 As shown, the starting time is set to the 0th time step and the end time is the 240th time step. In Figure (a), 2 tanks are set for drying operations, and the results show that the reduction in electricity costs ranges from 70.4% to 84.8%. For Figure (b), this experiment set up 3 tanks for drying operations, which can reduce electricity costs to between 73.4% and 84.8%. In Figure (c), 4 tanks are set for drying operations, and the reduction in electricity costs is observed to be between 39.0% and 43.2%. Finally, in Figure (d), 5 tanks are set for drying operations, and it is found that the reduction in electricity costs ranges from 15.8% to 19.9%.
[0220] In order to verify the effectiveness of the optimization algorithm at different starting times, we selected four different starting times for the experiment. The horizontal axis represents the group, and the vertical axis represents the total electricity cost generated by the dryer group performing the regeneration operation. We selected 1 control group and 7 experimental groups. The control group is marked as CG, and the conventional factory method is used for production, that is, the regeneration operation is performed immediately after the tank is dried. The experimental group is marked as EG, and the optimization algorithm is used to schedule the dryer group. The usage formula is as follows:
[0221] ROR=(CG-EG)÷CG×100%
[0222] As an indicator of the degree of electricity cost reduction, the larger the ROR value, the greater the reduction in electricity cost, and the more significant the effect of the optimization algorithm. Set 4 cans for drying operation, and take the 240th time step as the termination time. Figure 4 shown.
[0223] Figure (a) takes the first time step as the starting moment, and the electricity cost can be reduced by 34.5% to 44.2%. Figure (b) takes the 11th time step as the starting moment. This experiment reduces the electricity cost by 43.8% to 46.5%. Figure (c) takes the 14th time step as the starting moment, and the electricity cost can be reduced by 48.2% to 59.2%. Figure (d) takes the 21st time step as the starting moment, and the electricity cost can be reduced by 52.0% to 60.4%.
[0224] Compare the electricity expenditure and the completion of drying operations before and after the PPO algorithm optimization.
[0225] Summary of results:
[0226] The experimental data are organized into a table to show the changes in electricity expenditure.
[0227] The experimental results are summarized to evaluate the performance of the PPO algorithm in reducing electricity costs and meeting production needs.
[0228] Experimental summary:
[0229] Through the dryer group scheduling method based on the PPO algorithm, the experimental results show that:
[0230] Significant reduction in electricity costs: Within 240 time steps, the electricity costs of the regeneration operation of the dryer fleet were reduced by about 43.2%, verifying the effectiveness of this method in reducing electricity costs.
[0231] Improved scheduling efficiency: Intelligent scheduling strategies ensure efficient operation of the dryer fleet in different time periods to meet production needs.
[0232] Reduce manual operations: Through automated scheduling, the reliance on manual intervention is reduced and the degree of automation of the system is improved.
[0233] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.
Claims
1. A drying machine group scheduling method based on PPO algorithm, characterized in that: The following steps are involved: S1. Model the working state of the dryer group and the tank state as a Markov decision process; S2. Define state space, action space and reward function; S3. Use the PPO algorithm to train the policy network and value network to optimize the scheduling of the dryer fleet.
2. A method for scheduling a drying machine group based on a PPO algorithm according to claim 1, characterized in that: The S1 step specifically includes the following steps: S1.
1. Construct a tank status table to record whether each tank has completed the regeneration operation and whether it can be dried; S1.
2. Construct a working status table for the dryer group to record the current working status of each tank.
3. A method for scheduling a drying machine group based on a PPO algorithm according to claim 2, characterized in that: The step S1.2 checks that the current working state of each tank includes drying operation, regeneration operation and standby state.
4. The method for scheduling a drying machine group based on the PPO algorithm according to claim 1, characterized in that: The S2 step specifically includes the following steps: S2.
1. Define the state space, including the tank state table and the dryer group working state table; S2.2, define the action space, including the actions that each tank can choose at each time step; S2.
3. Design reward functions, including action reward function, time reward function and cooperation reward function.
5. The method for scheduling a drying machine group based on the PPO algorithm according to claim 1, characterized in that: The S3 step specifically includes the following steps: S3.
1. Initialize the policy network, value network and environment; S3.
2. Filter the actions that the tank can perform according to the tank status table and the dryer group working status table; S3.3, the dryer group executes the action and updates the environment; S3.4, determine whether the set working time has been reached, and if so, calculate the advantage function; S3.
5. Determine whether the number of iterations has been reached. If not, update the strategy network and value network according to the formula; S3.
6. Determine whether the total number of training times has been reached. If so, select the optimal strategy; otherwise, return to step S3.
1.
6. A method for scheduling a drying machine group based on a PPO algorithm according to claim 4, characterized in that: The reward function in step S2.3 is designed as follows: The action reward function is: The time reward function is: The cooperation reward function is:
7. The method for scheduling a drying machine group based on the PPO algorithm according to claim 5, characterized in that: The formula of the advantage function in step S3.4 is: Where γ represents the discount factor, λ represents the advantage function discount factor, γ and λ are used to balance the importance of current rewards and future rewards, t represents the current time step, T represents the final time step, δ t It is expressed as an estimate of the advantage function at time step t as follows: δ t =R t +γ * V(s t+1 )-V(s t ), where R t represents the reward obtained at time t, V(s t+1 ) means s t+1 The state value estimation, V(s t ) means s t The estimated state value.
8. The method for scheduling a drying machine group based on the PPO algorithm according to claim 5, characterized in that: In step S3.5, the objective function is used to update the policy network parameters, where the objective function is: where π θ represents the decision of the new policy network, represents the decision of the old policy network, π θ and The ratio of is called the new and old strategy ratio.
Citation Information
Patent Citations
Household energy management method combining LSTM and deep reinforcement learning and medium
CN114841409A
Equipment manufacturing workshop intelligent scheduling method and system based on deep reinforcement learning
CN116542445A
Dynamic economic dispatching method and system for power distribution network
CN117674114A
Flexible job shop scheduling method based on graph reinforcement learning under industrial demand response
CN118095753A
Service chain adaptive arrangement method based on deep reinforcement learning
CN118158112A
Cited By
Peak-averting and valley-averting energy-saving control method and system
CN121082037A
Peak-avoiding and valley-choosing energy-saving control method and system
CN121082037B
