Refrigeration control method and device for cold storage, computer device and storage medium

Abstract

The present application relates to the technical field of industrial intelligent control, and particularly relates to a refrigeration control method and device for a cold storage, computer equipment and a storage medium, wherein the refrigeration control task is decoupled into multiple warehouse agents and multiple compressor agents, action reasoning and action execution are respectively performed according to state space data of each warehouse agent and state space data of each compressor agent, reward values of each warehouse agent and each compressor agent after action execution are calculated, and the warehouse agent and the compressor agent are updated in combination with the reward values of each warehouse agent and each compressor agent after action execution, so that the comprehensive refrigeration energy efficiency is improved and the overall energy consumption is reduced.

Description

Technical Field

[0001] This invention relates to the field of industrial intelligent control technology, and in particular to a refrigeration control method, device, computer equipment, and storage medium for a cold storage facility. Background Technology

[0002] With the large-scale development of the cold chain logistics industry, large-scale cold chain logistics centers typically consist of dozens of heterogeneous cold rooms (such as freezers, refrigerated warehouses, and passageways) and parallel units composed of multiple compressors. These systems are characterized by strong nonlinearity, high thermal inertia, and numerous sources of interference. In practical engineering applications, existing control technologies are mainly divided into two categories: traditional control based on static rule logic and intelligent control based on deep reinforcement learning, but both have significant technical shortcomings.

[0003] Traditional control based on static rule logic lacks global spatiotemporal coordination for multiple cooling rooms and parallel units when dealing with large-scale complex systems. On the one hand, due to the randomness of heat load changes in each cooling room, multiple cooling rooms are prone to concentrated cooling at the same time, resulting in peak superposition of the total system heat load, significantly increasing the system's maximum power demand and operating electricity costs. On the other hand, the coefficient of performance (COP) of parallel units changes dynamically with the load rate, and traditional control based on static rule logic struggles to perceive and calculate energy efficiency differences under different operating conditions in real time, leading to rigid unit adjustment strategies and making it difficult to maintain parallel units in a high COP range. Summary of the Invention

[0004] Based on this, the purpose of this invention is to provide a refrigeration control method, device, computer equipment, and storage medium for cold storage. By decoupling the refrigeration control task into multiple warehouse intelligent agents and multiple compressor intelligent agents, action reasoning and action execution are performed based on the state space data of each warehouse intelligent agent and each compressor intelligent agent, and the reward values ​​of each warehouse intelligent agent and each compressor intelligent agent are calculated after the action execution. The warehouse intelligent agents and compressor intelligent agents are updated by combining the reward values ​​of each warehouse intelligent agent and each compressor intelligent agent after the action execution, thereby improving the overall refrigeration energy efficiency and reducing the overall energy consumption.

[0005] In a first aspect, embodiments of this application provide a refrigeration control method for a cold storage facility, comprising the following steps:

[0006] Based on the state space data of each warehouse intelligent agent and each compressor intelligent agent in the intelligent agent refrigeration control network of the cold storage at each target time in the preset time period, the action space data of each warehouse intelligent agent and each compressor intelligent agent at each target time are obtained. Based on the motion space data of each warehouse agent and each compressor agent at each target time, refrigeration control and reward calculation are performed to obtain the reward value of each warehouse agent and each compressor agent at each target time. The state space data, action space data, reward value, and state space data of the next time step of the same warehouse agent and the same compressor agent at the target time are combined to construct the quadruple data of each warehouse agent and each compressor agent at each target time. Based on the quadruple data of each warehouse agent and each compressor agent at each target time, the agent refrigeration control network is updated to obtain the target agent refrigeration control network. Based on the state space data of each warehouse agent and each compressor agent in the cold storage target agent refrigeration control network at the current time after the preset time period, the refrigeration control of the cold storage is executed.

[0007] Secondly, embodiments of this application provide a refrigeration control device for a cold storage facility, comprising: The action reasoning module is used to obtain the action space data of each warehouse intelligent agent and each compressor intelligent agent at each target time based on the state space data of each warehouse intelligent agent and each compressor intelligent agent in the intelligent agent refrigeration control network of the cold storage at each target time in a preset time period. The reward calculation module is used to perform refrigeration control and reward calculation based on the action space data of each warehouse intelligent agent and each compressor intelligent agent at each target time, so as to obtain the reward value of each warehouse intelligent agent and each compressor intelligent agent at each target time. The data processing module is used to combine the state space data, action space data, reward value, and state space data of the next moment of the same warehouse agent and the same compressor agent at the target time to construct the quadruple data of each warehouse agent and each compressor agent at each target time. The network update module is used to update the intelligent agent refrigeration control network based on the four-tuple data of each warehouse intelligent agent and the four-tuple data of each compressor intelligent agent at each target time, so as to obtain the target intelligent agent refrigeration control network. The refrigeration control module is used to perform refrigeration control of the cold storage based on the state space data of each warehouse agent and the state space data of each compressor agent in the target agent refrigeration control network of the cold storage at the current time after the preset time period.

[0008] Thirdly, embodiments of this application provide a computer device, including: a processor, a memory, and a computer program stored in the memory and executable on the processor; when the computer program is executed by the processor, it implements the steps of the refrigeration control method for the cold storage as described in the first aspect.

[0009] Fourthly, embodiments of this application provide a storage medium storing a computer program that, when executed by a processor, implements the steps of the refrigeration control method for a cold storage as described in the first aspect.

[0010] In this application embodiment, a refrigeration control method, apparatus, computer equipment, and storage medium for a cold storage are provided. By decoupling the refrigeration control task into multiple warehouse intelligent agents and multiple compressor intelligent agents, action reasoning and action execution are performed based on the state space data of each warehouse intelligent agent and each compressor intelligent agent, and the reward values ​​of each warehouse intelligent agent and each compressor intelligent agent are calculated after the action execution. The warehouse intelligent agents and compressor intelligent agents are updated by combining the reward values ​​of each warehouse intelligent agent and each compressor intelligent agent after the action execution, thereby improving the overall refrigeration energy efficiency and reducing the overall energy consumption.

[0011] To better understand and implement this invention, the following detailed description is provided in conjunction with the accompanying drawings. Attached Figure Description

[0012] Figure 1 A schematic flowchart of a refrigeration control method for a cold storage provided in one embodiment of this application; Figure 2 A schematic flowchart of step S2 in a refrigeration control method for a cold storage provided in one embodiment of this application; Figure 3 A schematic flowchart of step S22 in a refrigeration control method for a cold storage provided in one embodiment of this application; Figure 4 A schematic flowchart of step S2 in a refrigeration control method for a cold storage provided in another embodiment of this application; Figure 5 A schematic flowchart of step S4 in a refrigeration control method for a cold storage provided in one embodiment of this application; Figure 6 A schematic flowchart of step S41 in a refrigeration control method for a cold storage provided in one embodiment of this application; Figure 7 A schematic flowchart of step S42 in a refrigeration control method for a cold storage provided in one embodiment of this application; Figure 8 A schematic diagram of the structure of a refrigeration control device for a cold storage provided in one embodiment of this application; Figure 9 This is a schematic diagram of the structure of a computer device provided in one embodiment of this application. Detailed Implementation

[0013] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0014] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0015] It should be understood that although the terms first, second, third, etc., may be used in this application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0016] Please see Figure 1 , Figure 1 The following is a flowchart illustrating a refrigeration control method for a cold storage facility according to an embodiment of this application. The method includes the following steps: S1: Based on the state space data of each warehouse intelligent agent and each compressor intelligent agent in the intelligent agent refrigeration control network of the cold storage at each target time within a preset time period, obtain the action space data of each warehouse intelligent agent and each compressor intelligent agent at each target time.

[0017] The executor of the refrigeration control method for cold storage in this application is the control equipment for the refrigeration control method of cold storage (hereinafter referred to as the control equipment). In an optional embodiment, the control equipment may be a computer device, a server, or a server cluster composed of multiple computer devices.

[0018] In this embodiment, the control device uses the state space data of each warehouse intelligent agent and the state space data of each compressor intelligent agent in the intelligent agent refrigeration control network of the cold storage at each target time within a preset time period. The intelligent agent refrigeration control network includes a warehouse intelligent agent group and a compressor intelligent agent group; the warehouse intelligent agent group includes several warehouse intelligent agents.

[0019] Each warehouse intelligent agent corresponds one-to-one with each physical warehouse in the cold storage and is configured to independently perceive the local environmental state. The environmental state characteristics of the cold storage serve as the state space data of the corresponding warehouse intelligent agent, thereby obtaining the state space data of each warehouse intelligent agent. The state space data of the warehouse intelligent agent is used to provide feedback on the thermodynamic characteristics of the corresponding warehouse and the current status of refrigeration control execution.

[0020] Each warehouse intelligent agent outputs action space data based on the state space data of the corresponding warehouse intelligent agent, thereby obtaining the action space data of the warehouse intelligent agent. The action space data of the warehouse intelligent agent is used to indicate the combined control action of the internal cooling air unit of the warehouse.

[0021] The compressor intelligent agent group includes several compressor intelligent agents. Each compressor intelligent agent corresponds one-to-one with each physical compressor in the cold storage and is configured to independently sense the operating status of the compressor. The operating status characteristics of the compressor are matched with the state space data of the corresponding compressor intelligent agent to obtain the state space data of each compressor intelligent agent. The state space data of the compressor intelligent agent is used to indicate the current operating status of the compressor.

[0022] Each compressor agent outputs action space data based on its corresponding state space data to obtain the compressor agent's action space data. The action space data of the compressor agent is used to indicate the start-stop and loading / unloading control commands of the compressor.

[0023] The warehouse intelligent agent swarm and the compressor intelligent agent swarm do not communicate explicitly directly. Each agent performs local policy updates and distributed execution based on a shared global state and a hybrid reward function that includes global constraints. Both the warehouse and compressor intelligent agent swarms employ a parameter sharing mechanism to improve training efficiency and scalability.

[0024] Specifically, the state space data includes local state space data and global state space data; wherein, the local state space data of the warehouse agent includes the current temperature, the target set temperature, the deviation between the current temperature and the target set temperature, the temperature change rate, the temperature trajectory sequence within the historical time window, the total number of air coolers installed, the number of air coolers turned on, the speed of the turned-on air coolers, the warehouse load, and the warehouse type; the global state space data of the warehouse agent includes the total global power and the number of air coolers turned on in other warehouses.

[0025] The local state space data of the compressor agent includes the operating state at the previous target time, the load rate at the previous target time, the current continuous running time or continuous downtime, the current instantaneous coefficient of performance (COP), the current suction pressure, the target suction pressure, the deviation between the current suction pressure and the target suction pressure, and the suction pressure change rate. The global state space data of the compressor agent includes the global total power and the number of compressors currently being started at the current target time.

[0026] Specifically, each warehouse agent and each compressor agent invokes its corresponding action policy network. The action policy network performs action reasoning calculations based on the state space data to obtain the current action space data of each warehouse agent and each compressor agent. The action space data of the warehouse agents includes a cooler quantity control dimension and a cooler intensity control dimension. The cooler quantity control dimension indicates the number of coolers that need to be put into operation in the corresponding warehouse; the cooler intensity control dimension indicates the operating level of the coolers when they are in operation.

[0027] The action space data of the compressor agent includes compressor start / stop commands and capacity adjustment commands. The compressor start / stop commands indicate the operating status of the corresponding compressor; the capacity adjustment commands indicate the capacity adjustment status of the corresponding compressor. For variable frequency compressors, the compressor start / stop commands are mapped to frequency changes. For slide valve regulating compressors, the compressor start / stop commands represent loading / unloading commands.

[0028] S2: Execute refrigeration control and reward calculation based on the motion space data of each warehouse agent and each compressor agent at each target time, and obtain the reward value of each warehouse agent and each compressor agent at each target time.

[0029] In this embodiment, the control device performs refrigeration control and reward calculation based on the motion space data of each warehouse intelligent agent and each compressor intelligent agent at each target time, and obtains the reward value of each warehouse intelligent agent and each compressor intelligent agent at each target time.

[0030] Please see Figure 2 , Figure 2 The flowchart of step S2 in the refrigeration control method for a cold storage provided in one embodiment of this application includes steps S21 to S22, as follows: S21: Execute refrigeration control based on the action space data of each warehouse agent and each compressor agent at each target time, and obtain the refrigeration control execution index of the agent refrigeration control network at the current time.

[0031] In this embodiment, the control device performs refrigeration control based on the motion space data of each warehouse intelligent agent and the motion space data of each compressor intelligent agent at each target time, and obtains the refrigeration control execution index of the intelligent agent refrigeration control network at the current time. The refrigeration control execution index of the intelligent agent refrigeration control network includes the total global power, the number of compressors performing the start-up action, the current temperature of each warehouse intelligent agent, and the current suction pressure of each compressor intelligent agent.

[0032] S22: Calculate rewards based on the refrigeration control execution indicators of the agent refrigeration control network at each target time, and obtain the reward values ​​of each warehouse agent and each compressor agent at each target time.

[0033] In this embodiment, the control device calculates rewards based on the refrigeration control execution indicators of the intelligent agent refrigeration control network at each target time, and obtains the reward values ​​of each warehouse intelligent agent and each compressor intelligent agent at each target time.

[0034] For the reward value of the warehouse agent, please refer to [link / reference]. Figure 3 , Figure 3 The flowchart of step S22 in the refrigeration control method for a cold storage provided in one embodiment of this application includes steps S221 to S222, as follows: S221: Calculate the energy consumption cost and power over-limit penalty based on the global total power at each target time to obtain the energy consumption cost and power over-limit penalty value at each target time; calculate the compressor concurrent start-up penalty based on the number of compressors performing the start-up action at each target time to obtain the compressor concurrent start-up penalty value at each target time; sum up the energy consumption cost, power over-limit penalty value, and compressor concurrent start-up penalty value at each target time to obtain the global reward value at each target time.

[0035] In this embodiment, the control device calculates the energy consumption cost and power over-limit penalty based on the global total power at each target time, and obtains the energy consumption cost and power over-limit penalty value at each target time.

[0036] Specifically, the control device multiplies the current global total power with a preset energy consumption cost weighting coefficient to obtain the energy consumption cost at each target time, guiding all intelligent agents to reduce the load when not necessary, thereby reducing basic operating electricity costs.

[0037] The control device obtains the power over-limit penalty value at the current moment based on the current global total power, a preset safe power threshold, a power over-limit weighting coefficient, and a power over-limit penalty calculation algorithm. When the global total power approaches or exceeds the threshold, the power over-limit penalty value increases linearly or exponentially. The power over-limit penalty calculation algorithm is as follows:

[0038] In the formula, This is the penalty value for exceeding power limits. This is the power over-limit weighting coefficient. It is a non-linear activation function. This represents the total global power. This is the safe power threshold.

[0039] When the total global power approaches or exceeds the safe power threshold, the power over-limit penalty increases linearly or exponentially. This generates a strong gradient signal that suppresses the load-increasing actions of all warehouse agents and compressor agents, achieving automatic peak shaving.

[0040] The control equipment calculates the compressor concurrent start penalty based on the number of compressors performing start-up actions at each target time, obtaining the compressor concurrent start penalty value for each target time. Specifically, the control equipment obtains the compressor concurrent start penalty value for each target time based on the number of compressors performing start-up actions at each target time, a preset compressor concurrent start weight coefficient, and a preset compressor concurrent start penalty calculation algorithm. The compressor concurrent start penalty value represents the penalty imposed if more than one compressor is detected attempting to start at the same time. The compressor concurrent start penalty calculation algorithm is as follows:

[0041] In the formula, This is the penalty value for concurrent compressor startup. This is the weighting coefficient for concurrent compressor startup. It is a non-linear activation function. The number of compressors that perform the start-up action.

[0042] The control device accumulates the energy consumption cost, power over-limit penalty value, and compressor concurrent start-up penalty value at each target time to obtain the global reward value at each target time.

[0043] S222: Calculate the temperature penalty based on the current temperature of each warehouse agent at each target time and the target temperature of the warehouse, and obtain the temperature penalty value of each warehouse agent at each target time; calculate the reward value based on the global total power, global reward value, temperature penalty value of each warehouse agent at each target time and the preset first reward calculation algorithm, and obtain the reward value of each warehouse agent at each target time.

[0044] In this embodiment, the control device calculates the temperature penalty based on the current temperature of each warehouse intelligent agent at each target time and the target set temperature of the warehouse, and obtains the temperature penalty value of each warehouse intelligent agent at each target time.

[0045] Specifically, the control device obtains the temperature penalty value for each warehouse intelligent agent at each target time based on the current temperature of each warehouse intelligent agent at each target time, the target temperature of the warehouse, the preset temperature weighting coefficient, and the temperature penalty calculation algorithm. The temperature penalty calculation algorithm is as follows:

[0046] In the formula, This is the temperature penalty value. As the first weighting coefficient, The current temperature. Set the target temperature for the warehouse. The temperature penalty function is as follows:

[0047] In the formula, The deviation between the current temperature and the target set temperature, The penalty coefficient is... The temperature is set to a preset allowable range. If the deviation between the current temperature and the target set temperature is within the allowable range, the penalty is 0. When the temperature exceeds the allowable range, the penalty value increases non-linearly with the deviation.

[0048] By introducing a temperature penalty function, a "zero-penalty buffer" is constructed in the control logic. When the warehouse temperature is within this buffer, the warehouse agent not only avoids temperature deviation penalties but also receives a positive overall reward (i.e., a lower negative penalty) because shutting down the air cooler reduces the total global power. This mechanism grants the warehouse agent autonomous decision-making power in non-emergency situations, enabling it to utilize the warehouse's thermal inertia to proactively maintain shutdown during peak grid demand periods to offload loads, and then compensate for cooling during off-peak demand periods, thereby achieving system-level peak shaving and valley filling without compromising the safety of goods storage.

[0049] The control device calculates reward values ​​for each warehouse agent at each target time based on the global total power, global reward value, temperature penalty value of each warehouse agent, and a preset first reward calculation algorithm. The first reward calculation algorithm is as follows:

[0050] In the formula, The reward value for the warehouse agent. This is the global reward value. This is the second weighting coefficient. This is the third weighting coefficient.

[0051] The reward value of the warehouse intelligence is calculated by combining the global total power, global reward value, and temperature penalty value of the warehouse intelligence. By utilizing the thermal inertia of the physical warehouse, the system actively outputs control actions to shut down or maintain low-frequency operation, so as to transfer the limited power load quota to the warehouse intelligence with temperature deviation exceeding the preset threshold. In this way, the concurrent cooling demand that may have occurred simultaneously in the system can be staggered on the time axis, thereby avoiding demand overrun caused by the superposition of load peaks.

[0052] For the reward value of the warehouse agent, please refer to [link / reference]. Figure 4 , Figure 4 A flowchart illustrating step S2 of the refrigeration control method for a cold storage provided in another embodiment of this application includes step S223, as follows: S223: Calculate the intake pressure penalty based on the current intake pressure of each compressor agent at each target time and the target intake pressure of the compressor, and obtain the intake pressure penalty value of each compressor agent at each target time; calculate the reward value based on the global reward value at each target time, the intake pressure penalty value of each compressor agent, and the preset second reward calculation algorithm, and obtain the reward value of each compressor agent at each target time.

[0053] In this embodiment, the control device performs intake pressure penalty calculation based on the current intake pressure of each compressor agent at each target time and the target set intake pressure of the compressor, and obtains the intake pressure penalty value of each compressor agent at each target time.

[0054] Specifically, the control device subtracts the current suction pressure of each compressor intelligent body at each target time from the target set suction pressure of the compressor to obtain the suction pressure deviation of each compressor intelligent body at each target time; the control device multiplies the suction pressure deviation of each compressor intelligent body at each target time with a preset pressure weighting coefficient to obtain the suction pressure penalty value of each compressor intelligent body at each target time.

[0055] The control device calculates reward values ​​for each compressor agent at each target time based on the global reward value at each target time, the intake pressure penalty value of each compressor agent, and a preset second reward calculation algorithm. The second reward calculation algorithm is as follows:

[0056] In the formula, The reward value for the compressor agent. It is the fourth weighting coefficient. This is the suction pressure penalty value for the compressor's intelligent agent. For smooth action items, used to represent action smoothing penalties.

[0057] The reward value of the compressor agent is calculated by combining the global reward value and the suction pressure penalty value of the compressor agent, so that each compressor agent can make collaborative decisions by observing the concurrent startup status of the system in real time. When it is detected that there are other compressors in the startup process in the system, the current compressor agent actively suppresses its own startup control command to eliminate the risk of surge current superposition and circuit breaker protection caused by the concurrent startup of multiple high-power motors.

[0058] Furthermore, the concurrent start-up constraints and action smoothing penalties embedded in the reward value prompt the compressor agent to spontaneously execute the avoidance start-up strategy, which not only eliminates the impact of concurrent surge current on the microgrid from the source, but also significantly reduces the frequent start-up and shutdown of equipment and mechanical wear, thus providing a strong guarantee for the dual operational safety of the microgrid and the underlying hardware equipment.

[0059] S3: Combine the state space data, action space data, reward value, and state space data of the next time step of the same warehouse agent and the same compressor agent at the target time to construct the quadruple data of each warehouse agent and each compressor agent at each target time.

[0060] In this embodiment, the control device obtains the state space data of each warehouse intelligent agent and the state space data of each compressor intelligent agent at the next moment.

[0061] The control device combines the state space data, action space data, reward value, and state space data of the next moment of the same warehouse agent and the same compressor agent at the target time to construct the quadruple data of each warehouse agent and each compressor agent at each target time, and stores them in a preset shared experience replay pool.

[0062] S4: Based on the quadruple data of each warehouse agent and each compressor agent at each target time, update the agent refrigeration control network to obtain the target agent refrigeration control network.

[0063] In this embodiment, the control device updates the intelligent agent refrigeration control network based on the quadruple data of each warehouse intelligent agent and the quadruple data of each compressor intelligent agent at each target time, thereby obtaining the target intelligent agent refrigeration control network.

[0064] All warehouse agents in the aforementioned intelligent agent refrigeration control network invoke the same shared warehouse agent policy network, and all compressor agents invoke the same shared compressor agent policy network; both the warehouse agent policy network and the compressor agent policy network include an action policy network and a value network; please refer to [link / reference]. Figure 5 , Figure 5 The flowchart of step S4 in the refrigeration control method for a cold storage provided in one embodiment of this application includes steps S41 to S43, as follows: S41: Based on the quadruple data of each warehouse agent at each target time, update the action policy network and value network in the warehouse agent policy network to obtain the updated warehouse agent policy network.

[0065] In this embodiment, the control device updates the action policy network and value network in the warehouse agent policy network based on the quadruple data of each warehouse agent at each target time, thereby obtaining the updated warehouse agent policy network.

[0066] Please see Figure 6 , Figure 6 The flowchart of step S41 of the refrigeration control method for a cold storage provided in one embodiment of this application includes steps S411 to S414, as follows: S411: Input the quadruple data of each warehouse agent at each target time into the action policy network and value network in the warehouse agent policy network to obtain the action probability distribution and prediction value of each warehouse agent at each target time.

[0067] In this embodiment, the control device inputs the quadruple data of each warehouse agent at each target time into the action policy network and the value network in the warehouse agent policy network to obtain the action probability distribution and prediction value of each warehouse agent at each target time.

[0068] S412: Calculate the target value based on the reward value of each warehouse agent at each target time and the predicted value of the same warehouse agent at the next target time, to obtain the target value of each warehouse agent at each target time; calculate the error value based on the predicted value and target value of each warehouse agent at each target time, to obtain the error value of each warehouse agent at each time.

[0069] In this embodiment, the control device calculates the target value based on the reward value of each warehouse agent at each target time and the predicted value of the same warehouse agent at the next time step, thus obtaining the target value of each warehouse agent at each target time, as described below:

[0070] In the formula, For the first t The value of a goal at any given moment For the first t The reward value at each moment. This is the discount factor. For the first t +1 time-to-time predictive value.

[0071] The control equipment calculates the error value for each warehouse agent at each target time based on the predicted value and the target value of each agent at each target time. Specifically, the control equipment subtracts the target value and the predicted value of the same warehouse agent at the same time to obtain the error value for each warehouse agent at each time.

[0072] S413: Using gradient descent, the value network in the warehouse agent policy network is updated based on the error values ​​of each warehouse agent at each target time; using gradient ascent, the action policy network in the warehouse agent policy network is updated based on the error values ​​of each warehouse agent at each target time and the action probability distribution.

[0073] In this embodiment, the control device uses the gradient descent method to update the value network in the warehouse agent policy network based on the error values ​​of each warehouse agent at each target time.

[0074] Specifically, the control device calculates the mean squared error loss value based on the error values ​​of each warehouse agent at each time point, thus obtaining the mean squared error loss value corresponding to the warehouse agent. The control device uses gradient descent to update the value network in the warehouse agent's policy network based on the gradient of the mean squared error loss value corresponding to the warehouse agent, wherein the mean squared error loss value is:

[0075] In the formula, L This is the mean squared error loss value. B This represents the number of training samples, i.e., the total number of time points. The first intelligent agent for warehouses t The error value at each time point.

[0076] The control device uses the gradient ascent method to update the action policy network in the warehouse agent policy network based on the error value and action probability distribution of each warehouse agent at each target time.

[0077] Specifically, the control device employs the gradient ascent method to calculate the cumulative gradient of the action policy network based on the error values ​​and action probability distributions of each warehouse agent at each time point. This yields the cumulative gradient of the action policy network within the warehouse agent's policy network. The control device then updates the action policy network within the warehouse agent's policy network based on this cumulative gradient. The cumulative gradient is:

[0078] In the formula, For the first t The probability distribution of actions at each time point.

[0079] S414: Based on the value network and action policy network in the updated warehouse agent policy network, obtain the updated warehouse agent policy network.

[0080] In this embodiment, the control device obtains the updated warehouse agent policy network based on the value network and the action policy network in the updated warehouse agent policy network.

[0081] S42: Based on the quadruple data of each compressor agent at each target time, update the action policy network and value network in the compressor agent policy network to obtain the updated compressor agent policy network.

[0082] In this embodiment, the control device updates the action policy network and value network in the compressor agent policy network based on the quadruple data of each compressor agent at each target time, thereby obtaining the updated compressor agent policy network.

[0083] Please see Figure 7 , Figure 7 The flowchart of step S42 in the refrigeration control method for a cold storage provided in one embodiment of this application includes steps S421 to S424, as follows: S421: Input the quadruple data of each compressor agent at each target time into the action policy network and value network in the compressor agent policy network to obtain the action probability distribution and prediction value of each compressor agent at each target time. In this embodiment, the control device inputs the quadruple data of each compressor agent at each target time into the action policy network and the value network in the compressor agent policy network to obtain the action probability distribution and prediction value of each compressor agent at each target time.

[0084] S422: Calculate the target value based on the reward value of each compressor agent at each target time and the predicted value of the same compressor agent at the next time step of each target time, and obtain the target value of each compressor agent at each target time; the control device calculates the error value based on the predicted value and target value of each compressor agent at each target time, and obtains the error value of each compressor agent at each time step.

[0085] In this embodiment, the control device calculates the target value based on the reward value of each compressor agent at each target time and the predicted value of the same compressor agent at the next time step, thereby obtaining the target value of each compressor agent at each target time.

[0086] The control device calculates the error value of each compressor agent at each target time based on the predicted value and the target value of each target time, thus obtaining the error value of each compressor agent at each time. Specific implementation details can be found in step S412, and will not be repeated here.

[0087] S423: Using gradient descent, the value network in the compressor agent policy network is updated based on the error values ​​of each compressor agent at each target time; using gradient ascent, the action policy network in the compressor agent policy network is updated based on the error values ​​of each compressor agent at each target time and the action probability distribution.

[0088] In this embodiment, the control device uses the gradient descent method to update the value network in the compressor agent policy network based on the error values ​​of each compressor agent at each target time.

[0089] The control device employs a gradient ascent method to update the action policy network in the compressor agent policy network based on the error values ​​and action probability distributions of each compressor agent at each target time. Specific implementation details can be found in step S413, and will not be repeated here.

[0090] S424: Based on the value network and action policy network in the updated compressor agent policy network, obtain the updated compressor agent policy network.

[0091] In this embodiment, the control device obtains the updated compressor agent policy network based on the value network and the action policy network in the updated compressor agent policy network.

[0092] S43: Obtain the target agent refrigeration control network based on the updated warehouse agent policy network and the updated compressor agent policy network.

[0093] In this embodiment, the control device obtains the target agent refrigeration control network based on the updated warehouse agent policy network and the updated compressor agent policy network.

[0094] S5: Based on the state space data of each warehouse agent and the state space data of each compressor agent in the cold storage target agent refrigeration control network at the current time after the preset time period, execute the refrigeration control of the cold storage.

[0095] In this embodiment, the control device performs refrigeration control of the cold storage based on the state space data of each warehouse intelligent agent and the state space data of each compressor intelligent agent in the target intelligent agent refrigeration control network of the cold storage at the current time after the preset time period.

[0096] Please refer to Figure 8 , Figure 8 This is a schematic diagram of the structure of a refrigeration control device for a cold storage provided in one embodiment of this application. This device can be implemented in whole or in part through software, hardware, or a combination of both. The refrigeration control device 8 for the cold storage includes: The action reasoning module 81 is used to obtain the action space data of each warehouse intelligent agent and each compressor intelligent agent at each target time based on the state space data of each warehouse intelligent agent and the state space data of each compressor intelligent agent in the intelligent agent refrigeration control network of the cold storage at each target time in a preset time period. The reward calculation module 82 is used to perform refrigeration control and reward calculation based on the action space data of each warehouse intelligent agent and the action space data of each compressor intelligent agent at each target time, so as to obtain the reward value of each warehouse intelligent agent and the reward value of each compressor intelligent agent at each target time. The data processing module 83 is used to combine the state space data, action space data, reward value, and state space data of the next moment of the same warehouse agent and the same compressor agent at the target time to construct the quadruple data of each warehouse agent and each compressor agent at each target time. The network update module 84 is used to update the intelligent agent refrigeration control network based on the four-tuple data of each warehouse intelligent agent and the four-tuple data of each compressor intelligent agent at each target time, so as to obtain the target intelligent agent refrigeration control network. The refrigeration control module 85 is used to perform refrigeration control of the cold storage based on the state space data of each warehouse intelligent agent and the state space data of each compressor intelligent agent in the target intelligent agent refrigeration control network of the cold storage at the current time after the preset time period.

[0097] In this embodiment, the action reasoning module obtains the action space data of each warehouse intelligent agent and each compressor intelligent agent in the cold storage intelligent agent refrigeration control network at each target time within a preset time period. The reward calculation module performs refrigeration control and reward calculation based on the action space data of each warehouse intelligent agent and each compressor intelligent agent at each target time to obtain the reward value of each warehouse intelligent agent and each compressor intelligent agent at each target time. Finally, the data processing module processes the state space data of the same warehouse intelligent agent and the same compressor intelligent agent at the target time. The system combines state space data, action space data at the target time, reward value at the target time, and state space data at the next time step to construct four-tuple data for each warehouse agent and each compressor agent at each target time. Through the network update module, the system updates the agent refrigeration control network based on the four-tuple data for each warehouse agent and each compressor agent at each target time, thus obtaining the target agent refrigeration control network. Through the refrigeration control module, the system executes refrigeration control for the cold storage based on the state space data of each warehouse agent and each compressor agent in the target agent refrigeration control network of the cold storage at the current time step after the preset time period. By decoupling the refrigeration control task into multiple warehouse agents and multiple compressor agents, action reasoning and execution are performed based on the state space data of each warehouse agent and each compressor agent. The reward values ​​of each warehouse agent and each compressor agent are calculated after the action is executed. The warehouse agents and compressor agents are updated by combining the reward values ​​of each warehouse agent and each compressor agent after the action is executed, thereby improving the overall refrigeration efficiency and reducing the overall energy consumption.

[0098] Please refer to Figure 9 , Figure 9 This is a schematic diagram of the structure of a computer device provided in one embodiment of this application. The computer device 9 includes: a processor 91, a memory 92, and a computer program 93 stored in the memory 92 and executable on the processor 91; the computer device can store multiple instructions, which are adapted to be loaded and executed by the processor 91. Figures 1 to 7 For the method steps and specific execution process, please refer to [link / reference]. Figures 1 to 7 Specific details will not be elaborated here.

[0099] The processor 91 may include one or more processing cores. The processor 91 connects to various parts of the server using various interfaces and lines, and executes various functions and processes data of the cold storage's refrigeration control device 8 by running or executing instructions, programs, code sets, or instruction sets stored in the memory 92, and by calling data from the memory 92. Optionally, the processor 91 may be implemented using at least one hardware form of Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array (PLA). The processor 91 may integrate one or a combination of several of the following: a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), and a modem. The CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the content required to be displayed on the touch screen; and the modem handles wireless communication. It is understood that the modem may also not be integrated into the processor 91 and may be implemented as a separate chip.

[0100] The memory 92 may include random access memory (RAM) or read-only memory. Optionally, the memory 92 may include a non-transitory computer-readable storage medium. The memory 92 can be used to store instructions, programs, code, code sets, or instruction sets. The memory 92 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as touch instructions), instructions for implementing the various method embodiments described above, etc.; the data storage area may store data involved in the various method embodiments described above, etc. Optionally, the memory 92 may also be at least one storage device located remotely from the aforementioned processor 91.

[0101] This application embodiment also provides a storage medium that can store multiple instructions, which are adapted to be loaded and executed by a processor as described above. Figures 1 to 7 For the method steps and specific execution process, please refer to [link / reference]. Figures 1 to 7 Specific details will not be elaborated here.

[0102] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0103] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0104] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the algorithm. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0105] In the embodiments provided by this invention, it should be understood that the disclosed apparatus / terminal devices and methods can be implemented in other ways. For example, the apparatus / terminal device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0106] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0107] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0108] If the integrated module / unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of the present invention can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms.

[0109] This invention is not limited to the above-described embodiments. If any modifications or variations to this invention do not depart from the spirit and scope of this invention, and if such modifications and variations fall within the scope of the claims and equivalent technologies of this invention, then this invention also intends to include such modifications and variations.

Claims

1. A refrigeration control method for a cold storage facility, characterized in that, Includes the following steps: Based on the state space data of each warehouse intelligent agent and each compressor intelligent agent in the intelligent agent refrigeration control network of the cold storage at each target time within a preset time period, the action space data of each warehouse intelligent agent and each compressor intelligent agent at each target time are obtained; the target time is any time within the preset time period. Based on the motion space data of each warehouse agent and each compressor agent at each target time, refrigeration control and reward calculation are performed to obtain the reward value of each warehouse agent and each compressor agent at each target time. The state space data, action space data, reward value, and state space data of the next time step of the same warehouse agent and the same compressor agent at the target time are combined to construct the quadruple data of each warehouse agent and each compressor agent at each target time. Based on the quadruple data of each warehouse agent and each compressor agent at each target time, the agent refrigeration control network is updated to obtain the target agent refrigeration control network. Based on the state space data of each warehouse agent and each compressor agent in the cold storage target agent refrigeration control network at the current time after the preset time period, the refrigeration control of the cold storage is executed.

2. The refrigeration control method for cold storage according to claim 1, characterized in that: The step of performing refrigeration control and reward calculation based on the motion space data of each warehouse agent and each compressor agent at each target time, to obtain the reward value of each warehouse agent and each compressor agent at each target time, includes the following steps: Based on the motion space data of each warehouse agent and each compressor agent at each target time, refrigeration control is executed to obtain the refrigeration control execution index of the agent refrigeration control network at each target time. Rewards are calculated based on the refrigeration control execution indicators of the agent refrigeration control network at each target time, to obtain the reward values ​​of each warehouse agent and each compressor agent at each target time.

3. The refrigeration control method for cold storage according to claim 2, characterized in that: The cooling control execution indicators of the intelligent agent cooling control network include the total global power, the number of compressors that perform the start-up action, the current temperature of each warehouse intelligent agent, and the current suction pressure of each compressor intelligent agent. The step of calculating rewards based on the refrigeration control execution indicators of the agent refrigeration control network at each target time, to obtain the reward values ​​of each warehouse agent and each compressor agent at each target time, includes the following steps: Energy consumption cost and power over-limit penalty are calculated based on the global total power at each target time to obtain the energy consumption cost and power over-limit penalty value at each target time; compressor concurrent start-up penalty is calculated based on the number of compressors that perform the start-up action at each target time to obtain the compressor concurrent start-up penalty value at each target time; the energy consumption cost, power over-limit penalty value, and compressor concurrent start-up penalty value at each target time are summed to obtain the global reward value at each target time. Temperature penalty is calculated for each warehouse agent at each target time based on the current temperature of each warehouse agent and the target temperature of the warehouse, to obtain the temperature penalty value of each warehouse agent at each target time; reward value is calculated for each warehouse agent at each target time based on the global total power, global reward value, temperature penalty value of each warehouse agent, and the preset first reward calculation algorithm, to obtain the reward value of each warehouse agent at each target time.

4. The refrigeration control method for cold storage according to claim 3, characterized in that, The step of calculating rewards based on the refrigeration control execution indicators of the agent refrigeration control network at each target time, to obtain the reward values ​​of each warehouse agent and each compressor agent at each target time, includes the following steps: The suction pressure penalty is calculated based on the current suction pressure of each compressor agent at each target time and the target suction pressure of the compressor, to obtain the suction pressure penalty value of each compressor agent at each target time; the reward value is calculated based on the global reward value at each target time, the suction pressure penalty value of each compressor agent, and the preset second reward calculation algorithm, to obtain the reward value of each compressor agent at each target time.

5. The refrigeration control method for cold storage according to claim 1, characterized in that: All warehouse agents in the intelligent agent refrigeration control network call the same set of shared warehouse agent policy networks, and all compressor agents call the same set of shared compressor agent policy networks; both the warehouse agent policy networks and the compressor agent policy networks include action policy networks and value networks. The step of updating the agent refrigeration control network based on the four-tuple data of each warehouse agent and each compressor agent at each target time to obtain the target agent refrigeration control network includes the following steps: Based on the quadruple data of each warehouse agent at each target time, the action policy network and value network in the warehouse agent policy network are updated to obtain the updated warehouse agent policy network. Based on the quadruple data of each compressor agent at each target time, the action policy network and value network in the compressor agent policy network are updated to obtain the updated compressor agent policy network. Based on the updated warehouse agent policy network and the updated compressor agent policy network, the target agent refrigeration control network is obtained.

6. The refrigeration control method for cold storage according to claim 5, characterized in that, The step of updating the action policy network and value network in the warehouse agent policy network based on the four-tuple data of each warehouse agent at each time point to obtain the updated warehouse agent policy network includes the following steps: The quadruple data of each warehouse agent at each target time are input into the action policy network and value network in the warehouse agent policy network to obtain the action probability distribution and prediction value of each warehouse agent at each target time. The target value is calculated based on the reward value of each warehouse agent at each target time and the predicted value of the same warehouse agent at the next target time. The error value is calculated based on the predicted value and target value of each warehouse agent at each target time. The gradient descent method is used to update the value network in the warehouse agent policy network based on the error values ​​of each warehouse agent at each target time; the gradient ascent method is used to update the action policy network in the warehouse agent policy network based on the error values ​​of each warehouse agent at each target time and the action probability distribution. Based on the value network and action policy network in the updated warehouse agent policy network, the updated warehouse agent policy network is obtained.

7. The refrigeration control method for cold storage according to claim 5, characterized in that, The step of updating the action policy network and value network of the compressor agent policy network based on the four-tuple data of each compressor agent at each target time to obtain the updated compressor agent policy network includes the following steps: The quadruple data of each compressor agent at each target time are input into the action policy network and value network in the compressor agent policy network to obtain the action probability distribution and prediction value of each compressor agent at each target time. The target value is calculated based on the reward value of each compressor agent at each target time and the predicted value of the same compressor agent at the next time step. The error value is calculated based on the predicted value and the target value of each compressor agent at each target time. The gradient descent method is used to update the value network in the compressor agent policy network based on the error values ​​of each compressor agent at each target time. The gradient ascent method is used to update the action policy network in the compressor agent policy network based on the error value and action probability distribution of each compressor agent at each target time. Based on the value network and action policy network in the updated compressor agent policy network, the updated compressor agent policy network is obtained.

8. A refrigeration control device for a cold storage facility, characterized in that, include: The action reasoning module is used to obtain the action space data of each warehouse intelligent agent and each compressor intelligent agent at each target time based on the state space data of each warehouse intelligent agent and each compressor intelligent agent in the intelligent agent refrigeration control network of the cold storage at each target time in a preset time period. The reward calculation module is used to perform refrigeration control and reward calculation based on the action space data of each warehouse intelligent agent and each compressor intelligent agent at each target time, so as to obtain the reward value of each warehouse intelligent agent and each compressor intelligent agent at each target time. The data processing module is used to combine the state space data, action space data, reward value, and state space data of the next moment of the same warehouse agent and the same compressor agent at the target time to construct the quadruple data of each warehouse agent and each compressor agent at each target time. The network update module is used to update the intelligent agent refrigeration control network based on the four-tuple data of each warehouse intelligent agent and the four-tuple data of each compressor intelligent agent at each target time, so as to obtain the target intelligent agent refrigeration control network. The refrigeration control module is used to perform refrigeration control of the cold storage based on the state space data of each warehouse agent and the state space data of each compressor agent in the target agent refrigeration control network of the cold storage at the current time after the preset time period.

9. A computer device, characterized in that, It includes a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the refrigeration control method for a cold storage as described in any one of claims 1 to 7.

10. A storage medium, characterized in that: The storage medium stores a computer program, which, when executed by a processor, implements the steps of the refrigeration control method for the cold storage as described in any one of claims 1 to 7.

Images