Cold storage PID control parameter optimization method, device and equipment and storage medium

Abstract

The invention relates to the field of intelligent temperature control, and provides a refrigeration house PID control parameter optimization method and device, equipment and a storage medium. The method comprises the steps of collecting temperature data and energy consumption data of the refrigeration house at a constant-temperature stage; based on the temperature data and the energy consumption data, a mathematical model representing the mapping relation between the temperature and the energy consumption of the refrigeration house is established; constructing a deep reinforcement learning model based on a preset reward function and a mathematical model giving consideration to the temperature stability and energy consumption optimization of the refrigeration house; and through interactive learning of the intelligent agent of the deep reinforcement learning model and the refrigeration house environment, determining optimization parameters used for adjusting refrigeration house PID control. According to the refrigeration house PID control parameter optimization method, device and equipment and the storage medium provided by the invention, the mathematical model of the temperature and energy consumption mapping relation of the refrigeration house can be effectively established, the deep reinforcement learning model is constructed through the reward function and the mathematical model, and the optimization parameters of refrigeration house PID control are adjusted through deep reinforcement learning.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent temperature control, and in particular to a cold storage PID control parameter optimization method, device, equipment and storage medium. BACKGROUND

[0002] As a core node of cold chain logistics, the sustained stability of the internal temperature of the cold storage is the key to guaranteeing the quality of stored goods and reducing the loss rate. At present, a proportional-integral-derivative (PID) controller is generally used in industrial applications to implement cold storage temperature regulation. This control method corrects the temperature deviation through the synergistic effect of the proportional, integral and differential links to achieve closed-loop control.

[0003] In actual deployment, the parameters of the traditional PID controller are mostly set offline depending on engineering experience. Common methods include the critical proportion method and the response curve method. The critical oscillation parameters or step response curve of the system are obtained through experiments, and then the proportional coefficient, integral time and differential time are calculated according to the empirical formula. The parameters are fixed and unchanged during the running process after the parameter setting is completed. The above control strategy has the following significant technical defects: first, it is difficult to cope with multi-source time-varying thermal disturbances in the actual operation of the cold storage, such as heat leakage of the enclosure structure caused by the temperature difference between the inside and outside of the cold storage, additional heat sources generated by the respiration of goods such as fruits and vegetables, and transient heat load impact caused by frequent opening and closing of the cold storage door, as well as control lag introduced by the inertia of the refrigeration equipment start-stop, which leads to continuous fluctuations in the temperature in the cold storage during the constant temperature stage, threatening the quality and safety of the stored goods, and causing energy waste due to frequent adjustment of the refrigeration power. Second, the improved PID algorithms that have appeared in recent years are mostly designed for the whole process of cooling-constant temperature, and do not focus on the core demand of fluctuation suppression in the constant temperature stage. Moreover, the modeling mostly uses ideal simulation data and does not combine the dynamic disturbance characteristics of the actual working conditions, resulting in a large deviation between the model and the actual running state, making it difficult to achieve precise control. SUMMARY

[0004] The cold storage PID control parameter optimization method, device, equipment and storage medium provided by the embodiments of the present application can solve the technical problem of the fixed parameters of the traditional PID control in the constant temperature stage and improve the temperature stability in the constant temperature stage of the cold storage.

[0005] In a first aspect, the embodiments of the present application provide a cold storage PID control parameter optimization method, comprising: collecting temperature data and energy consumption data in the constant temperature stage of the cold storage; establishing a mathematical model representing the mapping relationship between the temperature and the energy consumption of the cold storage based on the temperature data and the energy consumption data; constructing a deep reinforcement learning model based on a preset reward function considering the temperature stability and energy consumption optimization of the cold storage and the mathematical model. The agent of the deep reinforcement learning model interacts with the cold storage environment to learn and determine the optimized parameters for adjusting the PID control of the cold storage.

[0006] In one embodiment, a deep reinforcement learning model is constructed based on a preset reward function and mathematical model that takes into account the temperature stability and energy consumption optimization of the cold storage, which can be implemented as follows: An environment interaction module is constructed based on the mathematical model, which is used to feedback the temperature dynamic changes and energy consumption of the cold storage under different PID control parameter inputs; An agent evaluation module is constructed based on the reward function; An agent learning module is constructed, which is used to learn the PID control parameter adjustment strategy suitable for multiple scenarios of the cold storage; The environment interaction module, agent evaluation module, and agent learning module are integrated to construct the deep reinforcement learning model.

[0007] In another embodiment, the above-mentioned environment interaction module is composed of a state space St and an action space At, which includes: The state space St is used to reflect the current running state of the cold storage; The state space St includes at least one of the temperature deviation, temperature deviation rate, thermal disturbance intensity estimate, refrigeration power, and cumulative energy consumption; The action space At refers to a set of control operations output to affect the cold storage environment; The action space At includes at least one of the proportional coefficient increment, integral coefficient increment, and differential coefficient increment of the cold storage PID.

[0008] In another embodiment, the above-mentioned reward function refers to a comprehensive value evaluation function composed of multiple single control target reward items combined through weight allocation; The reward function is the weighted sum of the temperature stability reward item, energy consumption reward item, and power smoothness reward item; The temperature stability reward item is a monotonic decreasing function of the absolute value of the current temperature deviation; The energy consumption reward item is a ratio function of the current cumulative energy consumption and the preset optimal energy consumption benchmark value; The power smoothness reward item is a monotonic decreasing function of the absolute value of the current refrigeration power change rate.

[0009] In another embodiment, the above-mentioned agent learning module includes an Actor network, a Critic network, a target Actor network, a target Critic network, and an experience replay pool; The Actor network is used to output a PID control parameter adjustment amount, the Critic network is used to evaluate an action value, the target Actor network is used to provide an action benchmark, the target Critic network is used to provide a value benchmark, and the experience replay pool is used to store interaction samples. The agent of the deep reinforcement learning model interacts with the cold storage environment to learn and determine the optimization parameters for adjusting the PID control of the cold storage, including: The preset iteration step is repeatedly executed to update the adjustment strategy of the PID control parameter until the reward corresponding to the PID control parameter adjustment amount obtained in the last iteration reaches a preset target value, and the PID controller parameter output by the Actor network parameter in the last iteration is taken as the optimization parameter for the PID control of the cold storage. The iteration step includes: Based on the next state space St+1 of the cold storage, the reward Rt and the transition samples stored in the experience replay pool obtained after the last iteration, the Critic network parameter and the Actor network parameter after the last iteration are updated to obtain the Critic network parameter and the Actor network parameter after the current iteration. The update process includes: updating the Critic network parameter in the current iteration by minimizing the time difference error, and updating the Actor network parameter in the current iteration in the direction of maximizing the expected cumulative reward based on the action value calculated by the Critic network output in the current iteration. After the update is completed, the parameters of the target Actor network and the target Critic network are synchronized by using a soft update strategy. Based on the Actor network parameter after the current iteration, the PID control parameter adjustment amount At is output for the current running state space St of the cold storage in the current iteration, and the PID controller parameter is updated based on At to generate a refrigeration power control signal u(t) and act on the cold storage to obtain the next state space St+1 of the cold storage and the reward Rt after the current iteration. The current running state of the cold storage used in the first iteration is the initial cold storage state space S0, and the initial PID control parameter is a preset initial value. The Critic network parameter and the Actor network parameter used in the first iteration are initialization parameters, and the action At output in the first iteration is the first PID control parameter adjustment amount output by the Actor network based on the initial state space S0.

[0010] In yet another embodiment, based on the temperature data and the energy consumption data, a mathematical model representing the mapping relationship between the temperature of the cold storage and the energy consumption is established, which can be implemented as follows: Based on the temperature data, a temperature transfer function representing the dynamic change rule of the temperature of the cold storage is constructed. The temperature transfer function is Laplace transformed to obtain a target temperature transfer function matched with the PID control; Based on the energy consumption data, an energy consumption function is constructed; Based on the target temperature transfer function and the energy consumption function, a mathematical model is constructed.

[0011] In a second aspect, an embodiment of the present application provides a cold storage PID control parameter optimization device, comprising: an acquisition module, a processing module, a construction module, and an optimization module.

[0012] The acquisition module is configured to: acquire temperature data and energy consumption data of the cold storage in a constant temperature stage. The processing module is configured to: based on the temperature data and the energy consumption data, establish a mathematical model representing a mapping relationship between the temperature and the energy consumption of the cold storage. The construction module is configured to: based on a preset reward function considering both the temperature stability and the energy consumption optimization of the cold storage and the mathematical model, construct a deep reinforcement learning model. The optimization module is configured to: through an agent of the deep reinforcement learning model and the cold storage environment, learn interactively to determine an optimization parameter for adjusting the PID control of the cold storage.

[0013] In a third aspect, an embodiment of the present application provides an electronic device, comprising a processor and a memory storing a computer program, and the processor implements the steps of the cold storage PID control parameter optimization method of the first aspect when executing the program.

[0014] In a fourth aspect, an embodiment of the present application provides a non-transitory computer readable storage medium, which stores a computer program, and the computer program, when executed by a processor, implements the steps of the cold storage PID control parameter optimization method of the first aspect.

[0015] In a fifth aspect, an embodiment of the present application provides a computer program product, comprising a computer program, and the computer program, when executed by a processor, implements the steps of the cold storage PID control parameter optimization method of the first aspect.

[0016] The cold storage PID control parameter optimization method, device, equipment, and storage medium provided by the present application can acquire temperature data and energy consumption data of the cold storage in a constant temperature stage in real time; based on the two types of data, a mathematical model accurately representing the mapping relationship between the temperature and the energy consumption is constructed; a deep reinforcement learning model is built in combination with a preset reward function considering both the temperature stability and the energy consumption optimization and the mathematical model; through an agent of the model and the dynamic environment of the cold storage, interactive learning is performed to dynamically output an optimization parameter for adjusting the PID control of the cold storage. The present application realizes intelligent dynamic setting of the PID parameter through the fusion of data driving and deep reinforcement learning, without human intervention, which not only improves the constant temperature precision of the cold storage, reduces temperature fluctuations, but also optimizes energy consumption configuration, and adapts to complex dynamic working conditions of the cold storage. BRIEF DESCRIPTION OF DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are some embodiments of the application, and other drawings can be obtained by those skilled in the art without creative effort on the basis of these drawings.

[0018] Figure 1 The flowchart of the cold storage PID control parameter optimization method provided by the embodiments of the application is shown in the figure. Figure 2 The structural diagram of the cold storage PID control parameter optimization device provided by the embodiments of the application is shown in the figure. Figure 3 The structural diagram of the electronic device provided by the embodiments of the application is shown in the figure. DETAILED DESCRIPTION

[0019] In order to make the purpose, technical solutions and advantages of the application clearer, the technical solutions in the application will be described clearly and completely in the following with reference to the drawings in the application. Obviously, the described embodiments are some embodiments of the application, not all the embodiments. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative effort belong to the protection scope of the application.

[0020] In the description of the application, it should be understood that the terms "upper", "lower", "left", "right", "front", "back", "inner", "outer" and the like indicate the orientation or positional relationship based on the orientation or relative position relationship shown in the drawings, and are only for the convenience of describing the application and simplifying the description, and do not indicate or imply that the device or element indicated must have a particular orientation, be constructed and operated in a particular orientation, therefore it cannot be understood as a limitation to the application. Unless otherwise specified, the above orientation description can be flexibly set in the process of actual application under the condition of meeting the relative position relationship shown in the drawings.

[0021] The terms "first", "second" are only for the purpose of description, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features limited by "first", "second" can explicitly or implicitly include one or more features. In the description of the application, unless otherwise specified, the meaning of "multiple" is two or more.

[0022] In the description of the present application, it is necessary to point out that, unless explicitly defined and limited, the terms "mounting", "connection", "linking", "communication" should be understood in a broad sense, for example, it can be fixed connection, or detachable connection, or integrally connected. It can be directly connected, or indirectly connected through an intermediate medium, or the communication inside two elements. For those skilled in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances.

[0023] In the embodiments of the present application, the terms "comprising", "containing" or any other variants thereof are intended to cover non-exclusive containing, so that the process, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed, or includes elements inherent to such process, article or device. Without more limitation, the element defined by the sentence "including a…" does not exclude the presence of other identical elements in the process, article or device including the element.

[0024] In the embodiments of the present application, the words such as "exemplary" or "for example" are used to represent an example, illustration or description. Any embodiment or design scheme described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as more preferred or more advantageous than other embodiments or design schemes. On the contrary, the words such as "exemplary" or "for example" are intended to present the relevant concept in a specific way.

[0025] In the embodiments of the present application, at least one can also be described as one or more, and the number can be two, three, four or more, which is not limited by the present application.

[0026] In the description of the present application, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0027] In order to facilitate understanding, the terms involved in the embodiments of the present application are first explained.

[0028] Proportional-Integral-Derivative (PID): It is an algorithm commonly used in industrial closed-loop control. Through the synergistic effect of proportional, integral and derivative three links, the deviation of controlled quantity and set value is corrected to realize accurate control.

[0029] Laplace transform: It is a mathematical transform that converts time domain functions into complex frequency domain functions, which can simplify the solution of differential equations and dynamic characteristic analysis of linear systems.

[0030] Figure 1 The flowchart of the cold storage PID control parameter optimization method provided in the embodiments of the present application is shown in FIG. 1.Figure 1 The embodiment of the application provides a cold storage PID control parameter optimization method, which can comprise the following steps: Step S101: collecting temperature data and energy consumption data of the cold storage constant temperature stage.

[0031] The temperature data refers to key correlation data capable of reflecting the internal temperature state of the cold storage, external environmental thermal disturbance and temperature stability.

[0032] Optionally, the temperature data comprises real-time temperature inside the cold storage (reflecting the actual temperature control state in the cold storage, and being a direct index for judging temperature deviation), external environment temperature of the cold storage (affecting the heat leakage intensity of the enclosure structure, and being a key parameter of external thermal disturbance), door opening and closing event data (recording the door opening and closing time and frequency, corresponding to instantaneous heat load impact, and being an important source of time-varying thermal disturbance), and the like, but is not limited thereto, and the embodiment of the application does not make a specific limitation in this regard.

[0033] The energy consumption data refers to characteristic data capable of reflecting the energy consumption of the refrigeration system for maintaining constant temperature and the correlation between energy consumption and temperature regulation.

[0034] Optionally, the energy consumption data comprises real-time refrigeration power of the refrigeration system (directly reflecting the energy consumption intensity of the refrigeration equipment, and being the basis for calculating total energy consumption), and fluctuation additional energy consumption basis data related to temperature deviation, but is not limited thereto, and the embodiment of the application does not make a specific limitation in this regard.

[0035] For example, the temperature data is collected by using an industrial-grade PT100 platinum resistance temperature sensor (measurement accuracy ±0.1℃, range -50℃~100℃), which is arranged at multiple storage areas in the cold storage and at a place outside the cold storage without direct sunlight. A door magnetic switch sensor (response ≤10ms) is arranged at the door. The temperature data is collected once per minute, and the door state is recorded in a triggered manner.

[0036] The energy consumption data is collected by using a Hall current sensor (±0.5% accuracy) and a voltage module (±0.2% accuracy), and the current and voltage are synchronously collected once per minute in series / parallel connection of the refrigeration main circuit.

[0037] Step S102: based on the temperature data and the energy consumption data, a mathematical model representing the mapping relationship between the temperature of the cold storage and the energy consumption is established.

[0038] The mathematical model refers to a temperature transfer function and energy consumption model coupling system constructed based on the temperature data and the energy consumption data collected in step S101, in combination with the law of conservation of energy, and is used for quantitatively describing the correlation rule between temperature dynamic change and energy consumption.

[0039] Specifically, the steps of constructing the mathematical model comprise the following steps: Step 1: based on the temperature data, a temperature transfer function representing the temperature dynamic change rule of the cold storage is constructed.

[0040] Specifically, based on the law of conservation of energy, a temperature transfer function is constructed in combination with temperature data, and a differential equation form of the temperature transfer function is as follows: wherein C represents an equivalent specific heat capacity inside the cold storage, represents a deviation of an actual temperature in the cold storage from a set temperature at t, represents a refrigeration power, represents a refrigeration coefficient, represents a refrigeration device start-stop inertia loss, represents a total heat disturbance heat at t.

[0041] It should be noted that the total heat disturbance heat is composed of envelope structure heat leakage, cargo respiration heat, periodic small heat disturbance and door opening disturbance.

[0042] Step 2: Laplace transform is performed on the temperature transfer function to obtain a target temperature transfer function matched with the PID control.

[0043] Specifically, the target temperature transfer function satisfies the following formula: wherein, represents a transfer function, represents a Laplace transform of the cold storage temperature deviation, represents a Laplace transform of the control input, K represents a system gain, and Ts represents a time constant.

[0044] Step 3: Based on the energy consumption data, an energy consumption function is constructed.

[0045] Specifically, the energy consumption function satisfies the following formula: wherein E represents total energy consumption, T represents constant temperature duration, u(t) represents a refrigeration power, represents a fluctuation additional energy consumption related to the temperature deviation.

[0046] Step 3: Based on the target temperature transfer function and the energy consumption function, a mathematical model is constructed.

[0047] The target temperature transfer function and the energy consumption function are coupled to form a mathematical model representing a temperature-energy consumption mapping relationship.

[0048] Step S103: Based on the preset reward function and the mathematical model which take into account the cold storage temperature stability and energy consumption optimization, a deep reinforcement learning model is constructed.

[0049] The reward function is a set of quantitative evaluation criteria that defines in mathematical form whether a certain operation of an intelligent agent (such as a temperature control decision system for cold storage) is excellent. The reward value (positive / negative, large / small) serves as a feedback signal to guide the model to learn the optimal control strategy that balances the temperature stability and energy consumption optimization of the cold storage.

[0050] In simple terms, the reward function is like a scoring rule for the model: if the temperature control operation performed by the model (such as adjusting the power of the refrigeration equipment, the start-stop frequency, the air supply speed, etc.) can make the cold storage temperature more stable and the energy consumption lower, it will be given a high positive reward; if it causes excessive temperature fluctuations (such as deviating too much from the set value) or wasted energy (such as the equipment running idle or over-cooling), it will be given a low reward or even a negative reward.

[0051] Specifically, the steps for building a deep reinforcement learning model include: Step 1: Construct an environment interaction module based on a mathematical model.

[0052] The environmental interaction module is used to provide feedback on the dynamic temperature changes and energy consumption of the cold storage under different PID control parameter inputs.

[0053] The environment interaction module consists of a state space St and an action space At, including: The state space St is used to reflect the current operating status of the cold storage.

[0054] Optionally, the state space St includes at least one of temperature deviation, temperature deviation rate of change, thermal disturbance intensity estimate, cooling power, and cumulative energy consumption.

[0055] Action space At refers to the set of control operations that affect the cold storage environment.

[0056] Optionally, the action space At includes at least one of the proportional coefficient increment, integral coefficient increment, and derivative coefficient increment of the cold storage PID.

[0057] Step 2: Construct an agent evaluation module based on the reward function.

[0058] A reward function is a comprehensive value assessment function that combines the reward items corresponding to multiple single control objectives by assigning weights.

[0059] The reward function is a weighted sum of temperature stability reward, energy consumption reward, and power smoothness reward.

[0060] Specifically, the reward function satisfies the following formula: in, This represents the total reward value at time t. This represents the weighting coefficient of the temperature stability reward item. represents a temperature stability reward term, represents a weight coefficient of an energy consumption reward term, represents an energy consumption reward term, represents a weight coefficient of a power smoothness reward term, represents a power smoothness reward term.

[0061] The temperature stability reward term refers to a monotonically decreasing function of an absolute value of a current temperature deviation; The energy consumption reward term refers to a ratio function of a current cumulative energy consumption and a preset optimal energy consumption reference value; The power smoothness reward term refers to a monotonically decreasing function of an absolute value of a current refrigeration power change rate.

[0062] Step 3: constructing an agent learning module.

[0063] The agent learning module is configured to learn a PID control parameter adjustment strategy suitable for multiple scenarios of the cold storage.

[0064] Specifically, the agent learning module includes an Actor network, a Critic network, a target Actor network, a target Critic network, and an experience replay pool.

[0065] The Actor network is configured to output a PID parameter adjustment amount At based on a current state space St.

[0066] The Critic network is configured to evaluate an action value Q(St, At) corresponding to the current state St and the action At.

[0067] The target Actor / target Critic network provides an action reference and a value reference, respectively, to avoid training oscillation.

[0068] The experience replay pool is configured to store transition samples of state St-action At-reward Rt-next state St+1, and is used for network parameter updating.

[0069] Step 3: integrating the environment interaction module, the agent evaluation module, and the agent learning module to construct a deep reinforcement learning model.

[0070] The environment interaction module, the agent evaluation module, and the agent learning module form a closed loop through a data interaction link to realize iterative optimization of state collection-action output-reward feedback-strategy updating.

[0071] Step S104: learning through the agent of the deep reinforcement learning model and the cold storage environment to determine an optimized parameter for adjusting the PID control of the cold storage.

[0072] Specifically, the adjustment strategy of the PID control parameter is updated by repeatedly performing the preset iteration step until the reward corresponding to the PID control parameter adjustment amount obtained in the last iteration reaches a preset target value, and the PID controller parameter output by the Actor network parameter in the last iteration is taken as the optimization parameter of the cold storage PID control.

[0073] The iteration step comprises: Based on the next state space St+1 of the cold storage, the reward Rt and the transition sample stored in the experience replay pool obtained after the last iteration, the Critic network parameter and the Actor network parameter after the last iteration are updated to obtain the Critic network parameter and the Actor network parameter after the current iteration.

[0074] The updating process comprises: updating the Critic network parameter in the current iteration by minimizing the time difference error, and updating the Actor network parameter in the current iteration in the direction of maximizing the expected cumulative reward based on the action value calculated by the Critic network in the current iteration.

[0075] After the updating is completed, the parameters of the target Actor network and the target Critic network are synchronized by using a soft updating strategy.

[0076] In the embodiments of the present application, the role of the target network is finally reflected in making the update of the PID control parameter more stable and efficient, the stable value evaluation of the target Critic network can more accurately feedback the pros and cons of the current PID parameter adjustment amount, and the direction of PID parameter optimization is avoided due to the fluctuation of the value estimation. The stable action generation of the target Actor network can ensure that the policy update of adjusting the PID parameter based on the current state is more reliable, and finally makes the PID control perform more robustly in the cold storage temperature-energy consumption collaborative optimization.

[0077] Based on the Actor network parameter after the current iteration, the PID control parameter adjustment amount At of the current running state space St of the cold storage is output, and the PID controller parameter is updated based on At to generate a refrigeration power control signal u(t) and act on the cold storage to obtain the next state space St+1 of the cold storage and the reward Rt after the current iteration.

[0078] The current running state of the cold storage used in the first iteration is the initial cold storage state space S0, and the initial PID control parameter is a preset initial value. The Critic network parameter and the Actor network parameter used in the first iteration are initialization parameters, and the action At output in the first iteration is the first PID control parameter adjustment amount output by the Actor network based on the initial state space S0.

[0079] The iteration step will be described in detail below.

[0080] Step 1: input the current state space St of the cold storage to the Actor network, and output the action space At, At represents the PID control parameter adjustment amount.

[0081] Step 2: update the current PID control parameters of the cold storage according to the PID control parameter adjustment amount At.

[0082] Step 3: generate a refrigeration power control signal u(t) based on the updated PID control parameters.

[0083] Step 3: collect feedback data of the cold storage under the action of the control signal u(t), and calculate the current reward Rt and the next state space St+1 based on the mathematical model and the reward function.

[0084] Step 5: store the transition sample (St, At, Rt, St+1) in the experience replay pool.

[0085] Step 6: randomly extract a batch of transition samples from the experience replay pool, and update the parameters of the Critic network by minimizing the time difference error.

[0086] Step 7: calculate the policy gradient based on the action value Q(St, At) output by the Critic network, and update the parameters of the Actor network using gradient ascent method.

[0087] Step 8: update the parameters of the target Actor network and the target Critic network by soft update strategy.

[0088] Step 9: iterative optimization: repeat steps 1-8 until the model converges (i.e. the cumulative reward value tends to be stable, and the control strategy meets the temperature control accuracy, energy consumption and operation smoothness requirements).

[0089] The cold storage PID control parameter optimization method provided by the embodiments of the present application adopts the Actor-Critic architecture (Actor-Critic Architecture, Actor-Critic): the Actor network is responsible for generating the optimal action according to the current state, outputting the current best PID parameter adjustment amount, and automatically determining the adjustment direction and amplitude of the PID parameter by learning the mapping relationship between the state and the action; while the Critic network is responsible for evaluating the value (Q value) of the action output by the Actor network, calculating the reward value corresponding to the current state-action pair according to the reward function, and feeding back the evaluation result to the Actor network for policy improvement.

[0090] In summary, the cold storage PID control parameter optimization method provided by the embodiments of the present application can collect temperature data and energy consumption data of the cold storage in the constant temperature stage in real time; a mathematical model accurately representing the mapping relationship between temperature and energy consumption is constructed based on the two types of data; a deep reinforcement learning model is built in combination with a preset reward function considering temperature stability and energy consumption optimization and the mathematical model; an agent of the model interacts with the dynamic environment of the cold storage to learn dynamically, and outputs optimization parameters for adjusting the PID control of the cold storage. The present application realizes intelligent dynamic setting of PID parameters through the fusion of data driving and deep reinforcement learning, without manual intervention, which not only improves the constant temperature accuracy of the cold storage, reduces temperature fluctuations, but also optimizes energy consumption configuration and adapts to complex dynamic working conditions of the cold storage.

[0091] The above mainly introduces the scheme provided by the present application. Correspondingly, the present application also provides a cold storage PID control parameter optimization device, which is used to implement the method embodiments described above.

[0092] The cold storage PID control parameter optimization device provided by the embodiments of the present application is described below, and the cold storage PID control parameter optimization device described below can be referred to the cold storage PID control parameter optimization method described above.

[0093] Figure 2 The structure diagram of the cold storage PID control parameter optimization device provided by the embodiments of the present application is shown in FIG. 1. As shown in FIG. 1, the present application provides a cold storage PID control parameter optimization device, which can include: Figure 2 The acquisition module 201 is configured to collect temperature data and energy consumption data of the cold storage in the constant temperature stage. The processing module 202 is configured to establish a mathematical model representing the mapping relationship between the temperature and the energy consumption of the cold storage based on the temperature data and the energy consumption data. The construction module 203 is configured to construct a deep reinforcement learning model based on a preset reward function considering the temperature stability and energy consumption optimization of the cold storage and the mathematical model. The optimization module 203 is configured to determine optimization parameters for adjusting the PID control of the cold storage by an agent of the deep reinforcement learning model interacting with the environment of the cold storage.

[0094] In one embodiment, the construction module 203 is specifically configured to construct an environment interaction module based on the mathematical model, and the environment interaction module is configured to feedback the temperature dynamic change and the energy consumption of the cold storage under different PID control parameter inputs. An agent evaluation module is constructed based on the reward function. An agent learning module is constructed, and the agent learning module is configured to learn a PID control parameter adjustment strategy adapting to multiple scenes of the cold storage. ​The environment interaction module, the agent evaluation module and the agent learning module are integrated to construct a deep reinforcement learning model.

[0095] In another embodiment, the environment interaction module comprises a state space St and an action space At, and includes: The state space St is used to reflect the current running state of the cold storage; The state space St includes at least one of the temperature deviation, the temperature deviation change rate, the thermal disturbance intensity estimate, the refrigeration power and the cumulative energy consumption; The action space At refers to a set of control operations output to affect the cold storage environment; The action space At includes at least one of the proportional coefficient increment, the integral coefficient increment and the differential coefficient increment of the cold storage PID.

[0096] In another embodiment, the reward function refers to a comprehensive value evaluation function composed of multiple single control target reward items combined by weight distribution; The reward function is a weighted sum of the temperature stability reward item, the energy consumption reward item and the power smoothness reward item; The temperature stability reward item is a monotonic decreasing function of the absolute value of the current temperature deviation; The energy consumption reward item is a ratio function of the current cumulative energy consumption and the preset optimal energy consumption reference value; The power smoothness reward item is a monotonic decreasing function of the absolute value of the current refrigeration power change rate.

[0097] In another embodiment, the agent learning module includes an Actor network, a Critic network, a target Actor network, a target Critic network and an experience replay pool; The Actor network is used to output the PID control parameter adjustment, the Critic network is used to evaluate the action value, the target Actor network is used to provide the action reference, the target Critic network is used to provide the value reference, and the experience replay pool is used to store the interaction samples; The agent of the deep reinforcement learning model interacts with the cold storage environment to learn and determine the optimization parameters for adjusting the PID control of the cold storage, including: The PID control parameter adjustment strategy is updated by repeatedly executing the preset iteration step until the reward corresponding to the PID control parameter adjustment obtained in the last iteration reaches the preset target value, and the PID controller parameter output based on the Actor network parameter in the last iteration is used as the optimization parameter for the PID control of the cold storage; The iteration step includes: Based on the next state space St+1 of the cold storage, the reward Rt and the transition samples stored in the experience replay pool after the last iteration, the Critic network parameters and the Actor network parameters after the last iteration are updated to obtain the Critic network parameters and the Actor network parameters after the current iteration; The updating process includes: updating the current Critic network parameters by minimizing the time difference error, and updating the current Actor network parameters in the direction of maximizing the expected cumulative reward based on the action value output by the current Critic network; After the updating is completed, the parameters of the target Actor network and the target Critic network are synchronized by using a soft updating strategy; Based on the Actor network parameters after the current iteration, the PID control parameter adjustment amount At of the current running state space St of the cold storage is output, and the PID controller parameters are updated based on At to generate a refrigeration power control signal u(t) and act on the cold storage to obtain the next state space St+1 of the cold storage and the reward Rt after the current iteration. The current running state of the cold storage used in the first iteration is the initial cold storage state space S0, the initial PID control parameters are preset initial values, the Critic network parameters and the Actor network parameters used in the first iteration are initialization parameters, and the action At output in the first iteration is the first PID control parameter adjustment amount output by the Actor network based on the initial state space S0.

[0098] In yet another embodiment, the processing module 202 is specifically configured to: based on the temperature data, construct a temperature transfer function representing the dynamic change rule of the temperature of the cold storage; Perform Laplace transform on the temperature transfer function to obtain a target temperature transfer function matched with the PID control; Based on the energy consumption data, construct an energy consumption function; Based on the target temperature transfer function and the energy consumption function, construct a mathematical model.

[0099] In some embodiments, the cold storage PID control parameter optimization device includes hardware structures and / or software modules corresponding to each function to achieve the above functions. Those skilled in the art should easily realize that the units and algorithm steps of the examples described in combination with the embodiments disclosed herein can be realized in the form of hardware or a combination of hardware and computer software. Whether a certain function is executed in hardware or computer software driven hardware depends on the specific application and design constraints of the technical solution. Professional technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0100] This application embodiment can divide the cold storage PID control parameter optimization device into functional modules according to the above method embodiment. For example, each function can be divided into its own functional module, or two or more functions can be integrated into a feature extraction module. The integrated module can be implemented in hardware or as a software functional module. It should be noted that the module division in this application embodiment is illustrative and only represents one logical functional division. In actual implementation, there may be other division methods.

[0101] Figure 3 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application, such as... Figure 3 As shown, the electronic device may include a processor 310, a communication interface 320, a memory 330, and a communication bus 340. The processor 310, communication interface 320, and memory 330 communicate with each other via the communication bus 340. The processor 310 can call logical instructions stored in the memory 330 to execute the steps of a cold storage PID control parameter optimization method. This method includes: collecting temperature data and energy consumption data during the constant temperature phase of the cold storage; establishing a mathematical model characterizing the mapping relationship between cold storage temperature and energy consumption based on the temperature and energy consumption data; constructing a deep reinforcement learning model based on a preset reward function and mathematical model that balances cold storage temperature stability and energy consumption optimization; and determining the optimization parameters for adjusting the cold storage PID control through interactive learning between the agent of the deep reinforcement learning model and the cold storage environment.

[0102] Furthermore, the logical instructions in the aforementioned memory 330 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0103] In another aspect, the embodiments of the present application also provide a computer program product, which comprises a computer program. The computer program can be stored in a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the steps of the cold storage PID control parameter optimization method provided by the above embodiments, for example, comprising: collecting temperature data and energy consumption data of the cold storage in a constant temperature stage; establishing a mathematical model representing the mapping relationship between the temperature and the energy consumption of the cold storage based on the temperature data and the energy consumption data; constructing a deep reinforcement learning model based on a preset reward function considering the temperature stability and energy consumption optimization of the cold storage and the mathematical model; and determining the optimization parameters for adjusting the PID control of the cold storage through the interaction learning between the agent of the deep reinforcement learning model and the cold storage environment.

[0104] In another aspect, the embodiments of the present application also provide a processor-readable storage medium, which stores a computer program. The computer program is used to make the processor execute the steps of the method provided by the above embodiments, for example, comprising: collecting temperature data and energy consumption data of the cold storage in a constant temperature stage; establishing a mathematical model representing the mapping relationship between the temperature and the energy consumption of the cold storage based on the temperature data and the energy consumption data; constructing a deep reinforcement learning model based on a preset reward function considering the temperature stability and energy consumption optimization of the cold storage and the mathematical model; and determining the optimization parameters for adjusting the PID control of the cold storage through the interaction learning between the agent of the deep reinforcement learning model and the cold storage environment.

[0105] The processor-readable storage medium can be any available medium or data storage device that the processor can access, including but not limited to a magnetic storage (such as a floppy disk, a hard disk, a magnetic tape, a magneto-optical disk (MO), etc.), an optical storage (such as a CD, a DVD, a BD, a HVD, etc.), and a semiconductor memory (such as a ROM, an EPROM, an EEPROM, a NAND FLASH, a solid state disk (SSD)), etc.

[0106] The device embodiments described above are only schematic, wherein the units shown as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, i.e., can be located in one place or distributed on multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the embodiments. Those skilled in the art can understand and implement without creative labor.

[0107] Those skilled in the art can clearly understand the implementation of the various embodiments by means of software and necessary general hardware platforms through the description of the above embodiments, and of course, the various embodiments can also be implemented by hardware. Based on such understanding, the above technical solutions, essentially or in other words, the part of the prior art that contributes to the technical solutions can be embodied in the form of a software product. The computer software product can be stored in a computer readable storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, and the like, and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods of the various embodiments or some parts of the embodiments.

[0108] Finally, it should be noted that: the above examples are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing examples, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing examples, or make equivalent replacement for some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the various embodiments of the present application.

Claims

1. A cold storage PID control parameter optimization method, characterized in that, The method comprises: Collecting temperature data and energy consumption data of the cold storage constant temperature stage; Based on the temperature data and the energy consumption data, a mathematical model representing the mapping relationship between the temperature of the cold storage and the energy consumption is established; Based on the preset reward function considering the temperature stability and energy consumption optimization of the cold storage and the mathematical model, a deep reinforcement learning model is constructed; An agent of the deep reinforcement learning model interacts with the cold storage environment to learn and determine the optimization parameters for adjusting the PID control of the cold storage.

2. The method of claim 1, wherein, The deep reinforcement learning model is constructed based on the preset reward function considering the temperature stability and energy consumption optimization of the cold storage and the mathematical model, comprising: Based on the mathematical model, an environment interaction module is constructed, which is used to feedback the temperature dynamic change and energy consumption of the cold storage under different PID control parameter inputs; Based on the reward function, an agent evaluation module is constructed; An agent learning module is constructed, which is used to learn the PID control parameter adjustment strategy suitable for the cold storage in multiple scenes; The environment interaction module, the agent evaluation module and the agent learning module are integrated to construct the deep reinforcement learning model.

3. The method of claim 2, wherein, The environment interaction module is composed of a state space St and an action space At, comprising: The state space St is used to reflect the current running state of the cold storage; The state space St includes at least one of temperature deviation, temperature deviation change rate, thermal disturbance intensity evaluation, refrigeration power and cumulative energy consumption; The action space At refers to a set of control operations output to affect the cold storage environment; The action space At includes at least one of the proportional coefficient increment, integral coefficient increment and differential coefficient increment of the cold storage PID.

4. The method of claim 2, wherein, The reward function is a comprehensive value evaluation function composed of multiple single control target reward items combined by weight distribution; The reward function is the weighted sum of temperature stability reward item, energy consumption reward item and power smoothness reward item; The temperature stability reward item is a monotonic decreasing function of the absolute value of the current temperature deviation; The energy consumption reward item is a ratio function of the current cumulative energy consumption and the preset optimal energy consumption benchmark value; The power smoothness reward item is a monotonic decreasing function of the absolute value of the current refrigeration power change rate.

5. The method of claim 2, wherein the PID control parameters are optimized for the cold storage. The agent learning module includes an Actor network, a Critic network, a target Actor network, a target Critic network and an experience replay pool; The Actor network is used to output the PID control parameter adjustment amount, the Critic network is used to evaluate the action value, the target Actor network is used to provide the action reference, the target Critic network is used to provide the value reference, and the experience replay pool is used to store interaction samples; The agent of the deep reinforcement learning model interacts with the cold storage environment to learn and determine the optimization parameters for adjusting the PID control of the cold storage, comprising: The preset iterative step is repeatedly performed to update the adjustment strategy of the PID control parameter until the reward corresponding to the PID control parameter adjustment amount obtained in the last iteration reaches a preset target value, and the PID controller parameter output by the Actor network parameter in the last iteration is taken as the optimization parameter of the cold storage PID control; The iterative step comprises: Based on the next state space St+1 of the cold storage, the reward Rt obtained after the last iteration, and the transition sample stored in the experience replay pool, the Critic network parameter and the Actor network parameter after the last iteration are updated to obtain the Critic network parameter and the Actor network parameter after the current iteration; Wherein, the updating process comprises: updating the current Critic network parameter by minimizing the time difference error, and then calculating the policy gradient based on the action value output by the current Critic network, and updating the current Actor network parameter in the direction of maximizing the expected cumulative reward; After updating, the parameters of the target Actor network and the target Critic network are synchronized by using a soft update strategy; Based on the Actor network parameter after the current iteration, the PID control parameter adjustment amount At of the current running state space St of the cold storage is output, and the PID controller parameter is updated based on the At to generate a refrigeration power control signal u(t) and act on the cold storage to obtain the next state space St+1 of the cold storage and the reward Rt after the current iteration. Wherein, the current running state of the cold storage used in the first iteration is the initial cold storage state space S0, the initial PID control parameter is the preset initial value, the Critic network parameter and the Actor network parameter used in the first iteration are the initialization parameters, and the action At output by the Actor network based on the initial state space S0 is the first PID control parameter adjustment amount.

6. The method of claim 1, wherein, The mathematical model representing the mapping relationship between the temperature and the energy consumption of the cold storage is established based on the temperature data and the energy consumption data, comprising: Based on the temperature data, a temperature transfer function representing the dynamic change rule of the temperature of the cold storage is constructed; The temperature transfer function is subjected to Laplace transformation to obtain a target temperature transfer function matched with the PID control; Based on the energy consumption data, an energy consumption function is constructed; Based on the target temperature transfer function and the energy consumption function, the mathematical model is constructed.

7. A cold storage PID control parameter optimization device, characterized in that, Comprise: The acquisition module is used for acquiring temperature data and energy consumption data in the constant temperature stage of the cold storage; The processing module is used for establishing a mathematical model representing the mapping relationship between the temperature and the energy consumption of the cold storage based on the temperature data and the energy consumption data; The construction module is used for constructing a deep reinforcement learning model based on a preset reward function considering the temperature stability and energy consumption optimization of the cold storage and the mathematical model; The optimization module is used for determining the optimization parameter for adjusting the PID control of the cold storage by interacting between the agent of the deep reinforcement learning model and the cold storage environment.

8. An electronic device comprising a processor and a memory having a computer program stored therein, characterized in that, The processor executes the computer program to realize the steps of the cold storage PID control parameter optimization method in any one of claims 1 to 6. 9.A non-transitory computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by a processor to implement the steps of the cold storage PID control parameter optimization method according to any one of claims 1 to 6.

10. A computer program product comprising a computer program, characterized in that, The computer program is executed by a processor to implement the steps of the cold storage PID control parameter optimization method according to any one of claims 1 to 6.

Images