An Energy-Saving Optimization Control Method for Low Dew Point Dual-Drum Dehumidifier Based on Offline Reinforcement Learning

Abstract

This invention discloses an energy-saving optimization control method for a low-dew-point dual-rotor dehumidifier based on offline reinforcement learning. The method includes: first, acquiring a raw dataset by deploying sensors on the low-dew-point dual-rotor dehumidifier; constructing a synthetic dataset based on the raw dataset; then, acquiring an integrated dataset using an offline experience replay method with adaptive dynamic adjustment of policy entropy; training a decision model for the low-dew-point dual-rotor dehumidifier based on the integrated dataset; and finally, deploying the trained decision model to the low-dew-point dual-rotor dehumidifier for control decision-making. A periodic training and update method is used to process the decision model, achieving continuous optimization of the low-dew-point dual-rotor dehumidifier. This invention reduces the model training's dependence on the raw dataset, enables the agent to adaptively adjust to the raw dataset's requirements, and reduces model complexity and computational resource consumption.

Description

Technical Field

[0001] This invention belongs to the field of dehumidification and energy-saving control technology, specifically involving an energy-saving optimization control method for a low dew point dual-rotor dehumidifier based on offline reinforcement learning. Background Technology

[0002] With the continuous improvement of industrial production processes and product quality, industries such as new energy, electronics, and pharmaceuticals are increasingly demanding strict requirements on environmental humidity in production and storage processes, leading to a growing demand for low dew point environment monitoring equipment. As a typical example of dehumidification equipment for low-humidity environments, the low dew point dual-rotor dehumidifier employs a series-connected two-stage dehumidification adsorption rotor and various stages of surface coolers and regenerative heaters connected by pipelines. Through processes such as cooling dehumidification, adsorption dehumidification, and regenerative heating, it ensures ultra-low dew point airflow at ambient temperature, achieving effective control of the low dew point environment in production workshops.

[0003] Low-dew-point dual-rotor dehumidifiers are complex in structure, exhibiting significant characteristics of nonlinearity, large inertia, and strong coupling. Current control strategies mostly employ PID control, requiring manual adjustment, leaving excessive control margins, and potentially leading to over-dehumidification. They lack refined, intelligent, and dynamic control methods. The dehumidification process in dual-rotor dehumidifier systems is typically energy-intensive, consuming large amounts of energy, including steam, cooling, and electricity, to maintain continuous dehumidification. Over-dehumidification translates to high production costs. Therefore, researching intelligent energy-saving control methods for low-dew-point dual-rotor dehumidifiers is crucial for achieving precise control of ambient humidity and optimizing the energy-efficient operation of dehumidifiers. This is significant for ensuring industrial production safety, saving production costs, and reducing carbon emissions in related processes. Summary of the Invention

[0004] To address the problems existing in the background technology, this invention provides an energy-saving optimization control method for low dew point dual-rotor dehumidifiers based on offline reinforcement learning, which solves the technical problems of high energy consumption of low dew point dual-rotor dehumidifiers and the lack of refined, intelligent and dynamic control methods in existing technologies.

[0005] The technical solution adopted in this invention includes:

[0006] I. An energy-saving optimization control method for low dew point dual-rotor dehumidifiers based on offline reinforcement learning:

[0007] S1. Sensors are deployed at key locations of the low dew point dual-rotor dehumidifier to acquire various data from the low dew point dual-rotor dehumidifier, and the original dataset of the low dew point dual-rotor dehumidifier is constructed based on all the acquired data.

[0008] S2. Store the original dataset in the constructed real playback buffer, construct a TimeDiff diffusion model for reinforcement learning based on the original dataset and perform data augmentation to obtain the augmented synthetic dataset.

[0009] S3. The offline experience replay method based on policy entropy adaptive dynamic adjustment is adopted to obtain the ratio of real data required from the original dataset for each round of training. Based on the ratio of real data, data is selected from the original dataset and the synthetic dataset to construct the integrated dataset required for each round of training.

[0010] S4. Construct a decision model for a low dew point dual-rotor dehumidifier. Input the integrated dataset required for each round of training into the decision model for training to obtain a trained decision model for a low dew point dual-rotor dehumidifier.

[0011] S5. Deploy the trained decision model of the low dew point dual-rotor dehumidifier to the low dew point dual-rotor dehumidifier for control decision-making. Use a periodic training and update method to process the decision model of the low dew point dual-rotor dehumidifier to achieve continuous optimization of the low dew point dual-rotor dehumidifier.

[0012] Step S1 specifically involves:

[0013] S11. Sensors are installed at key locations of the low dew point dual-rotor dehumidifier. Using empirical strategies or other equipment transfer strategies, sensors are deployed to the low dew point dual-rotor dehumidifier for control decision-making. Air state data at key locations is collected as environmental state data, and energy consumption data of the low dew point dual-rotor dehumidifier is collected. Temperature setting parameters of the air outlet of the internal control components are recorded as action data.

[0014] S12. Obtain the reward and termination status identifiers for each time step based on environmental status data, energy consumption data, action data, and the preset reward function.

[0015] S13. Based on the environmental state data of the current time step and the next time step, as well as the action data, reward, and termination status identifier of the current time step, construct the state transition tuple of the low dew point dual-rotor dehumidifier for the current time step, and use the state transition tuple of the current time step as a single time step data.

[0016] S14. Several consecutive time step data constitute a single dehumidifier continuous control data, and several dehumidifier continuous control data are summarized to obtain the original dataset of the low dew point dual-rotor dehumidifier.

[0017] The key locations include the fresh air inlet of the low dew point dual-rotor dehumidifier, the front surface cooler outlet, the fresh air rotor treatment outlet, the room return air outlet, the return air mixing outlet, the middle surface cooler outlet, the low humidity rotor treatment outlet, the room supply air outlet, the low humidity rotor cold air outlet, the low humidity regeneration heater outlet, the low humidity rotor regeneration exhaust outlet, the fresh air rotor cold air outlet, the regeneration fresh air outlet, the regeneration mixing outlet, the fresh air regeneration heater outlet, and the regeneration exhaust outlet.

[0018] The air condition data includes temperature, humidity, and wind speed; the internal control components of the equipment include a front surface cooler, a middle surface cooler, a low-humidity regenerative heater, and a fresh air regenerative heater.

[0019] The preset reward function is set according to the following formula:

[0020] R t =R episode ,Td t room >Td target ;R t =R continuous ,Td t room ≤Td target

[0021] R episode =R sparse +R auxialiary

[0022] R continuous =R maintain +β energy R energy

[0023] R sparse =R target ,Td t+1 room ≤Td target ;R sparse =0,Td t room >Td target

[0024] R auxialiary =R move ,Td t+1 room <Td t room ;R auxialiary =η episode R move ,Td t+1 room ≥Td t room

[0025] R maintain =R loop ,Td t+1 room ≤Td t room ;R maintain =η maintain R loop ,Td t+1 room ≤Td t room

[0026] R energy =Cost average –Cost t Cost t ≤Cost average ;R energy =η energy (Cost average –Cost t Cost t Cost average

[0027] Among them, R t R represents the reward function at time step t; episode The reward function for the scenario task indicating that the humidity in the controlled workshop is not within the target range and dehumidification treatment is required; R continuous R represents the reward function for a continuous task where the humidity in a controlled workshop is within the target range, requiring the maintenance of a balanced temperature and humidity environment. sparse This indicates sparse rewards for story missions; R auxialiary This indicates a supplementary reward for the plot mission; Td t room Td represents the controlled workshop dew point temperature at time step t. target Indicates the set target value for the dew point temperature of the controlled workshop; R maintain Indicates continuous mission sustain rewards; β energy R represents the energy consumption reward weighting coefficient; energy Indicates energy-saving rewards for continuous tasks; R target This indicates the preset plot mission objective reward; Td t+1 room R represents the controlled workshop dew point temperature at time step t+1; move Indicates the pre-set reward for advancing in the plot mission; η episode R represents the preset plot task penalty factor; loop Indicates the preset continuous task cycle reward; η maintain Cost represents the preset penalty factor for consecutive tasks. average Cost represents the average energy consumption of a low dew point dual-rotor dehumidifier per time step.t Indicates the energy consumption of the low dew point dual-rotor dehumidifier at time step t; η energy This represents the preset energy consumption penalty factor.

[0028] Step S2 specifically involves:

[0029] S21. Construct a real playback buffer and store the original dataset in the real playback buffer.

[0030] S22. Extract the corresponding state-action pair from each state transition tuple in the original dataset in the real playback buffer. Treat each state-action pair as a single-step original sequence. Several consecutive single-step original sequences constitute a state-action sequence. Divide each state-action sequence into a historical observation sequence and a target prediction sequence.

[0031] S23. Construct a TimeDiff diffusion model and train it using the divided state-action sequence to obtain the trained TimeDiff diffusion model.

[0032] S24. Each historical observation sequence is randomly perturbed and then input into the trained TimeDiff diffusion model to generate a corresponding target prediction sequence. The randomly perturbed historical observation sequence and the corresponding target prediction sequence are concatenated to obtain a synthetic state-action sequence. Each synthetic state-action sequence is then combined with the corresponding original state transition tuple to generate a corresponding synthetic state transition tuple. Several consecutive synthetic state transition tuples form a single synthetic dehumidifier continuous control data. Several synthetic dehumidifier continuous control data are summarized to obtain a synthetic dataset, and the synthetic dataset is stored in the constructed synthetic playback buffer.

[0033] The offline experience replay method based on policy entropy adaptive dynamic adjustment is as follows:

[0034] D1. Initialize the real data ratio, entropy temperature coefficient, and target entropy.

[0035] D2, N training sessions r After a certain number of time steps, the policy entropy of the reinforcement learning policy network is obtained.

[0036] D3. Update the target entropy according to the time decay strategy.

[0037] D4. Calculate the ratio of the actual data at the current time step using the following formula:

[0038] r real (t) =β ema r real (t-1) +(1-β ema )r clip(t)

[0039] r clip (t) =max(r min ,min(r max ,r^ real (t) ))

[0040] r^ real (t) =sigmoid(α r (H target –H(π)))

[0041] Where, r real (t) β represents the proportion of the actual data at the current time step t, indicating the percentage of data selected from the original dataset at time step t relative to the training batch samples; ema R represents the smoothing coefficient; real (t-1) This represents the ratio of the actual data at time step t-1 before the update; r clip (t) This indicates the ratio of the actual data to the estimated data after truncation at time step t; r max and r min These are the preset maximum and minimum values ​​for the actual data ratio used in the truncation operation; r^ real (t) α represents the ratio of the actual data to the estimated data at time step t; r H represents the entropy temperature coefficient; target H(π) represents the target entropy; H(π) represents the strategy entropy.

[0042] D5. If the policy entropy is continuously N T Each time step deviates from the target entropy, i.e., |H target -H(π)|>ξ T If the entropy temperature coefficient is not updated, then update the entropy temperature coefficient; otherwise, do not update the entropy temperature coefficient.

[0043] The updated entropy temperature coefficient is set according to the following formula:

[0044] α r (n+1) =α r (n) (1+η r sign(H target -H(π)))

[0045] Where, α r (n+1) The updated entropy temperature coefficient; α r (n) The entropy temperature coefficient before the update; η rξ is the learning rate for the entropy temperature coefficient; sign(·) is the sign function; T The threshold for adjusting the entropy temperature coefficient is used.

[0046] D6. Repeat steps D2-D5 to continuously obtain the true data ratio at each time step.

[0047] The decision model for the low dew point dual-rotor dehumidifier adopts an integrated diversified actor-critic algorithm based on a full covariance strategy. The policy network in the integrated diversified actor-critic algorithm based on a full covariance strategy adopts a spatiotemporal graph attention neural network, and both the Q-value network and the target Q-value network adopt multi-layer fully connected neural networks. The output of the policy network adopts a full covariance Gaussian strategy.

[0048] The improved integrated diverse actor-critic algorithm based on the full covariance strategy is implemented according to the following steps:

[0049] F1. Initialize the parameters of the policy network, Q-value network, and target Q-value network.

[0050] F2. Obtain the ratio of real data to r using the offline experience replay method based on policy entropy adaptive dynamic adjustment. real (t) According to real data, compared to r real (t) An offline ensemble dataset was obtained by batch sampling from the real playback buffer and the synthetic playback buffer for training.

[0051] F3. Calculate the target Q-value of the network.

[0052] F4. Based on the gradient descent strategy, first calculate the gradient of the parameters of each Q-value network according to the loss function, and then update the parameters of each Q-value network and the corresponding Q-value function according to the gradient.

[0053] F5. The policy network is updated every d steps based on the gradient descent strategy.

[0054] F6. Update the target Q-value network.

[0055] F7. Repeat steps F2-F6 for iterative training until the policy network converges.

[0056] The method for periodic training and updating is as follows:

[0057] H1. Real-time acquisition of air status data, energy consumption data, and setting parameters of internal control components of the low dew point dual-rotor dehumidifier within a preset time period, thereby obtaining the raw dataset within the preset time period.

[0058] H2. Based on the original dataset within the preset time period, obtain the integrated dataset within the preset time period using the same method as steps S2-S3.

[0059] H3. Input the integrated dataset within the preset time period into the trained low dew point dual-rotor dehumidifier decision model for retraining to obtain the retrained low dew point dual-rotor dehumidifier decision model.

[0060] H4. The low dew point dual-rotor dehumidifier makes control decisions in real time based on the newly completed decision model of the low dew point dual-rotor dehumidifier.

[0061] H5. Repeat steps H1-H4 according to the preset time interval to achieve periodic training and updating of the decision model of the low dew point dual-rotor dehumidifier.

[0062] II. A computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the above-described method.

[0063] The beneficial effects of this invention are:

[0064] 1. This invention uses the TimeDiff diffusion model to expand the original dataset, reducing the model training's dependence on the original dataset.

[0065] 2. This invention uses an offline experience replay method based on policy entropy adaptive dynamic adjustment to construct an integrated dataset, thereby enabling the agent to adaptively adjust to the original dataset according to its needs.

[0066] 3. The present invention adopts an integrated diversified actor-critic algorithm based on the improved full covariance strategy, which not only captures the spatiotemporal dependencies in the original dataset, but also overcomes the problem of overestimation of the Q-value of state-action pairs outside the distribution of the offline dataset by common offline reinforcement learning algorithms, and reduces the complexity of the model and the consumption of computational resources. Attached Figure Description

[0067] Figure 1 This is a flowchart of the method of the present invention.

[0068] Figure 2 This is a schematic diagram of the structure of the low dew point dual-rotor dehumidifier of the present invention.

[0069] Figure 3 This is an undirected graph showing the positional relationships of the low dew point dual-rotor dehumidifier structure of the present invention.

[0070] Figure 4 This is a structural diagram of the spatiotemporal graph attention neural network used in the strategy network of this invention. Detailed Implementation

[0071] The present invention will now be described in more detail with reference to the accompanying drawings and embodiments. However, the present invention is not limited thereto. For those skilled in the art, several improvements and modifications can be made without departing from the principles of the present invention, and these improvements and modifications are also considered to be within the scope of protection of the present invention. Contents not described in detail in this specification are prior art known to those skilled in the art.

[0072] The specific structure of the low dew point dual-rotor dehumidifier of this invention is as follows: Figure 2 and Figure 3 As shown, its structure includes: a fresh air impeller, a low-humidity impeller, a front surface cooler, a middle surface cooler, a rear surface cooler, a fresh air regeneration heater, a low-humidity regeneration heater, a rear heater, a primary filter, a medium-efficiency filter, a high-efficiency filter, a regenerated fresh air filter, a fresh air fan, a low-humidity fan, a regeneration fan, a differential pressure transmitter, valves at all levels, and air circulation ducts for connecting the various components, etc.

[0073] The fresh air decanter specifically includes a treatment zone, a cooling zone, and a regeneration zone. The air, after being cooled and dehumidified by the front surface cooling, is divided into two ducts. One duct passes through the fresh air treatment zone for adsorption and dehumidification and mixes with the room return air. It then passes through the two ducts for deep dehumidification via the middle surface cooling and is used for regeneration heating via the low-humidity decanter cooling zone. The other duct passes through the fresh air decanter cooling zone for heating and mixes with the low-humidity decanter regenerated exhaust air and regenerated fresh air. It is then heated by the fresh air regeneration heater and enters the fresh air decanter regeneration zone. The high-temperature regenerated mixed air desorbs moisture in the fresh air decanter regeneration zone, restoring the fresh air dehumidifier's dehumidification capacity. The resulting high-temperature, high-humidity regenerated exhaust air is discharged outdoors by the regeneration fan.

[0074] The low-humidity rotary dehumidifier specifically includes a treatment zone, a cooling zone, and a regeneration zone. The air dehumidified by the fresh air rotary dehumidifier is mixed with the room return air and then divided into two pipes. One pipe is cooled by the intermediate surface cooler and then enters the low-humidity rotary dehumidifier treatment zone for deep dehumidification. The air dehumidified by the low-humidity rotary dehumidifier is temperature-regulated by the post-cooler or post-heater and finally delivered to the room. The other pipe is heated by the low-humidity rotary dehumidifier cooling zone and then used as regeneration air to enter the low-humidity regeneration heater for regeneration heating. The high-temperature regeneration air after passing through the low-humidity regeneration heater desorbs moisture in the low-humidity rotary dehumidifier regeneration zone, restoring the dehumidifier's dehumidification capacity. The regeneration exhaust air from the low-humidity rotary dehumidifier regeneration zone is mixed with the regeneration fresh air and the low-humidity rotary dehumidifier cooling air and then entered into the fresh air regeneration heater for moisture desorption in the fresh air rotary dehumidifier regeneration zone.

[0075] The front, middle, and rear surface coolers are used to cool the air to be treated. The front surface cooler can cool and dehumidify air with excessive moisture content through temperature control. Specifically, each surface cooler is equipped with an electric regulating valve at the water outlet pipe, which can automatically adjust the water valve opening based on the set temperature, thereby controlling the outlet air temperature of the surface cooler.

[0076] The fresh air regeneration heater, low humidity regeneration heater, and post-heater are used to heat the air to be treated. Specifically, each heater is equipped with an electric regulating valve at the air inlet, which can automatically adjust the valve opening based on the set temperature, thereby controlling the heater outlet air temperature.

[0077] Furthermore, both the post-cooler and post-heater are installed in the air supply duct of the low dew point dual-rotor dehumidifier room. During dehumidifier operation, both are controlled by the same controller. Specifically, depending on the temperature of the air to be treated, the controller will automatically select either the post-cooler or the post-heater to process the supplied air based on the controlled workshop air temperature setpoint, thereby controlling the room supply air temperature. In actual dehumidifier operation, the controlled workshop air temperature setpoint is usually not changed arbitrarily; the set temperatures of the post-cooler and post-heater can be considered fixed values.

[0078] The fresh air fan, low-humidity fan, and regeneration fan are installed between the primary filter and the pre-cooler, the medium-efficiency filter and the intermediate cooler, and the regeneration exhaust duct and the fresh air impeller regeneration zone, respectively. Each fan is equipped with a frequency converter, which is used to control the fresh air volume, the room supply air volume, and the regeneration exhaust air volume. Each frequency converter is only used for on-site equipment commissioning and does not need to be reset during the actual operation of the dehumidifier.

[0079] In general, during the actual operation of a dehumidifier, the rotational speed of each impeller and fan, as well as the set temperature values ​​of the rear surface cooler and rear heater, can all be considered as fixed conditions. The internal control components used for air handling specifically include the front surface cooler, the middle surface cooler, the low-humidity regeneration heater, and the fresh air regeneration heater. The air volume handled at each point is mainly controlled by the pressure difference between the room air and the air in the duct, which is specifically manifested as the wind speed of the air being handled at each point.

[0080] like Figure 1 As shown, this embodiment is implemented according to the following steps:

[0081] S1. Sensors are deployed at key locations of the low dew point dual-rotor dehumidifier to acquire various data from the low dew point dual-rotor dehumidifier, and the original dataset of the low dew point dual-rotor dehumidifier is constructed based on all the acquired data.

[0082] S11. Sensors are installed at key locations of the low dew point dual-rotor dehumidifier. Using empirical strategies or other equipment transfer strategies, sensors are deployed to the low dew point dual-rotor dehumidifier for control decision-making. Air state data at key locations is collected as environmental state data, and energy consumption data of the low dew point dual-rotor dehumidifier is collected. Temperature setting parameters of the air outlet of the internal control components are recorded as action data.

[0083] The experience-based strategy is a decision-making method based on practical experience accumulated in the operation and control of low dew point dual-rotor dehumidifiers or similar equipment. The other equipment transfer strategy refers to a method of applying control strategies successfully applied to other similar equipment, after appropriate adjustments, to low dew point dual-rotor dehumidifiers.

[0084] The key locations include the fresh air inlet of the low dew point dual-rotor dehumidifier, the front surface cooler outlet, the fresh air rotor treatment outlet, the room return air outlet, the return air mixing outlet, the middle surface cooler outlet, the low humidity rotor treatment outlet, the room supply air outlet, the low humidity rotor cold air outlet, the low humidity regeneration heater outlet, the low humidity rotor regeneration exhaust outlet, the fresh air rotor cold air outlet, the regeneration fresh air outlet, the regeneration mixing outlet, the fresh air regeneration heater outlet, and the regeneration exhaust outlet.

[0085] In practical implementation, temperature, humidity, and wind speed sensors are installed at the fresh air inlet of the low dew point dual-rotor dehumidifier; temperature and humidity sensors are installed at the front surface cooler outlet; temperature and dew point temperature sensors are installed at the fresh air rotor treatment outlet; temperature, dew point temperature, and wind speed sensors are installed at the room return air inlet; temperature and dew point temperature sensors are installed at the return air mixing outlet; temperature and dew point temperature sensors are installed at the middle surface cooler outlet; temperature and dew point temperature sensors are installed at the low humidity rotor treatment outlet; and temperature and dew point temperature sensors are installed at the room supply air outlet. Temperature sensors and wind speed sensors are installed at the following locations: the cold air outlet of the low-humidity rotor, the air outlet of the low-humidity regeneration heater, the exhaust outlet of the low-humidity rotor regeneration system, the cold air outlet of the fresh air rotor, the fresh air inlet of the regeneration system, the mixed air outlet of the regeneration system, the air outlet of the fresh air regeneration heater, and the exhaust outlet of the regeneration system.

[0086] The air condition data includes temperature, humidity, and wind speed; the internal control components of the equipment include a front surface cooler, a middle surface cooler, a low-humidity regenerative heater, and a fresh air regenerative heater.

[0087] In practice, air state data is used as the environmental state s at time step t. t =[T n,t H n,t V n,t ], corresponding to the temperature, humidity, and wind speed of node n at time step t, respectively. Its state space S has a node dimension N=16 and a feature dimension F=3, where the humidity H n,tBecause the sensors involved are different, specifically relative humidity (RH) and dew point temperature (Td), for some locations where sensors are not installed due to pipeline limitations or insufficient measurement accuracy, wind speed (V) is also considered. n,t All values ​​are represented by a fixed value of 0. The specific data collected at time step t includes: fresh air temperature T1, fresh air humidity RH1, fresh air velocity V1, front surface cooler outlet air temperature T2, front surface cooler outlet air humidity RH2, fresh air rotary cooler outlet air temperature T3, fresh air rotary cooler outlet air dew point temperature Td3, room return air temperature T4, room return air dew point temperature Td4, room return air velocity V4, return air mixed air temperature T5, return air mixed air dew point temperature Td5, middle surface cooler outlet air temperature T6, middle surface cooler outlet air dew point temperature Td6, low humidity rotary cooler outlet air temperature T7, low humidity rotary cooler outlet air dew point temperature Td7, room supply air temperature T8, room supply air dew point temperature Td8, room supply air velocity V8, low humidity rotary cooler outlet air temperature T9, low humidity rotary cooler outlet air dew point temperature Td9, and low humidity regenerator outlet air temperature T... 10 Low humidity regenerative heater outlet dew point temperature Td 10 Low humidity rotor regeneration exhaust temperature T 11 Low humidity rotary regeneration exhaust air humidity RH 11 Fresh air rotor cold air outlet temperature T 12 Fresh air deflector, cold air outlet humidity (RH) 12 Regenerated fresh air temperature T 13 RH of regenerated fresh air 13 Regeneration air mixing temperature T 14 Regenerated mixed air humidity RH 14 Fresh air regeneration heater outlet air temperature T 15 Fresh air regeneration heater outlet air humidity RH 15 Regeneration exhaust temperature T 16 RH of regenerated exhaust air 16 Regeneration exhaust velocity V 16 As attached Figure 3 As shown.

[0088] The temperature setting parameter of the air outlet of the device's internal control component is used as the action data of the intelligent agent, specifically the intelligent agent's action a at time step t. t =[T c1,t ,T c2,t ,T h1,t ,T h2,t This includes the set temperature T of the surface cooler outlet air before time step t. c1 The set temperature value T of the outlet air of the intermediate surface cooler c2 Low humidity regenerative heater outlet air set temperature value T h1 , Fresh air regeneration heater outlet air set temperature value T h2 Each control component is subject to physical constraints and must operate within a safe temperature setting range, i.e., at ~[a max ,a min ], where a max =[T c1,t max ,T c2,t max ,T h1,t max ,T h2,t max ]、a min =[T c1,t min ,T c2,t min ,T h1,t min ,T h2,t min ] are the maximum and minimum value vectors of the action, respectively. The specific vector elements are the maximum and minimum value settings corresponding to each control component. The superscripts max and min represent the set maximum and minimum values, respectively. The dimension of the corresponding action space A is M=4.

[0089] S12. Obtain the reward and termination status identifiers for each time step based on environmental status data, energy consumption data, motion data, and a pre-designed reward function.

[0090] The pre-designed reward function is set according to the following formula:

[0091] R t =R episode ,Td t room >Td target ;R t =R continuous ,Td t room ≤Td target

[0092] R episode =R sparse +R auxialiary

[0093] R continuous =R maintain +β energy R energy

[0094] R sparse =R target ,Td t+1 room ≤Td target ;R sparse =0,Td t room >Td target

[0095] R auxialiary =R move ,Td t+1 room <Td t room ;R auxialiary =η episode R move ,Td t+1 room ≥Td t room

[0096] R maintain =R loop ,Td t+1 room ≤Td t room ;R maintain =η maintain R loop ,Td t+1 room ≤Td t room

[0097] R energy =Cost average –Cost t Cost t ≤Cost average ;R energy =η energy (Cost average –Cost t Cost t Cost average

[0098] Among them, R t R represents the reward function at time step t; episode The reward function for the scenario task, Td, indicates that the humidity in the controlled workshop is not within the target range and dehumidification is required. t room >Td target The plot task has a clearly defined decision termination state, and the specific decision objective is to find the termination state Td. episode room Satisfy Td episode room ≤Td target The optimal control strategy π* corresponding to the shortest time; R continuous Td represents the reward function for a continuous task where the humidity in a controlled workshop needs to be maintained within the target range to achieve a balance between temperature and humidity. t room ≤Tdtarget The continuous task does not have a clear termination state; the specific decision-making objective is to find the controlled workshop dew point temperature that satisfies the maximum value for the next Tt time steps. i=1 T-t Td t+i room ≤Td target At that time, the optimal control strategy π* corresponds to the lowest energy consumption cost; R sparse This indicates sparse rewards for story missions; R auxialiary This indicates a supplementary reward for the plot mission; Td t room Td represents the controlled workshop dew point temperature at time step t. target Indicates the set target value for the dew point temperature of the controlled workshop; R maintain Indicates continuous mission sustain rewards; β energy This represents the energy consumption reward weighting coefficient, and β energy >0;R energy Indicates energy-saving rewards for continuous tasks; R target This indicates the preset plot mission objective reward; Td t+1 room R represents the controlled workshop dew point temperature at time step t+1; move This indicates the pre-defined reward for advancing through the storyline missions, and R move >0;η episode Represents the plot task penalty factor, and η episode <0;R loop Indicates the preset continuous task cycle reward; η maintain Cost represents the penalty factor for consecutive tasks. average Cost represents the average energy consumption of a low dew point dual-rotor dehumidifier per time step. t Indicates the energy consumption of a low dew point dual-rotor dehumidifier at time step t; η energy Represents the energy consumption penalty factor, and η energy >0.

[0099] Rewards and termination status indicators for each time step d t The calculation process is as follows: the room return air dew point temperature Td4 at the current time step t is taken as the controlled workshop dew point temperature Td. t room Based on the controlled workshop dew point temperature Td t room The set target value Td of the dew point temperature in the controlled workshop target The relationship between them determines whether the current task is a phase task in the dehumidification stage or a continuous task in the maintenance stage, based on the controlled workshop dew point temperature Td at the next time step t+1. t+1 roomConfirm whether the state at the current time step t is a terminated state and generate the corresponding terminated state identifier d. t That is, if the current state is a terminated state, d t =1, otherwise d t =0. The corresponding reward function is selected based on the task type, taking into account the state at the current time step t, the state at the next time step t+1, and the current time step Cost. t Energy consumption generates reward r at current time step t t To simplify calculations, the Cost in the continuous task reward function is... average It is represented by the average energy consumption per time step of the first N consecutive tasks.

[0100] S13. Based on the environmental state data of the current time step and the next time step, as well as the action data, reward, and termination status identifier of the current time step, construct the state transition tuple of the low dew point dual-rotor dehumidifier for the current time step, and use the state transition tuple of the current time step as a single time step data.

[0101] The state transition tuple for a single time step is t ,a t ,r t ,s t+1 ,d t Specifically, it can be expressed as the environmental state data s of the agent at the current time step t. t Take action a t Receive reward r t And transfer to the next environmental state data. t+1 Generate termination status identifier d t To determine whether the task has been terminated.

[0102] S14. Several consecutive time step data constitute a single dehumidifier continuous control data, and several dehumidifier continuous control data are summarized to obtain the original dataset of the low dew point dual-rotor dehumidifier.

[0103] S2. Store the original dataset in the constructed real playback buffer, construct a TimeDiff diffusion model for reinforcement learning based on the original dataset and perform data augmentation to obtain the augmented synthetic dataset.

[0104] S21. Construct the real playback buffer D real The original dataset is stored in the actual playback buffer D. real In the middle. Actual playback buffer D real The data in the original dataset is also called real data.

[0105] In practice, the original dataset is stored in the database as real state transition tuples in the form of a real replay buffer D. real ​In this context, the real playback buffer D is used for subsequent reinforcement learning algorithm training. real A data structure for storing and managing real-world interaction data between reinforcement learning agents and their environment.

[0106] S22, from the real playback buffer D real Extract the corresponding state-action pair from each state transition tuple in the original dataset, treat each state-action pair as a single-step original sequence, and several consecutive single-step original sequences constitute a state-action sequence. Then, divide each state-action sequence into a historical observation sequence and a target prediction sequence.

[0107] In practical implementation, the single-step state transition tuple is represented as follows: t ,a t ,r t ,s t+1 ,d t > Extract state-action pairs from them t ,a t > and concatenate them into a single-step original vector x t 0 =concat(s t ,a t ), where concat(·) represents the concatenation operation.

[0108] The state-action sequence consists of L+H consecutive single-step original vectors, denoted as (x t-L+1 0 ,…,x t 0 ,…,x t+H 0 ), where L and H are the historical observation sequences x t-L+1:t 0 =(x t-L+1 0 ,…,x t 0 ) and the target prediction sequence x t+1:t+H 0 =(x t+1 0 ,…,x t+H 0 The length of ), where the superscript 0 corresponds to the diffusion step k in the diffusion model, and k=0 represents the real data or the synthetic data output by the TimeDiff diffusion model; for historical observation samples x i 0 ∈x t-L+1:t 0 Instance normalization is applied, i.e., the normalized value x in,i 0 =(x i 0 ​​-μ hist ) / σ hist , where μ hist σ hist These are historical observation sequences x t-L+1:t 0 The mean and standard deviation.

[0109] S23. Construct a TimeDiff diffusion model and train it using several divided state-action sequences to obtain a trained TimeDiff diffusion model.

[0110] In practice, the TimeDiff diffusion model is a type of diffusion model that incorporates conditional information to guide the generation process, formally expressed as p θ (x t+1:t+H 0 )=∫p θ (x t+1:t+H 0:K |c)dx t+1:t+H 1:K , where x t+1:t+H k Indicates the target prediction sequence x t+1:t+H 0 The corresponding latent variable sequence for the k-th diffusion step, x t+1:t+H 0:K This represents the complete diffusion trajectory consisting of K+1 sequences, where K is the preset maximum diffusion step; c is the condition vector, calculated and concatenated using historical observation sequences as the original condition information through two types of condition mechanisms: future mixing and autoregressive initialization; p θ (·|c) is a parameterized probabilistic model for a neural network, used to model the joint distribution of the observed sequence and the latent variable sequence given a conditional vector c. The integral operation represents the marginalization of the latent variable sequence, thereby obtaining the marginal distribution of the observed state, i.e., the prediction x of the target sequence in the next H steps. t+1:t+H 0 .

[0111] The TimeDiff model is used to model the state-action sequence, and its denoising generation process is performed by a neural network-parameterized target prediction sequence estimator x. θ (x t+1:t+H k The denoising network is implemented using the condition vector c, the diffusion step k, and the corresponding hidden variable sequence x. t+1:t+H k Perform the real sequence x t+1:t+H 0 It is estimated that the specific structure follows the encoder-decoder architecture adopted by the original model, and the initial denoised sample z obtained from the encoder output in the original model is used. denoisek The adaptive normalization layer AdaLN is used to extract self-conditional information, i.e., AdaLN(z) denoise k ,c)=c scale LayerNorm(z denoise k )+c shift Where LayerNorm(·) represents the normalization layer, c scale c shift The scaling factor and translation factor are calculated by inputting the condition vector c into two independent fully connected layers, respectively. Finally, the target prediction sequence estimate x is generated through the decoder structure. θ (x t+1:t+H k ,k|c), and update the latent variable sequence estimate x^ at diffusion step k-1. t+1:t+H k-1 =((1-α k-1 bar (α) k ) 1 / 2 ) / (1-α k bar ))x t+1:t+H k +((β k (α k-1 bar ) 1 / 2 ) / (1-α k bar ))x θ (x t+1:t+H k ,k|c)+σ k Ψ1, where α k The signal-to-noise ratio weight is defined as α. k =1–β k ,β k α is the preset diffusion step k noise variance coefficient. k bar The cumulative signal-to-noise ratio (SNR) weight for diffusion step k is defined as the SNR weight α for the forward k diffusion steps. k The product of σ k Let Ψ1 be the noise standard deviation corresponding to diffusion step k, and let Ψ1 be the random noise vector.

[0112] Specifically, the improvement of the TimeDiff diffusion model for reinforcement learning lies in defining the loss function during the training phase as L(θ)=E to guide the TimeDiff diffusion model to explore energy-saving action states within a safe action range. k [L k [θ], where the loss function L for the k-th diffusion step k (θ)=||xt+1:t+H 0 -x θ (x t+1:t+H k ,k|c)|| 2 Where ||·|| represents the L2 norm. When the diffusion step k=1, the denoising network needs to generate the predicted estimate output x^ in the output layer. t+1:t+H 0 Perform inverse instance normalization to ensure that the output conforms to the original sequence distribution, i.e., x. t+1:t+H 0 =σ hist x^ t+1:t+H 0 +μ hist And because the synthetic action is estimated as a est,t Due to the physical constraints of the equipment, a cutoff mechanism is introduced to ensure the output action a. t Within the safe range, i.e., a t =clip(a est,t ,a max ,a min ), where clip(·) represents the truncation operation, a max With a min Given the predefined maximum and minimum values ​​of the action in S11, the loss function L1(θ) = ||x t+1:t+H 0 -x θ (x t+1:t+H k ,k|c)|| 2 +L energy (θ), where the energy loss function L energy (θ)=λ1∑ i=1 H ((E(s t+i ,a t+i )-E opt ) / (E max -E min E(s) t+i ,a t+i The pre-trained energy consumption estimation model is used for the state action vector x at time step t+i. t+i 0 The energy consumption estimate is obtained by using a three-layer fully connected network structure in the pre-trained model to estimate the energy consumption of synthetic data based on state-action pairs, where E... opt The optimal energy consumption value for the historical dataset is specifically taken as the lowest 5th percentile, E max and E min λ1 represents the historical maximum and minimum energy consumption, H represents the weighting coefficient, and H represents the length of the target prediction sequence.

[0113] S24. Transfer the actual playback buffer Dreal Each historical observation sequence in the original dataset is randomly perturbed and input into the trained TimeDiff diffusion model to generate a corresponding target prediction sequence. The randomly perturbed historical observation sequence and the corresponding target prediction sequence are concatenated to obtain a synthetic state-action sequence. Each synthetic state-action sequence is then combined with its corresponding original state transition tuple to generate a corresponding synthetic state transition tuple. Several consecutive synthetic state transition tuples form a single synthetic dehumidifier continuous control data set. Several synthetic dehumidifier continuous control data sets are summarized to obtain a synthetic dataset, which is then stored in the constructed synthetic playback buffer D. synth .

[0114] In practice, the synthesized state transition tuple uses the environmental state data, actions, and next environmental state data from the synthesized state-action sequence, as well as the reward and termination state identifiers from the corresponding state transition tuple before synthesis.

[0115] In practice, data samples x from diffusion step K are randomly sampled from a pure Gaussian source. t+1:t+H K The denoising and generation process begins by randomly sampling historical observation sequences x from the original dataset. t-L+1:t 0 Random Gaussian noise Ψ~N(0,Σ) is sampled and superimposed onto the historical observation sequence to form a perturbed historical observation sequence, in order to simulate sensor measurement error, i.e., x in,t 0 =x t 0 +Ψ, where Σ represents the noise intensity corresponding to different sensor types, and the final result is based on the initial generation step input x. t+1:t+H K and conditional input x in,t-L+1:t 0 The target prediction sequence (future prediction fragment) is generated through a conditional denoising process. t+1:t+H 0 By concatenating the conditional inputs and the predicted outputs, a synthetic state-action sequence is obtained.

[0116] The construction of synthetic datasets specifically refers to constructing state transitions based on the continuous temporal relationships in the state-action sequences generated by the TimeDiff diffusion model. t ,a t ,s t+1 > When training with the TimeDiff diffusion model, the pre-trained energy consumption estimation model E(s) t ,a t Energy consumption (Cost) generated t And according to the reward function R t The corresponding reward r is generated according to the task termination rules. t and termination status identifier d​t To form a composite state transition tuple t ,a t ,r t ,s t+1 ,d t > Stored in the synthesis playback buffer D synth .

[0117] In practice, the obtained continuous control data of the synthetic dehumidifier is stored in the synthetic playback buffer D in the form of several consecutive synthetic state transition tuples. synth In the middle, the synthesis playback buffer D synth The data structure used to store and manage synthetic datasets is derived from the TimeDiff diffusion model trained on the original dataset, and the synthetic playback buffer D... synth There is no capacity limit, allowing for the storage of as much synthesized playback data as possible. If hardware resources are limited, a synthesized playback buffer D can be configured. synth To improve the model's ability to capture temporal patterns in data distribution within a limited capacity, the synthetic playback buffer D needs to be replaced with newly generated synthetic dehumidifier continuous control data under this limited capacity. synth It stores the earliest continuous control data of synthetic dehumidifiers.

[0118] S3. The offline experience replay method based on policy entropy adaptive dynamic adjustment is adopted to obtain the ratio of real data required from the original dataset for each round of training. Based on the ratio of real data, data is selected from the original dataset and the synthetic dataset to construct the integrated dataset required for each round of training.

[0119] The offline experience replay method based on policy entropy adaptive dynamic adjustment is as follows:

[0120] D1, Initialize the ratio of real data to r real Entropy temperature coefficient α r And the target entropy H of reinforcement learning target .

[0121] D2, N training sessions r After 1 time step, obtain the policy entropy H(π) = -E calculated by the reinforcement learning policy network. a~π(·|S) [logπ(a|s)]=(Mlog(2π)+∑ i=1 M logl ii +M) / 2, where N r The preset dynamic adjustment time step; π represents the strategy; E a~π(·|S) Let represent the expectation of an action; π(·|s) and π(a|s) represent the probabilities that the agent will take an action π and an action a given environmental state s, respectively; M usually represents the dimension of the action space A; l​ii Let represent the i-th diagonal element of the learnable lower triangular matrix L(s) obtained by parameterizing the policy distribution covariance matrix Σ(s) through Cholesky decomposition, where Σ(s) = L(s)L(s) T .

[0122] D3. Update the target entropy H according to the time decay strategy. target H target (t) =H target (t-1) e -βt , where e is a constant and β is a predefined target entropy decay coefficient. By introducing the target entropy decay coefficient, we can ensure that the agent's policy shifts from exploration to exploitation in the later stages of training, and finally achieve policy convergence.

[0123] D4. Calculate the ratio of the actual data at the current time step using the following formula:

[0124] r real (t) =β ema r real (t-1) +(1-β ema )r clip (t)

[0125] r clip (t) =max(r min ,min(r max ,r^ real (t) ))

[0126] r^ real (t) =sigmoid(α r (H target –H(π)))

[0127] Where, r real (t) β represents the proportion of data selected from the real playback buffer at time step t, relative to the total number of training batches acquired in each iteration. ema R represents the smoothing coefficient; real (t-1) r represents the ratio of the actual data at time step t-1 before the update; clip (t) This indicates the ratio of the actual data to the estimated data after time step t t; r max and r min These are the preset maximum and minimum values ​​for the actual data ratio used in the truncation operation; r^ real (t)α represents the ratio of the actual data to the estimated data at time step t; r H represents the entropy temperature coefficient, used to control the sensitivity of the real data comparison strategy to entropy changes; target H(π) represents the target entropy; H(π) represents the policy entropy.

[0128] Entropy temperature coefficient α r Apply the sigmoid function to compare the policy entropy H(π) at the current training time step with the target entropy H. target The difference between them maps to the true data before the update compared to r. real (t-1) The value range is (0,1), and the maximum value r of the predefined true data ratio is used. max and minimum value r min The output result is truncated to obtain r. clip (t) To avoid extreme values, and to prevent short-term fluctuations in the ratio of the actual data to the updated value from significantly impacting data quality, a moving average strategy is adopted to adjust the ratio of the actual data to the updated value in the output. real (t) Smoothing is performed, that is, according to the predefined smoothing coefficient β. ema Calculate the value r for the current training time step clip (t) Compared with the actual data before the update, r real (t-1) The weighted average is used to obtain the ratio of the actual data at the current time step to r. real (t) .

[0129] D5. If the strategy entropy H(π) is continuously N T Each time step deviates from the target entropy, i.e., |H target -H(π)|>ξ T Then, the entropy temperature coefficient α is adaptively updated. r (n+1) =α r (n) (1+η r ·sign(H target -H(π))), otherwise the entropy temperature coefficient is not updated; where α r (n+1) The updated entropy temperature coefficient is used as the entropy temperature coefficient for the next repeated step; α r (n) The entropy temperature coefficient before the update; η r ξ is the learning rate for the entropy temperature coefficient; sign(·) is the sign function; T The threshold for adjusting the entropy temperature coefficient; N TThe maximum time step is adjusted for the entropy temperature coefficient; adaptive adjustment rules are applied to ensure that the final goal and policy converge within a certain range, thereby improving policy stability and robustness.

[0130] D6. Repeat steps D2-D5 to continuously obtain the true data ratio at each time step and construct the ensemble dataset required for reinforcement learning training.

[0131] In specific implementation, the offline experience replay algorithm based on policy entropy adaptive adjustment, combined with the sampling mechanism of the experience replay algorithm applied to online reinforcement learning and the parameter dynamic adjustment strategy based on policy entropy, enables the agent to flexibly balance exploration and utilization, achieving full utilization of the limited real dataset while stably converging to a high-performance policy. Specifically:

[0132] Define the offline data playback buffer D off Data structures used for training reinforcement learning agents, storing data based on real data ratios r real Small batch sampling from the actual playback buffer D real With the synthesis playback buffer D synth The offline integrated dataset B={(s t ,a t ,r t ,s t+1 ,d t The samples used to train the agent's policy π are of batch size b, where b comes from the real playback buffer D. real For real data B real From the synthesis playback buffer D synth For synthetic data B synth That is, B=B real ∪B synth .

[0133] Real data compared to r real Defined as real data B selected from the real playback buffer. real The proportion of B samples in each training batch acquired, i.e., the size of the actual data, is b. real =r real b, the size of the synthesized data is b synth =(1-r real b, by dynamically controlling the ratio of the real data to r in each training batch B. real This is to balance the agent's tendency toward exploration and exploitation during the current training phase.

[0134] S4. Construct a decision model for a low dew point dual-rotor dehumidifier. Input the integrated dataset required for each round of training into the decision model for training until the decision model converges, and obtain the trained decision model for the low dew point dual-rotor dehumidifier.

[0135] The decision model for the low dew point dual-rotor dehumidifier adopts the Integrated Diversified Actor-Critic Algorithm (EDAC algorithm) based on the improved full covariance strategy. The policy network in the improved EDAC algorithm uses a spatiotemporal graph attention neural network, and both the Q-value network and the target Q-value network use multi-layer fully connected neural networks. The output of the policy network adopts the full covariance Gaussian strategy.

[0136] like Figure 4 As shown, the spatiotemporal graph attention neural network consists of a graph attention network, a bidirectional gated recurrent unit, a temporal Transformer network layer, and a multi-layer fully connected network for policy network output, connected sequentially. Specifically, the spatiotemporal graph attention neural network adopts the same network structure as the low dew point dual-rotor dehumidifier energy-saving prediction method based on spatiotemporal graph attention in patent publication number CN119557832A. The fully connected layer for specific output is used as the action Gaussian distribution modeling layer, adjusted to simultaneously output the mean μ(s) of the Gaussian distribution. t ) and the learnable lower triangular matrix L(s) t The dual-head output structure.

[0137] In practice, the actor network in the integrated diverse actor-critic network is the policy network, which is responsible for generating actions based on the current state, and the critic network is the Q-value network, which is used to evaluate the value of the state-action pair, i.e., the Q-value.

[0138] Specifically, the EDAC algorithm involves one policy network, N Q-value networks, and N target Q-value networks. All networks are parameterized using neural networks, denoted as θ, {φ} ... j} j=1 N 、{φ j tar} j=1 N , used to approximate the policy function π θ (a|s), Q-value function Q j (s,a) and the objective Q-value function Q j tar (s,a).

[0139] The policy network employs a spatiotemporal graph attention neural network, including a graph attention network, bidirectional gated recurrent units, and a temporal Transformer network layer. A multi-layer fully connected network is used for the actor network output to extract the spatial correlation, local temporal correlation, and global temporal correlation of the original data, and to output the data for action 'a'. t The sampled data are obtained from a multivariate Gaussian distribution, with mean μ and covariance Σ. The policy network input is X. in =[s t-T+1,…,s t-1 ,s t [ ], that is, the environmental state data s of the past T-1 time steps and the current time step. t The set of environment states is used to record the environment state data for the first T-1 time steps during each training session. For environment state data that cannot be obtained in the first T-1 time steps, the environment state data is assigned a value of 0. Note that the specific feature of the policy network structure used is to capture the spatiotemporal dependency characteristics of the policy, thereby helping the agent to better select action a. t .

[0140] The EDAC algorithm based on the full covariance policy improvement is specifically reflected in the improvement of the policy network output. It is typically applied to reinforcement learning methods in continuous action spaces, treating the action distribution π(a|s) as a multivariate Gaussian distribution and modeling it through neural network θ parameterization, i.e., π θ (a t |s t )=N(μ θ (s t ),Σ θ (s t )), where μ θ (s t ) and Σ θ (s t The mean μ(s) of the corresponding actions are respectively. t ) and variance Σ(s t ), and based on the current environmental state s t The action to be taken is obtained by randomly sampling from a Gaussian distribution. t , where a t ~N(μ θ (s t ),Σ θ (s t )), Σ θ (s t )=L θ (s t )L θ (s t ) T Existing EDAC algorithms employ a diagonal covariance Gaussian strategy, where the diagonal covariance matrix is ​​expressed as the variance σ of each action dimension. 2 (s) is a diagonal matrix composed of elements, ignoring the correlation between actions. Considering the high correlation between the dimensions of the action space mentioned in this embodiment, this invention improves the diagonal covariance Gaussian policy network into a full covariance Gaussian policy network, wherein the full covariance matrix Σ(s) is parameterized by Cholesky decomposition, i.e., Σ(s) = LL. T This yields a learnable lower triangular matrix L∈R M×M Its diagonal element lii The variance σ corresponding to each action dimension 2 (s), note that l needs to be ensured ii To ensure the positive definiteness of the lower triangular matrix L, the strategy entropy of the fully covariant Gaussian distribution is calculated as H(π) = (log(2πe)). M |Σ|) / 2, where |Σ| represents the determinant of the covariance matrix Σ, i.e., |Σ|=∏ i=1 M l ii The formula for calculating the corresponding strategy entropy simplifies to H(π) = (Mlog(2π) + ∑ i=1 M logl ii +M) / 2, where policy entropy reflects the uncertainty of an agent's action selection. Higher entropy indicates a greater tendency for the agent to explore, while lower entropy indicates a greater tendency to exploit. In this case, the action sampling is a. t =μ(s t )+L(s t Ψ2, where Ψ2 is the noise vector sampled from a standard multivariate Gaussian distribution.

[0141] Multiple Q-value networks employ a multi-layer fully connected neural network structure to output the Q-value estimate for each network. That is, the output layer of the neural network structure has only one unit, used to estimate the value corresponding to a given state-action pair. The Q-value networks have similar network structures, but their parameters are not shared.

[0142] The target Q-value network has the same neural network structure as the Q-value network, namely, a multi-layer fully connected neural network. The output layer also has only one unit, and the output is the target Q-value of the corresponding Q-value network. The neural network parameters of the target Q-value network are updated through a soft update mechanism to reduce fluctuations during training.

[0143] The advantage of the EDAC algorithm lies in its use of multiple Q-value networks to estimate the value of state-action pairs, thereby reducing the overestimation of out-of-distribution data. Each Q-value network has a loss function to encourage differentiation among multiple Q-value networks, reducing the number of Q-value networks required in practical applications and effectively reducing algorithm complexity and computational resource consumption.

[0144] The improved integrated diverse actor-critic algorithm based on the full covariance strategy is implemented according to the following steps:

[0145] F1. Initialize the policy network parameters θ and the Q-value network parameters {φ} j} j=1 N and target Q-value network parameters {φ j tar} j=1N .

[0146] F2. Obtain the ratio of real data to r using the offline experience replay method based on policy entropy adaptive dynamic adjustment. real (t) According to real data, compared to r real (t) From the real playback buffer D real and the synthesis playback buffer D synth Batch sampling offline ensemble dataset B={(s t ,a t ,r t ,s t+1 ,d t )} is used for training.

[0147] F3. Calculate the target Q-value of the network.

[0148] Specifically, the target Q value is y(r t ,s t+1 )=r t +γ(min j=1,…,N Q j tar (s t+1 ,a t+1 )-β en H t+1 (π)), H t+1 (π)=logπ θ (a t+1 |s t+1 ); where the next action a t+1 Sampled from policy network π θ (·|s t+1 ), that is, a t+1 =μ θ (s t+1 )+L(s t+1 )Ω, μ θ (s t+1 ) represents the policy network π θ (·|s t+1 The mean function of ), L(s) t+1 ) represents the covariance matrix of the policy distribution Σ(s t+1 The learnable lower triangular matrix obtained by Cholesky decomposition parameterization; Ω represents a random variable sampled from the standard normal distribution (0, I); β en y(r) is the entropy regularization coefficient. t ,s t+1 Let be the target Q value, representing the value in the current environmental state s. t Next, execute action a t Then, from the next environmental state s t+1Initial expected cumulative discount reward; r t Indicates the current environmental state s t Next, execute action a t The immediate reward obtained afterward; γ represents the discount factor; Q j tar (s t+1 ,a t+1 ) represents the Q-value of the j-th objective network in response to the next environmental state s. t+1 and the next action a t+1 Q-value estimation; H t+1 (π) represents the strategy π in environment state s. t+1 Take action a t+1 Entropy; π θ (a t+1 |s t+1 ) represents the policy network π θ In environmental state s t+1 Take action a t+1 The probability of.

[0149] F4. Based on the gradient descent strategy, first process the parameters φ of each Q-value network. j Solve for the gradient, and then update the parameters φ of each Q-value network based on the gradient. j and the corresponding Q-value function Q j .

[0150] Specifically, the gradient is:

[0151] ▽ φ_j (1 / |B|)∑ st∈B ((Q j (s t ,a t )-y(r t ,s t+1 )) 2 +η∑ i=1 |A| ∑ j=1 |A| (d 2 Q bar / (da i da j )) 2 ), Q bar =(1 / N)∑ j=1 N Q j ,▽ φ_j To adjust the neural network parameters φ j Find the gradient; η is the Hessian regularization coefficient; ∑ i=1 |A| ∑ j=1 |A| (d2 Q bar / (da i da j )) 2 For Hessian regularization terms, where d 2 Q bar Indicates Q bar The second-order partial derivative; da i and da j They represent the values ​​for a respectively i and a j First-order partial derivative; Q j (s t ,a t ) represents the j-th Q-value network response to environment state s. t and action a t Q-value estimation; (Q j (s t ,a t )-y(r t ,s t+1 )) 2 is the mean squared error loss term; |A| represents the dimension of the action space; B represents the offline ensemble dataset; |B| represents the number of samples in the offline ensemble dataset.

[0152] F5. Update the policy network π every d steps based on the gradient d d strategy. θ .

[0153] Specifically, the gradient is ▽ θ (1 / |B|)∑ s∈B (min j=1,…,N Q j (s,a θ (s))-β en H a_θ(s) (π)), for the states s involved in the offline ensemble dataset B, sampled from the policy network π θ The action a of (·|s) θ The policy entropy corresponding to (s) is H a_θ(s) (π)=logπ θ (a θ (s)|s);Q j (s,a θ (s) represents the j-th Q-value network's relationship to the environment state s and action a. θ Q-value estimation of (s).

[0154] F6. Update the target Q-value network.

[0155] Specifically, the target Q-value network is φ j tar,(n+1) =ρφ jtar,(n) +(1-ρ)φ j The hyperparameter ρ is used to control the update speed of the target network; φ j tar,(n) With φ j tar,(n+1) φ represents the parameters of the network for the j-th objective Q-value before and after the update, respectively; j This represents the parameters of the j-th Q-value network.

[0156] F7. Repeat steps F2-F6 for iterative training until the policy network based on agent behavior converges.

[0157] S5. Deploy the trained decision model of the low dew point dual-rotor dehumidifier to the low dew point dual-rotor dehumidifier for control decision-making. Use a periodic training and update method to process the decision model of the low dew point dual-rotor dehumidifier to achieve continuous optimization of the low dew point dual-rotor dehumidifier.

[0158] The method for periodic training and updating is as follows:

[0159] H1. Real-time acquisition of air status data, energy consumption data, and setting parameters of internal control components of the low dew point dual-rotor dehumidifier within a preset time period, thereby obtaining the raw dataset within the preset time period.

[0160] H2. Based on the original dataset within the preset time period, obtain the integrated dataset within the preset time period using the same method as steps S2-S3.

[0161] H3. Input the integrated dataset within the preset time period into the trained low dew point dual-rotor dehumidifier decision model for retraining to obtain the retrained low dew point dual-rotor dehumidifier decision model.

[0162] H4. The low dew point dual-rotor dehumidifier makes control decisions in real time based on the newly completed decision model of the low dew point dual-rotor dehumidifier.

[0163] H5. Repeat steps H1-H4 according to the preset time interval to achieve periodic training and updating of the decision model of the low dew point dual-rotor dehumidifier.

[0164] This invention combines the specific structural features and working objectives of a low dew point dual-rotor dehumidifier. It employs an offline reinforcement learning method to reduce the impact of agent training on the operation of the low dew point dual-rotor dehumidifier. Based on a periodic training and update approach, it achieves continuous optimization of the working strategy of the low dew point dual-rotor dehumidifier over time, thereby improving the energy-saving performance of the low dew point dual-rotor dehumidifier.

[0165] The above embodiments are merely preferred embodiments provided to fully illustrate the present invention, and the scope of protection of the present invention is not limited thereto. Equivalent substitutions or modifications made by those skilled in the art based on the present invention are all within the scope of protection of the present invention. The scope of protection of the present invention is defined by the claims.

Claims

1. A low dew point double rotary dehumidifier energy saving optimization control method based on offline reinforcement learning, characterized in that, The method comprises the following steps: S1, sensors are arranged at key positions of the low-dew-point double-rotor dehumidifier to obtain various data of the low-dew-point double-rotor dehumidifier, and an original data set of the low-dew-point double-rotor dehumidifier is constructed according to all the obtained data; S2, the original data set is stored in a constructed real playback buffer, a TimeDiff diffusion model oriented to reinforcement learning is constructed according to the original data set, and data enhancement is performed to obtain an enhanced synthetic data set; S3, an offline experience playback method based on adaptive dynamic adjustment of strategy entropy is adopted to obtain a real data ratio required by each round of training from the original data set, and data is selected from the original data set and the synthetic data set according to the real data ratio to construct an integrated data set required by each round of training; The offline experience playback method based on adaptive dynamic adjustment of strategy entropy is specifically: D1, initializing a real data ratio, an entropy temperature coefficient and a target entropy; D2, every N r After each training N r time step, obtain the policy entropy calculated by the reinforcement learning policy network. D3, updating the target entropy according to a time decay strategy; D4, calculating the real data ratio of the current time step according to the following formula: r real (t) =β ema r real (t-1) +(1-β ema )r clip (t) r clip (t) =max(r min ,min(r max ,r^ real (t) )) r^ real (t) =sigmoid(α r (H target –H(π))) where r real (t) is the real data ratio at the current time step t, indicating the proportion of data selected from the original data set in the training batch sample at time step t; β ema is the smoothing coefficient; r real (t-1) is the real data ratio at time step t-1 before updating; r clip (t) is the truncated real data ratio estimate at time step t; r max and r min are the preset maximum and minimum values of the real data ratio for the truncation operation; r^ real (t) is the real data ratio estimate at time step t; α r is the entropy temperature coefficient; H target is the target entropy; H(π) is the policy entropy; D5、if the policy entropy deviates from the target entropy at consecutive N T time steps, i.e., |H target -H(π)|>ξ T then update the entropy temperature coefficient, otherwise do not update the entropy temperature coefficient; The updated entropy temperature coefficient is set according to the following formula: a r (n+1) = a r (n) (1 + η r sign(H target - H(π)) wherein, α r (n+1) is the updated entropy temperature coefficient; α r (n) is the updated entropy temperature coefficient; η r is the entropy temperature coefficient learning rate; sign(·) is the sign function; ξ T is the entropy temperature coefficient adjustment threshold; D6, repeating steps D2-D5 to continuously obtain the real data ratio of each time step; S4, constructing a low-dew-point double-rotor dehumidifier decision model, inputting the integrated data set required by each round of training into the low-dew-point double-rotor dehumidifier decision model for training, and obtaining a trained low-dew-point double-rotor dehumidifier decision model; S5, deploying the trained low-dew-point double-rotor dehumidifier decision model to the low-dew-point double-rotor dehumidifier for control decision, and periodically updating the low-dew-point double-rotor dehumidifier decision model to realize continuous optimization of the low-dew-point double-rotor dehumidifier.

2. The low dew point dual rotary dehumidifier energy optimization control method of claim 1, wherein, Step S1 is specifically: S11, sensors are arranged at key positions of the low-dew-point double-rotor dehumidifier, an experience strategy or other device transplantation strategy is deployed into the low-dew-point double-rotor dehumidifier for control decision, air state data at the key positions are collected as environmental state data, energy consumption data of the low-dew-point double-rotor dehumidifier is collected, and temperature setting parameters of the air outlet of the internal control components of the device are recorded as action data; S12, rewards and termination state identifiers of each time step are obtained according to the environmental state data, the energy consumption data, the action data and a preset reward function; S13, a state transition tuple of the low-dew-point double-rotor dehumidifier at the current time step is constructed according to the environmental state data of the current time step and the next time step and the action data, the reward and the termination state identifier of the current time step, and the state transition tuple at the current time step is taken as single time step data; S14, a plurality of continuous time step data form a single dehumidifier continuous control data, and a plurality of dehumidifier continuous control data are aggregated to obtain the original data set of the low-dew-point double-rotor dehumidifier.

3. The low-dew-point double-rotor dehumidifier energy-saving optimization control method according to claim 2, characterized in that: The key positions include a fresh air inlet of the low-dew-point double-rotor dehumidifier, a front surface cooling outlet, a fresh air rotor processing outlet, a room return air outlet, a return air mixing outlet, a middle surface cooling outlet, a low-humidity rotor processing outlet, a room air supply outlet, a low-humidity rotor cold-blowing outlet, a low-humidity regeneration heater outlet, a low-humidity rotor regeneration exhaust outlet, a fresh air rotor cold-blowing outlet, a regeneration fresh air outlet, a regeneration mixing outlet, a fresh air regeneration heater outlet, and a regeneration exhaust outlet.

4. The low-dew-point double-rotor dehumidifier energy-saving optimization control method according to claim 2, wherein the air state data comprises temperature, humidity, and air speed; and the internal device control components comprise a front surface cooler, a middle surface cooler, a low-humidity regeneration heater, and a fresh air regeneration heater.

5. The low-dew-point double-rotor dehumidifier energy-saving optimization control method according to claim 2, wherein the preset reward function is set according to the following formula: The step S2 is specifically: S21, constructing a real playback buffer and storing the original data set in the real playback buffer; R t =R episode ,Td t room >Td target ;R t =R continuous ,Td t room ≤Td target R episode = R sparse + R auxialiary R continuous = R maintain + β energy R energy R sparse =0,Td target ,Td t+1 room ≤Td target ;R sparse =0,Td t room >Td target R auxialiary = R move , Td t+1 room <Td t room ; R auxialiary =η episode R move ,Td t+1 room ≥Td t room R maintain = R loop , Td t+1 room ≤ Td t room ; R maintain =η maintain R loop ,Td t+1 room ≤Td t room R energy = Cost average – Cost t , Cost t ≤ Cost average ; R energy = η energy (Cost average – Cost t ), Cost t > Cost average wherein R t represents the reward function of time step t; R episode represents the episodic task reward function for the scenario that the controlled workshop humidity is not in the target interval and needs to be dehumidified; R continuous represents the continuous task reward function for the scenario that the controlled workshop humidity is in the target interval and needs to maintain the balance of workshop temperature and humidity; R sparse represents the episodic task sparse reward; R auxialiary represents the episodic task auxiliary reward; Td t room represents the controlled workshop dew point temperature of time step t; Td target represents the set target value of the controlled workshop dew point temperature; R maintain represents the continuous task maintenance reward; β energy represents the energy consumption reward weight coefficient; R energy represents the continuous task energy saving reward; R target represents the preset episodic task target reward; Td t+1 room represents the controlled workshop dew point temperature of time step t+1; R move represents the preset episodic task forward reward; η episode represents the preset episodic task penalty factor; R loop represents the preset continuous task cycle reward; η maintain represents the preset continuous task penalty factor; Cost average represents the average energy consumption of the low-dew-point double-rotor dehumidifier in a single time step; Cost t represents the energy consumption of the low-dew-point double-rotor dehumidifier in time step t; η energy represents the preset energy consumption penalty factor.

6. The low dew point dual rotary dehumidifier energy optimization control method of claim 1 wherein, S22, extracting corresponding state-action pairs from each state transition tuple of the original data set in the real playback buffer, taking each state-action pair as a single-step original sequence, and dividing a plurality of continuous single-step original sequences into a history observation sequence and a target prediction sequence; S23, constructing a TimeDiff diffusion model, training the TimeDiff diffusion model using the divided state-action sequence, and obtaining the trained TimeDiff diffusion model; S24, inputting each history observation sequence after random disturbance into the trained TimeDiff diffusion model to generate a corresponding target prediction sequence, splicing the history observation sequence after random disturbance and the corresponding target prediction sequence to obtain a synthetic state-action sequence, and generating a corresponding synthetic state transition tuple by combining each obtained synthetic state-action sequence with the corresponding original state transition tuple, a plurality of continuous synthetic state transition tuples forming a single synthetic dehumidifier continuous control data, a plurality of synthetic dehumidifier continuous control data being aggregated to obtain a synthetic data set, and storing the synthetic data set in the constructed synthetic playback buffer.

7. The low-dew-point double-rotor dehumidifier energy-saving optimization control method according to claim 1, wherein the decision-making model of the low-dew-point double-rotor dehumidifier adopts an integrated diversified actor-critic algorithm improved based on a full covariance strategy, the strategy network in the integrated diversified actor-critic algorithm improved based on the full covariance strategy adopts a spatio-temporal graph attention neural network, the Q value network and the target Q value network both adopt a multi-layer fully connected neural network, and the output of the strategy network adopts a full covariance Gaussian strategy. The method for periodically training and updating is specifically: H1, collecting air state data, energy consumption data, and setting parameters of internal device control components of the low-dew-point double-rotor dehumidifier in a preset time period in real time, thereby obtaining an original data set in the preset time period; ​ 8. The low dew point dual rotary dehumidifier energy optimization control method of claim 1 wherein, ​ ​ H2, obtaining an integrated data set in the preset time period according to the same method as steps S2-S3 according to the original data set in the preset time period; H3, inputting the integrated data set in the preset time into the trained low-dew-point double-rotor dehumidifier decision model for retraining to obtain a retrained low-dew-point double-rotor dehumidifier decision model; H4, controlling and deciding in real time according to the retrained low-dew-point double-rotor dehumidifier decision model; H5, repeating steps H1-H4 according to a preset time interval, thereby realizing periodic training and updating of the low-dew-point double-rotor dehumidifier decision model. 9.A computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the computer device is characterized in that: The processor implements the steps of the method of any one of claims 1-8 when executing the computer program.

Images