Vacuum insulation thermal insulation core pipe intelligent regulation and control method based on reinforcement learning algorithm

Abstract

The application provides a vacuum heat preservation core pipe column intelligent regulation and control method based on a reinforcement learning algorithm, and comprises the following steps: S1: vacuum heat preservation core pipe column operation data acquisition; S2: collected data preprocessing and data set division; S3: reinforcement learning model construction; S4: reinforcement learning model training and verification; S5: real-time state acquisition and preprocessing; S6: reinforcement learning model inference and optimal regulation action generation; S7: regulation action execution; S8: regulation effect monitoring and feedback data acquisition; S9: reward calculation and reinforcement learning model incremental optimization; and S10: cyclic regulation and convergence judgment. Through a closed-loop process of data acquisition-fusion modeling-training optimization-real-time regulation-feedback iteration, the application solves the problem that heat preservation stability and energy consumption economy are difficult to balance in the regulation and control of a vacuum heat preservation core pipe column, and provides a feasible technical path for the intelligent operation of high-precision heat preservation equipment.

Description

Technical Field

[0001] This invention relates to the field of intelligent control technology for vacuum insulation equipment, and in particular to an intelligent control method for the core tube column in vacuum insulation based on reinforcement learning algorithms. Background Technology

[0002] Vacuum insulation core tubes are the core equipment that uses a sandwiched vacuum environment, a high-efficiency insulation layer, and auxiliary insulation devices to suppress heat loss. They are widely used in fields with stringent temperature stability requirements, such as petrochemicals, new energy, and precision manufacturing.

[0003] Existing control technologies mainly rely on two methods: one is manual control, where operators adjust the power of the vacuum pump and the setting of the insulation device based on experience. This method suffers from problems such as response lag, low control accuracy, reliance on human experience, and difficulty in adapting to dynamic changes such as ambient temperature fluctuations and equipment aging (e.g., the decay of the thermal conductivity coefficient of the insulation layer); the other is automatic control based on traditional control algorithms (such as PID control). This type of method requires preset fixed control parameters, can only cope with simple working conditions, and cannot learn the optimal control strategy under different conditions. This makes it difficult to achieve a dynamic balance between insulation effect and energy consumption—either excessive energy consumption is consumed to ensure insulation accuracy, or temperature stability is sacrificed to reduce energy consumption.

[0004] Meanwhile, the application of a single reinforcement learning algorithm in this scenario has limitations: using only deep Q-network algorithms can easily lead to excessive fluctuations in action policies, resulting in unstable equipment operation; using only proximal policy optimization algorithms lacks long-term value orientation and may get trapped in short-term optimal local solutions; using only deep deterministic policy gradient algorithms is prone to being out of touch with actual control needs, and action selection lacks specificity. In addition, existing technologies have not formed a complete closed loop of data acquisition-model training-real-time control-feedback optimization, and cannot adapt to the dynamic changes in equipment operating status, resulting in significant long-term decay in control effects. Summary of the Invention

[0005] This invention provides an intelligent control method for a vacuum insulation core tube based on a reinforcement learning algorithm. First, core state parameters throughout the entire operating cycle of the tube are collected and, after cleaning and normalization, divided into datasets. Second, a fusion model is constructed, in which a deep Q-network provides a long-term value benchmark for actions, a proximal policy optimization algorithm optimizes policy stability, and a deep deterministic policy gradient enhances action smoothness. The optimal action probability is output through weighted fusion. Subsequently, model training and validation are completed based on the dataset to obtain an optimal fusion model adapted to the operating characteristics of the tube. During the real-time control phase, the current state of the tube is continuously collected, pre-processed, and input into the model to infer the optimal control action, driving the actuator to complete the coordinated control of vacuum level, insulation device, and insulation layer adhesion. Finally, feedback data is generated through monitoring the control effect to achieve incremental optimization of the model. Through cyclical control and convergence judgment, the long-term stable and low-consumption operation of the tube is maintained.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] A method for intelligent control of the core tube in vacuum insulation based on reinforcement learning algorithms includes the following steps:

[0008] S1: Collect operational data of the core tube column in the vacuum insulation and heat preservation system to form the raw dataset;

[0009] S2: Clean and normalize the original dataset to obtain preprocessed data, and divide the preprocessed data into training set, validation set and test set;

[0010] S3: Construct a reinforcement learning fusion model integrating deep Q-networks, proximal policy optimization, and deep deterministic policy gradients. Define state space and action space. The deep Q-network outputs expected cumulative rewards for combined actions in the action space through a neural network. Proximal policy optimization receives input from the state space and outputs action selection probabilities for combined actions. Simultaneously, it calculates the advantage function based on the expected cumulative rewards and optimizes the action selection probabilities through a policy clipping mechanism. Deep deterministic policy gradients receive input from the state space and output continuous weights for combined actions through an Actor network. After being constrained by the action selection probabilities of the proximal policy optimization algorithm, the value of the continuous weights is evaluated by a Critic network to enhance the smoothness of action selection. The three algorithms output the optimal control action through a fusion function.

[0011] S4: Train the fusion model using the training set, calibrate the fusion weights of the fusion model using the validation set, test the performance of the fusion model using the test set, and finally obtain the optimal fusion model;

[0012] S5: Real-time acquisition of the current operating data of the vacuum insulation core column, and after the same preprocessing as in step S2, the real-time state vector is obtained;

[0013] S6: Input the real-time state vector into the optimal fusion model, and generate the optimal control action through collaborative reasoning and fusion function calculation of the fusion deep Q network, near-end policy optimization and deep deterministic policy gradient;

[0014] S7: The vacuum insulation core tube column control system executes optimal control actions, driving the vacuum pump, auxiliary insulation device, and insulation layer hydraulic mechanism to adjust the vacuum level, insulation power, and fit.

[0015] In this specification, the intelligent control method for vacuum insulation core tube based on reinforcement learning algorithm also includes S8: collecting new state data of the vacuum insulation core tube after control, calculating temperature fluctuation, energy consumption change and actual reward, and forming a feedback dataset.

[0016] In this specification, the intelligent control method for the core tube column of vacuum thermal insulation based on reinforcement learning algorithm also includes S9: updating the empirical samples of the fusion model based on the feedback dataset, incrementally optimizing the parameters of the deep Q network, the near-end policy optimization and the deep deterministic policy gradient to obtain the updated fusion model.

[0017] In this specification, the intelligent control method for the vacuum insulation core tube based on reinforcement learning algorithm also includes S10: repeatedly executing S5-S9 to realize the cyclic control of the vacuum insulation core tube until the preset convergence condition is met.

[0018] In this manual, S1 includes the following operating data: inlet temperature, outlet temperature, and mid-section temperature of the medium inside the vacuum insulation core tube; external ambient temperature of the vacuum insulation core tube; vacuum degree of the interlayer of the vacuum insulation core tube; internal wall pressure of the vacuum insulation core tube; thermal conductivity coefficient of the insulation layer; and current operating energy consumption.

[0019] In this specification, in S3, the interaction process between the deep Q-network algorithm and the proximal policy optimization algorithm is as follows: the proximal policy optimization algorithm associates the output of the deep Q-network algorithm with the advantage function, which is the difference between the expected cumulative reward of the deep Q-network algorithm output and the average state value of the proximal policy optimization algorithm. The average state value is the weighted sum of the action selection probability of the proximal policy optimization algorithm and the expected cumulative reward of the deep Q-network algorithm.

[0020] In this specification, in S3, the interaction between the proximal policy optimization algorithm and the deep deterministic policy gradient algorithm is as follows: the action continuous weights output by the Actor network of the deep deterministic policy gradient need to be multiplied by the action selection probability output by the proximal policy optimization algorithm to obtain the constrained action continuous weights, ensuring that the deep deterministic policy gradient tends to select actions that the proximal policy optimization considers stable.

[0021] In this specification, the specific parameters for training the fusion model in S4 include: an experience replay pool capacity of 10,000 records, 10,000 training iterations, a learning rate of 0.001 for the deep Q-network, a learning rate of 0.0003 for the proximal policy optimization, a learning rate of 0.0001 for the deep deterministic policy gradient Actor network, and a learning rate of 0.001 for the Critic network; the training stopping condition is that the test set insulation accuracy is ≥88% and the average energy consumption is ≤500W.

[0022] In this manual, in S7, the process of performing the optimal control action is as follows: the vacuum pump adjusts its power according to the vacuum level in the optimal control action; the auxiliary heat preservation device switches modes and power according to the level; and the heat insulation layer hydraulic mechanism adjusts the heat insulation layer adhesion according to the adhesion level.

[0023] In this manual, S10 defines the convergence condition as follows: temperature fluctuation ≤ ±0.5℃ and operating energy consumption ≤ rated energy consumption after 100 consecutive rounds of regulation. Once the convergence condition is met, only status monitoring and effect evaluation are performed every 10 rounds. If status fluctuations occur, the entire regulation process is restarted. If the convergence condition is not met, regulation continues in a loop until the target is met, with a maximum of 1000 consecutive rounds of regulation before triggering a manual intervention alarm.

[0024] In summary, the present invention has at least the following beneficial effects:

[0025] 1. Improve the accuracy and stability of thermal insulation control: By synergistically integrating three reinforcement learning algorithms, the defects of a single algorithm are avoided, the internal medium temperature of the tubing is kept stable, the temperature fluctuation caused by environmental changes and equipment aging is effectively suppressed, and the risk of thermal insulation failure due to improper control is reduced.

[0026] 2. Optimize energy consumption and economy: The integrated model is guided by long-term value. Under the premise of ensuring the insulation effect, it autonomously selects low-energy consumption control strategies to reduce unnecessary power consumption and energy consumption of action switching, so as to achieve a dynamic balance between insulation effect and energy consumption.

[0027] 3. Enhanced Adaptability: Through real-time data feedback and incremental optimization mechanisms, the model can autonomously adapt to dynamic operating conditions such as changes in ambient temperature and equipment aging, without the need for manual adjustment of control parameters, thus reducing operation and maintenance costs.

[0028] 4. Reduce reliance on manual intervention: The entire process of data collection, strategy reasoning, action execution, and model optimization is automated, with manual intervention alarms only triggered under extreme conditions, thereby improving the intelligence and reliability of equipment operation.

[0029] 5. Extend equipment lifespan: Through smooth control actions and stable operation, the frequent start-stop and drastic switching of actuators such as vacuum pumps, hydraulic mechanisms, and auxiliary insulation devices are reduced, thereby reducing mechanical wear and extending the overall service life of the equipment. Attached Figure Description

[0030] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0031] Figure 1 This is a flowchart illustrating the intelligent control method for the core tube column in vacuum thermal insulation based on reinforcement learning algorithms involved in this invention.

[0032] Figure 2 This is a schematic diagram illustrating the process of constructing and training the fusion model involved in this invention.

[0033] Figure 3 This is a schematic diagram of the real-time control execution process involved in this invention.

[0034] Figure 4 This is a schematic diagram of the feedback optimization and cyclic control process involved in this invention. Detailed Implementation

[0035] In the following description, only certain exemplary embodiments are briefly described. As those skilled in the art will recognize, the described embodiments can be modified in various ways without departing from the spirit or scope of the embodiments of the invention. Therefore, the drawings and description are considered to be exemplary in nature and not restrictive.

[0036] The following disclosure provides many different implementations or examples for carrying out different structures of the embodiments of the present invention. To simplify the disclosure of the embodiments of the present invention, specific examples of components and arrangements are described below. Of course, these are merely examples and are not intended to limit the embodiments of the present invention. Furthermore, reference numerals and / or reference letters may be repeated in different examples of the embodiments of the present invention; such repetition is for simplification and clarity and does not in itself indicate a relationship between the various implementations and / or arrangements discussed.

[0037] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0038] like Figure 1 As shown, this embodiment provides an intelligent control method for a vacuum insulation core column based on a reinforcement learning algorithm, including:

[0039] S1: Data Acquisition of Operational Data for Vacuum Insulation Core Column

[0040] To construct a fusion reinforcement learning model, it is necessary to collect key data that comprehensively reflects the operating status and control effect of the tubing, providing a real-world basis for model training. The specific data collection process is as follows:

[0041] Data collection parameters and selection criteria:

[0042] Based on the working principle of the vacuum insulation core tube column (suppressing heat exchange through vacuum jacket, insulation layer and auxiliary devices), eight core parameters were identified, covering the entire chain from input state to controlled object to output effect:

[0043] 1. Temperature of the medium inside the tubing: including inlet temperature (°C), outlet temperature (°C), mid-section temperature (°C) – This directly reflects the heat preservation effect and is the core target parameter for control.

[0044] 2. External ambient temperature of the tubing string (°C) — The temperature difference between the environment and the tubing is the main driver of heat loss and needs to be used as an important input for regulation;

[0045] 3. Vacuum degree V (Pa) of the tubular interlayer — The higher the vacuum degree, the weaker the heat conduction of the air, and it is the core control object (adjusted by a vacuum pump).

[0046] 4. Inner wall pressure P (MPa) of the tubing – Abnormal pressure may cause deformation of the tubing, indirectly affecting the adhesion of the insulation layer. Real-time monitoring is required to avoid mechanical damage.

[0047] 5. Thermal conductivity coefficient k (W / (m·K)) of the insulation layer — a fundamental parameter reflecting the performance of the insulation material (which slowly decreases with the duration of use) and affecting the insulation effect;

[0048] 6. Current operating energy consumption E (W) – including the real-time power consumption of vacuum pumps and auxiliary insulation devices, is a direct indicator of economic efficiency.

[0049] Data sources and collection methods:

[0050] Temperature parameters: collected by PT100 platinum resistance temperature sensors built into the tubing (one each at the inlet, outlet, and middle section, with a measurement accuracy of ±0.1℃).

[0051] Vacuum level: Employs a piezoelectric vacuum sensor (measuring range 10). -4 ~10Pa, accuracy ±0.001Pa);

[0052] Inner wall pressure: MEMS pressure sensor (measurement range 0~1MPa, accuracy ±0.005MPa).

[0053] Thermal conductivity coefficient: calculated by a combination of heat flow meter and temperature sensor installed on the outer wall of the tubing (automatically calibrated once per hour).

[0054] Energy consumption: extracted from the built-in power recorder of the vacuum pump and auxiliary insulation device (sampling frequency 1Hz).

[0055] All sensor data is transmitted in real time via industrial Ethernet, with a uniform acquisition frequency of 1 time per second to ensure the capture of subtle changes in status.

[0056] Historical Data and Storage Format: In addition to real-time acquisition, historical operational data for the past year (covering all four seasons and different time periods of day and night) is extracted from the pipeline maintenance database. This includes operational records under extreme environments (such as high temperatures in summer and severe cold in winter) to improve the model's generalization ability. Raw data is stored on the local data server in the format of acquisition time (accurate to the second) - parameter type - value - sensor number - data status (normal / abnormal), forming the raw dataset. This provides complete input for the preprocessing of S2.

[0057] S2: Data Preprocessing and Dataset Partitioning

[0058] Original dataset Issues such as noise and missing values ​​exist, necessitating preprocessing to transform the dataset into a format suitable for training fusion reinforcement learning models. Simultaneously, the dataset needs to be partitioned for model training, validation, and testing. The preprocessing process is closely related to the requirements of the S3 model (e.g., preserving temporal features to support state sequence learning in the policy network), and the specific steps are as follows:

[0059] 1. Data cleaning:

[0060] Outlier handling: Outliers are identified using the 3σ criterion (data values ​​exceeding the mean of the parameter by ±3 times the standard deviation are considered outliers). For example, if a certain period of time... If the mean is 50℃ and the standard deviation is 2℃, then data >56℃ or <44℃ are marked as outliers and replaced with the moving average of the five valid data points before and after the parameter (to avoid single-point outliers affecting the continuity of the time series).

[0061] Missing value handling: When ≤5 consecutive data points are missing (i.e., ≤5 seconds), linear interpolation is used to fill in the missing data (based on the trend of changes in the valid data before and after); when >5 consecutive data points are missing, the historical average value of the same date and time period (such as the historical average temperature of every Wednesday from 14:00 to 14:05) is used to replace the missing data to ensure the temporal correlation of the data.

[0062] 2. Data normalization:

[0063] The cleaned data is mapped to the [0,1] interval to eliminate the influence of unit differences on model training (e.g., vacuum degree is in Pa, energy consumption is in W; direct input will cause the model to be biased towards large numerical parameters). The min-max normalization formula is used: Where x is the original data, , The parameter is in The minimum and maximum values ​​in (e.g., V) =0.001Pa, =1Pa). The normalized parameter is denoted as , ... This forms the basis of the model input.

[0064] 3. Dataset partitioning:

[0065] The normalized data was divided into training and training sets in a 7:2:1 ratio. Validation set Test set Strict temporal continuity must be maintained during the partitioning process (e.g., the first 70% of the time series data is...). The middle 20% is The last 10% are To avoid disrupting temporal features (the temporal correlation learning strategy in the fusion model, which requires the optimization of proximal policies and the gradient of deep deterministic policies to depend on the state).

[0066] Preprocessing results: Output (Used for joint training of the three algorithms in S3) (used for calibrating fusion weights) and model hyperparameters), (Used for the final evaluation of the generalization ability of the fusion model), all are stored in the model training server to provide standardized input for the training process of S4.

[0067] S3: Building Reinforcement Learning Models

[0068] To achieve precise, stable, and low-energy control of the core tube in vacuum insulation, a reinforcement learning model fusing Deep Q-Network (DQN), Proximal Policy Optimization (PPO), and Deep Deterministic Policy Gradient (DDPG) is required. These three algorithms do not operate independently but work collaboratively through a clear interaction mechanism: DQN provides a long-term value benchmark for actions, PPO optimizes policy stability based on this benchmark, and DDPG enhances the smoothness of action selection to reduce energy consumption fluctuations. Finally, the optimal control decision is output through a fusion function. The model input is the S2 preprocessed dataset (training set, validation set, and test set). The fusion model construction and training process is as follows: Figure 2 As shown.

[0069] 3.1 Basic Definitions

[0070] The core of reinforcement learning is that an agent learns the optimal strategy through interaction with its environment, where the environment is the operating system of the core tube in a vacuum insulation center, and the interaction is manifested in the control actions and state feedback of the tube. Therefore, it is necessary to first define the state space and action space shared by all algorithms:

[0071] State space S: The state is a complete description of the current operating state of the tubing string, and must include key parameters that directly affect the insulation effect and energy consumption. Based on the data collected in S1 and the operating mechanism of the tubing string, the state vector is defined as:

[0072] ;

[0073] The state space has 8 dimensions (i.e., 8 parameters), denoted as All parameters were preprocessed by S2 (normalized to the [0,1] interval) before being input into the model.

[0074] Action Space A: Actions are the agent's control operations on the tubing, and must cover all core means that can change the state of the tubing. Combining the controllable mechanisms of the tubing (vacuum pump, auxiliary insulation device, hydraulic fine-tuning mechanism), the action space is defined as a combination of three types of discrete actions, as follows:

[0075] 1. Vacuum Adjustment Function (A1): Changes the vacuum level of the jacket by controlling the power of the vacuum pump, with 4 discrete levels (corresponding to different power outputs):

[0076] 0.001Pa (maximum vacuum level, corresponding to full power operation of the vacuum pump);

[0077] 0.01Pa (high vacuum, corresponding to 80% power of the vacuum pump);

[0078] 0.1 Pa (medium vacuum, corresponding to 50% power of vacuum pump);

[0079] 1Pa (low vacuum level, corresponding to 30% power of vacuum pump).

[0080] 2. Auxiliary heat preservation device control (A2): Supplements heat preservation by switching heating / cooling modes and adjusting power, with 3 discrete settings:

[0081] Cooling mode (500W power, used to suppress internal temperature rise of the tubing when the ambient temperature is too high);

[0082] : Insulation setting (300W power, used to maintain stable internal temperature of the tubing);

[0083] Heating mode (800W power, used to compensate for heat dissipation inside the tubing when the ambient temperature is too low).

[0084] 3. Insulation layer adhesion fine-tuning action (A3): The hydraulic mechanism changes the tightness of the adhesion between the insulation layer and the inner wall of the pipe column (the higher the adhesion, the less heat loss), and is divided into 4 discrete levels:

[0085] Fit 85% (relatively loose, suitable for use when the pressure on the inner wall of the tubing is slightly high to avoid structural stress);

[0086] Fit: 90% (Medium, the default setting for normal operation);

[0087] : Fit 95% (relatively tight, suitable for enhancing thermal insulation when there are large temperature differences in the environment);

[0088] 100% fit (tightest fit, suitable for maximizing insulation effect in extreme low temperature environments).

[0089] The total dimension of the action space is the number of combinations of the three types of actions: 4(A1) × 3(A2) × 4(A3) = 48 discrete actions, denoted as... Each action These are all specific combination operations (e.g.) It can represent " (0.01Pa)+ (300W keep-warm setting) + (Fitness 85%).

[0090] 3.2 Algorithm 1: Deep Q-Network (DQN) – Establishing Action Value Benchmark

[0091] The core function of DQN is to evaluate the cumulative reward that may be obtained in the future after performing an action in the current state, that is, the long-term value of the action. This value benchmark is the basis for the decision-making of the other two algorithms (PPO and DDPG), ensuring that all regulatory actions are aimed at long-term optimality, rather than just pursuing short-term gains.

[0092] Model building details:

[0093] DQN uses a deep neural network to fit the action value function. The network structure design needs to match the state space dimension and the action space dimension, as follows:

[0094] Input layer: 8 neurons, corresponding to the 8 parameters of the state vector S (normalized by S2).

[0095] Hidden layer 1: 64 neurons, using the ReLU activation function (to solve the gradient vanishing problem and enhance nonlinear fitting ability).

[0096] Hidden layer 2: 32 neurons, also using the ReLU activation function (to further extract the association between state features and action value);

[0097] Output layer: 48 neurons, using a linear activation function (outputting an unrestricted Q-value), corresponding to the value evaluation of 48 actions.

[0098] Network parameters are denoted as (The subscript "D" indicates DQN), the output action value function is defined as:

[0099] ;

[0100] in This represents the expected cumulative reward from the current moment to the future after performing action a in state s (the higher the value, the better the long-term benefit of the action).

[0101] Training process:

[0102] DQN is trained using the S2 training set. Using the experience replay mechanism as input, the relationships between states, actions, rewards, and the next state are learned. The specific steps are as follows:

[0103] 1. Empirical sample collection: from the training set Extract time series data and construct empirical samples ( ),in:

[0104] The state at time t (from (normalized data);

[0105] The action to be performed at time t (selected from action space A);

[0106] The reward at time t ( (where R1 is the insulation effect bonus and R2 is the energy consumption penalty).

[0107] To perform the action The state at time t+1 (from (Time series data).

[0108] All samples are stored in an experience replay pool with a capacity of 10,000.

[0109] 2. Loss Function Definition: The training objective is to minimize the error between the predicted Q-value and the target Q-value. The loss function is:

[0110] (3-1)

[0111] The target Q value (an approximation of the true value) is calculated using the following formula: ;

[0112] =0.9 is the discount factor (representing the degree of decay of future rewards; the further into the future, the smaller the impact of rewards).

[0113] For the target network parameters (from the current network parameters every 100 rounds) (Replication and updates to avoid training fluctuations)

[0114] Indicates the next state The maximum Q value (i.e. the optimal future reward) of all possible actions.

[0115] This means taking the expectation of 32 randomly selected samples from the experience replay pool (batch training to stabilize the gradient).

[0116] 3. Parameter Update: Employs the Adam optimizer, learning rate... =0.001, update network parameters using gradient descent. until the loss function Convergence (validation set) The Q-value prediction error is ≤5%.

[0117] 3.3 Algorithm 2: Proximity Policy Optimization (PPO) – Policy Stability Optimization Based on Value Benchmark

[0118] The core role of PPO is to optimize the strategy for selecting actions (i.e., the probability distribution of action selection) based on the value benchmark of DQN. At the same time, it limits the update range of the strategy through the strategy clipping mechanism to avoid drastic changes in control actions due to data fluctuations (such as frequent switching of vacuum pumps between high and low power), thereby improving the stability of the tubing operation.

[0119] Model building details:

[0120] The PPO consists of two sub-networks: a policy network (which outputs the probability of action selection) and a state-value network (which helps calculate the advantage function), with the following specific structure:

[0121] Policy Networks:

[0122] Input layer (8 neurons, corresponding to state s) → Hidden layer 1 (128 neurons, ReLU activation) → Hidden layer 2 (64 neurons, ReLU activation) → Output layer (48 neurons, Softmax activation).

[0123] Network parameters are denoted as (The subscript "P" indicates PPO), the output policy function is defined as:

[0124] ;

[0125] in Let represent the probability of choosing action 'a' in state 's', and satisfy . (The sum of the probabilities of all actions is 1).

[0126] State-Value Network:

[0127] Input layer (8 neurons, corresponding to state s) → Hidden layer (64 neurons, ReLU activation) → Output layer (1 neuron, linear activation).

[0128] Network parameters are denoted as The output state value function is defined as:

[0129] ;

[0130] in It represents the average value of state s (i.e., the weighted average of the values ​​of all actions in that state).

[0131] Interaction mechanism with DQN:

[0132] PPO strategy optimization relies on DQN's action value assessment, which measures the merits of a particular action relative to the average level using a dominance function, as follows:

[0133] Advantage function Defined as:

[0134] (3-2)

[0135] like A value greater than 0 indicates that action a has a higher value than the average value of the current state (and is worth prioritizing).

[0136] like <0 indicates that the value of action a is below average (the probability of choosing it should be reduced).

[0137] State value function The calculation requires combining the Q-value of DQN with the policy probability of PPO:

[0138] ;

[0139] That is, the action value of DQN is weighted by the current policy probability of PPO to ensure that the average value is consistent with the current policy.

[0140] Training process:

[0141] The training of PPO uses the training set of S2 and the output of DQN as the input. The specific steps are as follows:

[0142] 1. Advantage function calculation: For each sample in the training set , substitute it into formula (3-2) to calculate the advantage function .

[0143] 2. Loss function definition: To avoid excessive policy update amplitude, PPO uses a clipped loss function, defined as:

[0144] ; (3-3)

[0145] is the policy update ratio (the ratio of the new policy probability to the old policy probability);

[0146] is the policy parameter before update (fixed every 20 rounds as a benchmark);

[0147] = 0.2 is the clipping coefficient (restricting within the range of [0.8, 1.2] to avoid policy mutation);

[0148] is the clipping function (if < l, then output l, if then output , otherwise output x).

[0149] 3. Parameter update: Use the Adam optimizer with a learning rate = 0.0003. Maximize the loss function through gradient ascent (that is, increase the selection probability of high-advantage actions), and at the same time update the state value network parameters (minimize the error between and ).

[0150] 3.4 Algorithm 3: Deep Deterministic Policy Gradient (DDPG) - Enhancement of Action Smoothness Based on Stable Policies

[0151] The core function of DDPG is to transform the discrete action space into a continuous action weight distribution. By smoothly adjusting the weights, it reduces the frequency of action switching (e.g., avoiding repeated fluctuations in insulation layer adhesion between 90% and 95%), thereby reducing the additional energy consumption caused by action switching. Its action generation must be constrained by the PPO strategy to ensure consistency with the direction of the stable strategy.

[0152] Model building details:

[0153] DDPG consists of two sub-networks: the Actor network (which generates action weights) and the Critic network (which evaluates the value of the weights), with the following specific structure:

[0154] Actor Network:

[0155] Input layer (8 neurons, corresponding to state s) → Hidden layer 1 (64 neurons, ReLU activation) → Hidden layer 2 (32 neurons, ReLU activation) → Output layer (48 neurons, Sigmoid activation).

[0156] Network parameters are denoted as (The subscript "G" indicates Actor generation), the output action weight function is defined as:

[0157] ;

[0158] in Indicates the i-th action The continuous weights (the higher the weight, the more suitable the action is for the current state).

[0159] Critic Network:

[0160] Input layer (8+48 neurons, corresponding to the weights of state s and Actor output) → Hidden layer (64 neurons, ReLU activation) → Output layer (1 neuron, linear activation).

[0161] Network parameters are denoted as (The subscript "C" indicates Critic evaluation), the output value function is defined as:

[0162] ;

[0163] in This indicates that a weight distribution is used in state s. The expected cumulative reward at that time (evaluation of the overall merits and demerits).

[0164] Interaction mechanism with PPO:

[0165] To avoid conflicts between the action weights generated by DDPG and the stable policy of PPO (for example, PPO considers an action to have an extremely low probability, but DDPG assigns it a high weight), the output of DDPG needs to be constrained by the policy probability of PPO, as follows:

[0166] Constrained action weights Defined as:

[0167] (3-4)

[0168] Let be the weight of the i-th action after constraint;

[0169] The product term ensures that if PPO affects the action... Selection probability Extremely low (e.g., <0.05), then regardless of How big? All will be suppressed (avoid choosing actions that the PPO considers unstable).

[0170] Training process:

[0171] DDPG is trained using the S2 training set. PPO output For input, the specific steps are as follows:

[0172] 1. Constraint weight calculation: Calculate the constraint weights for the output of the Actor network. Substitute into formula (3-4) to calculate the weights after constraints. .

[0173] 2. Critic Network Training: The objective is to minimize the value prediction error, and the loss function is:

[0174] (3-5)

[0175] The target value of Critic is calculated using the following formula: ;

[0176] For the target Actor network in state Weights after constraints;

[0177] For the target network parameters (from every 100 rounds) (Copy and update).

[0178] 3. Actor Network Training: The goal is to maximize the value of the Critic rating (through gradient ascent), and the loss function is:

[0179] (3-6)

[0180] 4. Parameter Update: Critic uses the Adam optimizer ( =0.001), minimized by gradient descent The Actor uses the Adam optimizer. =0.0001), maximized by gradient ascent ).

[0181] 3.5 Fusion Mechanism of the Three Algorithms

[0182] The outputs of the three algorithms need to be integrated through a fusion mechanism to form a final decision that balances value, stability, and smoothness. The fusion process requires clearly defining the weight ratio of each algorithm and ensuring, through normalization, that the output represents a legal action probability distribution.

[0183] Determining the fusion weights:

[0184] Let the fusion weighting coefficient be These correspond to the contributions of DQN, PPO, and DDPG, respectively, satisfying... The validation set of S2 Multiple sets of comparative experiments were conducted (testing the insulation effect and energy consumption under different weight combinations), and the final calibration was as follows:

[0185] =0.4 (DQN has the highest weight, ensuring that long-term value is the core).

[0186] =0.3 (PPO weight is secondary to ensure strategy stability);

[0187] =0.3 (DDPG weight is consistent with PPO to balance smoothness requirements).

[0188] Fusion calculation of action selection probability:

[0189] The probability of choosing the final action a The outputs of the three algorithms need to be integrated. The specific formula is as follows:

[0190] (3-7)

[0191] : Convert the Q-value of DQN into a probability distribution (the higher the value of the action, the greater the probability).

[0192] : Constrained action weights output by DDPG (already aligned with the PPO strategy via formula 3-4);

[0193] The normalization constant ensures that the sum of the probabilities of all actions is 1.

[0194] Optimal action selection:

[0195] The action with the highest probability is selected as the final output from the fusion probability distribution, i.e.:

[0196] (3-8)

[0197] For example: if the action after fusion ( + (300W keep-warm setting) + The probability of (90% fit) is the highest. .

[0198] 3.6 Overall Output of the Fusion Model

[0199] Fusion Model The final build result includes:

[0200] DQN network parameters ;

[0201] PPO strategy parameters With state value parameters ;

[0202] Actor parameters of DDPG With Critic parameters ;

[0203] Fusion weights =0.4、 =0.3、 =0.3.

[0204] The model takes the state vector s preprocessed by S2 as input and outputs the optimal control action. The parameters are stored in the model training server and used as the initial parameters for model training in S4.

[0205] Key Contributions: The fusion of three algorithms overcomes the limitations of a single algorithm: DQN ensures that action selection is guided by long-term optimal value, avoiding sacrificing insulation performance for short-term energy savings; PPO suppresses abrupt action changes through policy editing, solving the operational instability problem caused by frequent adjustments to the tube column; DDPG reduces the additional energy consumption of action switching (e.g., energy loss from frequent vacuum pump start-stop) through continuous weights and PPO constraints. The three algorithms work synergistically to ultimately achieve the control goal of stable insulation performance and economical energy consumption in the core tube column of vacuum insulation.

[0206] S4: Training and Validation of Reinforcement Learning Models

[0207] Dataset based on S2 The fusion model built with S3 is jointly trained and validated in multiple dimensions to ensure that the model can output stable and efficient control actions in different scenarios. The training process requires simultaneous optimization of the parameters of DQN, PPO, and DDPG, and dynamic adjustment of the fusion weights using the validation set. The specific steps are as follows:

[0208] 1. Training initialization:

[0209] Model parameters: Initialization parameters for DQN PPO DDPG Random values ​​(following a normal distribution) );

[0210] Hyperparameter settings: The unified experience replay pool capacity is set to 10,000 records (storage). (sample), total training iterations 10000 times, learning rates respectively =0.001(DQN)、 =0.0003 (PPO) =0.0001 (DDPG) =0.001 (DDPG's Critic), Discount Factor =0.9, PPO's editing factor =0.2;

[0211] Initial values ​​for fusion weights: =0.4、 =0.3、 =0.3 (Based on prior knowledge, to be determined later) calibration).

[0212] 2. Joint iterative training:

[0213] Each round of training starts from Extract continuous time-series data and synchronously update the parameters of three algorithms:

[0214] Step 1: State and Action Sampling: Extracting the state at time t Generate actions using the current fusion model (According to formula 3-7) choose);

[0215] Step 2: Reward Calculation: Based on After the action Calculated according to the reward function of S3 (R1 is the insulation effect bonus, R2 is the energy consumption penalty);

[0216] Step 3: Experience Storage: [This section appears to be incomplete and requires further context.] Stored in the shared experience replay pool (for use by the three algorithms);

[0217] Step 4: Parameter Update

[0218] DQN: When the number of samples in the experience pool is ≥1000, 32 samples are randomly selected, and the loss is calculated according to formula (3-1). Update via Adam optimizer ;

[0219] PPO: 64 samples are drawn from the experience pool every 20 rounds, and the advantage function is calculated according to formula (3-2). Then calculate the editing loss according to formula (3-3). ,renew ;

[0220] DDPG: 32 samples are drawn from the experience pool every 50 rounds, and the constraint weights are calculated first according to formula (3-4). Then update the Critic using formula (3-5). Finally, update the Actor's... .

[0221] 3. Model Validation and Fusion Weight Calibration:

[0222] Every 1000 rounds of training, using To evaluate model performance, the metrics include:

[0223] Average reward value (A higher value indicates a better overall effect);

[0224] Regulation stability: Action switching frequency (number of switching times per hour, the lower the frequency, the more stable the operation);

[0225] Thermal insulation accuracy: The percentage of samples with temperature fluctuations ≤ ±0.5℃ (the higher the percentage, the better the accuracy).

[0226] The fusion weights are dynamically adjusted based on the verification results: if stability is insufficient (switching frequency > 5 times / hour), the weights are increased. (PPO weight) 0.05; if the insulation accuracy is low (percentage <85%), then increase the weight. (DQN weight) 0.05; if energy consumption fluctuates greatly (energy consumption standard deviation > 100W), then increase. (DDPG weight) 0.05, to ensure .

[0227] 4. Model testing and convergence determination:

[0228] When the training iteration reaches 10,000 rounds, or meets the condition for 200 consecutive rounds... Training should be stopped when the stability switching frequency is ≤3 times / hour and the heat preservation accuracy is ≥90%. Conduct final testing. If the test set indicators meet the standards (insulation accuracy ≥ 88%, average energy consumption ≤ 500W), then output the optimal fusion model. (Includes all algorithm parameters and calibrated fusion weights); otherwise, adjust the learning rate (e.g.) =0.0005), return to Step 2 to retrain.

[0229] Training Results: Optimal Fusion Model The data is stored on the real-time control server, which serves as the inference core of the S6, ensuring the accuracy and stability of subsequent real-time control.

[0230] S5: Real-time Status Acquisition and Preprocessing

[0231] In actual operation of the tubing, the current state needs to be captured in real time and converted into an input format recognizable by the model to provide real-time data support for the inference of the fusion model. This step is consistent with the data acquisition method of S1, but places greater emphasis on the timeliness of the data and the speed of preprocessing. The specific process is as follows:

[0232] 1. Real-time data acquisition:

[0233] The sensor array deployed via S1 collects eight status parameters of the tubing once per second. (V, P, k, E), the data is transmitted to the real-time control server through the 5G industrial gateway (the latency is controlled within 50ms to avoid the impact of status lag on decision-making).

[0234] 2. Real-time preprocessing:

[0235] Outlier handling: Real-time data is checked using the 3σ criterion. If an anomaly occurs (such as a jump value caused by sensor failure), the moving average value of the previous 10 seconds is immediately used as a replacement, and a sensor failure alarm is triggered (notifying maintenance personnel to perform repairs).

[0236] Normalization: using the values ​​calculated in S2 (Based on historical data), the real-time parameters are normalized using a min-max method to generate a real-time state vector: Ensure that the format is consistent with the state space in S3.

[0237] Processing result: The model inference module of S6 is transmitted in real time as a fusion model. The input supports the rapid generation of optimal actions.

[0238] S6: Reinforcement Learning Model Inference and Optimal Control Action Generation

[0239] Real-time status based on S5 Through the fusion model The optimal control action, balancing value, stability, and smoothness, is deduced to provide explicit execution instructions for the tubular control system. The deduction process requires an efficient synergistic mechanism involving three algorithms; the specific steps are as follows:

[0240] 1. Parallel inference using multiple algorithms:

[0241] Will Simultaneously input The DQN, PPO, and DDPG modules in the code compute their respective outputs in parallel.

[0242] DQN: Outputs the value of each action. And convert it into a probability distribution using softmax. ;

[0243] PPO: Probability of Output Action Selection ;

[0244] DDPG: Raw weights output by the Actor network After being constrained by the PPO strategy (Formula 3-4), we obtain .

[0245] 2. Fusion probability calculation:

[0246] Substitute the fusion weights after S4 calibration Calculate the final action probability distribution according to formula (3-7). For example: if a certain action of =0.2、 =0.3、 =0.25, and Then its fusion probability is:

[0247] ;

[0248] Z is a normalization constant that ensures the sum of the probabilities of all actions is 1.

[0249] 3. Optimal action selection:

[0250] Select The most probable action as This is then parsed into specific control commands. For example: if The corresponding instruction would be "Adjust the vacuum level to 0.1 Pa ( + Auxiliary device switched to heat preservation mode 300W ( + The insulation layer adhesion is adjusted to 95%. )".

[0251] Reasoning result: The standardized instruction format (including action type, target value, and execution time limit) is transmitted to the column control system to trigger the execution of the control action of S7.

[0252] S7: Execution of control actions

[0253] The tubular control system receives the output of S6. This drives the various actuators to coordinate and complete control operations, ensuring precise implementation of actions. The execution process requires coordinated control across multiple agencies, with feedback on the execution status to verify the effectiveness of the actions, and real-time control of the execution flow, such as... Figure 3 As shown.

[0254] 1. Instruction parsing and allocation:

[0255] The control system will The instructions are broken down into three categories and assigned to the corresponding execution mechanisms:

[0256] Vacuum adjustment command: sent to the vacuum pump controller, including the target vacuum level (e.g., 0.1 Pa) and the power adjustment curve (e.g., linear transition from the current power to the target power to avoid shock).

[0257] Auxiliary heat preservation device command: sent to the heating / cooling module, including the operating mode (cooling / heat preservation / heating) and target power (e.g., 300W);

[0258] Insulation layer fit instruction: sent to the hydraulic control system, including the target fit (e.g., 95%) and pressure adjustment parameters (to ensure a smooth fit process and avoid damage to the insulation layer).

[0259] 2. Multi-agency collaborative execution:

[0260] The actuators move synchronously according to instructions, and the current status is fed back in real time through sensors:

[0261] Vacuum pump: Feeds back the current vacuum level every 0.5 seconds until the target value is reached (allowable error ±0.005Pa);

[0262] Auxiliary device: Real-time feedback of output power to ensure stability within ±50W of the target value;

[0263] Hydraulic system: Feeds back the current fit (calculated by a displacement sensor) until the target value ±1% is reached.

[0264] 3. Execution status confirmation:

[0265] Once the execution errors of all sub-instructions meet the requirements, the actuator returns an "execution completed" signal to the control system. The control system records the execution time of the action (e.g., 2.3 seconds from the issuance of the instruction to its completion) and the final state, providing a benchmark for monitoring the effect of S8.

[0266] Execution result: The status of the tubing string changed from... Transition to the new state The new state data is collected in real time by the sensors of S1, providing a basis for subsequent effect evaluation and model optimization.

[0267] S8: Monitoring and Feedback Data Collection of Regulation Effects

[0268] After the control action is executed, the new state of the tubing and the control effect need to be comprehensively monitored to form a feedback data closed loop, providing real interactive samples for incremental optimization of the fusion model. The monitoring content needs to cover all state parameters and key performance indicators defined in S3. The feedback optimization and cyclic control process is as follows: Figure 4 As shown.

[0269] 1. New Status Acquisition:

[0270] After the action is completed, continuously collect the tubing status data for 3 seconds (once every 0.5 seconds), and take the average value as... To ensure data stability (avoiding the impact of instantaneous fluctuations on the assessment).

[0271] 2. Calculation of performance indicators:

[0272] based on Compared with before execution Calculate the core performance indicators:

[0273] Temperature fluctuations : (Reflects thermal insulation stability);

[0274] Energy consumption changes : (Reflecting energy efficiency);

[0275] Action execution error: such as the deviation between the actual vacuum level and the target value (to assess the accuracy of the actuator).

[0276] 3. Feedback dataset construction:

[0277] Organize the following information into a feedback dataset. :

[0278] Adjusting timestamps , , ;

[0279] Calculated , and execution error;

[0280] The actual reward for this step (Calculated according to the reward function of S3).

[0281] Feedback results: Synchronous transmission to two modules: one is the S2 dataset update module (used for expansion). The first is to enhance the historical data coverage of the model; the second is the reward calculation and model optimization module of S9 (used to generate new experience samples and drive the adaptive evolution of the model).

[0282] S9: Reward Calculation and Incremental Optimization of Reinforcement Learning Models

[0283] Based on S8 Calculate the actual reward and update the parameters of the fusion model to adapt to changes in the state of the tubing during operation (such as an increase in k due to insulation aging). Incremental optimization requires simultaneous updates to the parameters of DQN, PPO, and DDPG to ensure the fusion mechanism remains effective. The specific steps are as follows:

[0284] 1. Actual reward calculation:

[0285] from Extract and Calculated according to the reward function of S3 :

[0286] like If, then R1 = 10; if If so, then R1=5; otherwise, R1=0.

[0287] (Rated power consumption is 500W), if ,but =0.5, otherwise =1.2;

[0288] final .

[0289] 2. Expanding the empirical sample:

[0290] Will The experience replay pool added to the fusion model, together with the historical samples from S4 training, constitutes the data source for incremental training.

[0291] 3. Multi-algorithm incremental update:

[0292] Incremental training is initiated when the number of new samples is ≥500.

[0293] DQN: Draw 32 new samples and update according to formula (3-1). (Learning rate drops to) =0.0005 (to avoid overfitting to new data).

[0294] PPO: Draw 64 new samples and recalculate the advantage function. (Based on the updated version) Update according to formula (3-3) ;

[0295] DDPG: Extract 32 new samples and update Critic. And Actor (Formulas 3-5 and 3-6) ensure constraint weights With the latest Alignment.

[0296] 4. Model replacement and validation:

[0297] After incremental training is completed, an updated model is generated. Validate its performance using the most recent 100 feedback data points (if...) If the improvement is ≥5%, then the original model in the real-time control server will be replaced. If the sample is used for inference in S6, then the update is abandoned and samples are accumulated.

[0298] Optimization results: Through continuous incremental optimization, It can adapt to factors such as column aging and environmental changes, ensuring stable long-term control effects.

[0299] S10: Cyclic Regulation and Convergence Judgment

[0300] By repeating the S5-S9 process, continuous intelligent control of the tubing is achieved, and the convergence of the control state (i.e., reaching a balanced state of stable insulation and economical energy consumption) is dynamically determined, avoiding ineffective control actions that waste resources. The specific loop mechanism is as follows:

[0301] 1. Loop triggering mechanism:

[0302] After each S5-S9 process is completed (taking approximately 10 seconds, including data acquisition, inference, execution, and feedback), the next control cycle is automatically initiated. If the control column is in a stable operating phase (e.g., no significant changes in status for 10 consecutive minutes), the cycle can be extended to 30 seconds to reduce unnecessary calculation and execution costs.

[0303] 2. Definition of convergence condition:

[0304] The core indicator for convergence is set as "satisfying the following after 100 consecutive rounds of adjustments": ±0.5℃ and "This means that the insulation effect meets the standard and the energy consumption is within the rated range."

[0305] 3. Post-convergence regulation strategy:

[0306] If the convergence condition is met: maintain the current state. Every 10 rounds, only S5-S8 (state monitoring and effect evaluation) are executed, without triggering model optimization in S9; if monitoring detects... >0.5℃ or If the voltage exceeds 500W, immediately re-enter the complete S5-S9 process;

[0307] If the convergence condition is not met: continue to execute the S5-S9 cycle adjustment until the convergence condition is met (maximum of 1000 consecutive adjustments; if it still does not converge, a manual intervention alarm will be triggered).

[0308] Cyclic Results: Through dynamic cycling and convergence judgment, the tubing can be adaptively controlled under different working conditions, which can ensure the insulation accuracy, minimize energy consumption, and extend the service life of the equipment.

[0309] In some embodiments, the S3 fusion model introduces two algorithms: A2C (Advantage Actor-Critic) and SAC (Soft Actor-Critic) to further optimize the model's exploration capabilities, multi-objective balancing ability, and adaptability to complex dynamic environments. The fusion of these two algorithms with the existing DQN, PPO, and DDPG algorithms needs to form a complementary mechanism: A2C enhances the real-time collaborative learning of strategies and values, while SAC promotes exploration through the maximum entropy objective (avoiding getting trapped in local optima), and is deeply coupled with existing algorithms through an interactive formula. The specific details are as follows:

[0310] Algorithm 4: A2C (Advantage Actor-Critic) – Real-time Co-optimization of Strategy and Value

[0311] Core function: To compensate for the latency of PPO batch updates, improve the policy's response speed to state changes (such as rapid adjustment when the ambient temperature rises / falls suddenly) through real-time interaction between Actors and Critics, and provide real-time correction for DQN value assessment.

[0312] Model building details:

[0313] A2C includes two sub-networks (similar to PPO, but with a different update mechanism):

[0314] Actor Network (Policy Generation):

[0315] Input layer (8 neurons, corresponding to state s) → Hidden layer (64 neurons, ReLU activation) → Output layer (48 neurons, Softmax activation).

[0316] Network parameters are denoted as (The subscript "A" indicates A2C), output the policy function. , where represents the probability of choosing action a in state s.

[0317] Critic Network (Advantages Assessment):

[0318] Input layer (8 neurons, corresponding to state s) → Hidden layer (32 neurons, ReLU activation) → Output layer (1 neuron, linear activation).

[0319] Network parameters are denoted as Output state value , used to calculate the dominance function.

[0320] Interaction mechanism with DQN and PPO:

[0321] 1. Cross-algorithm adjustment of the dominance function: A2C dominance function By combining the Q-value of DQN with the policy probability of PPO, a more accurate action value assessment can be achieved.

[0322] (3-9)

[0323] in =0.6 is the weighting coefficient (through calibration), The state value function of PPO (Formula 3-2) ensures that the advantage assessment integrates the long-term value of DQN with the stable strategy perspective of PPO.

[0324] 2. Real-time supplement to policy updates: A2C uses single-step updates (instead of PPO's batch processing), and its Actor loss function references PPO's advantage function, but updates at a higher frequency.

[0325] (3-10)

[0326] in The advantage function of PPO (Formula 3-2) ensures that the strategy update of A2C is consistent with the stable strategy direction of PPO, but with a faster response.

[0327] Training process:

[0328] Each step draws one sample from the experience pool (instead of a batch) and updates the Actor and Critic synchronously:

[0329] Critic loss: (Mean squared error);

[0330] Actor Loss: According to formula (3-10), the RMSprop optimizer (learning rate) is used. =0.0005), the parameters are updated once per step to compensate for the delay defect of PPO during state transitions.

[0331] Algorithm 5: SAC (SoftActor-Critic) – Maximum Entropy Exploration and Multi-Objective Equilibrium

[0332] Core function: By using the maximum entropy objective, the model is encouraged to explore more potential effective actions (avoiding the neglect of better strategies due to over-reliance on historical experience), while enhancing the dynamic balance between the dual objectives of heat preservation accuracy and energy consumption cost, which is especially suitable for extreme environments (such as the need to balance heating energy consumption and heat preservation effect in severe winter).

[0333] Model building details:

[0334] SAC consists of an Actor network, two Critic networks (to avoid overestimation), and a target network.

[0335] Actor Network (Random Policy Generation):

[0336] Input layer (8 neurons, corresponding to state s) → Hidden layer (64 neurons, ReLU activation) → Output layer (48×2 neurons, mean and standard deviation of output action probability).

[0337] Network parameters are denoted as (The subscript "S" indicates SAC), output the random policy. (Based on Gaussian distribution sampling actions, enhancing the exploratory nature).

[0338] Critic Network (Soft Value Assessment):

[0339] Input layer (8+48 neurons, corresponding to state s and action a) → Hidden layer (64 neurons, ReLU activation) → Output layer (1 neuron, linear activation).

[0340] Network parameters are denoted as Output soft Q value This is used to evaluate the entropy enhancement value of an action.

[0341] Interaction mechanism with DDPG and DQN:

[0342] 1. Fusion of Maximum Entropy Rewards: SAC's soft Q-value incorporates entropy (exploration reward) into its value assessment, thus modifying DQN's reward function:

[0343] (3-11)

[0344] in: The strategy entropy (the higher the entropy, the stronger the exploratory nature); =0.2 is the entropy weight; Feedback was sent to DQN for correction. value( This enhances the exploratory orientation of value assessment.

[0345] 2. Smoothness constraint for continuous actions: The SAC Actor output and the DDPG Actor work together, and the policy differences are constrained by KL divergence.

[0346] (3-12)

[0347] in =0.1 is the KL divergence threshold to ensure that the exploration action of SAC does not deviate from the smooth constraint direction of DDPG (avoiding drastic fluctuations in action due to over-exploration).

[0348] Training process:

[0349] Critic loss: Minimizes the error between the soft Q value and the objective value (the objective value includes an entropy term):

[0350] ;

[0351] The target Critic network is updated every 200 rounds;

[0352] Actor loss: Maximizes the sum of soft Q-value and entropy.

[0353] ;

[0354] Using the Adam optimizer (Critic learning rate) =0.001, Actor learning rate =0.0003).

[0355] Optimization of the fusion mechanism of the five algorithms

[0356] After introducing A2C and SAC, the fusion weights are expanded to 5 ( , , , , ,satisfy ),pass Calibration is as follows:

[0357] =0.3 (DQN, basic value); =0.2 (PPO, stability); =0.2 (DDPG, smoothness); =0.15 (A2C, real-time response); =0.15 (SAC, exploratory).

[0358] Final action selection probability Integrating the output of five algorithms:

[0359] (3-13)

[0360] Where Z' is the normalization constant. For the A2C strategy probability, Let SAC be the random policy probability. The optimal action is still based on... choose.

[0361] Core optimization contributions:

[0362] A2C compensates for the batch processing latency of PPO by updating in real time, thereby improving the model's response speed to sudden situations (such as sudden changes in ambient temperature).

[0363] SAC's maximum entropy mechanism promotes exploration and avoids the model from ignoring better solutions (such as the novel thermal insulation-energy balance mode) due to over-reliance on fixed strategies during long-term operation.

[0364] After the five algorithms are integrated, the model forms a closed-loop optimization in five dimensions: value accuracy, strategy stability, action smoothness, response real-time performance, and exploration comprehensiveness, making it more adaptable to the complex dynamic operating environment of vacuum insulated tubular columns.

[0365] The embodiments described above are for illustrative purposes only and are not intended to limit the invention. Therefore, any changes in numerical values ​​or substitutions of equivalent elements should still fall within the scope of this invention.

[0366] The above detailed description will enable those skilled in the art to understand that the present invention can indeed achieve the aforementioned objectives and has complied with the provisions of the Patent Law.

[0367] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the invention. The above descriptions are merely preferred embodiments of the invention and are not intended to limit the invention. It should be noted that any modifications, equivalent substitutions, and improvements made within the spirit and principles of the invention should be included within the scope of protection of the invention.

[0368] It should be noted that the above description of the process is for illustrative purposes only and does not limit the scope of this specification. Those skilled in the art can make various modifications and changes to the process under the guidance of this specification. However, these modifications and changes remain within the scope of this specification.

[0369] The basic concepts have been described above. Obviously, for those skilled in the art who have read this application, the above disclosure is merely illustrative and does not constitute a limitation of this application. Although not explicitly stated herein, those skilled in the art may make various modifications, improvements, and corrections to this application. Such modifications, improvements, and corrections are suggested in this application, and therefore, such modifications, improvements, and corrections still fall within the spirit and scope of the exemplary embodiments of this application.

[0370] Furthermore, this application uses specific terms to describe its embodiments. For example, "an embodiment," "one embodiment," and / or "some embodiments" refer to a particular feature, structure, or characteristic related to at least one embodiment of this application. Therefore, it should be emphasized and noted that "an embodiment," "one embodiment," or "an alternative embodiment" mentioned twice or more in different positions in this specification do not necessarily refer to the same embodiment. In addition, certain features, structures, or characteristics in one or more embodiments of this application can be appropriately combined.

[0371] Furthermore, those skilled in the art will understand that aspects of this application can be described and illustrated through several patentable types or situations, including any new and useful combination of processes, machines, products, or substances, or any new and useful improvements thereof. Therefore, aspects of this application can be implemented entirely in hardware, entirely in software (including firmware, resident software, microcode, etc.), or a combination of hardware and software. All of the above hardware or software can be referred to as a “unit,” “module,” or “system.” Furthermore, aspects of this application can take the form of a computer program product embodied in one or more computer-readable media, wherein computer-readable program code is contained therein.

[0372] Furthermore, unless expressly stated in the claims, the order of processing elements and sequences, the use of numbers and letters, or other names described in this application are not intended to limit the order of the processes and methods of this application. Although some currently considered useful embodiments of the invention have been discussed in the foregoing disclosure by way of various examples, it should be understood that such details are for illustrative purposes only, and the appended claims are not limited to the disclosed embodiments; rather, the claims are intended to cover all modifications and equivalent combinations that conform to the substance and scope of the embodiments of this application. For example, although the implementation of the various components described above can be embodied in a hardware device, it can also be implemented as a purely software solution, such as an installation on an existing server or mobile device.

Claims

1. A method for intelligent control of a core tube column in vacuum thermal insulation based on reinforcement learning algorithm, characterized in that, Includes the following steps: S1: Collect operational data of the core tube column in the vacuum insulation and heat preservation system to form the raw dataset; S2: Clean and normalize the original dataset to obtain preprocessed data, and divide the preprocessed data into training set, validation set and test set; S3: Construct a reinforcement learning fusion model integrating deep Q-networks, proximal policy optimization, and deep deterministic policy gradients. Define state space and action space. The deep Q-network outputs expected cumulative rewards for combined actions in the action space through a neural network. Proximal policy optimization receives input from the state space and outputs action selection probabilities for combined actions. Simultaneously, it calculates the advantage function based on the expected cumulative rewards and optimizes the action selection probabilities through a policy clipping mechanism. Deep deterministic policy gradients receive input from the state space and output continuous weights for combined actions through an Actor network. After being constrained by the action selection probabilities of the proximal policy optimization algorithm, the value of the continuous weights is evaluated by a Critic network to enhance the smoothness of action selection. The three algorithms output the optimal control action through a fusion function. S4: Train the fusion model using the training set, calibrate the fusion weights of the fusion model using the validation set, test the performance of the fusion model using the test set, and finally obtain the optimal fusion model; S5: Real-time acquisition of the current operating data of the vacuum insulation core tube column, and after the same preprocessing as in step S2, the real-time state vector is obtained; S6: Input the real-time state vector into the optimal fusion model, and generate the optimal control action through collaborative reasoning and fusion function calculation of the fusion deep Q network, near-end policy optimization and deep deterministic policy gradient; S7: The vacuum insulation core tube column control system performs optimal control actions, driving the vacuum pump, auxiliary insulation device and insulation layer hydraulic mechanism to adjust the vacuum degree, insulation power and fit. In S3, the interaction process between the deep Q-network algorithm and the proximal policy optimization algorithm is as follows: the proximal policy optimization algorithm associates the output of the deep Q-network algorithm with the advantage function. The advantage function is the difference between the expected cumulative reward of the deep Q-network algorithm and the average state value of the proximal policy optimization algorithm. The average state value is the weighted sum of the action selection probability of the proximal policy optimization algorithm and the expected cumulative reward of the deep Q-network algorithm. In S3, the interaction between the proximal policy optimization algorithm and the deep deterministic policy gradient algorithm is as follows: the action continuous weights output by the Actor network of the deep deterministic policy gradient are multiplied by the action selection probability output by the proximal policy optimization algorithm to obtain the constrained action continuous weights, ensuring that the deep deterministic policy gradient tends to select actions that the proximal policy optimization considers stable.

2. The intelligent control method for vacuum insulation core tube based on reinforcement learning algorithm according to claim 1, characterized in that, It also includes S8: collecting new state data of the vacuum insulation core column after regulation, calculating temperature fluctuations, energy consumption changes and actual rewards, and forming a feedback dataset.

3. The intelligent control method for vacuum insulation core tube based on reinforcement learning algorithm according to claim 2, characterized in that, It also includes S9: based on the feedback dataset, the empirical samples of the fusion model are updated, and the parameters of the deep Q network, the near-end policy optimization and the deep deterministic policy gradient are incrementally optimized to obtain the updated fusion model.

4. The intelligent control method for vacuum insulation core tube based on reinforcement learning algorithm according to claim 3, characterized in that, It also includes S10: Repeat S5-S9 to achieve cyclical control of the core tube column in vacuum insulation until the preset convergence condition is met.

5. The intelligent control method for vacuum thermal insulation core tube based on reinforcement learning algorithm according to claim 1, characterized in that, In S1, the operating data includes the inlet temperature, outlet temperature, and middle section temperature of the medium inside the vacuum insulation core tube, the external ambient temperature of the vacuum insulation core tube, the vacuum degree of the interlayer of the vacuum insulation core tube, the pressure on the inner wall of the vacuum insulation core tube, the thermal conductivity coefficient of the insulation layer, and the current operating energy consumption.

6. The intelligent control method for vacuum insulation core tube based on reinforcement learning algorithm according to claim 1, characterized in that, In S4, the specific parameters for training the fusion model include: an experience replay pool capacity of 10,000 records, 10,000 training iterations, a learning rate of 0.001 for the deep Q-network, a learning rate of 0.0003 for the proximal policy optimization, a learning rate of 0.0001 for the deep deterministic policy gradient Actor network, and a learning rate of 0.001 for the Critic network; the training stopping condition is that the test set insulation accuracy is ≥88% and the average energy consumption is ≤500W.

7. The intelligent control method for vacuum thermal insulation core column based on reinforcement learning algorithm according to claim 1, characterized in that, In S7, the optimal control action process is as follows: the vacuum pump adjusts its power according to the vacuum level in the optimal control action; the auxiliary heat preservation device switches modes and power according to the level; and the heat insulation layer hydraulic mechanism adjusts the heat insulation layer adhesion according to the adhesion level.

8. The intelligent control method for vacuum insulation core tube based on reinforcement learning algorithm according to claim 4, characterized in that, In S10, the convergence condition is: the temperature fluctuation of 100 consecutive rounds of regulation is ≤ ±0.5℃ and the operating energy consumption is ≤ rated energy consumption. After the convergence condition is met, only status monitoring and effect evaluation are performed every 10 rounds. If the status fluctuates, the complete regulation process is restarted. If the convergence condition is not met, the regulation continues in a loop until the target is reached. The maximum number of consecutive regulation rounds is 1000 before triggering a manual intervention alarm.

Images