An optimization control method for parallel multi-devices in process industry and related device

Abstract

The application provides an optimization control method for parallel multi-devices in a process industry and related equipment, relates to the technical field of industrial process control, and comprises the following steps: acquiring operation data of each reaction unit; using a double-branch decoupling variational autoencoder to encode the operation data of the reaction unit, obtaining shared features and private features, and generating a unit context representation vector based on the private features of the reaction unit; performing cross-unit feature reorganization on the shared features and the private features, generating virtual conversion samples and constructing an enhanced data set; training a context-conditioned strategy network and a value network using the unit context representation vector and the enhanced data set, obtaining a trained strategy network; and inputting the operation data of a newly added reaction unit into the trained strategy network after encoding for decision optimization, to generate an optimized control action of the newly added reaction unit, thereby improving the control efficiency and reliability of parallel multi-devices in the process industry.

Description

Technical Field

[0001] This invention relates to the field of industrial process control technology, and in particular to an optimized control method and related equipment for parallel multi-device operation in process industries. Background Technology

[0002] Process industries, also known as continuous process industries, refer to production methods based on continuous processes. Specifically, they utilize a single solid, liquid, or gaseous material as raw material, undergoing a series of chemical, physical, and biological reactions and different technological steps to ultimately transform it into a finished product. The production process is continuous and stable. Typical process industries include petrochemicals, chemicals, pharmaceuticals, and food processing, which typically require a high degree of automation and precise control to ensure the stability of the production process and the consistency of product quality.

[0003] Operational optimization in process industries focuses on improving production efficiency, product quality, and energy utilization by adjusting key parameter settings. Reinforcement learning has been extensively explored in this field due to its adaptive decision-making capabilities. However, in industrial systems composed of multiple identical reaction units operating in parallel, the scalability of reinforcement learning is severely limited.

[0004] In real-world industrial scenarios, parallel reaction units are designed identically. However, due to differences in deployment time, usage levels, maintenance plans, or operating conditions, different units often exhibit different operational characteristics. These differences mean that optimization strategies learned for one unit cannot be directly transferred to other units. Each unit often needs to be independently modeled and retrained, resulting in significant consumption of computing resources and increased operating costs.

[0005] Meta-reinforcement learning offers a potential solution to the aforementioned problems. Among them, context-based offline meta-reinforcement learning methods, by learning from pre-collected multi-task historical datasets, completely eliminate the need for online environment interaction, significantly improving the security of industrial applications and reducing testing costs. However, existing offline contextual meta-reinforcement learning methods still face many challenges when applied to complex industrial scenarios.

[0006] First, in industrial scenarios, different units may exhibit highly similar or even identical dynamic characteristics under specific operating conditions. Standard context encoders struggle to extract meaningful distinguishing features from the original operational trajectories, leading to feature entanglement. This entanglement severely weakens the effectiveness of task-conditioning strategies and significantly hinders cross-unit policy transfer. Second, due to the prohibitive costs and safety risks of continuous testing, industrial systems typically employ a rotational testing scheme, with each reaction unit taking turns for short-term testing and data recording. This inevitably results in the datasets of each unit exhibiting sparsity and fragmentation over time. While generative models such as variational autoencoders can alleviate the problem of insufficient samples through data augmentation techniques, when the original data distribution itself is poorly captured due to excessively long time intervals and insufficient coverage, the generated synthetic samples may deviate from actual physical laws and dynamic characteristics, thus misleading the policy learning process. Furthermore, traditional reinforcement learning methods rely on extensive online interactions with the environment to collect training data. However, in high-risk industrial production environments, online exploration may lead to production anomalies or even safety accidents, severely limiting its deployment. While using offline historical data for policy learning is a necessary prerequisite for ensuring the security of industrial applications, offline learning also presents unique technical challenges, such as overestimation of the value of out-of-distribution actions. Finally, if each reaction unit is modeled and trained independently, the overall cost of this independent modeling approach becomes prohibitive in large-scale industrial systems containing dozens or even hundreds of parallel units, and the similarities and commonalities between different units are not effectively utilized. Therefore, the reliability and efficiency of current cross-unit operation optimization in process industries are relatively low. Summary of the Invention

[0007] This invention provides an optimized control method and related equipment for parallel multiple devices in the process industry, with the aim of improving the control efficiency and reliability of parallel multiple devices in the process industry.

[0008] To achieve the above objectives, this invention provides an optimization control method for parallel multi-device systems in process industries. The process industry includes multiple parallel and identically designed reaction units. The optimization control method includes:

[0009] Step 1: Obtain the operational data for each reaction unit. The operational data includes multiple transition samples, and each sample includes state, action, reward, and next state.

[0010] Step 2: For each reaction unit, the operation data of the reaction unit is encoded using a dual-branch decoupled variational autoencoder to obtain the shared features and private features of each reaction unit, and the unit context representation vector of each reaction unit is generated based on the private features of the reaction unit.

[0011] Step 3: Perform cross-unit feature recombination on the shared and private features of each reaction unit to generate virtual transformation samples and construct an enhanced dataset;

[0012] Step 4: Use the unit context representation vector of each reaction unit and the augmented dataset to train the context-conditional policy network and value network to obtain the trained policy network.

[0013] Step 5: Encode the operation data of the newly added reaction unit and input it into the trained policy network for decision optimization to generate the optimized control action of the newly added reaction unit.

[0014] Furthermore, the dual-branch decoupled variational autoencoder includes a shared feature encoder, a private feature encoder, and a decoder;

[0015] Shared feature encoders are used to extract shared features that are invariant across units;

[0016] The private feature encoder is used to extract the behavioral features of each reaction unit;

[0017] The decoder reconstructs virtual transformation samples based on shared and behavioral features.

[0018] Furthermore, the loss function of the two-branch decoupled variational autoencoder consists of the variational autoencoding loss function, the distance metric learning loss function, and the orthogonality constraint loss function, and its expression is:

[0019] ;

[0020] in, This represents the total loss value of the two-branch decoupled variational autoencoder. This represents the variational autoencoder loss function. This represents the distance metric learning loss function. Represents the orthogonality constraint loss function. , , Represents the learnable uncertainty parameter. It represents the natural logarithm.

[0021] Furthermore, based on the private features of the reaction units, a unit context representation vector is generated for each reaction unit, including:

[0022] The unit context representation vector for each reaction unit is obtained by averaging the private features extracted from all operational data of each reaction unit. The calculation expression is as follows:

[0023] ;

[0024] in, Indicates the first The unit context representation vector of each reaction unit, This indicates calculating the mean. This represents a private feature encoder. Indicates the first Number of converted samples per reaction unit Indicates the first In the first reaction unit Private features of each transformed sample.

[0025] Furthermore, cross-unit feature recombination is performed on the shared and private features of each reaction unit to generate virtual transformation samples, including:

[0026] Obtain the shared and private characteristics of any two reaction units;

[0027] The shared and private features of two reaction units are combined with cross-features to construct a cross-potential representation, and the construction expression is as follows:

[0028] ;

[0029] ;

[0030] in, , Representing reaction units and Shared features , Representing reaction units and Private characteristics, Indicates by Shared features and Cross-potential representations of private feature combinations Indicates by Shared features and Cross-potential representation of private feature combinations Indicates feature concatenation operation;

[0031] The cross-potential representation is input into the decoder for decoding to obtain a virtual transformed sample.

[0032] Furthermore, the policy network is trained using the following loss function, the expression of which is:

[0033] ;

[0034] in, This represents the loss value of the policy network. Indicates state, Indicates an action, Indicates the first Augmented dataset of each reaction unit Represents the policy network, Represents the current value network. Parameters representing the value network, Indicates the first The unit context representation vector of each reaction unit, Indicates temperature parameter, Represents the given state of the policy network and unit context vector Choose action The probability of.

[0035] Furthermore, the value network is trained using a conservative Q-learning objective, and the expression for the training loss function is:

[0036] ;

[0037] in, This represents the conservative Q-learning loss value. Indicates state, Indicates an action, Indicates the first Augmented dataset of each reaction unit The Q-value function represents context-conditionalization. Indicates the first The unit context representation vector of each reaction unit, This represents the result of the Bellman operator acting on the Q-value function. For conservative weighting, This represents the current policy network.

[0038] This invention also provides an optimized control device for parallel multi-device applications in process industries, where the process industry includes multiple parallel and identically designed reaction units. The optimized control device includes:

[0039] The acquisition module is used to acquire the operational data of each reaction unit. The operational data includes multiple transition samples, and each sample includes the state, action, reward and next state.

[0040] The encoding module is used to encode the operation data of each reaction unit using a dual-branch decoupled variational autoencoder, to obtain the shared features and private features of each reaction unit, and to generate the unit context representation vector of each reaction unit based on the private features of the reaction unit.

[0041] The building module is used to perform cross-unit feature recombination of shared and private features of each reaction unit, generate virtual transformation samples, and build an augmented dataset;

[0042] The training module is used to train the context-conditional policy network and value network using the unit context representation vector of each reaction unit and the augmented dataset, resulting in the trained policy network.

[0043] The decision optimization module is used to encode the operation data of newly added reaction units and input it into the trained policy network for decision optimization, generating optimized control actions for the newly added reaction units.

[0044] The present invention also provides a terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements an optimized control method for multiple parallel devices in the process industry.

[0045] The present invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements an optimized control method for parallel multi-device operation in process industries.

[0046] The above-described solution of the present invention has the following beneficial effects:

[0047] This invention acquires the operational data of each reaction unit; encodes the operational data of the reaction units using a two-branch decoupled variational autoencoder to obtain the shared and private features of each reaction unit, and generates a unit context representation vector for each reaction unit based on the private features; performs cross-unit feature recombination on the shared and private features of each reaction unit to generate virtual transformation samples and construct an augmented dataset; trains a context-conditionalized policy network and a value network using the unit context representation vector of each reaction unit and the augmented dataset to obtain the trained policy network; and encodes the operational data of newly added reaction units and inputs it into the trained policy network for decision optimization to generate optimized control actions for the newly added reaction units. Compared with existing technologies, this invention utilizes a two-branch decoupled variational autoencoder to systematically separate the general features shared across units from the unit-specific behavioral features. This approach effectively solves the feature entanglement problem in high-similarity scenarios, generating discriminative task representation vectors. The cross-unit feature recombination strategy fully leverages the structural similarity between units, generating virtual transformation samples by cross-combining shared and private features from different units. This effectively alleviates data temporal sparsity caused by rotational testing. The generated virtual samples serve as intermediate tasks, filling gaps in the task space and promoting the generalization of meta-policies across a wider range of task variations. The context-conditional policy network encodes policies from multiple units through a single network, achieving parameter sharing and knowledge transfer. The conservative Q-learning objective ensures the safety and robustness of offline learning. It enables efficient cross-unit policy transfer under completely offline conditions, significantly reducing repetitive modeling costs and improving the control efficiency and reliability of parallel multi-device systems in the process industry, providing effective technical support for the intelligent upgrading of the process industry.

[0048] Other beneficial effects of the present invention will be described in detail in the following detailed description section. Attached Figure Description

[0049] Figure 1 This is a flowchart illustrating an embodiment of the present invention;

[0050] Figure 2 This is a schematic diagram of a system of multiple parallel continuous stirred tank reactors;

[0051] Figure 3 This is a schematic diagram of the control results of the Offline PEARL model;

[0052] Figure 4 This is a schematic diagram of the control results of the FOCAL model;

[0053] Figure 5 A schematic diagram of the control results of the CORRO model;

[0054] Figure 6This is a schematic diagram of the control results according to an embodiment of the present invention;

[0055] Figure 7 This is a schematic diagram of the optimized control device in an embodiment of the present invention;

[0056] Figure 8 This is a schematic diagram of the structure of the terminal device in an embodiment of the present invention. Detailed Implementation

[0057] To make the technical problems, solutions, and advantages of this invention clearer, a detailed description will be provided below with reference to the accompanying drawings and specific embodiments. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0058] In the description of this invention, it should be noted that the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0059] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a locking connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0060] Furthermore, the technical features involved in the different embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0061] This invention addresses existing problems by providing an optimized control method and related equipment for parallel multi-device operation in process industries.

[0062] like Figure 1 As shown, embodiments of the present invention provide an optimization control method for parallel multi-device systems in process industries. The process industry includes multiple parallel reaction units with identical designs. The optimization control method includes:

[0063] Step 1: Obtain the operational data for each reaction unit. The operational data includes multiple transition samples, and each sample includes state, action, reward, and next state.

[0064] Step 2: For each reaction unit, the operation data of the reaction unit is encoded using a dual-branch decoupled variational autoencoder to obtain the shared features and private features of each reaction unit, and the unit context representation vector of each reaction unit is generated based on the private features of the reaction unit.

[0065] Step 3: Perform cross-unit feature recombination on the shared and private features of each reaction unit to generate virtual transformation samples and construct an enhanced dataset;

[0066] Step 4: Use the unit context representation vector of each reaction unit and the augmented dataset to train the context-conditional policy network and value network to obtain the trained policy network.

[0067] Step 5: Encode the operation data of the newly added reaction unit and input it into the trained policy network for decision optimization to generate the optimized control action of the newly added reaction unit.

[0068] In this embodiment of the invention, the operating data of the reaction unit is denoted as follows: Each reaction unit shares the same state-action space, but has different system dynamics, and each set of operational data includes multiple transition samples. ,in For state, For action, As a reward, This is the next state.

[0069] In this embodiment of the invention, historical operation data of the reaction unit can be obtained by using devices such as sensors or control systems. Multiple reaction units are parallel units with similar structures but different dynamic characteristics, such as continuous stirred tank reactor systems with the same design but different operating years, distillation columns with different maintenance plans, etc., that is, they share the same state-action space definition, but due to factors such as equipment aging and differences in operating conditions, there are differences in the dynamics of the system.

[0070] The preferred embodiment is that the dual-branch decoupled variational autoencoder includes a shared feature encoder, a private feature encoder, and a decoder;

[0071] The shared feature encoder is used to extract the cross-unit invariant operating patterns of each reaction unit, i.e., shared features;

[0072] The private feature encoder is used to extract the dynamic behavior of each reaction unit, i.e., behavioral features;

[0073] The decoder reconstructs virtual transformation samples based on shared and behavioral features.

[0074] In this embodiment of the invention, for the first The encoding process of a shared feature encoder for each reaction unit is as follows:

[0075] ;

[0076] The encoding process of the private feature encoder is as follows:

[0077] ;

[0078] in, Indicates the first Shared characteristics of each reaction unit Indicates the first The private characteristics of each reaction unit Indicates the first A conversion sample of each reaction unit. Indicates a shared feature encoder. This indicates a private feature encoder.

[0079] It should be noted that the embodiments of the present invention do not limit the structure of shared feature encoders and private feature encoders, and can be selected according to the characteristics of the data. When there is continuous trajectory data, a sequence model such as GRU is used, but when there is only discrete transformation data, a fully connected network MLP can be used. It is worth mentioning that the shared features capture the common operating mode and physical laws of all reaction units, while the private features accurately characterize the specific behaviors of each reaction unit caused by differences in equipment, aging degree, etc. Through explicit decoupling, discriminability can still be maintained even in areas where units are highly similar.

[0080] Specifically, to avoid manually adjusting the loss weights and improve training stability, the loss function of the dual-branch decoupled variational autoencoder in this embodiment of the invention consists of a variational autoencoding loss function, a distance metric learning loss function, and an orthogonality constraint loss function, expressed as follows:

[0081] ;

[0082] in, This represents the total loss value of the two-branch decoupled variational autoencoder. This represents the variational autoencoder loss function. This represents the distance metric learning loss function. Represents the orthogonality constraint loss function. , , Represents the learnable uncertainty parameter. It represents the natural logarithm.

[0083] Specifically, the expression for the variational autoencoder loss function is:

[0084] ;

[0085] in, Indicates the first A conversion sample of each reaction unit. Indicates the first Shared characteristics of each reaction unit Indicates the first The private characteristics of each reaction unit Indicates decoder, Indicates a shared feature encoder. This represents a private feature encoder. Denotes KL divergence, Represents the standard normal distribution. , These are weighting coefficients used to balance reconstruction accuracy and distribution regularization; Let denot be the vector norm. The first term is the reconstruction loss, which ensures the fidelity of encoding and decoding. The second and third terms are KL divergence regularization terms, which constrain the feature distribution to approximate the standard normal distribution.

[0086] Specifically, the expression for the metric learning loss function is:

[0087] ;

[0088] in, Indicates an indicator function, , These represent the identifiers of the reaction units to which the two converted samples belong. , Representing reaction units and Private characteristics, As a scale factor, To prevent small constants from being divided by zero; the first term promotes the close clustering of private features of the same unit, and the second term promotes the separation of private features of different units; this loss is based on unit identifiers and enhances the discriminativeness of private features in the latent space.

[0089] Specifically, the expression for the orthogonality constraint loss function is:

[0090]

[0091] in, This represents the transpose of shared features. This loss penalizes the correlation between shared features and private features, forces feature decoupling, prevents feature space entanglement, and improves the interpretability of task representation.

[0092] Specifically, a unit context representation vector for each reaction unit is generated based on the private features of the reaction unit, including:

[0093] The unit context representation vector for each reaction unit is obtained by averaging the private features extracted from all operational data of each reaction unit. The calculation expression is as follows:

[0094] ;

[0095] in, Indicates the first The unit context representation vector of each reaction unit, This indicates calculating the mean. This represents a private feature encoder. Indicates the first Number of converted samples per reaction unit Indicates the first In the first reaction unit Private features of each transformed sample.

[0096] In this embodiment of the invention, the cell context representation vector encodes the specific dynamic characteristics of the reaction cell for subsequent policy and value network conditionation.

[0097] Specifically, cross-unit feature recombination is performed on the shared and private features of each reaction unit to generate virtual transformation samples, including:

[0098] Obtain the shared and private characteristics of any two reaction units;

[0099] The shared and private features of two reaction units are combined with cross-features to construct a cross-potential representation, and the construction expression is as follows:

[0100] ;

[0101] ;

[0102] in, , Representing reaction units and Shared features , Representing reaction units and Private characteristics, Indicates by Shared features and Cross-potential representation of private feature combinations Indicates by Shared features and Cross-potential representation of private feature combinations Indicates feature concatenation operation;

[0103] The cross-latent representation is input to the decoder for decoding, resulting in a virtual transformed sample, expressed as:

[0104] ;

[0105] in, Represents reaction unit and Virtual transformation samples between them.

[0106] The embodiments of the present invention employ a cross-unit feature recombination method to generate a virtual unit representation between two real units.

[0107] Specifically, an augmented dataset is constructed based on virtual transformation samples, that is, the operational data is mixed with the virtual transformation samples using the following formula:

[0108] ;

[0109] The proportion of virtual samples is an adjustable parameter. .

[0110] In this embodiment of the invention, the cross-unit feature recombination strategy can alleviate temporal sparsity; the shared features gather information from the hen sampling regions of all units, and the generated virtual transformation fills the time gap of the data in a single unit. Compared with simply interpolating within a single unit, it utilizes cross-unit information; the virtual transformation represents the transition dynamic between real units, forming a more continuous distribution in the task space, and promoting the generalization of the meta-strategy in a wider range of task transformations; unlike generative models that are entirely based on statistics, the virtual samples inherit the physical constraints learned by the decoder, and the combination of private features and shared features ensures the rationality of the method.

[0111] This invention utilizes an augmented dataset to train a context-conditional policy network and a value network. The policy network and value network treat each reaction unit as a task, with its latent task represented as a unit context vector. The input is the current state. and unit context vector The output is the operation action. The probability distribution is used to generate specific control strategies; value network The input is the state. ,action and unit context vector The output is a Q-value estimate of the state-action pair, which is used to evaluate the long-term value of taking an action in a specific unit.

[0112] Specifically, the policy network is trained using the following loss function, the expression of which is:

[0113] ;

[0114] in, This represents the loss value of the policy network. Indicates state, Indicates an action, Indicates the first Augmented dataset of individual reaction units Represents the policy network, Represents the current value network. Parameters representing the value network, Indicates the first The unit context representation vector of each reaction unit, The first term represents the temperature parameter, used to balance utilization and exploration; the second term is the entropy regularization term, which encourages strategy diversity. Represents the given state of the policy network and unit context vector Choose action The probability of.

[0115] Specifically, the value network is trained using a conservative Q-learning objective, and the expression for the training loss function is as follows:

[0116] ;

[0117] in, This represents the conservative Q-learning loss value. The first term is the standard temporal difference TD loss function, and the second term is a conservative regularization term used to penalize the expected Q-value under the current policy, encouraging the dataset to perform conservative learning. The Q-value of the action is used to ensure that the Q-function provides a conservative estimate of actions outside the data set. Indicates state, Indicates an action, Indicates the first Augmented dataset of individual reaction units The Q-value function represents context-conditionalization. Indicates the first The unit context representation vector of each reaction unit, This represents the result of the Bellman operator acting on the Q-value function. , Indicates an immediate reward. Indicates the discount factor. Indicates the next state. Indicates the next action. This is a conservative weight, used to control the degree of conservatism. This represents the current policy network.

[0118] In this embodiment of the invention, when faced with a newly added reaction unit, only a small amount of historical operation data of the reaction unit needs to be collected. Then, the unit context vector is inferred using a two-branch decoupled variational autoencoder. Then, the meta-policy is directly applied to generate the optimized control action of the reaction unit.

[0119] The method provided in this invention will be illustrated by way of example with reference to a specific example:

[0120] Taking the operation process of a multi-group continuous stirred tank reactor system with parallel structural features as an example, the specific structure of the system is as follows: Figure 2 As shown, including There are 10 reaction units, with reaction unit 1 being Unit 1, reaction unit 2 being Unit 2, ..., and reaction unit 10 being Unit 10. The feed flow rate for each reactor unit is [value]. The outlet concentration of each reactor is [value missing]. The temperature of each reactor, The feed concentration is... For feed temperature, The heat of the coolant in each reactor. The reaction mechanism of this reactor system is as follows: Figure 2 As shown in the top left corner, the reaction kinetic equation is:

[0121] ;

[0122] ;

[0123] in, ; ; ; ; ; ; ; .

[0124] like Figure 2 As shown in the lower left corner, different reaction units have different activation energies due to differences in their deployment time and degree of use. ,activation energy The range of values ​​is Among them, the reaction unit Reaction unit and reaction unit This represents units with different activation energies E. It is worth noting that some reaction units (such as reaction units) Reaction unit They exhibit the same reaction kinetics in specific concentration regions (shared regions), making it difficult for standard context encoding methods to extract meaningful distinguishing features.

[0125] The system state space is 2-dimensional and includes the reactor outlet concentration. and temperature The control space is two-dimensional, including the heat of the coolant in each reactor. and feed flow rate ,in , .

[0126] The optimization goal is to adjust the heat of the coolant. and feed flow rate This will increase the outlet concentration of each reactor. Stabilize at the target value .

[0127] The average reward for each round is used as the main performance indicator to evaluate the control performance of the obtained cross-unit operation optimization model, and the calculation is as follows:

[0128] ;

[0129] in, Indicates the number of test rounds. Indicates the first The cumulative reward for each round. The reward function is defined as follows: The deviation from the target concentration is the reward value; the closer the reward value is to 0, the better the control performance.

[0130] To systematically evaluate the superiority of the proposed method, the following three representative offline meta-reinforcement learning algorithms are selected as benchmarks:

[0131] Offline PEARL (Offline Meta-Reinforcement Learning with Probabilistic Context Encoding): A probabilistic context-based meta-reinforcement learning method that conditions policies by inferring latent task representations. In this embodiment, it is adapted to an offline setting for fair comparison.

[0132] FOCAL (Offline Meta-Reinforcement Learning Based on Distance Metric Learning): A fully offline meta-reinforcement learning algorithm that integrates distance metric learning and behavior regularization to improve task representation learning and policy generalization.

[0133] CORRO (Offline Meta-Reinforcement Learning Based on Contrastive Representation Learning): A method based on contrastive representation learning to enhance the robustness of task encoding to adapt to offline meta-reinforcement learning scenarios.

[0134] The control results of the reactor system controlled using the Offline PEARL model are as follows: Figure 3 As shown, the control results of the reactor system controlled using the FOCAL model are as follows: Figure 4As shown, the control results of the reactor system controlled using the CORRO model are as follows: Figure 5 As shown, the control results of the reactor system controlled by the method of this embodiment of the invention are as follows: Figure 6 As shown, Figures 3-6 The horizontal axis represents the training period, and the vertical axis represents the average reward. The average reward per round is shown in Table 1.

[0135] Table 1

[0136] In a simulation environment, the method proposed in this invention demonstrates significant performance advantages. Specifically, the method achieves an average cumulative reward of -0.0714, representing an improvement of approximately 65% ​​compared to the best-performing baseline algorithm, CORRO. This significant improvement validates the effectiveness of the feature decoupling and cross-cell virtual sample generation mechanism driven by the dual-branch decoupled variational autoencoder in optimizing complex cross-cell operations.

[0137] Of particular note is that the traditional probabilistic contextual reinforcement learning method, Offline PEARL, exhibits significant performance degradation in this environment, with an average reward of -1.0201, far lower than the method proposed in this embodiment. This result indicates that traditional contextual encoding mechanisms may lead to feature entanglement and task representation failure in high-similarity unit scenarios, while the method proposed in this embodiment, relying on the structured feature representation obtained by a dual-branch decoupled variational autoencoder, effectively alleviates the above problems.

[0138] Further analysis from the perspective of variance reveals that the method proposed in this embodiment exhibits superior learning stability. Its standard deviation is 0.0163, which is 89.5% lower than Offline PEARL's 0.1554, 90.7% lower than FOCAL's 0.1754, and 69.9% lower than CORRO's 0.0541. This indicates that the feature decoupling mechanism implemented by introducing a multi-objective joint loss function and the conservative Q-learning objective not only improves the expected performance but also significantly enhances the stability and robustness of the algorithm during policy convergence.

[0139] From the perspective of the learning process, such as Figure 6 As shown, the method proposed in this embodiment of the invention exhibits superior learning efficiency and convergence stability. The method demonstrates a stable upward trend in the early stages of training and rapidly converges to a high reward level of approximately -0.0714 after about 50 training epochs, with minimal overall fluctuation. This rapid and stable convergence process indicates that feature decoupling based on a two-branch decoupled variational autoencoder provides the agent with explicit structural priors, effectively accelerating policy learning.

[0140] Based on the comprehensive experimental results, the performance advantage of the proposed method mainly stems from the organic coupling between its three core modules: the dual-branch decoupled variational autoencoder can effectively extract the shared and private features of the system, providing a structured representation for knowledge transfer and task differentiation between units; the cross-unit virtual sample generation module captures the transition dynamics between units through feature recombination, enhancing the data coverage; and the context-conditional Actor-Critic network, combined with the conservative Q-learning objective, achieves robust offline policy optimization and rapid cross-unit adaptation.

[0141] This invention, in its embodiments, acquires operational data for each reaction unit; encodes the operational data of the reaction units using a two-branch decoupled variational autoencoder to obtain shared and private features for each reaction unit, and generates a unit context representation vector for each reaction unit based on its private features; performs cross-unit feature recombination on the shared and private features of each reaction unit to generate virtual transformation samples and construct an augmented dataset; trains a context-conditionalized policy network and a value network using the unit context representation vector of each reaction unit and the augmented dataset to obtain a trained policy network; and encodes the operational data of newly added reaction units and inputs it into the trained policy network for decision optimization to generate optimized control actions for the newly added reaction units. Compared with existing technologies, this invention, using a two-branch decoupled variational autoencoder, can systematically separate cross-unit shared general features from unit-specific actions. The system effectively solves the feature entanglement problem in high-similarity scenarios, generating discriminative task representation vectors. The cross-unit feature recombination strategy fully utilizes the structural similarity between units, generating virtual transformation samples by cross-combining shared and private features from different units. This effectively alleviates the data temporal sparsity caused by rotational testing. The generated virtual samples serve as intermediate tasks to fill gaps in the task space, promoting the generalization of meta-policies across a wider range of task variations. The context-conditional policy network encodes policies from multiple units through a single network, achieving parameter sharing and knowledge transfer. The conservative Q-learning objective ensures the safety and robustness of offline learning. It enables efficient cross-unit policy transfer under completely offline conditions, significantly reducing repetitive modeling costs and improving the control efficiency and reliability of parallel multi-device systems in the process industry, providing effective technical support for the intelligent upgrading of the process industry.

[0142] Corresponding to the optimization control method for parallel multiple devices in process industries described in the above embodiments, such as Figure 7 As shown, this embodiment of the invention also provides an optimization control device 100 for parallel multi-device systems in process industries. The process industry includes multiple parallel and identically designed reaction units. The optimization control device 100 includes:

[0143] The acquisition module 101 is used to acquire the operation data of each reaction unit. The operation data includes multiple transformation samples, and each sample includes state, action, reward and next state.

[0144] The encoding module 102 is used to encode the operation data of each reaction unit using a dual-branch decoupled variational autoencoder to obtain the shared features and private features of each reaction unit, and to generate the unit context representation vector of each reaction unit based on the private features of the reaction unit.

[0145] Module 103 is used to perform cross-unit feature recombination on the shared and private features of each reaction unit, generate virtual transformation samples, and construct an enhanced dataset;

[0146] Training module 104 is used to train the context-conditional policy network and value network using the unit context representation vector of each reaction unit and the augmented dataset, so as to obtain the trained policy network.

[0147] The decision optimization module 105 is used to encode the operation data of the newly added reaction unit and input it into the trained policy network for decision optimization, thereby generating the optimized control action of the newly added reaction unit.

[0148] It should be noted that the information interaction and execution process between the above-mentioned devices / units are based on the same concept as the method embodiments of the present invention. For details on their specific functions and technical effects, please refer to the method embodiments section, which will not be repeated here.

[0149] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of the embodiments of the present invention. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0150] This invention also provides a terminal device, such as... Figure 8 As shown, the terminal device D10 of this embodiment includes: at least one processor D100 ( Figure 8The diagram shows only one processor, a memory D101, and a computer program D102 stored in the memory D101 and executable on the at least one processor D100. When the processor D100 executes the computer program D102, it implements the above-described optimized control method for parallel multiple devices in process industries.

[0151] The terminal device D10 can be a desktop computer, laptop, handheld computer, server, server cluster, or cloud server, etc. This terminal device may include, but is not limited to, a processor D100 and a memory D101. Those skilled in the art will understand that... Figure 7 This is merely an example of terminal device D10 and does not constitute a limitation on terminal device D10. It may include more or fewer components than shown in the figure, or combine certain components, or different components, such as input / output devices, network access devices, etc.

[0152] The processor D100 can be a central processing unit (CPU), or it can be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0153] In some embodiments, the memory D101 may be an internal storage unit of the terminal device D10, such as a hard disk or memory of the terminal device D10. In other embodiments, the memory D101 may be an external storage device of the terminal device D10, such as a plug-in hard disk, smart media card (SMC), secure digital card (SD), flash card, etc., equipped on the terminal device D10. Furthermore, the memory D101 may include both internal and external storage units of the terminal device D10. The memory D101 is used to store the operating system, applications, bootloader, data, and other programs, such as the program code of the computer program. The memory D101 can also be used to temporarily store data that has been output or will be output.

[0154] It should be noted that the information interaction and execution process between the above-mentioned devices / units are based on the same concept as the method embodiments of the present invention. For details on their specific functions and technical effects, please refer to the method embodiments section, which will not be repeated here.

[0155] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of the embodiments of the present invention. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0156] This invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements an optimized control method for parallel multi-device operation in process industries.

[0157] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the embodiments of the present invention can implement all or part of the processes in the methods described above by instructing related hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include at least: any entity or device capable of carrying the computer program code to a building device / terminal device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks.

[0158] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. An optimization control method for parallel multi-device operation in process industries, characterized in that, The process industry comprises multiple parallel and identically designed reaction units, and the optimization control method includes: Step 1: Obtain the operation data for each reaction unit. The operation data includes multiple transition samples, and each sample includes a state, action, reward, and next state. Step 2: For each reaction unit, the operation data of the reaction unit is encoded using a dual-branch decoupled variational autoencoder to obtain the shared features and private features of each reaction unit. Based on the private features of the reaction unit, a unit context representation vector of each reaction unit is generated. The dual-branch decoupled variational autoencoder includes a shared feature encoder, a private feature encoder, and a decoder. The shared feature encoder is used to extract shared features that are invariant across units; The proprietary feature encoder is used to extract the behavioral features of each reaction unit; The decoder reconstructs virtual conversion samples based on shared features and behavioral features; Step 3 involves cross-unit feature recombination of the shared and private features of each reaction unit to generate virtual transformation samples, including: Obtain the shared and private characteristics of any two reaction units; The shared and private features of two reaction units are combined with cross-features to construct a cross-potential representation, and the construction expression is as follows: ； ； in, , Representing reaction units and Shared features , Representing reaction units and Private characteristics, Indicates by Shared features and Cross-potential representation of private feature combinations Indicates by Shared features and Cross-potential representation of private feature combinations Indicates feature concatenation operation; The cross-potential representation is input into the decoder for decoding to obtain virtual transformed samples and construct an augmented dataset; Step 4: Use the unit context representation vector of each reaction unit and the augmented dataset to train the context-conditional policy network and value network to obtain the trained policy network. Step 5: Encode the operation data of the newly added reaction unit and input it into the trained policy network for decision optimization to generate the optimized control action of the newly added reaction unit.

2. The optimization control method for parallel multi-device operation in process industries according to claim 1, characterized in that, The loss function of the dual-branch decoupled variational autoencoder consists of a variational autoencoder loss function, a distance metric learning loss function, and an orthogonality constraint loss function, and its expression is: ； in, This represents the total loss value of the two-branch decoupled variational autoencoder. This represents the variational autoencoder loss function. This represents the distance metric learning loss function. Represents the orthogonality constraint loss function. , , Represents the learnable uncertainty parameter. It represents the natural logarithm.

3. The optimization control method for parallel multi-device operation in process industries according to claim 1, characterized in that, Based on the private features of the reaction units, a unit context representation vector is generated for each reaction unit, including: The unit context representation vector for each reaction unit is obtained by averaging the private features extracted from all operational data of each reaction unit. The calculation expression is as follows: ； in, Indicates the first The unit context representation vector of each reaction unit, This indicates calculating the mean. This represents a private feature encoder. Indicates the first Number of converted samples per reaction unit Indicates the first In the first reaction unit Private features of each transformed sample.

4. The optimization control method for parallel multi-device operation in process industries according to claim 1, characterized in that, The policy network is trained using the following loss function, the expression of which is: ； in, This represents the loss value of the policy network. Indicates state, Indicates an action, Indicates the first Augmented dataset of individual reaction units Represents the policy network, Represents the current value network. Parameters representing the value network, Indicates the first The unit context representation vector of each reaction unit, Indicates temperature parameter, Represents the given state of the policy network and unit context vector Choose action The probability of.

5. The optimization control method for parallel multi-device operation in process industries according to claim 1, characterized in that, The value network is trained using a conservative Q-learning objective, and the expression for the training loss function is as follows: ； in, This represents the conservative Q-learning loss value. Indicates state, Indicates an action, Indicates the first Augmented dataset of individual reaction units The Q-value function represents context-conditionalization. Indicates the first The unit context representation vector of each reaction unit, This represents the result of the Bellman operator acting on the Q-value function. For conservative weighting, This represents the current policy network.

6. An optimized control device for parallel multi-device operation in process industries, characterized in that, The process industry comprises multiple parallel and identically designed reaction units, and the optimization control device includes: The acquisition module is used to acquire the operation data of each reaction unit. The operation data includes multiple transition samples, and each sample includes a state, action, reward and next state. The encoding module is used to encode the operation data of each reaction unit using a dual-branch decoupled variational autoencoder to obtain the shared features and private features of each reaction unit, and to generate the unit context representation vector of each reaction unit based on the private features of the reaction unit. The dual-branch decoupled variational autoencoder includes a shared feature encoder, a private feature encoder, and a decoder. The shared feature encoder is used to extract shared features that are invariant across units; The proprietary feature encoder is used to extract the behavioral features of each reaction unit; The decoder reconstructs virtual conversion samples based on shared features and behavioral features; A construction module is used to perform cross-unit feature recombination on the shared and private features of each reaction unit to generate virtual transformation samples, including: Obtain the shared and private characteristics of any two reaction units; The shared and private features of two reaction units are combined with cross-features to construct a cross-potential representation, and the construction expression is as follows: ； ； in, , Representing reaction units and Shared features , Representing reaction units and Private characteristics, Indicates by Shared features and Cross-potential representation of private feature combinations Indicates by Shared features and Cross-potential representation of private feature combinations Indicates feature concatenation operation; The cross-potential representation is input into the decoder for decoding to obtain virtual transformed samples and construct an augmented dataset; The training module is used to train the context-conditional policy network and value network using the unit context representation vector of each reaction unit and the augmented dataset, so as to obtain the trained policy network. The decision optimization module is used to encode the operation data of newly added reaction units and input it into the trained policy network for decision optimization, generating optimized control actions for the newly added reaction units.

7. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the optimized control method for parallel multiple devices in process industries as described in any one of claims 1 to 5.

8. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the optimized control method for parallel multiple devices in process industries as described in any one of claims 1 to 5.

Images