A cross-device control policy migration and online fine-tuning method based on latent space alignment

Abstract

The present application relates to a kind of cross-device control strategy migration and online fine-tuning method based on latent space alignment.The present application includes the offline dataset of source device and target device, configures observation encoder and action mapping layer for each device, generates the action vector available for each device by uniform latent space and a set of strategy parameters shared across devices;Strategy learning is carried out using offline trajectory data of source device, while aligning the latent space distribution of source side and target side on the working condition sub-target divided by scene semantic anchor point, obtaining the initial strategy that can be migrated to target device;Initial strategy is deployed on target device side, and the target device running process is observation encoding-shared strategy-device action mapping, to obtain candidate action, and then output terminal execution instruction after security constraint verification and necessary correction, online constraint fine-tuning is carried out when trigger condition is met, and online running strategy is updated.The method can reduce the cost of repeated training for each device under similar control tasks.

Description

Technical Field

[0001] This invention relates to the field of intelligent control technology in the process industry, and in particular to a method for cross-device control strategy migration and online fine-tuning based on implicit space alignment. Background Technology

[0002] Process industries are widely present in sectors such as chemical engineering, metallurgy, energy, power generation, water treatment, pharmaceuticals, and equipment manufacturing. For typical process variables such as level, flow rate, temperature, and pressure, there are often multiple devices with similar structures in the field, but differences in measuring point configuration, actuators, sampling cycles, and measurement ranges. These devices usually belong to the same type of control task, such as level setpoint control, but in engineering implementation, they exhibit the characteristics of "similar scenarios, heterogeneous equipment."

[0003] With the development of data-driven control and reinforcement learning technologies, training control strategies using historical operational data has become an important technical approach. However, directly transferring a neural network strategy trained on one device to another often results in network incompatibility due to differences in input / output dimensions and measurement point sets. Even if inputs are forcibly aligned using simple zero-padding or truncation, the original feature semantics are easily destroyed, leading to a significant performance degradation. On the other hand, if large-scale offline or online learning is performed independently from scratch for each device, there are problems of high data and computing power costs and long deployment cycles.

[0004] Currently, existing related work mainly falls into three categories. First, it discusses the application prospects of deep reinforcement learning in process control from the perspective of transfer learning, analyzing engineering obstacles such as sample efficiency and safety. However, it does not provide a feasible implementation scheme integrating latent space alignment and safety closed-loop fine-tuning under conditions of heterogeneous input and output across multiple devices, constrained by comparable operating condition semantics. Second, it combines model predictive control with inverse reinforcement learning, focusing on learning or optimization paths under specific control structures. This differs from the overall technical path of this case, which addresses heterogeneous observation and action spaces across multiple devices in similar process scenarios, achieving policy transfer through latent space alignment and combining target-side safety execution and constraint fine-tuning. Furthermore, there are also deep reinforcement learning adaptive control techniques for specific objects such as motors. These typically revolve around specific controlled objects and single-system control structures, differing from the combination of cross-device latent space alignment, semantic anchor constraints, and online closed-loop safety and acceptance mechanisms emphasized in this case.

[0005] Some existing transfer learning methods focus on feature alignment in general domains. However, in process industry control, without explicit definition of "similar scenarios," data with different control semantics can easily be mixed and aligned, leading to negative transfer. Furthermore, without joint consideration of operational constraints and the distribution of historical possible actions, actions that do not meet safety requirements may still be output in the early stages of target device deployment. In addition, purely offline transfer strategies often have limited data coverage on the target device, and may still be insufficiently adapted when operating conditions change or disturbances increase, requiring low-risk online correction. However, if online correction lacks constraints and acceptance mechanisms, it may lead to excessively rapid strategy drift, affecting stability.

[0006] Therefore, existing technologies suffer from at least the following problems: First, in similar process scenarios, the heterogeneity of multiple devices makes it difficult to directly reuse strategies; second, there is a lack of a systematic method that combines cross-device implicit space alignment with process industry safety constraints and historical behavior constraints; and third, there is a lack of an offline-to-online fine-tuning mechanism on the target device that balances adaptability, safety, and stability. Therefore, it is necessary to propose a cross-device control strategy migration and online fine-tuning method for similar process scenarios to solve the above-mentioned technical problems. Summary of the Invention

[0007] To address this, the present invention provides a method for cross-device control strategy migration and online fine-tuning based on latent space alignment. This method or system, in similar process scenarios, addresses multiple devices with different observation and action dimensions. It achieves offline control strategy migration through unified latent space alignment and combines it with closed-loop control for online constraint fine-tuning of the target device. This method is applicable to the deployment and rapid commissioning of multiple devices in process industries, including those related to liquid level, flow rate, pressure, and temperature. By defining similar process scenarios and learning shared control strategies within a unified latent space, and by restricting cross-device alignment to "comparable operating conditions" using scenario semantic anchors, heterogeneous observations can achieve controllable alignment under these anchors. This reduces the cost of collecting large amounts of data and repeated training per device, while improving the safety and transferability of the target device in the initial deployment phase. Furthermore, the constrained online fine-tuning and acceptance mechanisms enhance long-term operational adaptability.

[0008] To address the aforementioned technical problems, this invention provides a method for cross-device control policy migration and online fine-tuning based on latent space alignment, comprising: Historical operating data of the source device and the target device are collected respectively, and offline datasets of the source device and the target device are constructed based on the historical operating data. A control task model is established for the source device and the target device respectively, and the state space, action space, performance evaluation index and safety constraint set of the corresponding device are determined. An observation encoder is configured for the source device and the target device respectively. The observation encoder maps the observation vector of the corresponding device to a unified latent space with a fixed dimension. A shared control strategy is defined in the unified latent space, and the output of the shared control strategy is mapped to the device action mapping of the corresponding device action space. The shared control strategy uses the same set of strategy parameters for all devices. The observation encoder, the shared control strategy, and the device action mapping are learned offline using the source device offline dataset. During the offline learning process, cross-device latent space alignment terms under similar process scenarios are introduced so that the latent space representation of the source device and the latent space representation of the target device meet the preset consistency criteria under the preset scenario semantic anchor point conditions, and the initial strategy parameters suitable for the deployment of the target device are obtained. The initial strategy parameters are loaded into the target device-side controller to complete the deployment initialization of the target device observation encoder, shared control strategy, and target device action mapping; During the actual operation of the target device, the observation vector of the target device at the current moment is collected, the latent space vector is obtained by the target device observation encoder, the latent space vector is input into the shared control strategy to obtain the intermediate control quantity, and the candidate control action is obtained by the target device action mapping. The candidate control actions are checked for safety constraints. When the candidate control action meets the safety constraints, it is applied to the target device as the final execution action. When the candidate control action does not meet the safety constraints, the candidate control action is corrected to obtain the final execution action, or the final execution action is output by the preset backup controller. Based on the interactive data collected during the online operation of the target device and verified by safety constraints or after anomaly screening, the online constraint fine-tuning is performed on the target device's observation encoder and / or the adaptation module in the shared control strategy. After the fine-tuning is completed, acceptance is performed based on preset safety indicators and control performance indicators. Only when the acceptance is passed will the fine-tuned strategy parameters be replaced with the current online operation strategy parameters.

[0009] In one embodiment of the present invention, the latent variables in the unified latent space are decomposed into cross-device shared latent variables and device-specific latent variables; during offline learning, cross-device consistency constraints are applied to the cross-device shared latent variables, while device-specific differences are allowed for the device-specific latent variables; during online constraint fine-tuning, the device-specific latent variables and the low-rank adaptation parameters inserted between the observation encoder and the shared control strategy are updated first, and strong regularization constraints or partial freezing are applied to the parameters related to the cross-device shared latent variables.

[0010] In one embodiment of the present invention, offline learning is performed on the observation encoder, the shared control policy, and the device action mapping using the source device offline dataset, including: An objective function for offline learning is introduced, which includes a control performance optimization term, a behavioral constraint term, a security constraint penalty term, and a cross-device latent space alignment term; wherein, the behavioral constraint term is used to limit the deviation between the output of the shared control strategy and the device action mapping and the historical distribution of possible actions in the corresponding device offline dataset.

[0011] In one embodiment of the present invention, the objective function for offline learning is expressed as: ; in, The source device's offline dataset; This represents optimization terms related to control performance; This represents behavioral constraints based on offline data from the source device; Indicates safety constraint penalties; This refers to the cross-device implicit space alignment item; , , These are the weighting coefficients.

[0012] In one embodiment of the present invention, the cross-device latent space alignment term is achieved by minimizing the distribution distance between the source device and the target device in the unified latent space; the distribution distance includes one or more of the following: maximum mean difference, adversarial domain classification loss, contrastive learning loss, and Euclidean distance penalty on paired anchor samples.

[0013] In one embodiment of the present invention, the scene semantic anchor points include one or more of the following: the interval to which the dimensionless controlled deviation belongs, the load segment label, the set value neighborhood label, the start-stop stage label, and the operating condition switching label; the cross-device latent space alignment term is calculated on the sample subset divided by at least one type of scene semantic anchor points.

[0014] In one embodiment of the present invention, the online constraint fine-tuning is performed when a preset triggering condition is met. The triggering condition includes one or a combination of the following: the control error exceeds a set error threshold for multiple consecutive control cycles; the change in the amplitude of the external disturbance exceeds a set change threshold; the operating condition of the controlled object changes; the deviation between the control strategy output and the actual control requirements exceeds a set deviation threshold; and during the online fine-tuning process, the update frequency, the number of update steps per time, and the maximum allowable parameter change are limited.

[0015] In one embodiment of the present invention, the objective function of the online constraint fine-tuning is expressed as: ; in, This represents the loss term sampled from the source device offline dataset and / or the target device offline dataset; This represents the loss term based on online interaction samples from the target device. Regular expressions representing parameter changes relative to the strategy parameters before deployment; , These are the weighting coefficients.

[0016] In one embodiment of the present invention, the parameter variation regularization term is represented as: ; in, This represents the set of parameters that are currently to be updated. The set of parameters that indicates the moment when deployment is complete.

[0017] In one embodiment of the present invention, the source device and the target device both belong to the same type of process scenario. The same type of process scenario refers to a scenario category in which the control task category is consistent and the controlled variable type, main control target structure and safety constraint semantics are consistent among the devices. The same type of process scenario is one of the following: liquid level setpoint control scenario, flow regulation scenario, pressure regulation scenario or temperature regulation scenario.

[0018] In one embodiment of the present invention, the source device is a dual-tank coupled liquid level experimental device, and the target device is a single-tank liquid level device, both of which belong to the liquid level setpoint control scenario.

[0019] In one embodiment of the present invention, the historical operating data includes the current moment observation vector with device identification, control action, next moment observation vector and constraint-related information. The observation space dimension and / or action space dimension of the source device and the target device are allowed to be different. The source device has historical control data under multiple operating conditions, and the target device has partial historical data under a small number of operating conditions.

[0020] The technical solution of the present invention has the following advantages compared with the prior art: The present invention discloses a cross-device control strategy migration and online fine-tuning method based on latent space alignment. By defining similar process scenarios and learning shared control strategies in a unified latent space, the method transforms the problem of inconsistent input-output dimensions caused by "similar scenarios but heterogeneous devices" into a trainable encoding-policy-device mapping structure. This significantly reduces the cost of training policies from scratch for multiple devices and improves engineering replicability.

[0021] This invention introduces a cross-device latent space alignment term constrained by scene semantic anchors during the offline phase, which enables the alignment process to focus on comparable operating conditions under similar control tasks, generalize to operating conditions where the source device is known but the target device is unknown, and to more unknown target devices, which helps to suppress negative migration and improve the control rationality in the early stage of target device deployment.

[0022] This invention incorporates behavioral constraints and safety constraint penalties into the offline learning objective, making the initial strategy obtained from the transfer more closely resemble the historical distribution of possible actions and possessing constraint awareness, which helps reduce the risk of exceeding limits in the early stages of the target device's operation.

[0023] This invention sets up a safety constraint verification, action correction, and backup controller switching mechanism during the online operation phase of the target device, and uses online data that has passed the safety screening for fine-tuning under regular constraints. Combined with update frequency and parameter change range limits and acceptance replacement mechanism, it can improve adaptability while suppressing excessively rapid strategy drift, thus taking into account safety, stability, and dynamic performance.

[0024] This invention is applicable to various process industrial control scenarios such as liquid level, flow rate, pressure, and temperature, and has good engineering application value and promotion prospects. Attached Figure Description

[0025] To make the content of this invention easier to understand, the invention will be further described in detail below with reference to specific embodiments and accompanying drawings.

[0026] Figure 1 This is a flowchart of the overall process of a cross-device control strategy migration and online fine-tuning method based on latent space alignment according to the present invention, corresponding to the main sequence of the overall process links and the first-level steps of the specific implementation method described in the invention content.

[0027] Figure 2 This is a schematic diagram of the cross-device offline learning and latent space alignment training architecture in this invention, illustrating the meaning of source and target device observation encoding, unified latent space representation, shared control strategy, device action mapping, and scene semantic anchor related alignment and joint optimization.

[0028] Figure 3 This is a schematic diagram of the collaborative mechanism for online closed-loop control, safety verification, and online constraint fine-tuning of the target device in this invention, illustrating the relationship between online inference, safety constraint output, controlled object feedback, and constraint fine-tuning and acceptance update driven by the online experience pool. Detailed Implementation

[0029] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, so that those skilled in the art can better understand and implement the present invention. However, the embodiments described are not intended to limit the present invention.

[0030] In this invention, when directions (up, down, left, right, front, and back) are described, it is only for the convenience of describing the technical solution of this invention, and does not indicate or imply that the technical features referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, it should not be construed as a limitation of this invention.

[0031] In this invention, "several" means one or more, "multiple" means two or more, "greater than," "less than," "exceeding," etc., are understood to exclude the stated number; "above," "below," "within," etc., are understood to include the stated number. In the description of this invention, the terms "first" and "second" are used only to distinguish technical features and should not be construed as indicating or implying relative importance, or implicitly indicating the number of indicated technical features, or implicitly indicating the order of the indicated technical features.

[0032] In this invention, unless otherwise explicitly defined, the terms "setting," "installing," and "connecting" should be interpreted broadly. For example, they can refer to a direct connection or an indirect connection through an intermediate medium; a fixed connection, a detachable connection, or an integrally formed connection; a mechanical connection, an electrical connection, or a connection capable of mutual communication; or the internal connection of two components or the interaction between two components. Those skilled in the art can reasonably determine the specific meaning of the above terms in this invention based on the specific content of the technical solution.

[0033] This embodiment takes a similar liquid level setpoint control scenario as an example. The source device is a dual-tank coupled liquid level experimental device, and the target device is a single-tank liquid level device. The proposed cross-device control strategy migration and online fine-tuning method based on latent space alignment is explained. This method is also applicable to other device combinations in the same scenario, as well as similar process scenarios involving flow, pressure, and temperature, as long as all devices belong to the same control task category and maintain consistency in safety constraint semantics.

[0034] In this embodiment, similar process scenarios are defined as liquid level setpoint control: the main controlled variable of each device is the liquid level, and the control objective is to make the liquid level track the setpoint and meet the safety boundary between the liquid level and the actuator, thus constituting the scenario category. Both the source device and the target device belong to... However, the observation vector dimension is different from the action dimension: the source device observation can include the liquid level of two water tanks, the related quantities of two valves or pumps and disturbance estimation, while the target device observation only includes the liquid level of a single water tank and related quantities; the source device action can be the adjustment quantity of two actuators, while the target device action is the adjustment quantity of a single actuator.

[0035] From an overall process perspective, this method includes: first, defining the scenario and constructing data; then, establishing the task model and security constraints; subsequently, building a shared policy within a unified latent space and completing offline alignment training; next, deploying and performing online inference on the target device; and finally, achieving long-term stable operation through secure execution and online constraint fine-tuning. The overall process described above can be found in the appendix. Figure 1 Overall flowchart.

[0036] During the offline training phase, observations from the source and target devices are fed into a unified latent space via their respective encoders. The shared policy outputs a fixed-dimensional intermediate control intent in the latent space, which is then converted into the actual usable action vector for each device via a dedicated action mapping. Statistical expectation, which is strongly correlated with control performance, is estimated primarily using large-scale offline trajectories from the source device to obtain a more stable gradient source. The cross-device latent space alignment term consumes samples from both the source and target sides simultaneously on a subset defined by scene semantic anchors, bringing the latent representations of both sides closer together or reducing distribution differences under comparable operating conditions. Behavioral constraint terms and soft safety penalty terms are jointly optimized with the above terms to suppress policy output deviations from the historical feasible region and soft constraints. This training architecture can be found in the appendix. Figure 2 Training architecture diagram.

[0037] During the online operation phase of the target device, current observations are mapped to candidate actions via the target-side encoder, shared strategy, and target-side action mapping. These candidate actions are then processed by the safety constraint processing module, which outputs the final execution value. Simultaneously, the safety-filtered interaction trajectory is written into the online experience pool. When triggering conditions are met, constraint fine-tuning with upper limits on parameter changes is initiated. This closed-loop and safety fine-tuning coordination mechanism is detailed in the appendix. Figure 3 A schematic diagram of online closed-loop control and fine-tuning. The following describes the detailed steps of this method.

[0038] The cross-device control policy migration and online fine-tuning method based on latent space alignment includes the following steps: S1. Define scene categories and construct an offline dataset for device labeling.

[0039] First, define similar process scenarios. and determine the source device. With target equipment All belong to Historical operating data of both the source and target devices are collected. This historical operating data can originate from manual operation records, traditional controller operation records, historical experimental data, or high-reliability simulation data. Each sample carries a device identifier. And includes at least the observation vector at the current time. Controlling actions Next time observation vector Real-time performance evaluation value and related constraints Offline data samples can be represented as: ; in, For equipment The total number of offline samples. The source device offline dataset consists of samples from the source device. The target device offline dataset is composed of target device samples. It should be noted that, It can be much smaller This means that the target device is allowed to have only a small amount of historical data for alignment or cold start assistance. In other words, the source device has a large amount of control data under multiple operating conditions, while the target device has a small amount of partial data under a few operating conditions.

[0040] To ensure data quality, historical operational data can be preprocessed, including removing outliers, filling in missing values, labeling scene semantic anchor points according to operating conditions, and normalizing or dimensionlessizing according to device range. Scene semantic anchor points can come from process rule segments, set operating conditions, etc., or can be supplemented by process personnel based on experience, as long as subsequent alignment steps can consistently divide subsets on both the source and target sides according to the same label system. The scene semantic anchor points include one or more of the following: the interval to which the dimensionless controlled deviation belongs, load segment labels, set value neighborhood labels, start-up and shutdown stage labels, and operating condition switching labels.

[0041] Preferably, for liquid level control scenarios, a dimensionless liquid level deviation feature can be constructed for subsequent anchor point subset partitioning.

[0042] S2. Establish equipment-related task models and safety constraints.

[0043] A control task model is established for each device to determine the observation space. Action space A set of performance evaluation indicators and safety constraints. Due to equipment heterogeneity, [the following is permissible]: and / or This means that the source device and the target device are allowed to have different state dimensions or action dimensions.

[0044] In a preferred embodiment, the real-time performance evaluation employs a weighted sum related to control error, error variation, control energy consumption, and motion smoothness, which can be generally expressed as: ; in, For equipment At any moment The controlled deviation vector; For deviation changes; For changes in action; , , , The weighting coefficient is greater than zero.

[0045] The set of safety constraints includes at least one or more of the following: range constraints of the controlled variable, amplitude constraints of the control quantity, rate of change constraints of the control quantity, and equipment operating boundary constraints. During the policy learning phase, soft constraint penalty terms can be further introduced. For example, hinged penalties or barrier penalties can be applied to the amount of constraint violations to enhance the constraint awareness of the policy.

[0046] S3. Construct a unified hidden space, share policies and device action mappings.

[0047] Configure an observation encoder for each device. , Represents function separator; Will Mapping to a unified latent space : ; Define shared control strategies within the implicit space Output intermediate control quantity ( (For fixed dimensions). Further, configure device action mapping for each device. ,Will Mapped to the real motion space : ; In one alternative implementation, This includes affine transformations and saturated nonlinearities to calibrate intermediate control quantities to the effective stroke of the target device actuator; in another alternative implementation, It is a small neural network used to compensate for the nonlinear characteristics of the device.

[0048] In a further alternative implementation, Decompose into cross-device shared implicit variables Device private hidden variables ,Right now ; And in offline learning and Apply stronger alignment constraints to Allow for device differences to reduce the risk of negative migration.

[0049] S4, cross-device offline learning and latent space alignment.

[0050] This step maps the offline objective components (interpreted through input data, representing the various parts of the offline learning objective function) to the data source: To align with the main body of the expected control performance value; use the alignment term simultaneously. and Sample pairing or grouping within the anchor point subset; if, in engineering practice, there are still a small number of traceable, tagged, and quality-inspected offline trajectories on the target side, these can be incorporated with a smaller sampling weight. This can enhance the consistency between the target-side representation and the sharing strategy, but it is not a necessary implementation path for this invention.

[0051] Utilizing offline datasets from source devices As the primary training data, joint optimization The offline learning objective function can take the following form: ; in, The source device's offline dataset; This represents optimization terms related to control performance; This represents behavioral constraints based on offline data from the source device; Indicates safety constraint penalties; This refers to the cross-device implicit space alignment item; , , These are the weighting coefficients.

[0052] in This includes, but is not limited to, a critic (value network) objective based on Bellman residuals, an actor (policy network) objective based on the advantage function, or a combination of behavioral cloning aids based on rewards; It can be written as: ; Cross-device implicit space alignment items Compute on a subset of scene semantic anchor points. For example, minimize the statistical difference in the latent space distribution for source and target samples whose dimensionless biases fall within the same interval: ; in, For anchor index, Number of anchor point categories; Indicates at the anchor point The sample mean of the corresponding subset.

[0053] In another implementation, a maximum mean difference or adversarial domain classifier can also be used to compute on the anchor subset. .

[0054] It should be noted that this invention emphasizes alignment within similar process scenarios. Anchor labels are defined by scenario semantics, rather than arbitrary unrelated statistical groupings, thereby avoiding incorrect alignment of data with different control semantics.

[0055] S5, Target Device Deployment Initialization and Online Inference.

[0056] After completing offline learning, the parameters (And necessary shared modules) are loaded into the target device's controller. During online operation, current observations from the target device are collected. ,calculate: ; If the action is incremental, it can be made And update the actual execution control quantity according to the following formula: ; S6. Target equipment online closed-loop operation, safety constraints and strategy fine-tuning.

[0057] S601, Safety constraint verification and action correction.

[0058] A safety constraint processing module is set up between the strategy output and the execution mechanism. This module handles candidate actions. or Perform amplitude clipping and rate of change limitation; project candidate actions to the safe and feasible region if necessary. Inside: ; in, This indicates the candidate execution quantity before pruning. If a high risk of constraint violation is detected or the correction exceeds the threshold, the system switches to the safety controller. The guaranteed controller is any one of a proportional-integral-derivative controller, a model prediction controller, or a rule-based controller.

[0059] S602, Online constraint fine-tuning and acceptance replacement.

[0060] The interactive data generated during the online operation of the target device is written into the online experience pool after passing security verification or anomaly filtering. Online fine-tuning is performed when a triggering condition is met, which can be consistent with claim 7. The objective function for online fine-tuning takes the following form: ; in, This represents the loss term sampled from the source device offline dataset and / or the target device offline dataset; This represents the loss term based on online interaction samples from the target device. Regular expressions representing parameter changes relative to the strategy parameters before deployment; , These are the weighting coefficients.

[0061] in It can be a weighted average of Bellman residuals, behavioral cloning loss, or both on the target device; from , and The data is obtained through hybrid sampling to suppress forgetting and ensure stable updates.

[0062] Preferably, only the encoder of the target device Low-rank adaptation parameters and strategies Update some parameters and... relatively Apply the regularization constraint as described in claim 9. After fine-tuning, conduct acceptance testing. Only replace the online operating parameters if both safety and control performance indicators meet the requirements; otherwise, maintain the settings. Continue running.

[0063] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0064] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0065] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0066] These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable apparatus for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0067] Finally, it should be noted that the above specific embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to examples, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A method for cross-device control strategy migration and online fine-tuning based on latent space alignment, characterized in that, include: Historical operating data of the source device and the target device are collected separately, and offline datasets of the source device and the target device are constructed based on the historical operating data. A control task model is established for the source device and the target device respectively, and the state space, action space, performance evaluation index and safety constraint set of the corresponding device are determined. An observation encoder is configured for the source device and the target device respectively. The observation encoder maps the observation vector of the corresponding device to a unified latent space with a fixed dimension. A shared control strategy is defined in the unified latent space, and the output of the shared control strategy is mapped to the device action mapping of the corresponding device action space. The shared control strategy uses the same set of strategy parameters for all devices. The observation encoder, the shared control strategy, and the device action mapping are learned offline using the source device offline dataset. During the offline learning process, cross-device latent space alignment terms under similar process scenarios are introduced so that the latent space representation of the source device and the latent space representation of the target device meet the preset consistency criteria under the preset scenario semantic anchor point conditions, and the initial strategy parameters suitable for the deployment of the target device are obtained. The initial strategy parameters are loaded into the target device-side controller to complete the deployment initialization of the target device observation encoder, shared control strategy, and target device action mapping; During the actual operation of the target device, the observation vector of the target device at the current moment is collected, the latent space vector is obtained by the target device observation encoder, the latent space vector is input into the shared control strategy to obtain the intermediate control quantity, and the candidate control action is obtained by the target device action mapping. The candidate control actions are checked for safety constraints. When the candidate control action meets the safety constraints, it is applied to the target device as the final execution action. When the candidate control action does not meet the safety constraints, the candidate control action is corrected to obtain the final execution action, or the final execution action is output by the preset backup controller. Based on the interactive data collected during the online operation of the target device and verified by safety constraints or after anomaly screening, the online constraint fine-tuning is performed on the target device's observation encoder and / or the adaptation module in the shared control strategy. After the fine-tuning is completed, acceptance is performed based on preset safety indicators and control performance indicators. Only when the acceptance is passed will the fine-tuned strategy parameters be replaced with the current online operation strategy parameters.

2. The method for cross-device control strategy migration and online fine-tuning based on latent space alignment according to claim 1, characterized in that, The latent variables in the unified latent space are decomposed into cross-device shared latent variables and device-specific latent variables. During offline learning, cross-device consistency constraints are applied to the cross-device shared latent variables, while device-specific differences are allowed for the device-specific latent variables. During online constraint fine-tuning, the device-specific latent variables and the low-rank adaptation parameters inserted between the observation encoder and the shared control strategy are updated first, and strong regularization constraints or partial freezing are applied to the parameters related to the cross-device shared latent variables.

3. The method for cross-device control strategy migration and online fine-tuning based on latent space alignment according to claim 1, characterized in that, The observation encoder, the shared control policy, and the device action mapping are learned offline using the source device offline dataset, including: An objective function for offline learning is introduced, which includes a control performance optimization term, a behavioral constraint term, a security constraint penalty term, and a cross-device latent space alignment term; wherein, the behavioral constraint term is used to limit the deviation between the output of the shared control strategy and the device action mapping and the historical distribution of possible actions in the corresponding device offline dataset.

4. The method for cross-device control strategy migration and online fine-tuning based on latent space alignment according to claim 3, characterized in that, The objective function for offline learning is expressed as: ； in, The source device's offline dataset; This represents optimization terms related to control performance; This represents behavioral constraints based on offline data from the source device; Indicates safety constraint penalties; This refers to the cross-device implicit space alignment item; , , These are the weighting coefficients.

5. A method for cross-device control strategy migration and online fine-tuning based on latent space alignment according to any one of claims 3-4, characterized in that, The cross-device latent space alignment term is achieved by minimizing the distribution distance between the source device and the target device in the unified latent space; the distribution distance includes one or more of the following: maximum mean difference, adversarial domain classification loss, contrastive learning loss, and Euclidean distance penalty on paired anchor samples.

6. A method for cross-device control strategy migration and online fine-tuning based on latent space alignment according to any one of claims 3-4, characterized in that, The scene semantic anchor points include one or more of the following: the interval to which the dimensionless controlled deviation belongs, the load segment label, the set value neighborhood label, the start-stop stage label, and the operating condition switching label; the cross-device implicit space alignment term is calculated on the sample subset divided by at least one of the scene semantic anchor points.

7. The method for cross-device control strategy migration and online fine-tuning based on latent space alignment according to claim 1, characterized in that, The online constraint fine-tuning is performed when preset triggering conditions are met. The triggering conditions include one or a combination of the following: the control error exceeds a set error threshold for multiple consecutive control cycles; the change in the amplitude of external disturbance exceeds a set change threshold; the operating condition of the controlled object changes; the deviation between the control strategy output and the actual control requirements exceeds a set deviation threshold; during the online fine-tuning process, the update frequency, the number of update steps per time, and the maximum allowable parameter change are limited.

8. A method for cross-device control strategy migration and online fine-tuning based on latent space alignment according to claim 1 or 7, characterized in that, The objective function for the online constraint fine-tuning is expressed as: ； in, This represents the loss term sampled from the source device offline dataset and / or the target device offline dataset; This represents the loss term based on online interaction samples from the target device. Regular expressions representing parameter changes relative to the strategy parameters before deployment; , These are the weighting coefficients.

9. The method for cross-device control strategy migration and online fine-tuning based on latent space alignment according to claim 8, characterized in that, The parameter change regularization term is expressed as follows: ； in, This represents the set of parameters that are currently to be updated. The set of parameters representing the moment when deployment is completed.

10. The method for cross-device control strategy migration and online fine-tuning based on latent space alignment according to claim 1, characterized in that, The source device and the target device both belong to the same type of process scenario. The same type of process scenario refers to a scenario category in which the control task category is the same and the controlled variable type, main control target structure and safety constraint semantics are consistent among the devices. The same type of process scenario is one of the following: liquid level setpoint control scenario, flow regulation scenario, pressure regulation scenario or temperature regulation scenario.

Images