Intelligent factory fault prediction self-repair system based on digital twin and deep reinforcement learning

Abstract

The present application relates to the field of intelligent factory fault diagnosis, adaptive control and artificial intelligence, in particular to an intelligent factory fault prediction self-repair system based on digital twinning and deep reinforcement learning. The system comprises a multi-source state data synchronous acquisition unit for real-time acquisition of physical state vectors and obtaining virtual state vectors; a cognitive divergence quantization unit for calculating model-reality deviation rate and strategy confidence; a system mode switching judgment unit for synthesizing cognitive entropy and generating utilization mode signals or exploration mode signals; an online physical law identification and twinning model reconstruction unit for activating exploratory action strategies, identifying new physical law model parameters and reconstructing the digital twinning model in response to the exploration mode signals; and an adaptive deep reinforcement learning strategy regeneration unit for generating a new optimal strategy and driving the system to switch to the utilization mode. The present application realizes accurate, efficient and stable mode switching, and improves the overall system operation efficiency.

Description

Technical Field

[0001] This invention relates to the fields of smart factory fault diagnosis, adaptive control, and artificial intelligence technology, specifically a smart factory fault prediction and self-repair system based on digital twins and deep reinforcement learning. Background Technology

[0002] In the current field of smart factory technology, high-fidelity digital twin models are constructed to simulate the operating state of physical entities, and deep reinforcement learning agents are used to train the optimal control strategy on this model. The effectiveness of this model depends heavily on the consistency between the digital twin model and physical reality. However, physical equipment can undergo dynamic changes due to wear, changes in operating conditions, or sudden failures, leading to a mismatch between the pre-established digital twin model and the actual situation.

[0003] In existing technologies, when model mismatch occurs, the control strategy formulated based on the erroneous model will no longer be optimal and may even damage the equipment. Existing monitoring systems have limited ability to handle this problem. These systems generally lack a mechanism for real-time, quantitative evaluation of the deviation between the model and reality, as well as the confidence level of the control strategy itself.

[0004] Therefore, how to provide a smart factory fault prediction and self-healing system that can dynamically quantify cognitive bias and autonomously trigger model reconstruction and strategy relearning based on this bias, in order to solve the technical problem that the system cannot effectively adapt and repair itself when faced with model mismatch and unknown faults, is a problem that urgently needs to be solved by those skilled in the art.

[0005] The information disclosed in the background section above is only intended to enhance the understanding of the background of this disclosure, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0006] To address the aforementioned technical problems, this invention discloses a smart factory fault prediction and self-healing system based on digital twins and deep reinforcement learning. Specifically, the technical solution of this invention includes:

[0007] The multi-source state data synchronous acquisition unit is used to acquire the physical state vector of physical devices in real time and synchronously obtain the virtual state vector predicted by the digital twin model.

[0008] The cognitive divergence quantification unit is used to calculate the model-reality deviation rate and policy confidence. The model-reality deviation rate is calculated based on the physical state vector and the virtual state vector. The policy confidence is calculated based on the state-action value function extracted from the deep reinforcement learning agent.

[0009] The system mode switching determination unit is configured to synthesize the cognitive entropy, compare and analyze the cognitive entropy with a preset critical switching threshold, and generate a utilization mode signal or an exploration mode signal.

[0010] The online physical law identification and twin model reconstruction unit is configured to activate an exploratory action strategy in response to the exploration mode signal, identify a new physical law model parameter, and reconstruct the digital twin model.

[0011] The adaptive deep reinforcement learning strategy regeneration unit is configured to perform strategy retraining based on the reconstructed digital twin model, generate a new optimal strategy, and deploy the new optimal strategy to drive the system to switch to the utilization mode.

[0012] Preferably, the model-reality deviation rate is calculated as follows:

[0013] The dimensionless relative deviation is calculated by calculating the absolute deviation of each component of the physical state vector and the virtual state vector, and dividing by the corresponding normalization reference;

[0014] The dimensionless relative deviations of each component are multiplied by the corresponding weights and summed to generate a comprehensive deviation rate;

[0015] The comprehensive deviation rate is set as the model-reality deviation rate;

[0016] The normalization reference is derived from the statistics in the historical normal operation data of the physical device; the weights are pre-set according to the importance of each physical quantity to the overall health status of the device.

[0017] Preferably, the strategy confidence is calculated as follows:

[0018] The state-action value function calculated by the value network of the deep reinforcement learning agent is obtained;

[0019] The variance of the state-action value function is calculated;

[0020] The reciprocal of the variance is set as the strategy confidence.

[0021] Preferably, the cognitive entropy is synthesized as follows:

[0022] The model-reality deviation rate is divided by a preset reference deviation threshold to perform normalization processing to obtain a relative deviation measure;

[0023] The strategy confidence is subjected to nonlinear transformation processing;

[0024] The relative deviation measure and the result of the nonlinear transformation processing of the strategy confidence are multiplied to generate the cognitive entropy.

[0025] Preferably, the determination process of the system mode switching determination unit is as follows:

[0026] If the cognitive entropy is less than or equal to the critical switching threshold, a utilization mode signal is generated;

[0027] If the cognitive entropy is greater than the critical switching threshold, an exploration mode signal is generated.

[0028] Preferably, the reconstruction process of the digital twin model is as follows:

[0029] An exploratory action strategy is activated;

[0030] A state data sequence under the exploratory action is collected;

[0031] According to the state data sequence, a deviation model function is identified and solved;

[0032] The deviation model function is integrated with the original digital twin model to generate an updated digital twin state transition function.

[0033] Preferably, the generation process of the new optimal strategy is as follows:

[0034] The reconstructed digital twin model is set as a simulation environment;

[0035] The original strategy network is incrementally trained in the simulation environment;

[0036] By running the strategy optimization algorithm to solve the objective function that can maximize the long-term return, a new optimal strategy is obtained.

[0037] Preferably, the deployment process of the new optimal strategy is as follows:

[0038] The new optimal strategy is loaded to the controller of the physical system;

[0039] A mode switching signal is sent to switch the system from the exploration mode back to the utilization mode.

[0040] Compared with the prior art, the present application has the following beneficial effects:

[0041] 1、The present application creates a cognitive entropy index, which nonlinearly couples the model-reality deviation rate and the strategy confidence. This design can accurately capture and amplify the critical dangerous state of blind confidence of the system to the wrong model, ensuring that self-repair is triggered only when the cognitive failure risk reaches the threshold, avoiding false positives and unnecessary exploration caused by non-critical parameter fluctuations, achieving accurate, efficient and stable mode switching, and improving the overall system efficiency.

[0042] 2、The application constructs a complete cognitive self-repairing closed loop from state perception, cognitive quantification, mode switching, to model reconstruction and strategy regeneration. The system can actively switch from the utilization mode to the exploration mode, online identify unknown physical laws to reconstruct the digital twin, and automatically switch back based on the new model to retrain the optimal strategy. This kind of unmanned full-process automation fundamentally improves the self-adaptation and self-learning ability of the intelligent factory when facing unknown faults, and realizes higher-order system autonomy.

[0043] 3、The application proposes a method for online identification and reconstruction of digital twin model. In the exploration mode, the system identifies the deviation model describing the unknown dynamics by applying a safe disturbance and collecting response data, integrates it with the original model, and realizes online and automatic update of the digital twin. This makes the digital twin no longer a static mirror, but a dynamic model that can evolve with the physical entity, providing a continuous and reliable high-fidelity simulation environment for subsequent strategy generation, enhancing the robustness of the system.

[0044] 4、The application uses the reconstructed high-fidelity digital twin model as a simulation environment, combined with transfer learning technology, to realize the rapid and safe regeneration of control strategy. The new optimal strategy has been fully trained and evaluated offline before being deployed in the physical system, ensuring its adaptability to the changed system dynamics. This eliminates suboptimal control decisions caused by model mismatch from the root, maintains the system performance at the optimal level for a long time, and ensures production efficiency and equipment health. BRIEF DESCRIPTION OF DRAWINGS

[0045] The application will be further explained in conjunction with the drawings and examples:

[0046] Figure 1 is a system structure diagram of the application. DETAILED DESCRIPTION

[0047] In order to make the purpose, technical scheme and advantages of the application clearer and more apparent, the application will be further described in detail in conjunction with specific examples.

[0048] Example 1:

[0049] Please refer to Figure 1 , the intelligent factory fault prediction self-repairing system of digital twin and deep reinforcement learning, including:

[0050] A multi-source state data synchronous acquisition unit is used to acquire the physical state vector of the physical device in real time, and synchronously acquire the virtual state vector predicted by the digital twin model;

[0051] A cognitive divergence quantification unit is configured to calculate a model-reality deviation rate and a policy confidence; the model-reality deviation rate is calculated based on a physical state vector and a virtual state vector; the policy confidence is calculated based on a state-action value function extracted from the deep reinforcement learning agent;

[0052] A system mode switching determination unit is configured to synthesize cognitive entropy, compare the cognitive entropy with a preset critical switching threshold, and generate a utilization mode signal or an exploration mode signal;

[0053] An online physical law identification and twin model reconstruction unit is configured to activate an exploratory action policy in response to the exploration mode signal, identify a new physical law model parameter, and reconstruct a digital twin model;

[0054] An adaptive deep reinforcement learning policy regeneration unit is configured to perform policy retraining based on the reconstructed digital twin model, generate a new optimal policy, and deploy the new optimal policy to drive the system to switch to the utilization mode;

[0055] The embodiment provides a smart factory fault prediction self-repair system based on digital twin and deep reinforcement learning;

[0056] The system aims to solve the technical problem that an existing smart factory monitoring system cannot effectively adaptively repair when facing model mismatch and unknown faults; the system quantifies the deviation of the physical and virtual and the confidence of the control policy, and based on the coupling effect of the two, actively triggers the mode switching from utilization to exploration at the right time, drives the system to identify unknown laws online, reconstructs the digital twin model, and re-trains the optimal control policy that adapts to the new law, forming a complete cognitive level self-repairing closed loop;

[0057] In a high-precision numerical control machine tool processing process application scenario, the units of the system work cooperatively;

[0058] A multi-source state data synchronous acquisition unit provides real-time and comparative state observation basis for the system, which acquires a physical state vector constituting the running state of the machine tool through a sensor array deployed on key components of the machine tool The vector is a set containing multiple physical measurements, for example The unit obtains a predicted virtual state vector from the digital twin simulation environment based on the state at the previous moment and the executed control instructions;

[0059] A cognitive divergence quantification unit quantifies the abstract divergence between the physical reality and the digital model into two accurate and calculable indicators; the unit receives and calculates a model-reality deviation rate With strategy confidence ; is a dimensionless scalar index for measuring the comprehensive fidelity of digital twin model, which is calculated according to the difference between the physical state vector and the virtual state vector ; is an index for quantifying the degree of certainty of the DRL agent about the optimality of its current decision, which is calculated according to the state-action value function extracted from the value network of the DRL agent

[0060] The system mode switching decision unit makes key decisions about the system operation mode based on the above quantitative indicators; this unit will receive the and synthesized into a composite index cognitive entropy ; can amplify both the model has wrong and the strategy stubborn two signals, its role is to judge whether the system is in the state of wrong cognition of the wrong model; this unit compares and analyzes the with the preset critical switching threshold ; if , the utilization mode signal is generated, and the system maintains normal operation; if , the exploration mode signal is generated, triggering the subsequent self-repair process;

[0061] The online physical law identification and twin model reconstruction unit responds to the exploration mode signal to actively repair the cognitive defects of the digital twin model; after receiving the exploration mode signal, this unit activates the exploratory action strategy, such as injecting a small perturbation into the control command of the machine tool; this unit collects system response data under perturbation, which is used to identify new physical law model parameters, the core of which is to establish a bias model function to describe the unknown dynamics; by integrating the identified bias model with the original model, the digital twin model is reconstructed, and an updated model with higher fidelity is output;

[0062] The adaptive deep reinforcement learning strategy regeneration unit generates and deploys an optimal control strategy that adapts to the new law based on the updated model; this unit takes the reconstructed digital twin model as a new simulation environment, in which the original DRL strategy network is retrained, and a new optimal strategy is quickly generated through transfer learning ; After the new strategy is verified to meet the performance requirements, it is deployed to the controller of the physical machine tool, and the system is switched back to the utilization mode;

[0063] The system constructed in this embodiment can dynamically quantify its own cognitive limitations and actively trigger model reconstruction and strategy relearning when necessary, thereby improving the robustness and self-adaptive ability of the smart factory system, enabling it to realize self-diagnosis and self-repair when facing gradual or sudden failures that are not fully modeled in the initial model, rather than just providing early warning or shutdown; the system can continuously maintain an optimal or near-optimal operating state without human intervention, improving production efficiency and equipment health level.

[0064] Embodiment 2:

[0065] The calculation process of the model-reality deviation rate is as follows:

[0066] Calculate the dimensionless relative deviation, wherein the dimensionless relative deviation is obtained by calculating the absolute deviation of each component of the physical state vector and the virtual state vector, and dividing by the corresponding normalization reference;

[0067] Multiply the dimensionless relative deviation of each component by the corresponding weight to obtain the comprehensive deviation rate;

[0068] Set the comprehensive deviation rate as the model-reality deviation rate;

[0069] The normalization reference is a statistical quantity in the historical normal operation data of the physical device; the weight is pre-set according to the importance of each physical quantity to the overall health status of the device;

[0070] In this embodiment, the model-reality deviation rate in the cognitive divergence quantification unit is Further limitation of the specific calculation process; this calculation process aims to standardize and aggregate the state deviations of multiple dimensions and different physical units into an objective scalar index;

[0071] The calculation process is as follows: for each corresponding component in the state vector and , such as the th component and , calculate the absolute deviation , and divide the absolute deviation by the corresponding normalization reference to obtain the dimensionless relative deviation; here, is a reference value for standardizing the deviation of the corresponding physical quantity, which is obtained by statistical analysis of a large amount of historical normal operation data of the physical device, and is usually the range of the physical quantity or its standard deviation to ensure the stability and comparability of the calculation; multiply the dimensionless relative deviation of each component by its corresponding weight to obtain the comprehensive deviation rate, and set it as the model-reality deviation rate ; here, It is pre-set and corresponds to the first The coefficient of importance of each physical quantity to the overall health status of the equipment, satisfying It is usually determined by combining prior knowledge from domain experts or through data-driven sensitivity analysis.

[0072] The calculation process is precisely expressed by the following formula:

[0073]

[0074] in, Model-reality deviation rate, dimensionless, calculated in this step;

[0075] The total number of components in the state vector, an integer, preset by the system design;

[0076] : No. The weights of each component are dimensionless and are preset based on expert experience or sensitivity analysis.

[0077] : The first physical state vector Each component, consistent with the specific physical dimensions, is collected in real time by the multi-source state data synchronous acquisition unit.

[0078] : The first virtual state vector Each component, consistent with the specific physical dimensions, is predicted by the digital twin model based on the state and control commands of the previous moment.

[0079] : No. The normalization reference for each component is consistent with the specific physical dimensions and is obtained through statistical analysis of historical normal operating data of the equipment.

[0080] The above-described weight normalization calculation method not only solves the problem of incompatibility caused by inconsistent dimensions of multi-source heterogeneous data, but also introduces importance assessment; this makes the calculated model-reality deviation rate more accurate. Compared to simple Euclidean distance or mean square error, it can more accurately and robustly reflect the fidelity of the digital twin model on key health dimensions, thus providing a more reliable basis for subsequent mode switching decisions and avoiding misjudgments caused by normal fluctuations in non-critical parameters.

[0081] Example 3:

[0082] The process of calculating policy confidence is as follows:

[0083] Obtain the state-action value function calculated by the value network of a deep reinforcement learning agent;

[0084] the variance of the state-action value function;

[0085] set the inverse of the variance as the policy confidence;

[0086] In this embodiment, the policy confidence Further limitation of the specific calculation process; the technical purpose of this calculation process is to quantify how confident the reinforcement learning agent is about its output decision;

[0087] The calculation process is as follows: obtain the state-action value function ; to quantify the confidence, the value network is constructed to output a probability distribution rather than a deterministic Q value, such as a Gaussian distribution, so that the mean and variance of the value evaluation can be obtained; The policy confidence is defined as the ratio of the square of the mean value of the value evaluation to the variance, i.e. the square of the signal-to-noise ratio;

[0088]

[0089]

[0090] wherein, : policy confidence, dimensionless, calculated by this step;

[0091] : mean value of state-action value function distribution, consistent with the dimension of value unit, calculated by the value network of DRL agent;

[0092] : variance of state-action value function distribution, consistent with the dimension of square of value unit, calculated by the value network of DRL agent;

[0093] By defining the policy confidence as the square of the dimensionless signal-to-noise ratio, this embodiment provides a unique introspective perspective and rigorous physical meaning for the system; it makes the system not only know what to do, but also know how confident it is about the decision; this design enables the subsequent cognitive entropy to judge a most dangerous situation: when the model has made a mistake, i.e. is very high, while the DRL policy is abnormally confident about the decision based on this erroneous model, i.e. is very high; this quantification of blind confidence in unknown situations is the key to triggering the system to explore and self-repair, enhancing the safety of the system's decision-making.

[0094] Embodiment 4:

[0095] The synthesis process of cognitive entropy is as follows:

[0096] The model-reality deviation rate is divided by a preset reference deviation threshold to perform normalization processing to obtain a relative deviation metric;

[0097] The strategy confidence is subjected to nonlinear transformation processing;

[0098] The relative deviation metric is multiplied by the result of the nonlinear transformation processing of the strategy confidence to generate cognitive entropy;

[0099] In the system mode switching determination unit, the cognitive entropy of the embodiment is further limited in the specific synthesis process; the cognitive entropy is the core self-defined index of the present application, and its mathematical form is a coupling function specially constructed to solve the technical problem, aiming to create a composite index that can amplify both the model error and the strategy stubbornness signals;

[0100] The synthesis process is to divide the model-reality deviation rate by a preset reference deviation threshold to perform normalization processing to obtain a relative deviation metric; here, is a key adjustable parameter for defining incidental noise and real model deviation, which is usually set to the 99th percentile of the historical distribution through statistical analysis of the operation data of the equipment under a large number of normal conditions; the strategy confidence is subjected to nonlinear transformation processing, that is, ; the relative deviation metric is multiplied by the result of the nonlinear transformation processing of the strategy confidence to generate cognitive entropy ;

[0101] The synthesis process is accurately represented by the following formula:

[0102]

[0103] wherein, : cognitive entropy, dimensionless, calculated by the present step;

[0104] : model-reality deviation rate, dimensionless, calculated by the cognitive divergence quantification unit;

[0105] : reference deviation threshold, dimensionless, preset by statistical analysis of the distribution under historical normal conditions;

[0106] : strategy confidence, dimensionless, calculated by the cognitive divergence quantification unit;

[0107] This unique multiplicative coupling design, compared to simple weighted summation, exhibits a non-linear amplification effect; it only becomes effective when the model bias is significant. The value is much greater than 1, and the policy is abnormally confident in its decision-making under this flawed model, i.e. When the value is large, cognitive entropy Only then will it increase dramatically; if it's merely a slight deviation in the model, or the strategy itself is uncertain, then... It will remain at a low level; this logical construction accurately captures the critical state where the system most needs to self-repair, ensuring the accuracy and necessity of mode switching, avoiding unnecessary exploration operations, and improving the overall operating efficiency of the system.

[0108] Example 5:

[0109] The determination process of the system mode switching determination unit is as follows:

[0110] If the cognitive entropy is less than or equal to the critical switching threshold, then a mode utilization signal is generated;

[0111] If the cognitive entropy is greater than the critical switching threshold, an exploration mode signal is generated;

[0112] This embodiment further defines the specific determination process of the system mode switching determination unit based on embodiment 1; the determination process determines whether the system continues to trust the current model for production or initiates the self-repair process based on the magnitude of cognitive entropy.

[0113] The determination process involves using the real-time synthesized cognitive entropy. With the preset critical switching threshold Comparison; here It is a decision boundary used to trigger mode switching, and its determination method is as follows: by injecting different levels of known unmodeled perturbations into the digital twin model on an offline simulation platform, and recording the system before entering an unstable state. The value of is taken as its statistical expectation under a certain safety margin. To ensure the data-driven and reproducible nature of decision-making; the judgment logic is: if The system mode switching determination unit generates a utilization mode signal, and the system maintains a utilization mode based on the current model and DRL strategy for efficient production; if The system mode switching determination unit generates an exploration mode signal, which will directly activate the subsequent online physical law identification and twin model reconstruction unit.

[0114] This is achieved by introducing a method based on offline simulation and safety margin analysis. This embodiment provides an objective, quantitative, and verifiable decision-making benchmark for mode switching; this overcomes the drawbacks of traditional fault diagnosis systems that rely on fixed thresholds or manual rules, making switching decisions more intelligent and situation-adaptive; it ensures that the system only initiates resource-consuming exploration and reconstruction processes when it is truly necessary, maximizing the utilization time of normal production while ensuring safety and adaptability.

[0115] Example 6:

[0116] The reconstruction process of the digital twin model is as follows:

[0117] Activate exploratory action strategies;

[0118] Collect state data sequences during exploratory actions;

[0119] Based on the state data sequence, identify and solve the deviation model function;

[0120] The deviation model function is integrated with the original digital twin model to generate an updated digital twin state transition function;

[0121] Based on Example 1, this embodiment further defines the specific process of digital twin model reconstruction in the online physical law identification and twin model reconstruction unit; this process is activated after receiving the exploration mode signal and is the core technical step for the system to achieve cognitive self-repair.

[0122] The reconstruction process involves activating an exploratory strategy that applies a series of safe, information-rich micro-perturbation control sequences to the physical system to maximize its dynamic response. Simultaneously, the system records the state evolution data of the physical devices, forming a dataset containing... The data sequence; based on this data sequence, the deviation model function is identified and solved; the core of this step is to model and estimate the parameters of the unknown dynamic changes of the system, and express them through an extended state-space model:

[0123]

[0124] in, The updated model predicts the state at the next time step, which has the same dimensions as the state vector.

[0125] The original model function represents the known dynamic part of the original digital twin model, and its parameters are... It is fixed;

[0126] The deviation model function represents the unknown dynamic introduced by the new physical law, whose structure can be pre-defined based on prior knowledge, and whose parameters... is the unknown to be solved;

[0127] With the collected data sequence, the optimal parameters that minimize the error between the predicted state and the true state are solved by an online parameter estimation algorithm, such as recursive least squares method. When the estimated value of the parameters converges and the error of prediction using the updated model is lower than the preset accuracy, the identified deviation model is fixed and integrated with the original model to form an updated digital twin state transition function with higher fidelity.

[0128] The embodiment realizes an online and automatic model reconstruction mechanism through the process of applying disturbance-collecting data-identifying deviation model-integrating update. Compared with offline model update which needs manual intervention, the method can dynamically capture and learn new changes in the system without interrupting the main production process, and integrate these new knowledge into the digital twin model. This makes the digital twin not a static mirror, but a dynamic model that can evolve with the physical entity, thereby ensuring the effectiveness of subsequent policy retraining.

[0129] Embodiment 7:

[0130] The generation process of the new optimal policy is as follows:

[0131] The reconstructed digital twin model is set as a simulation environment.

[0132] The original policy network is incrementally trained in the simulation environment.

[0133] The new optimal policy is obtained by solving the objective function that can maximize the long-term return by running the policy optimization algorithm.

[0134] The embodiment further limits the specific process of generating a new optimal policy in the adaptive deep reinforcement learning policy regeneration unit based on embodiment 1. The process uses the reconstructed and more accurate model to generate an optimal control policy that can adapt to the current real dynamics of the system.

[0135] The generation process is to set the updated digital twin state transition function output by the previous unit as a brand new and high-fidelity simulation environment, i.e., a new world model . The DRL agent uses transfer learning to incrementally train the original policy network in this new environment to quickly adapt to the new system dynamics. The goal of the training is to find a policy that can maximize the future cumulative return under the new physical laws This is achieved by solving the Bellman optimality equation in the reinforcement learning domain, whose objective function is as follows:

[0136]

[0137] where, : new optimal policy, which is the output of the DRL agent, defines the mapping from state to action;

[0138] : policy, which represents a candidate control policy;

[0139] : expectation, which represents the mathematical expectation under the policy and the new world model ;

[0140] : new world model, i.e., the reconstructed high-fidelity simulation environment, is the key input of this step;

[0141] : discount factor, which is a hyperparameter between 0 and 1, e.g., 0.99, used to weigh the importance of immediate rewards and long-term rewards;

[0142] : reward, which is a pre-defined reward function according to the specific factory task;

[0143] By running the policy optimization algorithm, such as PPO, SAC, etc., in the new simulation environment, the target function is iteratively solved, and the obtained policy is the new optimal policy that adapts to the new system dynamics;

[0144] This embodiment realizes an efficient policy adaptation generation mechanism by using the reconstructed high-fidelity model as the training environment and combining transfer learning technology; it ensures that the newly generated policy can adapt to the real world, because the dynamic rules it learns have been online identified and corrected; compared with continuing to execute the old policy on the mismatched model, this method can fundamentally eliminate the suboptimal or even harmful control decisions caused by model errors, thereby restoring and maintaining the system performance at the optimal level in the long term.

[0145] Embodiment 8:

[0146] The deployment process of the new optimal policy is as follows:

[0147] Load the new optimal policy to the controller of the physical system;

[0148] Send a mode switching back signal to switch the system from the exploration mode back to the utilization mode;

[0149] The embodiment further limits the deployment process of the new optimal strategy in the adaptive deep reinforcement learning strategy regeneration unit based on the embodiment 1; the process is responsible for applying the newly generated strategy to the physical reality and restoring the system to normal;

[0150] The deployment process is as follows: after sufficient training and offline evaluation in the new simulation environment, when the expected performance of the new optimal strategy meets the preset deployment requirements, the network weight file of the strategy is loaded into the controller of the physical system to replace the original old strategy; after successful deployment of the strategy, the adaptive deep reinforcement learning strategy regeneration unit sends a mode switching signal to the system mode switching judgment unit to automatically switch the system from the exploration mode back to the utilization mode; at this point, the system not only responds to potential failures through the new strategy at the physical level, but also completes the repair and iteration of the model and strategy at the cognitive level, forming a complete adaptive and self-learning closed-loop control logic;

[0151] The embodiment defines a clear and safe strategy deployment and mode switching process; it ensures that the new strategy is applied to the physical device only after being fully verified, ensuring the reliability of the switching process; the automatic mode switching mechanism enables the entire self-repairing process to be closed-loop without human intervention, achieving high autonomy of the system; this enables the system proposed in the present application to run continuously, stably and unattended, and in the dynamic and uncertain industrial environment, it always maintains the best production efficiency and equipment health status, achieving higher-order intelligentization of the smart factory.

[0152] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present application and not to limit it, although the present application has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present application can be modified or replaced equivalently without departing from the spirit and scope of the technical solutions of the present application.

Claims

1. A smart factory failure prediction self-repair system using digital twin and deep reinforcement learning, characterized in that, The method comprises the following steps: A multi-source state data synchronous acquisition unit is used to acquire the physical state vector of the physical device in real time and synchronously acquire the virtual state vector predicted by the digital twin model; A cognitive discrepancy quantification unit is used to calculate the model-reality deviation rate and the strategy confidence; wherein, the model-reality deviation rate is calculated based on the physical state vector and the virtual state vector; the strategy confidence is calculated based on the state-action value function extracted from the deep reinforcement learning agent; A system mode switching judgment unit is used to synthesize the cognitive entropy and compare and analyze the cognitive entropy with the preset critical switching threshold to generate a utilization mode signal or an exploration mode signal; An online physical law identification and digital twin model reconstruction unit is used to activate the exploratory action strategy in response to the exploration mode signal, identify the new physical law model parameters, and reconstruct the digital twin model; An adaptive deep reinforcement learning strategy regeneration unit is used to perform strategy retraining based on the reconstructed digital twin model, generate a new optimal strategy, and deploy the new optimal strategy to drive the system to switch to the utilization mode; The calculation process of the model-reality deviation rate is as follows: Calculate the dimensionless relative deviation; wherein, the dimensionless relative deviation is obtained by calculating the absolute deviation of each component of the physical state vector and the virtual state vector, and dividing by the corresponding normalization reference; Sum the dimensionless relative deviations of each component after multiplying by the corresponding weight to generate a comprehensive deviation rate; Set the comprehensive deviation rate as the model-reality deviation rate; Wherein, the normalization reference is derived from the statistics in the historical normal operation data of the physical device; the weight is pre-set according to the importance of each physical quantity to the overall health status of the device; The calculation process of the strategy confidence is as follows: Obtain the state-action value function calculated by the value network of the deep reinforcement learning agent; Calculate the variance of the state-action value function; Set the reciprocal of the variance as the strategy confidence; Policy confidence The calculation process is precisely represented by the following formula: wherein : policy confidence, dimensionless, calculated by this step; : mean of the state-action value function distribution, in the dimension of the value unit, computed by the value network of the DRL agent; : variance of the state-action value function distribution, with dimension of square of the value unit, calculated by the value network of the DRL agent; The synthesis process of the cognitive entropy is as follows: Normalize the model-reality deviation rate by dividing it by the preset reference deviation threshold to obtain the relative deviation measure; Perform nonlinear transformation processing on the strategy confidence; Multiply the relative deviation measure by the result of the nonlinear transformation processing of the strategy confidence to generate the cognitive entropy; Cognitive entropy H c The synthesis process is exactly represented by the following equation: H c : cognitive entropy, dimensionless, calculated in this step; : model-reality deviation rate, dimensionless, calculated by the cognitive discrepancy quantification unit; : reference deviation threshold, dimensionless, preset by statistical analysis on historical normal operating condition distribution; : policy confidence.

2. The digital twin and deep reinforcement learning based smart factory failure prediction and self-repair system of claim 1, wherein, The judgment process of the system mode switching judgment unit is as follows: If the cognitive entropy is less than or equal to the critical switching threshold, generate a utilization mode signal; If the cognitive entropy is greater than the critical switching threshold, generate an exploration mode signal.

3. The digital twin and deep reinforcement learning based smart factory failure prediction and self-repair system of claim 1, wherein, The reconstruction process of the digital twin model is as follows: Activate the exploratory action strategy; Acquire the state data sequence under the exploratory action; Identify and solve the deviation model function based on the state data sequence; Integrate the deviation model function with the original digital twin model to generate an updated digital twin state transition function.

4. The digital twin and deep reinforcement learning based smart factory failure prediction and self-repair system of claim 1, wherein, The generation process of the new optimal strategy is as follows: Set the reconstructed digital twin model as a simulation environment; Incrementally train the original strategy network in the simulation environment; Solve the objective function that can maximize the long-term return by running the strategy optimization algorithm to obtain the new optimal strategy.

5. The digital twin and deep reinforcement learning based smart factory failure prediction and self-repair system of claim 1, wherein, The deployment process of the new optimal strategy is as follows: Load the new optimal strategy to the controller of the physical system; A send mode switch back signal switches the system from the explore mode back to the exploit mode.

Images