Real-time transfer learning method based on reinforcement learning framework

Abstract

The invention discloses a real-time transfer learning method based on a reinforcement learning framework, and the method comprises the steps: collecting original data at an offline stage, constructing a semi-physical mechanism simulation model for generating simulation data, training a reinforcement learning model through a data set containing the simulation data and the actual operation data of a mechanical system, and in the online stage, real-time migration learning is realized based on the to-be-migrated feature space through the trained reinforcement learning model, and a fault diagnosis result is generated. According to the method, the parameter feature space combination condition for constructing the movable frame is considered, the updating process of the feature space is integrated through the reinforcement learning framework, and feature matching is carried out on different time data during actual operation of a mechanical system through real-time transfer learning; the mode and rule of mechanical system fault data are obtained; and the limitation of the existing deep learning model is overcome.

Description

Technical Field

[0001] The present invention relates to a technology in the field of fault diagnosis, specifically a real-time transfer learning method based on a reinforcement learning framework. Background Art

[0002] Existing deep learning models rely heavily on large amounts of labeled data. In practical applications, obtaining sufficient labeled data is often costly and time-consuming. Deep learning models generally perform well on the distribution of training data, but their generalization capabilities are often insufficient when faced with new, unseen data. Existing models are difficult to migrate and adapt to new tasks in different environments or conditions, especially in real-time environments, where this ability to transfer learning is particularly important. For application scenarios that require real-time response, existing deep learning models often cannot meet real-time requirements because they take a long time to process and make decisions. Deep learning models are often viewed as "black boxes," meaning their decision-making process lacks transparency, which is a significant issue in fields such as fault diagnosis. Summary of the Invention

[0003] In response to the above-mentioned deficiencies in the prior art, the present invention proposes a real-time transfer learning method based on a reinforcement learning framework. Taking into account the parameter feature space combination of constructing the active frame, the update process of the feature space is integrated through the reinforcement learning framework, and the different time data during the actual operation of the mechanical system are matched with features through real-time transfer learning, thereby obtaining the patterns and laws of the mechanical system fault data and overcoming the limitations of the existing deep learning models.

[0004] The present invention is achieved through the following technical solutions:

[0005] The present invention relates to a real-time transfer learning method based on a reinforcement learning framework. In the offline stage, raw data is collected and a semi-physical mechanism simulation model is constructed to generate simulation data. The reinforcement learning model is trained with a data set containing simulation data and actual operation data of the mechanical system. In the online stage, real-time transfer learning is achieved based on the feature space to be transferred through the trained reinforcement learning model to generate fault diagnosis results.

[0006] The reinforcement learning model includes an action space, a state space, and a reward function.

[0007] The data set includes simulation operation data generated by the semi-physical mechanism simulation model and actual operation data of the mechanical system.

[0008] The simulation data is calculated by setting the bearing stiffness coefficient to 1.2×10 6 N / m to 2.5×10 6 N / m, and the damping coefficient is 1.8×10 3N·s / m to 3.5×10 3 After obtaining the semi-physical mechanism simulation model with N·s / m, the simulated vibration signal was generated by inputting the operating parameters of the speed of 800-2000 rpm and the temperature of 20-80℃. The sampling frequency was 10.24 kHz and the simulation time was 6 minutes.

[0009] The real-time migration is achieved in the following ways:

[0010] Step 1: Select the historical operation data of the mechanical system and its feature space as the source domain, and the operation data of the mechanical system at another time point (current or future) and its feature space as the target domain, and generate the feature spaces of the source domain and target domain through dynamic time window partitioning.

[0011] The length of the time window is 10 seconds, and the step length is 5 seconds.

[0012] The main difference between the target domain and the source domain lies in the difference in the time dimension, that is, the target domain focuses on current or future operating data, while the source domain focuses on historical operating data.

[0013] Step 2: Adjust the model strategy and feature space online through the dynamic update mechanism of the reinforcement learning framework, that is, respond to environmental changes in the target domain in real time and quickly update the model's decision strategy and feature representation.

[0014] Step 3: Ensure real-time performance through priority experience replay and dynamic learning rate adjustment.

[0015] The priority experience replay refers to sorting historical migration samples by migration loss L2, prioritizing replaying samples with the top 10% loss, and accelerating strategy convergence.

[0016] The dynamic learning rate adjustment means that if the migration fails for three consecutive times, the learning rate α is decayed according to α←0.95α, and vice versa, it is increased according to α←1.05α to balance exploration and utilization.

[0017] The present invention relates to a system for implementing the above-mentioned method, comprising: a data acquisition and preprocessing unit for collecting operating data of a mechanical system at different time points, and preprocessing the collected data to generate an input feature vector suitable for a reinforcement learning model and a data set containing source domain and target domain features, for subsequent model training and transfer learning; a reinforcement learning model training unit for constructing and training a reinforcement learning model based on the data set generated by the data acquisition and preprocessing unit, defining an action space, a state space and a reward function, and training the model using a reinforcement learning algorithm to obtain a trained reinforcement learning model and its policy network and value function; a real-time transfer learning unit for realizing real-time migration of feature space through a reinforcement learning model, adapting to new environments or tasks, and dynamically adjusting the migration strategy using the trained reinforcement learning model according to the feature data of the current target domain, matching the feature patterns of the source domain and the target domain, and updating the model parameters in real time to generate a migrated feature representation and a corresponding decision result.

[0018] The operating data includes but is not limited to multi-dimensional sensor data such as vibration signals, temperature, and rotation speed.

[0019] The preprocessing includes data cleaning, feature extraction and standardization. Technical Effects

[0020] The present invention uses positive and negative reward signals to stimulate the reinforcement learning model to dynamically adjust its strategy during the migration process, ensuring the efficient migration of knowledge from the source domain to the target domain; by quantifying the changes in the target domain features and the source domain features, it optimizes the feature matching in the transfer learning process and improves the accuracy and efficiency of the migration; in a dynamic environment, the reinforcement learning framework is used to update the strategy in real time, so that the model can quickly adapt to new tasks or environmental changes, meet real-time requirements, and integrate the update process of the feature space through the reinforcement learning framework to achieve feature matching between the source domain and the target domain, significantly improving the generalization ability of the model. Compared with the existing technology, the present invention uses a reinforcement learning framework, so that the model can more efficiently utilize limited data, reduce dependence on large amounts of labeled data, and significantly reduce data collection and labeling costs; the real-time transfer learning mechanism enables the model to respond quickly in a dynamic environment, meeting the needs of application scenarios such as real-time fault diagnosis; through the dynamic adjustment of the feature space and the migration correctness reward mechanism, the model can quickly adapt to new environments or tasks, and significantly improve the generalization ability on different data distributions; through a transparent reward mechanism and feature matching process, the decision-making process of the model is clearer, which helps to improve the reliability and credibility of fault diagnosis; through the feature loss function to optimize the feature matching in the migration process, it is ensured that the performance of the model in the target domain is stable and efficient. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 Flowchart of the present invention;

[0022] Figure 2 This is a flowchart for reinforcement learning. DETAILED DESCRIPTION

[0023] like Figure 1 As shown, this embodiment relates to a real-time transfer learning method based on a reinforcement learning framework, including:

[0024] Step 1: Establish a reinforcement learning model, including action space, state space, and reward function.

[0025] The action space is a discrete set of values used to indicate whether a fault occurs, specifically including two elements: 0 and 1, where the digital label 0 indicates normal operation without any fault, and the digital label 1 indicates a fault exists.

[0026] The state space is the characteristic space of the mechanical system operation data (such as the bearing vibration signal), including the various input parameters and acquisition parameters in the mechanism model, the semi-physical mechanism simulation model, and the actual operation data-driven model, the transformation combination of various parameters, and the network structure, weight parameters and their gradients used in various models.

[0027] The reward function R = k1R1-k2L2, where: the reward for the correctness of the transfer of knowledge from one domain (source domain) to another domain (target domain) is used to measure whether the model successfully transfers knowledge from one domain (source domain) to another domain (target domain) during the transfer learning process. That is, when the model successfully migrates, R1 will provide a positive reward signal to encourage the model to maintain or enhance this migration ability; otherwise, when the migration fails, R1 will provide a negative reward signal to encourage the model to adjust its strategy to improve the migration effect; the feature loss function used to quantify the change or loss of the model's features during the migration process is Where: i is the weight, is the target domain feature, is the source domain feature, ||·|| is the feature change calculation; the two weight parameters k1 and k2 are used to adjust the relative importance of the transfer correctness reward and feature loss in the total reward function.

[0028] The weight parameters are determined by grid search optimization, and the final values are k1=0.8, k2=0.2.

[0029] Step 2: Collect the characteristics of the same mechanical system at different operating times and create a data set, including:

[0030] 2.1 Determine the acquisition object and feature type: Select the key operating parameters of the mechanical system as the feature acquisition objects, including vibration signals (acquisition location: drive end bearing, sensor model: PCB 352C33), temperature (acquisition location: inside the temperature control box, sensor model: PT100), and speed (acquisition location: motor output shaft, sensor model: HENGSTLERRI58-O / 5000AR.12KB). The sampling frequency is uniformly set to 10.24kHz.

[0031] 2.2 Data Collection and Preprocessing: Collect historical operating data of the mechanical system at different time points, including data under normal operating conditions and fault conditions. Preprocess the collected data, including data cleaning (removing noise and outliers), normalization (unifying the data scale), and feature extraction (such as time domain features and frequency domain features).

[0032] 2.3 Feature Space Construction: The preprocessed data is organized into feature vectors to form feature spaces for the source and target domains. The source domain feature space contains feature representations of historical operating data, while the target domain feature space contains feature representations of current or future operating data.

[0033] 2.4 Data Organization and Storage: Sort data by timestamp to ensure temporal continuity and consistency between source and target domain feature data. Store the dataset to facilitate subsequent reinforcement learning model training and transfer learning implementation. Label each data point with its corresponding timestamp, operating status (normal or faulty), and feature label.

[0034] Step 3: Use the data set obtained in step 2 to train the reinforcement learning model constructed in step 1, thereby realizing real-time transfer learning. Specifically, the agent of the reinforcement learning model selects a feature transfer action in the action space according to the current state and executes it. The environment will update the state and calculate the reward based on the action and the current state. The goal of the agent is to maximize the cumulative reward through learning. The whole process is cyclical so that the agent continuously learns from the environment and optimizes its strategy to complete the training and output the feature transfer results of the mechanical system.

[0035] The agent selects action a according to the strategy π(a|s), which defines the probability of taking each possible action in a given state, specifically: π(a|s) = P(A t =a|S t =s), where: P is the probability, A t is the action taken at time step t, S tis the state at time step t. The policy network is a 3-layer fully connected neural network (hidden layer dimension 256, activation function ReLU), and the value function network structure is the same; the training hyperparameters are set to learning rate α = 3e-4, discount factor γ = 0.99, batch size 256, and total training steps 10e5.

[0036] The agent updates its decision-making process after selecting an action, specifically: Q(s,a)←Q(s,a)+α[Rt+1+γmaxa′Q(s′,a′)-Q(s,a)], where Q(s,a) is the action value function, α is the learning rate, and R t+1 is the reward obtained at time step t+1, γ is the discount factor, and s′ is the state at time step t+1.

[0037] The agent applies the selected action to the environment to execute the action, specifically: t+1 ,R t+1 ←Environment(St,At) is state S t+1 and reward R t+1 The environment is based on the current state S t and action A t Decide.

[0038] The state is the characteristic space of the bearing vibration signal, including the various input parameters and acquisition parameters in the mechanism model, semi-physical mechanism simulation model and actual operation data-driven model, the transformation combination of various parameters, and also the network structure, weight parameters and their gradients used in various models.

[0039] The reward function is R=k1R1-k2L2.

[0040] The mechanical system feature migration results are as follows: an acceleration sensor is placed at the drive end, the mechanical system is placed in a heating hood, and temperature data is collected using a temperature sensor with a sampling frequency of 10.24 kHz and a sampling time of 6 minutes. The following mechanical system data is collected. Collection data type Data Example Vibration acceleration <![CDATA[0.0042m / s 2 、0.0053m / s 2 ]]> temperature 25℃、50℃ Speed 900rpm、1800rpm

[0041] The experimental results of real-time transfer learning using the present invention are as follows: Performance indicators Parameter value Accuracy 94.86％ Live migration speed 12.7s

[0042] Compared with the existing technology, the present invention utilizes data more effectively through the reinforcement learning framework, reduces the dependence on large amounts of labeled data, and allows the model to update strategies in real time in a dynamic environment, thereby meeting real-time requirements; through the real-time transfer learning mechanism, the model can quickly adapt to the new environment, improve the model's generalization ability on different data distributions, and make the decision-making process more transparent, which helps to understand the behavior of the model and improve the reliability of fault diagnosis.

[0043] The above-mentioned specific implementation can be partially adjusted in different ways by those skilled in the art without departing from the principles and purpose of the present invention. The scope of protection of the present invention shall be based on the claims and shall not be limited by the above-mentioned specific implementation. All implementation schemes within its scope shall be subject to the constraints of the present invention.

Claims

1. A real-time transfer learning method based on a reinforcement learning framework, characterized in that: In the offline phase, raw data is collected and a semi-physical mechanism simulation model is constructed to generate simulation data. A reinforcement learning model is trained with a dataset containing simulation data and actual operation data of the mechanical system. In the online phase, the trained reinforcement learning model is used to implement real-time transfer learning based on the feature space to be transferred to generate fault diagnosis results. The data set includes simulation operation data generated by the semi-physical mechanism simulation model and actual operation data of the mechanical system.

2. The real-time transfer learning method based on the reinforcement learning framework according to claim 1 is characterized in that: The simulation data is calculated by setting the bearing stiffness coefficient to 1.2×10 6 N / m to 2.5×10 6 N / m, and the damping coefficient is 1.8×10 3 N·s / m to 3.5×10 3 After obtaining the semi-physical mechanism simulation model with N·s / m, the simulated vibration signal was generated by inputting the operating parameters of the speed of 800-2000 rpm and the temperature of 20-80℃. The sampling frequency was 10.24 kHz and the simulation time was 6 minutes.

3. The real-time transfer learning method based on the reinforcement learning framework according to claim 1 is characterized in that: The reinforcement learning model includes an action space, a state space, and a reward function; The action space is a discrete set of values used to indicate whether a fault has occurred. Specifically, it includes two elements: 0 and 1, where the digital label 0 indicates normal operation without any fault, and the digital label 1 indicates a fault. The state space is the characteristic space of the mechanical system operation data, including the various input parameters and acquisition parameters in the mechanism model, semi-physical mechanism simulation model and the actual operation data-driven model, the transformation combination of various parameters, and also includes the network structure, weight parameters and their gradients used in various models; The reward function R = k1R1-k2L2, where: the reward for measuring whether the model successfully transfers knowledge from the source domain to the target domain during the transfer learning process The feature loss function used to quantify the change or loss of the model's features during the migration process ω i is the weight, W i t is the target domain feature, W i o is the source domain feature, ||·|| is the feature change calculation; the two weight parameters k1 and k2 are used to adjust the relative importance of the transfer correctness reward and feature loss in the total reward function.

4. The real-time transfer learning method based on the reinforcement learning framework according to claim 1 is characterized in that: The real-time migration is achieved in the following ways: Step 1: Select the historical operation data of the mechanical system and its feature space as the source domain, and the operation data of the mechanical system at another time point and its feature space as the target domain, and generate the feature spaces of the source domain and target domain through dynamic time window partitioning. Step 2: Adjust the model strategy and feature space online through the dynamic update mechanism of the reinforcement learning framework, that is, respond to environmental changes in the target domain in real time and quickly update the model's decision strategy and feature representation. Step 3: Ensure real-time performance through priority experience replay and dynamic learning rate adjustment.

5. The real-time transfer learning method based on the reinforcement learning framework according to claim 4 is characterized in that: Priority experience replay refers to sorting historical migration samples by migration loss L2, prioritizing replaying samples with the top 10% loss, and accelerating strategy convergence. The dynamic learning rate adjustment means that if the migration fails for three consecutive times, the learning rate α is decayed according to α←0.95α, and vice versa, it is increased according to α←1.05α to balance exploration and utilization.

6. A real-time transfer learning system for implementing the method according to any one of claims 1 to 5, characterized in that: include: Data acquisition and preprocessing unit, reinforcement learning model training unit and real-time transfer learning unit, wherein: the data acquisition and preprocessing unit collects the operating data of the mechanical system at different time points, and preprocesses the collected data to generate an input feature vector suitable for the reinforcement learning model and a data set containing source domain and target domain features for subsequent model training and transfer learning; the reinforcement learning model training unit constructs and trains the reinforcement learning model based on the data set generated by the data acquisition and preprocessing unit, defines the action space, state space and reward function, and uses the reinforcement learning algorithm to train the model and obtain the trained reinforcement learning model and its policy network and value function; the real-time transfer learning unit realizes real-time migration of feature space through the reinforcement learning model, adapts to the new environment or task and dynamically adjusts the migration strategy based on the feature data of the current target domain using the trained reinforcement learning model, matches the feature patterns of the source domain and target domain, and updates the model parameters in real time to generate the migrated feature representation and corresponding decision results.

Images