Rolling bearing health index construction method based on deep reinforcement learning

Abstract

This invention belongs to the field of mechanical equipment condition monitoring and fault prediction technology, specifically relating to a method for constructing a health index for rolling bearings based on deep reinforcement learning. The method includes: acquiring features to be fused from sensor data of the mechanical equipment and performing normalization processing; utilizing deep reinforcement learning technology to simultaneously perform two stages of tasks: feature selection and feature weight allocation, to construct a comprehensive health index; wherein the action space of deep reinforcement learning includes discrete feature selection actions and continuous feature weight allocation actions, and the reward function is based on the information gain ratio of the health index; by training a deep reinforcement learning agent, it learns to select the most effective subset of features and allocate optimal weights, thereby generating a health index. This invention can automatically and adaptively construct health indices, avoiding the problems of relying on expert knowledge and manual design in traditional methods, and improving the quality and generalization ability of health indices.

Description

Technical Field

[0001] This invention belongs to the field of mechanical equipment condition monitoring and fault prediction technology, specifically involving a method for constructing health indicators for rolling bearings based on deep reinforcement learning. Background Technology

[0002] In the field of industrial equipment health monitoring, the construction of Health Indicators (HIs) is crucial for predicting the remaining service life of equipment and preventing failures. Traditional methods for constructing HIs mainly rely on unsupervised feature extraction techniques, such as Principal Component Analysis (PCA) and autoencoders. These methods fuse multi-sensor data into a single health indicator through linear or nonlinear dimensionality reduction. However, these methods have significant limitations: PCA, based on linear assumptions, struggles to capture complex nonlinear degradation processes; while autoencoders can handle nonlinearity, they rely on large amounts of labeled data and suffer from unstable training; furthermore, traditional methods cannot dynamically adapt to the temporal characteristics of equipment degradation sequences, resulting in insufficient monotonicity, trend-based analysis, smoothness, and predictability of the HIs. Methods such as Bayesian optimization can optimize feature weights, but they are computationally expensive and prone to getting trapped in local optima. Therefore, there is an urgent need for an intelligent method that can automatically learn the optimal feature subset and fusion weights while ensuring the quality of the HIs.

[0003] In recent years, reinforcement learning has demonstrated its powerful capabilities in sequential decision-making problems, but its direct application to the construction of health indicators faces challenges such as excessive action space and unstable training. The inventors aim to effectively solve these problems by innovatively decomposing the problem into two stages and combining it with a maximum entropy reinforcement learning framework. Summary of the Invention

[0004] The purpose of this invention is to overcome the above-mentioned problems in traditional technologies and provide a method for constructing rolling bearing health indicators based on deep reinforcement learning.

[0005] To achieve the above-mentioned technical objectives and effects, the present invention is implemented through the following technical solution:

[0006] This invention provides a method for constructing health indicators for rolling bearings based on deep reinforcement learning. The method adopts a cascaded two-stage architecture, which decomposes the health indicator construction problem into two sub-tasks executed sequentially: feature selection and weight optimization.

[0007] Phase 1: Identify the most discriminative feature subset from the original feature set and output a binary feature selection mask;

[0008] The second stage involves learning the optimal weighted fusion strategy based on the selected feature subset to generate the final health indicator sequence.

[0009] Both stages employ the maximum entropy reinforcement learning framework, and training stability is ensured through experience replay mechanisms and target network strategies.

[0010] Further, the steps are as follows:

[0011] S1. Problem modeling;

[0012] S2, a two-stage reinforcement learning architecture;

[0013] S3, Neural Network Construction and Training in the Proxy Model.

[0014] Furthermore, the specific steps of step S1 are as follows:

[0015] S101, State Space Design

[0016] State representation: State From the recent l Each time step d The data consists of dimensional features, in which d This represents the original number of features. l The time window size; the state matrix is ​​flattened as 3D vector as network input:

[0017] ;

[0018] in, For the entire feature sequence, For sequence length, The feature dimension of the sequence;

[0019] S102, Motionspace Design

[0020] The action space is a hybrid action space, containing discrete binary feature selection actions and continuous weight assignment actions:

[0021] ;

[0022] The total dimension is 2d.

[0023] For a discrete-continuous mixed action space, the Gumbel-Softmax reparameterization technique is used to handle discrete actions, ensuring gradient backtracking:

[0024] .

[0025] Furthermore, the specific steps of step S2 are as follows:

[0026] S201, Feature Selection

[0027] State transition: The environment changes from Initially, advance one time step after each action, until... End at time;

[0028] Reward function design:

[0029] ;

[0030] in, As a penalty for average relevance, ; As a reward for average information entropy, , To select the feature index set, Features and The correlation coefficient; and The weight values ​​corresponding to the two penalty and reward items;

[0031] S202, Weight Optimization

[0032] State Reconstruction: Feature Selection Mask Based on First-Stage Output Reconstructing state Maintaining 48 input dimensions through zero padding;

[0033] Reward function design:

[0034] ;

[0035] in, , , and The corresponding weight coefficients for the monotonicity, trend, smoothness, and predictiveness reward functions are used; the comprehensive quality assessment values ​​of the reward functions and health indicators are consistent.

[0036] Furthermore, the specific steps of step S3 are as follows:

[0037] S301, Network Structure Design

[0038] Each stage is equipped with an independent SAC algorithm instance, which includes an Actor network and a Critic network. The Actor network has a shared feature extraction layer and two output heads that generate feature selection probabilities and weight assignments, respectively. The Critic network has a dual Q-network structure, with the input being a state-action concatenation vector and the output being a Q-value estimate.

[0039] S302 and SAC algorithm implementation

[0040] Optimization goal:

[0041] ;

[0042] Soft Bellman equation:

[0043] SAC extends the standard Bellman equations by including an entropy regularization term:

[0044] ;

[0045] The soft-state value function is defined as follows:

[0046] ;

[0047] S303, Training strategies include Critic network update, Actor network update, and target network soft update:

[0048] Critic Network Update:

[0049] ;

[0050] Among them, the target network Q̄ provides a stable learning objective, avoiding the divergence of the value function; For the target Q value, For a moment Instant rewards As a termination marker, and These are the Q-functions of the first and second target Critic networks, respectively. Sample the next action based on the current strategy;

[0051] Critic loss function:

[0052] ;

[0053] Actor Network Update:

[0054] The Actor's objective is to maximize the expected Q value plus the policy entropy:

[0055] ;

[0056] Target network soft update:

[0057] The target network slowly tracks the changes in the parameters of the main network using an exponential moving average method.

[0058] ;

[0059] This lazy update strategy allows for incremental improvement of the strategy while maintaining the stability of the learning objectives.

[0060] S304, Training Process and Convergence Analysis

[0061] Experience replay mechanism: Each stage maintains an independent experience replay buffer, employing a first-in-first-out (FIFO) management strategy to store five-tuples. .

[0062] Furthermore, in step S303, reparameterization is used to reduce variance. ,in For the noise vector:

[0063] .

[0064] Furthermore, in S304, multi-timescale update coordination: the system has three timescales:

[0065] 1) Critic network update: Q-value function learning;

[0066] 2) Actor Network Update: Policy Improvement;

[0067] 3) Target network update: parameter tracking.

[0068] 8. The method for constructing rolling bearing health indicators based on deep reinforcement learning according to claim 7, characterized in that, in S304, the convergence of network training is guaranteed through three aspects:

[0069] 1) The strategy improvement theorem ensures improved monotonic performance;

[0070] 2) Gradient clipping prevents gradient explosion;

[0071] 3) Adaptive learning rate balances convergence speed and stability.

[0072] The beneficial effects of this invention are:

[0073] 1. This invention simplifies the high-dimensional action space into two low-dimensional sub-problems through a two-stage task decomposition, avoiding the "curse of dimensionality" and improving training efficiency and stability. Each stage designs a dedicated reward function. The feature selection stage focuses on feature independence and information content, while the weight optimization stage directly optimizes the quality of health indicators, ensuring that the generated indicators have strong monotonicity, trend, smoothness, and predictability.

[0074] 2. This invention utilizes the sequential decision-making characteristics of reinforcement learning to adaptively capture the dynamic changes in the equipment degradation process, outperforming static fusion methods. The maximum entropy framework and experience replay mechanism enhance the algorithm's exploratory capabilities and convergence stability, making it suitable for various industrial scenarios.

[0075] 3. This invention provides a universal and efficient solution for equipment health monitoring, which can be widely applied in aerospace, intelligent manufacturing and other fields.

[0076] Of course, any product implementing this invention does not necessarily need to achieve all of the above advantages at the same time. Attached Figure Description

[0077] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0078] Figure 1 This is a schematic diagram of the eight features to be fused after normalization of the first unit of the FD001 turbofan engine.

[0079] Figure 2 This is a schematic diagram of a two-stage machine health index construction method based on reinforcement learning;

[0080] Figure 3 This is a schematic diagram of a two-stage training process based on reinforcement learning.

[0081] Figure 4 A diagram illustrating the feature selection mask and fusion weights obtained from reinforcement learning;

[0082] Figure 5 A schematic diagram of the structure of the Actor-Critic network for reinforcement learning;

[0083] Figure 6 A diagram illustrating the constructed health indicators;

[0084] Figure 7 A comparison chart showing the normalized features to be fused, the comparison methods (principal component analysis, autoencoder, Bayesian optimization), and the information gain index obtained by this invention. Detailed Implementation

[0085] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0086] This invention provides a method for constructing a health index for rolling bearings based on deep reinforcement learning. The method includes: acquiring features to be fused from sensor data of mechanical equipment and performing normalization processing; utilizing deep reinforcement learning technology to simultaneously perform two stages of tasks: feature selection and feature weight allocation, to construct a comprehensive health index; wherein the action space of deep reinforcement learning includes discrete feature selection actions and continuous feature weight allocation actions, and the reward function is based on the information gain ratio of the health index, which is calculated by comprehensively considering monotonicity, trend, and predictability. By training a deep reinforcement learning agent, it learns to select the most effective subset of features and allocate optimal weights, thereby generating a health index with high monotonicity, trend, and predictability, used to accurately assess the health status of mechanical equipment and predict its remaining service life. This invention can automatically and adaptively construct health indices, avoiding the problems of relying on expert knowledge and manual design in traditional methods, and improving the quality and generalization ability of health indices.

[0087] Example

[0088] I. Implementation Environment and Data Preparation

[0089] This embodiment uses the turbofan engine degradation simulation dataset (FD001) released by NASA for verification. This dataset contains the degradation process of multiple engine units under constant operating conditions, with time-series data recorded by 21 sensors for each unit. This example uses the first unit of the FD001 subset, and the data covers 192 operating cycles.

[0090] Data preprocessing steps:

[0091] (1) Sensor selection: Eight sensors with obvious degradation trends were selected from 21 sensors (numbered 3, 4, 5, 6, 7, 11, 12, 13). Their monitoring data included parameters such as temperature, pressure, and flow rate, which were used as the original feature set.

[0092] (2) Data normalization: Z-score normalization is applied to the time-series data of each sensor to eliminate the influence of dimensions. The normalization formula is:

[0093]

[0094] in, The original data, The mean, The standard deviation is 1. After normalization, each feature has a mean of 0 and a standard deviation of 1, which is convenient for reinforcement learning. The normalized feature sequence is as follows: Figure 1 As shown, the data from eight sensors exhibits trends over 192 periods, providing input for subsequent feature selection.

[0095] II. Parameter Settings and System Configuration

[0096] The key parameters in this embodiment are set based on algorithm requirements and dataset characteristics to ensure training efficiency and stability.

[0097] Reinforcement learning parameters:

[0098] (1) Discount factor: γ=0.99, to balance the importance of current and future rewards.

[0099] (2) Entropy weight: α=0.2, to encourage exploratory strategies.

[0100] (3) Experience replay buffer size: 10 6 Store training samples.

[0101] (4) Batch size: 256, the number of samples is randomly sampled from the buffer each time it is updated.

[0102] (5) Target network update coefficient: τ=0.005, soft update rate.

[0103] Reward function weights:

[0104] (1) Feature selection stage: relevance penalty weight Information entropy reward weight To balance feature independence and information content.

[0105] (2) Weight optimization stage: monotonic weights Trend weight Smoothness weights Predictive weights It highlights the dominant role of monotony and trends.

[0106] (3) Optimizer and network parameters: Critic network learning rate: 1×10 −3 The Adam optimizer is used. Actor network learning rate: 1×10⁻⁶ −4 To avoid overly rapid policy updates. Network structure: Both the Actor and Critic networks use fully connected layers, with an input dimension of 48 (corresponding to the flattened state vector), hidden layer dimensions of 128-64, and an output layer adapted to the action space.

[0107] III. Two-Phase Intensive Learning and Training Process

[0108] The training process is divided into two stages: feature selection and weight optimization. Each stage runs an independent SAC algorithm instance. The overall process is as follows: Figure 2 As shown, this illustrates the data flow and decision-making logic of a cascaded architecture.

[0109] Phase 1: Feature Selection

[0110] (1) State construction: construct the state with a time window of l=6. The vector is flattened into a 48-dimensional vector. Training begins at t=5 and the window is gradually slid until the end of the sequence.

[0111] (2) Action execution: The agent outputs an 8-dimensional binary action vector, with each element corresponding to the selection of a feature (1 for selection, 0 for non-selection). Action generation is based on the probability distribution output by the Actor network, and gradient backtracking is ensured through Gumbel-Softmax sampling.

[0112] (3) Reward Calculation: Based on immediate rewards The update strategy, which penalizes feature correlation (CorrP) and rewards information entropy (EntroP) in the reward function, drives the agent to select a subset of features with strong discriminative power.

[0113] (4) Network Update: The Critic network updates by minimizing the temporal difference error, while the Actor network maximizes the entropy-regularized expected reward. Each training step involves experience replay and soft updates of the target network, as follows: Figure 3 As shown.

[0114] The output feature selection mask in this stage indicates that features 2, 5, 7, and 8 (index starts from 1) are selected, and redundant sensors are discarded. Figure 4 As shown.

[0115] Phase Two: Weight Optimization

[0116] (1) State reconstruction: Based on the feature selection mask, only the data of the selected features are retained to reconstruct the state. (4 selected features) and maintain 48-dimensional input dimensions through zero padding.

[0117] (2) Action execution: The agent outputs a 4-dimensional continuous action vector, representing the fusion weight of each feature. Softmax activation is used to ensure that the weight sum is 1.

[0118] (3) Reward Calculation: Reward The quality of health indicators is directly optimized by weighting the scores for monotonicity, trend, smoothness, and predictability.

[0119] (4) Strategy Optimization: The same SAC update mechanism as in the first stage is adopted, but the reward function focuses on health indicator characteristics. The Actor-Critic network structure is as follows: Figure 5 As shown, the dual output heads process the mixed motion space.

[0120] The output fusion weights in this stage correspond to the weight allocation for features 2, 5, 7, and 8, with fusion weights of 0.220, 0.305, 0.243, and 0.232 for each feature, respectively. Figure 4 As shown.

[0121] IV. Generation and Result Analysis of Health Indicators

[0122] The selected features are weighted and fused according to optimized weights to generate the final health indicator sequence:

[0123]

[0124] in, ={2,5,7,8} is the selected feature index set. As weight, These are normalized eigenvalues. The generated health indicators are as follows: Figure 6 As shown, it exhibits a clear monotonic downward trend, which is consistent with the law of engine performance degradation.

[0125] To verify its effectiveness, it is compared with the following traditional methods:

[0126] Principal Component Analysis (PCA-HI): Linear dimensionality reduction to generate health indicators.

[0127] Deep Autoencoder (DAE-HI): A nonlinear fusion of neural networks, with an encoder structure of 8-20-10-5-2-5-10-20-8 and a loss function of cross-entropy.

[0128] Bayesian Optimization (BO-HI): Probabilistic Random Forest surrogate model, with weights optimized in 500 iterations.

[0129] The comparison results are as follows Figure 7 As shown, the overall quality assessment values ​​of the normalized features are all low, while the overall quality assessment values ​​of the integrated health indicators obtained by the comparative method are improved. Among them, the proposed method (DRL-HIs) obtained the best overall quality assessment value, which illustrates the effectiveness and advantages of the proposed two-stage reinforcement learning method.

[0130] The core of this embodiment lies in providing a two-stage reinforcement learning framework for automatically constructing high-performance health indicators from multi-sensor time-series data. This framework decomposes the complex health indicator construction problem into two sequentially executed subtasks: feature selection and weight optimization, each optimized using a customized Soft Actor-Critic (SAC) algorithm. The method can automatically learn the optimal feature subset and its fusion weights, generating health indicators with strong monotonicity, trend characteristics, smoothness, and predictability.

[0131] The preferred embodiments of the present invention disclosed above are merely illustrative of the invention. These preferred embodiments do not exhaustively describe all details, nor do they limit the invention to specific implementations. Clearly, many modifications and variations can be made based on the content of this specification. This specification selects and specifically describes these embodiments to better explain the principles and practical applications of the invention, thereby enabling those skilled in the art to better understand and utilize the invention. The invention is limited only by the claims and their full scope and equivalents.

Claims

1. A method for constructing health indicators for rolling bearings based on deep reinforcement learning, characterized in that, This method adopts a cascaded two-stage architecture, which decomposes the health indicator construction problem into two sub-tasks executed sequentially: feature selection and weight optimization. Phase 1: Identify the most discriminative feature subset from the original feature set and output a binary feature selection mask; The second stage involves learning the optimal weighted fusion strategy based on the selected feature subset to generate the final health indicator sequence. Both stages employ the maximum entropy reinforcement learning framework, and training stability is ensured through experience replay mechanisms and target network strategies.

2. The method for constructing rolling bearing health indicators based on deep reinforcement learning according to claim 1, characterized in that, The steps are as follows: S1. Problem modeling; S2, a two-stage reinforcement learning architecture; S3, Neural Network Construction and Training in the Proxy Model.

3. The method for constructing rolling bearing health indicators based on deep reinforcement learning according to claim 2, characterized in that, The specific steps of step S1 are as follows: S101, State Space Design State representation: State From the recent l Each time step d The data consists of dimensional features, in which d This represents the original number of features. l The time window size; the state matrix is ​​flattened as 3D vector as network input: ； in, For the entire feature sequence, For sequence length, The feature dimension of the sequence; S102, Motionspace Design The action space is a hybrid action space, containing discrete binary feature selection actions and continuous weight assignment actions: ； The total dimension is 2d. For a discrete-continuous mixed action space, the Gumbel-Softmax reparameterization technique is used to handle discrete actions, ensuring gradient backtracking: 。 4. The method for constructing rolling bearing health indicators based on deep reinforcement learning according to claim 3, characterized in that, The specific steps of step S2 are as follows: S201, Feature Selection State transition: The environment changes from Initially, advance one time step after each action, until... End at time; Reward function design: ； in, As a penalty for average relevance, ; As a reward for average information entropy, , To select the feature index set, Features and The correlation coefficient; and The weight values ​​corresponding to the two penalty and reward items; S202, Weight Optimization State Reconstruction: Feature Selection Mask Based on First-Stage Output Reconstructing state Maintaining 48 input dimensions through zero padding; Reward function design: ； in, , , and The corresponding weight coefficients for the monotonicity, trend, smoothness, and predictiveness reward functions are used; the comprehensive quality assessment values ​​of the reward functions and health indicators are consistent.

5. The method for constructing rolling bearing health indicators based on deep reinforcement learning according to claim 4, characterized in that, The specific steps of step S3 are as follows: S301, Network Structure Design Each stage is equipped with an independent SAC algorithm instance, which includes an Actor network and a Critic network. The Actor network has a shared feature extraction layer and two output heads that generate feature selection probabilities and weight assignments, respectively. The Critic network has a dual Q-network structure, with the input being a state-action concatenation vector and the output being a Q-value estimate. S302 and SAC algorithm implementation Optimization goal: ； Soft Bellman equation: SAC extends the standard Bellman equations by including an entropy regularization term: ； The soft-state value function is defined as follows: ； S303, Training strategies include Critic network update, Actor network update, and target network soft update: Critic Network Update: ； Among them, the target network Q̄ provides a stable learning objective, avoiding the divergence of the value function; For the target Q value, For a moment Instant rewards As a termination marker, and These are the Q-functions of the first and second target Critic networks, respectively. Sample the next action based on the current strategy; Critic loss function: ； Actor Network Update: The Actor's objective is to maximize the expected Q value plus the policy entropy: ； Target network soft update: The target network slowly tracks the changes in the parameters of the main network using an exponential moving average method. ； This lazy update strategy allows for incremental improvement of the strategy while maintaining the stability of the learning objectives. S304, Training Process and Convergence Analysis Experience replay mechanism: Each stage maintains an independent experience replay buffer, employing a first-in-first-out (FIFO) management strategy to store five-tuples. .

6. The method for constructing rolling bearing health indicators based on deep reinforcement learning according to claim 5, characterized in that, In step S303, reparameterization is used to reduce variance. ,in For the noise vector: 。 7. The method for constructing rolling bearing health indicators based on deep reinforcement learning according to claim 6, characterized in that, In S304, multi-timescale update coordination: the system has three timescales: 1) Critic network update: Q-value function learning; 2) Actor Network Update: Policy Improvement; 3) Target network update: parameter tracking.

8. The method for constructing rolling bearing health indicators based on deep reinforcement learning according to claim 7, characterized in that, In S304, the convergence of network training is guaranteed through three aspects: 1) The strategy improvement theorem ensures improved monotonic performance; 2) Gradient clipping prevents gradient explosion; 3) Adaptive learning rate balances convergence speed and stability.

Images