Rolling bearing deep reinforcement learning residual life prediction method and system fusing msrae and sac-ere

Abstract

The application relates to the technical field of mechanical fault diagnosis, in particular to a rolling bearing deep reinforcement learning residual life prediction method and system fusing MSRAE and SAC-ERE. Time series prediction is modeled as a sequential decision process, a composite state space containing a historical prediction value sequence is constructed, the continuity of a historical track is considered by the model at each decision step, an instant reward function with a penalty term based on error change amount smoothing processing is introduced, an agent is guided to learn a residual life curve close to a real rolling bearing, a multi-scale feature fusion and reinforcement learning strategy are used to suppress the common "sawtooth" sharp fluctuation in non-stationary time series prediction, and an experience replay strategy with an adaptive shrinkage sampling window is used to adapt to the latest data distribution by preferentially sampling recent interaction data according to the inherent distribution offset characteristics of non-stationary time series data, so that the interference of old offset data on the current strategy is effectively relieved. The application aims to solve the problem of how to realize reliable rolling bearing residual life prediction under a non-stationary degradation scene.

Description

Technical Field

[0001] This application relates to the field of mechanical fault diagnosis technology, and in particular to a method and system for predicting the remaining life of rolling bearings by integrating MSRAE and SAC-ERE deep reinforcement learning. Background Technology

[0002] As a core component in modern industrial equipment, the health status of rolling bearings directly affects the operational safety and production efficiency of rotating machinery. With the rapid development of industrial IoT and artificial intelligence technologies, data-driven Remaining Useful Life (RUL) prediction methods have become a core means of solving health management (PHM) problems for complex electromechanical equipment because they do not rely on complex physical mechanism models and can directly extract performance degradation patterns from massive monitoring data.

[0003] Most existing data-driven RUL prediction methods employ deep learning (DL) techniques, such as convolutional neural networks (CNNs) or long short-term memory networks (LSTMs). These methods typically use a supervised learning paradigm, constructing deep networks to extract features from the monitored signal and establishing a mapping relationship from the monitored signal to the remaining lifetime by minimizing the instantaneous prediction error (such as mean squared error) at the current moment. Although deep learning methods greatly improve prediction accuracy, they are essentially still static, single-step regression fitting.

[0004] However, in practical applications, the aforementioned prediction methods based on traditional deep learning have significant limitations—large trajectory fluctuations and inaccuracies in later stages under non-stationary scenarios. Specifically, the traditional supervised learning paradigm ignores the fact that equipment degradation is a continuous, monotonous, and physically inertial dynamic process. The model struggles to effectively capture the inherent logical connections between the predicted values ​​of the same device at adjacent time steps. Due to the lack of constraints on the continuity of the predicted trajectory, when the input signal is disturbed by noise, the prediction results often exhibit severe sawtooth fluctuations, lacking physical smoothness and consistency, making it difficult to meet the requirements of high-reliability industrial scenarios.

[0005] In view of this, this application proposes a novel deep reinforcement learning method for predicting the remaining life of rolling bearings, aiming to achieve reliable prediction of the remaining life of rolling bearings under non-stationary degradation scenarios. Summary of the Invention

[0006] The main objective of this application is to provide a deep reinforcement learning method for predicting the remaining life of rolling bearings that integrates MSRAE and SAC-ERE, aiming to solve the problem of how to achieve reliable prediction of the remaining life of rolling bearings in non-stationary degradation scenarios.

[0007] To achieve the above objectives, this application provides a deep reinforcement learning method for predicting the remaining life of rolling bearings that integrates MSRAE and SAC-ERE, the method comprising:

[0008] S10, by constructing a multi-scale residual autoencoder (MSRAE), the potential health feature vector at the current moment is obtained from the time series data of the rolling bearing vibration sample, and the potential health feature vector is fused with the remaining life prediction value sequence at historical moments to construct a composite state vector containing historical prediction feedback.

[0009] S20, based on the composite state vector, a policy decision is made through a deep reinforcement learning prediction model based on soft actor-critic SAC-ERE that emphasizes the replay of recent experience, and the degenerate step size adjustment value in the continuous action space is output.

[0010] S30, determine the remaining lifetime prediction value at the current moment based on the change between the degradation step size adjustment value and the historical prediction result value, determine the immediate reward function containing the error change smoothing penalty term according to the smooth error gradient between the remaining lifetime prediction value and the true value, and store the interactive data generated during the training process under the constraint of the immediate reward function into the experience playback buffer.

[0011] S40, by prioritizing the sampling of recent data in the experience replay buffer in the soft actor-critic SAC-ERE reinforcement learning algorithm that emphasizes recent experience replay to optimize the agent's prediction strategy, the agent obtained after learning the prediction model is used to predict the remaining life of the rolling bearing.

[0012] Optionally, the method is characterized in that, before S10, it further includes:

[0013] Calculate the mean and standard deviation of the kurtosis of the first M samples of rolling bearing vibration samples in the healthy stage;

[0014] The judgment threshold is determined based on the weighted sum of the mean kurtosis and the standard deviation.

[0015] When the signal kurtosis value of n consecutive sampling times exceeds the determination threshold, the first time among the n times is determined as the degradation start point.

[0016] The time-series data collected before the degradation initiation point is used as healthy time-series data, and the time-series data collected after the degradation initiation point is used as lifetime decay time-series data.

[0017] Optionally, prior to S10, it also includes:

[0018] A multi-scale residual autoencoder (MSRAE) is constructed, specifically consisting of multiple convolutional layers, residual blocks, dilated convolutions, SE modules, and temporal attention pooling. For each vibration segment, the encoder outputs a low-dimensional latent feature vector. The decoder reconstructs waveforms starting from a fixed-length time grid and is optimized during the training phase by calculating a weighted sum of the temporal mean square error and the logarithmic magnitude spectrum loss. During the prediction phase, the autoencoder parameters are fixed, and feature extraction is performed using only the encoder.

[0019] Optionally, in step S30, based on the smoothed error gradient between the predicted and actual remaining lifetime values, an immediate reward function including a smoothing penalty term for error change is determined, specifically including:

[0020] The absolute difference between the predicted remaining lifetime and the actual lifetime is calculated as the prediction error at the current moment;

[0021] Calculate the instantaneous change in prediction error relative to the prediction error at the previous moment;

[0022] The instantaneous change is smoothed to obtain a smoothed value of the error change.

[0023] The error change smoothing penalty term is generated based on the smoothed value of the error change, and the immediate reward is determined by combining it with the error amplitude reward term.

[0024] Optionally, in step S30, the expression for calculating the smoothed value of the error change is:

[0025]

[0026] In the formula, This is the smoothing coefficient.

[0027] Optionally, the error change smoothing penalty term is generated based on the smoothed value of the error change, and combined with the error amplitude reward term to determine the immediate reward, specifically including:

[0028] When the prediction error exceeds the preset allowable threshold, a quadratic penalty is applied to the excess portion.

[0029] A linear reward is given when the prediction error is less than or equal to the preset allowable threshold.

[0030] The instant reward is determined according to the following formula:

[0031]

[0032] Where R(t) is the instant reward, For the error amplitude reward item, The term represents the penalty for smoothing the change in error, and k_3 is the penalty weight. This is the smoothed value of the error variation.

[0033] Optionally, S40 includes:

[0034] The sampling window size for experience replay is calculated based on the current training progress, and the sampling window size adaptively shrinks as the training progresses.

[0035] Within the sampling window, recent interaction data is sampled first for strategy optimization.

[0036] Optionally, the sampling window size for the k-th update is calculated using the following formula:

[0037]

[0038] in, Let N be the sampling window size at the k-th update, and N be the total capacity of the empirical replay buffer. The recent experience emphasis coefficient is represented by K, which is the total number of updates. Minimum sampling range;

[0039] The sampling window size adaptively shrinks as the training progresses, and also includes:

[0040] The recent experience emphasis coefficient Using a linear annealing strategy, the recent empirical emphasis coefficient for the current training time step is calculated according to the following formula:

[0041]

[0042] in, The recent experience emphasis coefficient for the current training time step t. The initial recent experience emphasis coefficient, The final recent experience emphasis coefficient is given, and T is the total training time step.

[0043] Optionally, based on the composite state vector, a policy decision is made using a constructed soft actor-critic (SAC_ERE) deep reinforcement learning prediction model that emphasizes the replay of recent experiences, outputting a degradation step size adjustment value in the continuous action space, including:

[0044] The predicted value adjustment amount at the current moment is output based on the composite state vector;

[0045] Based on the predicted value from the previous time step and the adjustment amount of the predicted value, the degradation step size adjustment value for the current time step is determined.

[0046] In addition, to achieve the above objectives, this application also provides a computer system, the computer system comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the computer program is executed by the processor, it implements the steps of the rolling bearing deep reinforcement learning method for predicting the remaining life of rolling bearings based on deep reinforcement learning, as described in any of the preceding claims.

[0047] This application has at least the following beneficial effects:

[0048] 1. Model time series prediction as a sequential decision-making process. By constructing a composite state space containing a sequence of historical prediction values, the model can consider the continuity of historical trajectories at each decision step, thereby avoiding sudden changes in results due to noise interference when the deep learning model makes single-point predictions.

[0049] 2. This invention introduces a multi-scale residual autoencoder (MSRAE) as the feature extraction front end. Combined with dilated convolution and attention mechanisms, it can deeply decouple degradation-sensitive features from the original vibration signal, providing high-quality state representation for reinforcement learning agents and significantly improving the model's perception ability under complex working conditions.

[0050] 3. A penalty term based on error change smoothing is introduced. The real oscillation trend is extracted by smoothing the instantaneous change of the prediction error, and a penalty is applied accordingly to constrain the smoothness of the prediction curve and suppress the "sawtooth" violent fluctuations that are common in non-stationary time series prediction.

[0051] 4. To address the inherent distribution shift characteristics of non-stationary time series data, an empirical replay strategy with an adaptive shrinking sampling window was adopted. This strategy prioritizes sampling recent interaction data to adapt to the latest data distribution and mitigates the interference of older shift data on the current strategy. Attached Figure Description

[0052] Figure 1 This is a flowchart illustrating the rolling bearing remaining life prediction method based on deep reinforcement learning, which integrates MSRAE and SAC-ERE, according to an embodiment of this application.

[0053] Figure 2 This is a schematic diagram of the architecture of the MSRAE model involved in the embodiments of this application;

[0054] Figure 3 This is a schematic diagram comparing the lifetime prediction results of the method proposed in this application with the actual lifetime curves in the embodiments of this application;

[0055] Figure 4 This is a schematic diagram of the hardware operating environment of the computer system involved in the embodiments of this application.

[0056] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0057] To better understand the above technical solutions, exemplary embodiments of this disclosure will be described in more detail below with reference to the accompanying drawings. While exemplary embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of this disclosure to those skilled in the art.

[0058] First Embodiment

[0059] Reference Figure 1 This embodiment provides a deep reinforcement learning method for predicting the remaining life of rolling bearings based on deep reinforcement learning, which integrates MSRAE and SAC-ERE. The method includes the following steps:

[0060] S10, by constructing a multi-scale residual autoencoder (MSRAE), the potential health feature vector of the current moment is extracted from the time series data of the rolling bearing vibration sample, and the potential health feature vector is fused with the remaining life prediction value sequence of historical moments to construct a composite state vector containing historical prediction feedback.

[0061] In non-stationary time series prediction scenarios, the evolution of the target object often exhibits physical inertia and continuity; for example, the degradation of mechanical parts is a gradual process rather than a discrete jump. However, traditional supervised learning models rely solely on the observed features at the current moment for single-point prediction, ignoring the inherent logical connections between predicted values ​​at adjacent time steps, making them highly susceptible to noise interference and abrupt changes. This embodiment constructs a closed-loop state space by introducing the remaining lifetime prediction values ​​from historical moments, forcing the model to explicitly refer to the predicted trajectory from the previous moment when making decisions at each step, thereby constraining the continuity of the predicted trajectory at the mechanism level.

[0062] In some optional implementations, the acquisition of time-series data for rolling bearing vibration samples can be configured with a sampling frequency of 25.6 kHz, a sampling interval of 1 minute, and a single sampling duration of 1.28 seconds. Based on the short-time stationarity assumption, a sliding window technique is used to slice the original signal. The window length is set to L = 2560 (corresponding to 0.1 seconds), and the step size is set to 2560 (non-overlapping sampling). For each slice sample... Its dimensions are (2,2560).

[0063] Furthermore, and optionally, to eliminate the differences in dimensions between different sensors and accelerate the convergence of the neural network, channel-independent min-max normalization is performed on the sliced ​​data, mapping it to... Interval.

[0064] The calculation formula is:

[0065]

[0066] In the formula: For the first slice In the nth sample, the nth The raw data of the channel, and These are the statistical extreme values ​​of the channel on the training set.

[0067] Specifically, the potential health feature vector can be extracted using a pre-built multi-scale residual autoencoder (MSRAE) for deep feature extraction.

[0068] As an optional implementation method in this embodiment, refer to Figure 2 The diagram shows the architecture of the MSRAE model, which includes an encoder and a decoder, and extracts degradation-sensitive latent features through unsupervised pre-training.

[0069] The encoder structure includes:

[0070] (1) Multi-scale convolution: First, downsampling is performed through a convolutional layer with a kernel size of 3 and a stride of 2.

[0071] (2) Multi-dilation rate residual block: After each downsampling stage, three different dilation rates are connected in parallel. The residual block of the design significantly expands the receptive field without increasing the number of parameters, enabling it to simultaneously capture short-term high-frequency impacts (small expansion rate) and long-term low-frequency trends (large expansion rate).

[0072] (3) SE attention module (Squeeze-and-Excitation): Embedded in the residual block. Channel statistics are obtained through global average pooling (Squeeze), and then channel weights are generated through two fully connected layers (Excitation, intermediate ReLU activation, and output Sigmoid activation) to adaptively recalibrate the feature map.

[0073] (4) Temporal Attention Pooling: At the end of the encoder, an attention mechanism is used instead of global average pooling. The weight distribution in the time dimension is calculated and weighted aggregation is used to obtain the final latent feature vector. The dimension is 32.

[0074] For the decoder structure: In order to preserve temporal structure information, the decoder does not directly reconstruct from single-point features, but instead uses latent features... It is mapped to a fixed-length temporal grid (length set to 32), and then gradually restored to the original length of 2560 through upsampling and convolutional layers.

[0075] In this step, the physical feature vector (i.e., the potential health feature vector) extracted by MSRAE at the current time is concatenated with the remaining life prediction sequence of the past few time steps to obtain a composite state vector.

[0076] For example, suppose the remaining lifetime prediction values ​​collected over the past three time steps constitute a remaining lifetime prediction value sequence, the mathematical expression for the composite state vector is:

[0077]

[0078] In the formula: For the extracted potential health feature vector, A sequence of predicted remaining lifetime values ​​of length 3.

[0079] S20, based on the composite state vector, a policy decision is made through a constructed deep reinforcement learning prediction model based on soft actor-critic (SAC_ERE) that emphasizes the replay of recent experience, and the degenerate step size adjustment value in the continuous action space is output.

[0080] In this embodiment, the composite state vector constructed in step S10 is received, and the predicted action at the current moment is output through the policy network. Then, the predicted value at the current moment, i.e., the degradation step size adjustment value, is calculated. Since the state vector already contains historical prediction feedback, when the policy network outputs the current action, it is actually fine-tuning and correcting based on the historical trajectory, rather than regressing from zero, thereby ensuring the continuity of the predicted trajectory.

[0081] S30, determine the remaining lifetime prediction value at the current moment based on the change between the degradation step size adjustment value and the historical prediction result value, determine the instant reward function containing the error change smoothing penalty term according to the smooth error gradient between the remaining lifetime prediction value and the true value, and store the interactive data generated during the training process under the constraint of the instant reward function into the experience playback buffer.

[0082] In non-stationary evolution processes, not only does the absolute amplitude of the prediction error need to be controlled, but the drastic fluctuations in the error are also the culprit behind the "sawtooth" oscillations in the predicted trajectory. Traditional reward functions are usually designed based only on the instantaneous error amplitude and cannot distinguish between the natural decay of the error and high-frequency oscillations. This embodiment innovatively introduces a smoothing value for the error change and generates a smoothing penalty term accordingly.

[0083] Specifically, the change in error reflects the instantaneous fluctuation trend of the prediction error, while smoothing it filters out the instantaneous jitter caused by random noise, extracting the essential features that reflect the true oscillation trend of the predicted trajectory. By penalizing this smoothing trend, this embodiment explicitly and stably constrains the smoothness of the prediction curve at the reward level, suppressing the violent fluctuations common in non-stationary time series predictions and guiding the agent to output a smoothly transitioning predicted trajectory. It should be understood that the smoothing method for the change in error is not limited to a specific mathematical filter; any smoothing method that can extract the oscillation trend and filter out high-frequency noise should be considered an equivalent substitute.

[0084] Specifically, and optionally, the mathematical expression for the remaining life prediction is:

[0085]

[0086] In the formula, These are historical prediction results. This is the degenerate step size adjustment value within the continuous motion space.

[0087] In this step, based on the smoothed error gradient between the predicted and actual remaining lifetime values, an immediate reward function including a smoothing penalty term for error change is determined, specifically including:

[0088] S31, calculate the difference between the predicted remaining lifespan and the actual lifespan as the prediction error at the current moment;

[0089] For example, let the current estimation error be... The measured true value is :

[0090]

[0091] S32, calculate the instantaneous change of the prediction error relative to the prediction error at the previous moment;

[0092] For example, let the instantaneous change be... , The prediction error from the previous time step:

[0093]

[0094] S33, Smooth the instantaneous change to obtain a smoothed value of the error change;

[0095] Further, and optionally, the expression for calculating the smoothed value of the error change is:

[0096]

[0097] In the formula, This is the smoothing coefficient.

[0098] S34, generate the error change smoothing penalty term based on the smoothed value of the error change amount, and combine it with the error amplitude reward term to determine the instant reward.

[0099] Further and optionally, S34 includes:

[0100] When the prediction error exceeds the preset allowable threshold, a quadratic penalty is applied to the excess portion.

[0101] A linear reward is given when the prediction error is less than or equal to the preset allowable threshold.

[0102] The instant reward is determined according to the following formula:

[0103]

[0104] Where R(t) is the instant reward, For the error amplitude reward item, The term represents the penalty for smoothing the change in error, and k_3 is the penalty weight. This is the smoothed value of the error variation.

[0105] For example,

[0106] In the formula: This is the tolerance threshold. As the weight, in this embodiment, The tolerance threshold, It is used to impose stronger penalties when there are large errors and to encourage finer predictions within small error ranges.

[0107] S40, by prioritizing the sampling of recent data in the experience replay buffer in the soft actor-critic SAC-ERE reinforcement learning algorithm that emphasizes recent experience replay to optimize the agent's prediction strategy, the agent obtained after learning the prediction model is used to predict the remaining life of the rolling bearing.

[0108] The core challenge of non-stationary time series data lies in the continuous shift in its data distribution over time, meaning that the evolutionary patterns reflected by early interaction data differ significantly from those of later data. Traditional uniform experience replay strategies treat all historical data equally during sampling, leading to frequent sampling of early, shifted data in later training stages. This severely interferes with the agent's adaptation to and learning of the new distribution, resulting in inaccurate predictions in later stages. Therefore, in this embodiment, during the agent's learning process, the sampling focus is gradually shifted to recent interaction data. This effectively overcomes the interference of the old data distribution shift on the new strategy, enabling the model to maintain good predictive stability and accuracy even in the later stages of time series evolution.

[0109] Specifically and optionally, to implement the aforementioned priority sampling, an adaptive shrinking sampling window strategy can be adopted, dynamically adjusting the sampling range according to the training progress: in the early stages of training, the sampling window is wider; as the training progresses, the sampling window adaptively shrinks, that is:

[0110] S41, calculate the sampling window size for experience replay based on the current training progress, and the sampling window size adaptively shrinks as the training progresses;

[0111] S42, within the sampling window, recent interaction data is sampled first for strategy optimization.

[0112] Specifically, the sampling window size for the k-th update is calculated using the following formula:

[0113]

[0114] in, Let N be the sampling window size at the k-th update, and N be the total capacity of the empirical replay buffer. The recent experience emphasis coefficient is represented by K, which is the total number of updates. Minimum sampling range;

[0115] It should be noted that, , The smaller the sample size, the faster the sampling range shrinks, and the higher the emphasis on recent data; when hour, This degenerates into uniform sampling of SAC; Minimum sampling range: This prevents the sampling range from being too small, which would lead to insufficient data diversity and cause overfitting.

[0116] To further balance the exploration efficiency in the early stages of training with the stability in the later stages, the sampling window size adaptively shrinks as the training progresses, and also includes:

[0117] The recent experience emphasis coefficient Using a linear annealing strategy, the recent empirical emphasis coefficient for the current training time step is calculated according to the following formula:

[0118]

[0119] in, The recent experience emphasis coefficient for the current training time step t. The initial recent experience emphasis coefficient, The final recent experience emphasis coefficient is given, and T is the total training time step.

[0120] In the technical solution provided in this embodiment, on the one hand, time series prediction is modeled as a sequential decision-making process. By constructing a composite state space containing a sequence of historical predicted values, the model considers the continuity of historical trajectories at each decision step, thereby avoiding sudden changes in results due to noise interference when the deep learning model makes single-point predictions. On the other hand, a penalty term based on error change smoothing is introduced. By smoothing the instantaneous change in prediction error to extract the true oscillation trend, a penalty is applied accordingly, thereby constraining the smoothness of the prediction curve and suppressing the "sawtooth" violent fluctuations common in non-stationary time series prediction. Furthermore, considering the inherent distribution offset characteristics of non-stationary time series data, an empirical replay strategy with an adaptive shrinking sampling window is adopted, allowing the model to prioritize sampling recent interaction data to adapt to the latest data distribution and alleviate the interference of old offset data on the current strategy.

[0121] Second Embodiment

[0122] Based on the first embodiment, this embodiment provides a specific implementation method for policy decision-making based on the composite state vector, outputting the degenerate step size adjustment value in the continuous action space. Specifically, this embodiment uses the soft actor-critic (SAC-ERE) algorithm, which emphasizes the replay of recent experiences, for policy decision learning.

[0123] First, the Actor Network is based on the current state. By using reparameterization techniques in a limited range Internal output optimal degenerate step size This action is essentially a prediction of the previous time step. The adjustment amount is determined by:

[0124]

[0125] This iterative logic, at the initial moment This enables a dynamic approximation of the remaining lifetime. This incremental prediction method ensures the physical continuity and rationality of the prediction curve.

[0126] The core objective of the Soft Actor-Critic (SAC) network construction is maximum entropy reinforcement learning. The Soft Actor-Critic (SAC) algorithm maximizes both the cumulative reward and the policy entropy (thus encouraging agent exploration), calculated using the following formula:

[0127]

[0128] In the formula, This represents the current policy, i.e., the given state. Action distribution during time ; It is a strategy The induced trajectory distribution corresponds to the state-action marginal distribution, which describes the agent's access to the environment. The probability of being correct; It is a state Next action The instant reward received; For the strategy in the state Entropy measures the randomness of action choices; the higher the entropy, the stronger the exploration ability. The temperature parameter is used to balance the weights of reward maximization and entropy maximization.

[0129] In practical applications, SAC achieves soft policy iteration through the alternating training of three core neural networks (function approximators):

[0130] (1) State-value network The optimization objective is to fit the soft value of the current state (i.e., Q-value minus policy entropy), and the loss function is calculated as follows:

[0131]

[0132] (2) Soft Q network (also known as critic network) The optimization objective is to evaluate the value of the current "state-action pair" (minimizing the soft Bellman residual). To mitigate the overestimation problem, two independent Q-networks are typically trained, and the minimum of their outputs is taken during computation. The loss function is calculated as follows:

[0133]

[0134] In the formula, This is a discount factor used to weigh immediate rewards against future rewards; The state transition probability of the environment; It is the output of the target value network, used to stabilize Q network updates.

[0135] (3) Strategy Network (also known as Actor Network) The optimization objective is to optimize the probability distribution of the output action. This is achieved through reparameterization techniques. (Introducing noise) To make the gradient transitive, we can directly minimize the expected KL divergence. The loss function is calculated as follows:

[0136]

[0137] In the formula, This is an experience replay buffer that stores trajectory data generated by the agent's interaction with the environment. ; It is based on a reparameterized skill action generation function. To fix the noise, this technique transforms the action sampling process into a differentiable operation, which facilitates gradient descent optimization. The state-action value output by the Q network guides the direction of policy updates.

[0138] Third Embodiment

[0139] Based on any of the above embodiments, in this embodiment, to avoid invalid predictions during the healthy phase, it is necessary to accurately locate the degradation initiation point. This embodiment provides an adaptive identification of the degradation initiation point (FPT) based on kurtosis. Before executing S10, the specific steps include:

[0140] S50, calculate the mean and standard deviation of the kurtosis of the first M samples of rolling bearing vibration samples in the healthy stage;

[0141] S60, determine the judgment threshold based on the weighted sum of the mean kurtosis and the standard deviation;

[0142] S70, when the signal kurtosis value of n consecutive sampling times exceeds the determination threshold, the first time among the n times is determined as the degradation start point;

[0143] S80, the time series data collected before the degradation start point is used as healthy time series data, and the time series data collected after the degradation start point is used as lifetime decay time series data.

[0144] In some alternative implementations, n is 3 and M is greater than n.

[0145] Further and optionally, based on the identified degradation initiation point Construct normalized RUL tags The calculation formula is as follows:

[0146]

[0147] In the formula: The moment when the rolling bearing completely fails. At the current sampling time, To pass the determined FPT time.

[0148] Verification Implementation Examples

[0149] To verify the effectiveness of the method of this invention, experiments were conducted using the publicly available XJTU-SY rolling bearing dataset. Rolling bearing data under operating condition 2 (speed 37.5 Hz, radial load 11 kN) was selected. In each experiment, the full life cycle data of one rolling bearing under the same operating condition was selected as the test set, and the remaining rolling bearing data were used as the training set. This process was repeated until all rolling bearings were tested.

[0150] To quantitatively evaluate the performance and robustness of the proposed MSRAE-SAC-ERE framework in predicting the remaining service (RUL) of rolling bearings, this embodiment employs widely used statistical metrics as evaluation standards: root mean square error (RMSE) and coefficient of determination (COP). ).

[0151] RMSE and The calculation formula is as follows:

[0152]

[0153]

[0154] In the formula: This represents the total number of test samples. Representing the Normalized true RUL value at each time step The predicted RUL value generated for the model, and This represents the average of the true values.

[0155] The method of this application (MSRAE-SAC-ERE) is compared with existing mainstream methods, and the results are shown in Table 1 below:

[0156] Table 1. Comparison of experimental results for rolling bearing datasets

[0157]

[0158] Experiments were conducted on the Xi'an Jiaotong University XJTU-SY rolling bearing dataset, and referenced... Figure 3 The diagram showing the comparison between the life prediction results and the actual life curves of the method proposed in this application can be seen to show that the two curves are quite similar and match well at the end of the rolling bearing's life.

[0159] As one implementation scheme, Figure 4This is a schematic diagram of the hardware operating environment of the computer system involved in the embodiments of this application.

[0160] like Figure 4 As shown, the computer system may include: a processor 1001, such as a CPU; a memory 1005; a user interface 1003; a network interface 1004; and a communication bus 1002. The communication bus 1002 is used to enable communication between these components. The user interface 1003 may include a display screen or an input unit such as a keyboard; optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface). The memory 1005 may be high-speed RAM or non-volatile memory, such as a disk drive. Optionally, the memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0161] Those skilled in the art will understand that Figure 4 The computer system architecture shown does not constitute a limitation on the computer system and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0162] like Figure 4 As shown, the memory 1005, as a storage medium, may include an operating system, a network communication module, a user interface module, and computer programs. The operating system is a program that manages and controls the hardware and software resources of the computer system, as well as the operation of the computer programs and other software or programs.

[0163] exist Figure 4 In the computer system shown, the user interface 1003 is mainly used to connect to the terminal and communicate with the terminal; the network interface 1004 is mainly used to communicate with the backend server; and the processor 1001 can be used to call the computer program stored in the memory 1005.

[0164] In this embodiment, the computer system includes: a memory 1005, a processor 1001, and a computer program stored in the memory and executable on the processor, wherein:

[0165] When processor 1001 calls a computer program stored in memory 1005, it performs the following operations:

[0166] S10, by constructing a multi-scale residual autoencoder (MSRAE), the potential health feature vector at the current moment is obtained from the time series data of the rolling bearing vibration sample, and the potential health feature vector is fused with the remaining life prediction value sequence at historical moments to construct a composite state vector containing historical prediction feedback.

[0167] S20, based on the composite state vector, a policy decision is made through a deep reinforcement learning prediction model based on soft actor-critic SAC-ERE that emphasizes the replay of recent experience, and the degenerate step size adjustment value in the continuous action space is output.

[0168] S30, determine the remaining lifetime prediction value at the current moment based on the change between the degradation step size adjustment value and the historical prediction result value, determine the immediate reward function containing the error change smoothing penalty term according to the smooth error gradient between the remaining lifetime prediction value and the true value, and store the interactive data generated during the training process under the constraint of the immediate reward function into the experience playback buffer.

[0169] S40, by prioritizing the sampling of recent data in the experience replay buffer in the soft actor-critic SAC-ERE reinforcement learning algorithm that emphasizes recent experience replay to optimize the agent's prediction strategy, the agent obtained after learning the prediction model is used to predict the remaining life of the rolling bearing.

[0170] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0171] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A method for predicting the remaining life of rolling bearings using deep reinforcement learning, integrating MSRAE and SAC-ERE, characterized in that, The method includes the following steps: S10, by constructing a multi-scale residual autoencoder (MSRAE), the potential health feature vector at the current moment is obtained from the time series data of the rolling bearing vibration sample, and the potential health feature vector is fused with the remaining life prediction value sequence at historical moments to construct a composite state vector containing historical prediction feedback. S20, based on the composite state vector, a policy decision is made through a deep reinforcement learning prediction model based on soft actor-critic SAC-ERE that emphasizes the replay of recent experience, and the degenerate step size adjustment value in the continuous action space is output. S30, determine the remaining lifetime prediction value at the current moment based on the change between the degradation step size adjustment value and the historical prediction result value, determine the immediate reward function containing the error change smoothing penalty term according to the smooth error gradient between the remaining lifetime prediction value and the true value, and store the interactive data generated during the training process under the constraint of the immediate reward function into the experience playback buffer. S40, by prioritizing the sampling of recent data in the experience replay buffer in the soft actor-critic SAC-ERE reinforcement learning algorithm that emphasizes recent experience replay to optimize the agent's prediction strategy, the agent obtained after learning the prediction model is used to predict the remaining life of the rolling bearing.

2. The rolling bearing deep reinforcement learning remaining life prediction method integrating MSRAE and SAC-ERE as described in claim 1, characterized in that, Prior to S10, it also included: Calculate the mean and standard deviation of the kurtosis of the first M samples of rolling bearing vibration samples in the healthy stage; The judgment threshold is determined based on the weighted sum of the mean kurtosis and the standard deviation. When the signal kurtosis value of n consecutive sampling times exceeds the determination threshold, the first time among the n times is determined as the degradation start point. The time-series data collected before the degradation initiation point is used as healthy time-series data, and the time-series data collected after the degradation initiation point is used as lifetime decay time-series data.

3. The rolling bearing deep reinforcement learning remaining life prediction method integrating MSRAE and SAC-ERE as described in claim 1, characterized in that, Prior to S10, it also included: A multi-scale residual autoencoder (MSRAE) is constructed, specifically as follows: the encoder consists of multiple convolutional layers, residual blocks, dilated convolutions, SE modules, and temporal attention pooling. For each vibration segment, the encoder outputs a low-dimensional latent feature vector. The decoder reconstructs waveforms starting from a fixed-length time grid. During the training phase, it is optimized by calculating a weighted sum of the time-domain mean square error and the logarithmic amplitude spectrum loss. During the prediction phase, the autoencoder parameters are fixed, and only the encoder is used for feature extraction.

4. The rolling bearing deep reinforcement learning remaining life prediction method integrating MSRAE and SAC-ERE as described in claim 1, characterized in that, In step S30, based on the smoothed error gradient between the predicted and actual remaining lifetime values, an immediate reward function including a smoothing penalty term for error change is determined, specifically including: The absolute difference between the predicted remaining lifetime and the actual lifetime is calculated as the prediction error at the current moment; Calculate the instantaneous change in prediction error relative to the prediction error at the previous moment; The instantaneous change is smoothed to obtain a smoothed value of the error change. The error change smoothing penalty term is generated based on the smoothed value of the error change, and the immediate reward is determined by combining it with the error amplitude reward term.

5. The rolling bearing deep reinforcement learning remaining life prediction method integrating MSRAE and SAC-ERE as described in claim 3, characterized in that, In step S30, the expression for calculating the smoothed value of the error change is: ； In the formula, This is the smoothing coefficient.

6. The rolling bearing deep reinforcement learning remaining life prediction method integrating MSRAE and SAC-ERE as described in claim 4, characterized in that, Based on the smoothed value of the error change, an error change smoothing penalty term is generated, and combined with the error amplitude reward term, the immediate reward is determined, specifically including: When the prediction error exceeds the preset allowable threshold, a quadratic penalty is applied to the excess portion. A linear reward is given when the prediction error is less than or equal to the preset allowable threshold. The instant reward is determined according to the following formula: ； Where R(t) is the instant reward, For the error amplitude reward item, The term represents the penalty for smoothing the change in error, and k_3 is the penalty weight. This is the smoothed value of the error variation.

7. The rolling bearing deep reinforcement learning remaining life prediction method integrating MSRAE and SAC-ERE as described in claim 1, characterized in that, S40 includes: The sampling window size for experience replay is calculated based on the current training progress, and the sampling window size adaptively shrinks as the training progresses. Within the sampling window, recent interaction data is sampled first for strategy optimization.

8. The rolling bearing deep reinforcement learning remaining life prediction method integrating MSRAE and SAC-ERE according to claim 6, characterized in that, The sampling window size for the k-th update is calculated using the following formula: ； in, Let N be the sampling window size at the k-th update, and N be the total capacity of the empirical replay buffer. The recent experience emphasis coefficient is represented by K, which is the total number of updates. Minimum sampling range; The sampling window size adaptively shrinks as the training progresses, and also includes: The recent experience emphasis coefficient Using a linear annealing strategy, the recent empirical emphasis coefficient for the current training time step is calculated according to the following formula: ； in, The recent experience emphasis coefficient for the current training time step t. The initial recent experience emphasis coefficient, The final recent experience emphasis coefficient is given, and T is the total training time step.

9. The rolling bearing deep reinforcement learning remaining life prediction method integrating MSRAE and SAC-ERE according to claim 1, characterized in that, S20 includes: The predicted value adjustment amount at the current moment is output based on the composite state vector; Based on the predicted value from the previous time step and the adjustment amount of the predicted value, the degradation step size adjustment value for the current time step is determined.

10. A computer system, characterized in that, The computer system includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the computer program is executed by the processor, it implements the steps of the rolling bearing deep reinforcement learning remaining lifetime prediction method that integrates MSRAE and SAC-ERE as described in any one of claims 1 to 9.

Images