A dynamic reinforcement learning gear fault diagnosis method and system and a medium

Abstract

The application discloses a kind of dynamic reinforcement learning gear fault diagnosis method, system and medium, relating to mechanical equipment fault diagnosis and intelligent operation and maintenance technical field, input after pre-processing gear vibration signal into the dynamic reinforcement learning model that has been trained, to generate gearbox fault diagnosis category;The training process of the dynamic reinforcement learning model is as follows: constructing training set;Interactive data is generated using strategy network processing training set and is stored in hierarchical experience replay pool;Combining state uncertainty measure and sample rarity weight factor constructs uncertainty perception Bellman equation;Adopt double deep Q network architecture, strategy network receives frequency domain feature vector and outputs current value distribution and optimal action, and optimal action is used as fault diagnosis category;Target network calculates target value distribution based on the uncertainty perception Bellman equation, and gradually optimizes parameters using soft update method;The method improves the value distribution accuracy and diagnostic accuracy.

Description

Technical Field

[0001] This invention relates to the field of mechanical equipment fault diagnosis and intelligent operation and maintenance technology, and in particular to a gear fault diagnosis method, system and medium based on dynamic reinforcement learning. Background Technology

[0002] Gearboxes are core power components of industrial transmission systems, widely used in critical fields such as wind power, metallurgy, chemical industry, and rail transportation. Their operating status directly affects the safety, reliability, and life-cycle maintenance costs of equipment. During gear operation, vibration signals are easily affected by load fluctuations, speed, and noise interference. Furthermore, the probability of early minor faults is low, and the acquisition cost is high, resulting in actual diagnostic data exhibiting characteristics such as small sample size, class imbalance, and strong noise.

[0003] To address issues such as insufficient sample size, class imbalance, and strong noise, data augmentation methods such as synthetic minority class oversampling can balance the number of samples in each class. However, there are significant differences in the feature complexity and noise interference levels of vibration signals in different fault categories, leading to an imbalance in feature information.

[0004] In existing intelligent gearbox diagnostic methods based on reinforcement learning, the traditional Categorical DQN model improves diagnostic accuracy by estimating the value distribution. However, when directly applied to imbalanced scenarios with small samples, it suffers from two limitations: First, the value support point setting lacks adaptability. The Categorical DQN model uses a fixed number and range of support points, which is difficult to adapt to the sparsity characteristics of small sample data, easily leading to excessive diagnostic errors or overfitting risks. Second, the value update of the traditional reinforcement learning model does not consider state uncertainty, and the traditional Bellman equation does not adapt to the strong noise characteristics of gear signals, resulting in value distribution deviation and affecting diagnostic accuracy. Summary of the Invention

[0005] Based on the technical problems existing in the background technology, this invention proposes a gear fault diagnosis method, system and medium using dynamic reinforcement learning, which improves the accuracy of value distribution and diagnostic precision.

[0006] The present invention proposes a gear fault diagnosis method based on dynamic reinforcement learning, which inputs the preprocessed gear vibration signal into a trained dynamic reinforcement learning model to generate a gearbox fault diagnosis category.

[0007] The training process of the dynamic reinforcement learning model is as follows:

[0008] Gear vibration signals from various operating states are acquired and preprocessed to generate frequency domain feature vectors to construct a training set;

[0009] The training set is processed using a policy network to generate interaction data and stored in a hierarchical experience replay pool; based on the degree of dispersion of the value distribution corresponding to the interaction data in the hierarchical experience replay pool, an adaptive set of value support points is generated.

[0010] The uncertainty measure of the state is calculated based on the value distribution represented by the adaptive support point set output by the policy network, and the uncertainty-aware Bellman equation is constructed by combining the preset sample sparseness weight factor.

[0011] A dual-deep Q-network architecture is adopted. The policy network receives the frequency domain feature vector and outputs the current value distribution and the optimal action, with the optimal action serving as the fault diagnosis category. The target network calculates the target value distribution based on the uncertainty-aware Bellman equation and uses a soft update method to progressively optimize the parameters.

[0012] Based on the current value distribution, the loss function and Shannon entropy are calculated synchronously, and the learning rate and model parameters are dynamically adjusted accordingly.

[0013] Furthermore, the generation process of the frequency domain feature vector is as follows:

[0014] The gear vibration signal is segmented into non-overlapping segments according to a set length to obtain vibration fragments;

[0015] The frequency domain features of each vibration segment are extracted by performing a fast Fourier transform and then standardized to obtain the frequency domain feature vector.

[0016] Furthermore, the hierarchical experience replay pool is divided into sub-pools according to fault categories, and the number of sub-pools corresponds to the number of fault categories. During sampling, the sampling weight is allocated inversely proportional to the proportion of interactive data in each type of sub-pool.

[0017] The interactive data includes the current state, actions representing the gear's operating state, immediate rewards corresponding to gearbox fault diagnosis categories, the next state, and termination state markers.

[0018] Furthermore, the process of generating an adaptive set of value support points based on the dispersion of the value distribution corresponding to the interaction data in the hierarchical experience replay pool is specifically as follows:

[0019] (1) When the amount of data in the hierarchical experience replay pool reaches the set lower limit capacity threshold, extract the value distribution V of all interaction data in the hierarchical experience replay pool and calculate the variance. ;

[0020] (2) If Then, the K-means clustering algorithm is used to cluster the value distribution V, and the silhouette coefficient of the value distribution V corresponding to each interaction data is calculated to evaluate the rationality of the clustering results, thereby generating each value support point. The preset variance threshold is used;

[0021] (3) If Then, a linear interval is used to generate each value support point;

[0022] (4) Repeat steps (1) to (3) according to the preset cycle to dynamically update the values ​​of each value support point and generate an adaptive value support point set accordingly.

[0023] Furthermore, the value distribution of the interactive data The calculation formula is as follows:

[0024] ;

[0025] in, The current state The corresponding value distribution results, As a discount factor, Indicates the next state Follows the state transition probability distribution Expectation calculation, For policy network prediction Expected value for The corresponding current optimal action.

[0026] Furthermore, the uncertainty measure of the state is calculated based on the value distribution represented by the adaptive support point set output by the policy network, and an uncertainty-aware Bellman equation is constructed by combining a preset sample sparseness weight factor, specifically:

[0027] Calculate the next state The uncertainty measure, the state Corresponding to the frequency domain eigenvectors;

[0028] For the value distribution represented by the adaptive support point set output by the policy network, the logarithm of the probability value corresponding to each value support point is added to the zero-prevention factor, and then multiplied by the corresponding probability value. The sum of all the multiplication results is then negative to obtain the Shannon entropy value.

[0029] Based on the Shannon entropy value and the preset threshold range, the state uncertainty is classified and determined, and a state uncertainty metric is generated accordingly.

[0030] The uncertainty-perceived Bellman equation is as follows:

[0031] ;

[0032] in, This represents the discrete probabilistic target value distribution that matches the target network output with the adaptive value support point. It is the immediate reward value corresponding to the current fault. As a discount factor, It is a dynamic control function based on the level of state uncertainty, triggered by the state uncertainty measure. It is the Shannon entropy value of the value distribution corresponding to the next state. This is a state-action weight adjustment function based on a sample rarity weight factor. Fault category, The optimal action for the next state. For optimal action The expected distribution of the corresponding next-state value distribution. For the target network based on parameters The target value distribution is calculated. For policy networks based on parameters The defined strategy is used to select the optimal action in the current state. for The state at any given moment, for The optimal action to be performed at any given time is the fault category output by the policy network.

[0033] Furthermore, the step of simultaneously calculating the loss function and Shannon entropy based on the current value distribution, and dynamically adjusting the model's learning rate and model parameters accordingly, specifically involves:

[0034] Using the weighted Wasserstein distance as the loss function, the loss value between the current value distribution and the target value distribution output by the target network is calculated, and the model parameters are adjusted accordingly.

[0035] The learning rate of the model is dynamically adjusted based on the rate of change of Shannon entropy, combined with the Warmup and cosine annealing mechanisms.

[0036] Furthermore, the step of dynamically adjusting the model's learning rate based on the rate of change of Shannon entropy, combined with the Warmup and cosine annealing mechanisms, specifically involves:

[0037] Calculate the rate of change of Shannon entropy :

[0038] Set the number of warm-up steps, and during the warm-up phase, the learning rate increases linearly from the initial value to the target value.

[0039] After preheating, the learning rate of the model is dynamically adjusted using the Shannon entropy change rate. .

[0040] A computer system includes a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the method described above.

[0041] A computer-readable storage medium storing a plurality of classification programs, the plurality of classification programs being invoked by a processor to execute the method described above.

[0042] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media that can store program code, such as ROM, RAM, magnetic disk, or optical disk.

[0043] The advantages of the dynamic reinforcement learning-based gear fault diagnosis method, system, and medium provided by this invention are as follows: It solves the problem of poor adaptability of fixed value support points by dynamically updating the value support point values ​​through K-means clustering; it constructs an uncertainty-aware Bellman equation that integrates state uncertainty measurement and sample sparseness weights to improve the noise resistance and balance of value updates; it dynamically adjusts the learning rate based on Shannon entropy values ​​of value distribution combined with preheating and cosine annealing mechanisms to optimize training efficiency in small sample scenarios; and it adopts weighted Wasserstein distance loss and dual-depth Q-networks to balance the adaptability and generalization ability of imbalanced data, providing a novel integrated technical solution for small-sample imbalanced gearbox fault diagnosis. Attached Figure Description

[0044] Figure 1 This is a schematic diagram of the process of the present invention;

[0045] Figure 2a A schematic diagram of gear vibration signals in a healthy gear state;

[0046] Figure 2b A schematic diagram of gear vibration signals due to gear wear;

[0047] Figure 2c A schematic diagram of the vibration signal of a gear with a broken tooth;

[0048] Figure 2d A schematic diagram of gear vibration signals for pitting corrosion.

[0049] Figure 2e A schematic diagram of the vibration signal of a gear with a missing tooth;

[0050] Figure 2f This is a schematic diagram of gear vibration signals caused by gear tooth cracks.

[0051] Figure 2g A schematic diagram of gear vibration signals when the inner ring gear breaks.

[0052] Figure 2h This is a schematic diagram of the vibration signal of a broken outer ring gear.

[0053] Figure 3 The FFT (Fast Fourier Transform) frequency domain amplitude heatmap of the gear fault samples is shown in this embodiment. The samples recorded in this embodiment correspond to the frequency domain feature vectors.

[0054] Figure 4 A heatmap of the confusion matrix for fault diagnosis categories in the model gearbox;

[0055] Figure 5 A frontal view of the cluster diagram for gearbox health status diagnosis;

[0056] Figure 6 This is a reverse diagram of the clustering graph used for gearbox health status diagnosis. Detailed Implementation

[0057] The technical solution of the present invention will now be described in detail through specific embodiments. Many specific details are set forth in the following description to provide a thorough understanding of the invention. However, the present invention can be implemented in many other ways different from those described herein, and those skilled in the art can make similar modifications without departing from the spirit of the invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0058] like Figures 1 to 6 As shown, the present invention proposes a gear fault diagnosis method based on dynamic reinforcement learning, which inputs the preprocessed gear vibration signal into a trained dynamic reinforcement learning model to generate a gearbox fault diagnosis category.

[0059] The training process of the dynamic reinforcement learning model is as follows:

[0060] Step 1: Acquire gear vibration signals from various operating states and preprocess them to generate frequency domain feature vectors to construct a training set;

[0061] Step 2: Use a policy network to process the training set to generate interaction data and store it in a hierarchical experience replay pool; based on the dispersion of the value distribution corresponding to the interaction data in the hierarchical experience replay pool, generate an adaptive set of value support points.

[0062] Step 3: Calculate the state uncertainty measure based on the value distribution represented by the adaptive support point set output by the policy network, and construct the uncertainty-aware Bellman equation by combining the preset sample sparseness weight factor.

[0063] Step 4: A dual-deep Q-network architecture is adopted. The policy network receives the frequency domain feature vector and outputs the current value distribution and the optimal action. The optimal action is used as the fault diagnosis category. The target network calculates the target value distribution based on the uncertainty-aware Bellman equation and uses a soft update method to progressively optimize the parameters.

[0064] Step 5: Based on the current value distribution, simultaneously calculate the loss function and Shannon entropy value, and dynamically adjust the model's learning rate and model parameters accordingly.

[0065] This embodiment addresses the poor adaptability of fixed value support points by dynamically updating their values ​​using K-means clustering. It constructs an uncertainty-aware Bellman equation that integrates state uncertainty metrics and sample sparseness weights to improve the noise resistance and balance of value updates. Based on the Shannon entropy value of the value distribution, it dynamically adjusts the learning rate using preheating and cosine annealing mechanisms to optimize training efficiency in small-sample scenarios. Finally, it employs a weighted Wasserstein distance loss and a dual-depth Q-network to balance the adaptability and generalization capabilities of imbalanced data, providing a novel integrated technical solution for fault diagnosis of imbalanced gearboxes with small samples.

[0066] In one embodiment, step one, acquiring gear vibration signals from various operating states and preprocessing them to generate frequency domain feature vectors to construct a training set, specifically involves:

[0067] The training and performance verification of the fault diagnosis model in this embodiment can be carried out using the multi-mode fault dataset of gearbox under variable working conditions jointly developed by Tsinghua University and MCC5 Group (original data set: Chen, Shijin; Liu, Zeyi; He, Xiao; Zou, Dongliang; Zhou, Donghua (2024), “Multi-mode FaultDiagnosis Datasets of Gearbox Under Variable Working Conditions”, MendeleyData, V1, doi: 10.17632 / p92gj2732w.1).

[0068] Extract the vibration signals of the three core gears along the x / y / z axes of the gearbox from the fault dataset, or acquire them using a triaxial vibration accelerometer to obtain gear vibration signals for eight operating states: gear health, gear pitting, gear wear, broken teeth, missing teeth, cracked teeth, inner gear fracture, and outer gear fracture. The criteria were used to remove data points that deviated from the mean by three times the standard deviation to avoid interference from outliers, and a gearbox vibration dataset was constructed.

[0069] Then, for the gearbox vibration dataset, the gear vibration signal is segmented without overlap according to a fixed length of 1024 (configurable) to obtain vibration segments; each vibration segment has a duration of 80ms, calculated based on the typical meshing frequency of the gearbox. This duration can completely cover at least 43 gear meshing cycles, avoiding feature redundancy caused by excessively long segments, ensuring that fault features are fully captured, and avoiding feature loss due to the sampling vibration segments not covering the complete fault cycle. Figure 3 It is a visual representation of frequency domain feature vectors, which experimentally verifies the rationality of parameters such as "segmentation length, duration, and frequency domain truncation" in the data preprocessing stage, providing solid feature support for the high-accuracy diagnosis of subsequent models.

[0070] Perform a Fast Fourier Transform (FFT) on each vibrating element, with N=1024 points (adjustable), to extract frequency domain features. Take the feature amplitude and truncate the 0-5000Hz (adjustable) frequency band, retaining 256 (adjustable) frequency domain points to obtain 256-dimensional single-channel frequency domain features. Perform StandardScaler processing on the single-channel frequency domain features, i.e.:

[0071] ;

[0072] in, These are the original values ​​of the single-channel frequency domain characteristics; These are the mean and standard deviation, respectively. This represents the standardized single-channel frequency domain characteristics.

[0073] By concatenating the standardized single-channel frequency domain features of the three axes (256 dimensions), a 768-dimensional frequency domain feature vector is obtained. , where settings These are the single-channel frequency domain characteristics of the triaxial sensor.

[0074] The 768-dimensional frequency domain feature vector is divided into training, validation, and test sets in a 7:2:1 ratio for the training, validation, and testing of the dynamic reinforcement learning model. This ensures that the sample distribution of the eight operating states within each dataset is consistent, avoiding model bias caused by data imbalance.

[0075] In one embodiment, step two involves using a policy network to process the training set to generate interaction data and storing it in a hierarchical experience replay pool; based on the dispersion of the value distribution corresponding to the interaction data in the hierarchical experience replay pool, an adaptive set of value support points is generated, specifically:

[0076] Initialize the hierarchical experience replay pool D. The hierarchical experience replay pool is divided into sub-pools according to fault categories, with the number of sub-pools corresponding to the number of fault categories (e.g., setting 8 fault categories corresponds to setting 8 sub-pools). During sampling, sampling weights are allocated inversely proportional to the proportion of interaction data in each sub-pool to ensure that the sampling probability of rare fault samples is not less than 30%, avoiding training data bias. A hierarchical experience replay pool of preset capacity is set to store historical interaction data. It contains State vector at time step Characterizing the operating state of gears Moment of action Gearbox fault diagnosis categories correspond to Instant rewards for moments State vector at time t+1 and termination status marker The tiered experience replay pool adopts a first-in, first-out (FIFO) strategy. When the amount of stored data exceeds the preset capacity, the earliest stored data is automatically deleted.

[0077] It should be noted that the policy network is used to process the training set to generate interaction data and store it in a hierarchical experience replay pool, specifically as follows:

[0078] Prepare the training set samples (frequency domain feature vectors + true fault category labels); traverse the samples in the training set through the initialized policy network, with the frequency domain feature vector serving as the state for each sample. The actual fault category label is used as the basis for calculating the reward; the policy network is used to analyze the state. Make predictions and obtain actions. (i.e., a fault category); comparison actions Calculate instant rewards based on actual fault category labels. Obtain the frequency domain feature vector of the next sample in the dataset as the state. ; construct the interactive data Based on their actual fault category labels, the data is stored in the sub-pool corresponding to the hierarchical experience replay pool. When the amount of data in the hierarchical experience replay pool reaches a certain level (e.g., 10,000 experiences), a batch of interaction data is sampled from the hierarchical experience replay pool, input into the policy network for forward propagation, and the value distribution V corresponding to this batch of interaction data is output for subsequent generation of value support points.

[0079] It should be noted that the state in this embodiment... It corresponds to the frequency domain feature vector, that is, each state corresponds to a frequency domain feature vector; each frequency domain feature vector is extracted from the vibration signal of a certain gear running state, and each state is associated with a gear running state (i.e., one of the eight categories such as health, pitting, wear, etc.) through the corresponding frequency domain feature vector.

[0080] (1) When the amount of data in the hierarchical experience replay pool reaches the set lower limit capacity threshold, extract the value distribution V of all interaction data in the hierarchical experience replay pool and calculate the variance. .

[0081] Specifically, in this embodiment, when the data volume of the hierarchical experience replay pool D reaches 20% of its capacity (configurable), the value distribution results of all interactive data in the hierarchical experience replay pool are extracted, and the value distribution formula is:

[0082] ;

[0083] in, The current state The corresponding value distribution results, As a discount factor, Indicates the next state Follows the state transition probability distribution Expectation calculation, For policy network prediction Expected value for The corresponding current optimal action.

[0084] Calculate the variance of the value distribution V This is used to quantify the dispersion of the value distribution results.

[0085] (2) If Then, the K-means clustering algorithm is used to cluster the value distribution V, and the silhouette coefficient of the value distribution V corresponding to each interaction data is calculated to evaluate the rationality of the clustering results, thereby generating each value support point. For example, to preset the variance threshold. ;

[0086] In clustering the value distribution V using the K-means clustering algorithm, the number of clusters k is first set to a range of [50, 150], covering a fixed number of value support points of 101 (optional). For each cluster size k, clustering is repeated 5 to 10 times (optional), with cluster centers randomly initialized each time. Then, the silhouette coefficient of the value distribution V corresponding to each interactive data point is calculated. The silhouette coefficient formula is: , The silhouette coefficient is used to evaluate the reasonableness of clustering results for interactive data. The average Euclidean distance between the interacting data and other interacting data in the same cluster. The average Euclidean distance between the interactive data and the nearest heterogeneous interactive data is given.

[0087] (3) If Then, linear intervals are used to generate each value support point. ,in, It is the first The generated value support point value. For the minimum and maximum values ​​of the value distribution results, A fixed number of value support points.

[0088] (4) Repeat steps (1) to (3) according to the preset cycle to dynamically update the values ​​of each value support point and generate an adaptive value support point set accordingly.

[0089] Finally, set the training cycle, for example, 50 cycles in the early stage of training and 200 cycles in the later stage. Re-execute the value distribution, variance calculation, and value support point generation process, dynamically updating the numerical range of the value support points. The update formula is: ,in, It is the current set of value supports. It is an updated set of value support points. This is the fault tolerance coefficient.

[0090] Using steps (1) to (4) has the following advantages:

[0091] To ensure a reliable data foundation, a lower limit capacity threshold of 20% (configurable) of the stratified experience replay pool is set to ensure that the value distribution and variance calculation are supported by sufficient samples, which can reduce the bias caused by early sparse data.

[0092] When the variance of the value distribution V When the data distribution is large, i.e., discrete, K-means clustering plus silhouette coefficient is used to select the optimal cluster, generating support points that fit the data distribution and adapt to the sparsity characteristics of small samples; when the variance of the value distribution V is large... When the data is small and concentrated, linear intervals are used to generate support points, balancing efficiency and accuracy, and the value support points have strong adaptability. The silhouette coefficient quantifies the clustering effect, which can avoid the support point deviation caused by invalid clustering and improve the accuracy of value distribution representation.

[0093] The support points are dynamically updated according to a preset cycle, always keeping pace with the dynamic characteristics of data distribution. This solves the problem of poor adaptability of traditional fixed support points and enhances the generalization ability of the model.

[0094] In one embodiment, step three involves calculating a state uncertainty measure based on the value distribution represented by the adaptive support point set output by the policy network, and constructing an uncertainty-aware Bellman equation by combining a preset sample sparseness weight factor; specifically:

[0095] S31. Calculate the next state. The uncertainty measure is as follows: for the value distribution represented by the adaptive support point set output by the policy network, a zero-prevention factor is added to the probability value corresponding to each value support point, the logarithm is taken, and then multiplied by the corresponding probability value. The sum of all multiplication results is then negative to obtain the Shannon entropy value. The Shannon entropy value is used to quantify state uncertainty. The larger the value, the more ambiguous the value distribution and the higher the state uncertainty. Based on the Shannon entropy value and the preset threshold range, the state uncertainty is classified and judged to generate a state uncertainty measure.

[0096] S32. Calculate the state-action weight adjustment function based on the sample rarity weight factor. This function is used to dynamically adjust the value update weights based on the distribution characteristics of fault samples.

[0097] Set the weight adjustment coefficient =0.02 (can be set), dynamically calculate the sample rarity weighting factor based on the proportion of sample size. According to the formula:

[0098] ;

[0099] in, The total number of normal samples in the dataset. The fault category to which the current sample belongs The number of samples, This is the error parameter.

[0100] Weighting factors for sample rarity Normalization yields the current fault category Sample proportion .

[0101] State-action weight adjustment function based on sample rarity weight factor:

[0102] ;

[0103] in, Current fault category Sample proportion This indicates a low-frequency fault. This indicates an intermediate frequency fault. This indicates a high-frequency fault.

[0104] S33. This embodiment improves the traditional Bellman equation and constructs an uncertainty-aware Bellman equation. Based on the Shannon entropy value of the value distribution in the next stage, the state uncertainty is divided into three levels: high, medium, and low, and the value propagation process is controlled in a hierarchical manner.

[0105] When in a state of high uncertainty, an inhibitory factor is introduced to reduce the propagation weight of the value of the next state; when in a state of medium uncertainty, the standard value update is maintained and the weight is balanced by combining the sample rarity weight factor; when in a state of low uncertainty, the value propagation weight corresponding to rare fault categories is enhanced.

[0106] Understandably, the inhibition factor is a preset control hyperparameter based on the level of state uncertainty, and its value range is determined by the characteristics of the gear fault diagnosis task. For example, in a high uncertainty state, the inhibition factor ranges from [0.3, 0.6], in a medium uncertainty state, the inhibition factor is 1.0, and in a low uncertainty state, the inhibition factor ranges from [1.1, 1.3]. The inhibition factor is embedded in the uncertainty-aware Bellman equation in the form of a multiplicative factor and directly affects the value propagation term. In a high uncertainty state, the inhibition factor is less than 1, which directly reduces the value of the multiplication formula, decreases the contribution of the next state value to the current target value, and avoids fuzzy value information from misleading model parameter updates.

[0107] An uncertainty-aware Bellman equation is constructed to calculate the value update result (i.e., the discrete probabilistic target value distribution matching the target network output and the adaptive value support point). For high uncertainty states caused by overlapping fault features and noise interference, the model's judgment of the current fault value is ambiguous, and the weight of the next state value is automatically reduced. A dedicated sample sparseness weight factor is configured for rare fault categories with a small number of samples to amplify their gradient contribution and avoid being dominated by high-frequency fault samples in the learning process.

[0108] The specific formula for the uncertainty-perceived Bellman equation is as follows:

[0109] ;

[0110] in, This represents the discrete probabilistic target value distribution that matches the target network output with the adaptive value support point. It is the immediate reward value corresponding to the current fault. As a discount factor, It is a dynamic control function based on the level of state uncertainty, triggered by the state uncertainty measure. It is the Shannon entropy value of the value distribution corresponding to the next state. This is a state-action weight adjustment function based on a sample rarity weight factor. Fault category, The optimal action for the next state. For optimal action The expected distribution of the corresponding next-state value distribution. For the target network based on parameters The target value distribution is calculated. For policy networks based on parameters The defined strategy is used to select the optimal action in the current state. for The state at any given moment, for The optimal action to be performed at any given time is the fault category output by the policy network.

[0111] For example, in the specific implementation process, Shannon entropy value >1.2, judged as a high uncertainty state, 0.6< ≤1.2, is classified as a state of moderate uncertainty. ≤0.6, judged as a low uncertainty state; dynamic control function In medium to high uncertainty states That is, introducing a suppression factor, under conditions of medium uncertainty. That is, maintaining standard value under low uncertainty. This means enhancing the value propagation weight corresponding to rare fault categories.

[0112] This embodiment constructs an uncertainty-aware Bellman equation that integrates value distribution and sample rarity weighting factors, overcoming the limitations of linear updates in traditional equations. It enhances the noise resistance of the dynamic reinforcement learning model through value distribution, using the set logarithmic weighting formula (…). This helps alleviate data imbalance and improve the recall rate for diagnosing rare faults.

[0113] In one embodiment, step four involves employing a dual-depth Q-network architecture. The policy network receives the frequency domain feature vector and outputs the current value distribution and optimal action. The target network calculates the target value distribution based on the uncertainty-aware Bellman equation and progressively optimizes the parameters using a soft update method. Specifically:

[0114] S41. Construct a dual-depth Q-network architecture. The policy network adopts a four-layer fully connected neural network: the input layer has a dimension of 768, and the hidden layers have dimensions of 256, 128, and 64 respectively, using the ReLU activation function. Each hidden layer is followed by a Dropout regularization of 0.2 to suppress overfitting. The output layer has a dimension of the number of fault categories × the number of value support points, and uses the Softmax activation function. The policy network parameters are denoted as follows. The target network structure is completely identical to the policy network, and the parameters are denoted as follows: ,and Initialize to .

[0115] S42, Policy Network Adopts - Greedy strategy for choosing actions , The initial value is 0.9 (configurable), and it decays exponentially to 0.05 (configurable). During training, exponential decay is used, and the decay formula is as follows:

[0116] ;

[0117] in, =0.9 is the initial exploration rate. =0.05 represents the final exploration rate. For the training period; generate random numbers between 0 and 1. When the random number < When the random number is greater than or equal to 1, a random action is selected; when the random number is greater than or equal to 1, the action is selected randomly. At that time, choose the action with the highest expected value distribution to balance exploration and utilization.

[0118] S43. Target network calculates target value distribution based on uncertainty-aware Bellman equation. And it is mapped to an adaptive set of value support points through a projection operation. superior, For the first There are three value support points, K=101 (which can be set), satisfying the following formula:

[0119] ;

[0120] in, The output of the target network The probability value of each target value support point For instant rewards, =0.99 is the discount factor. The target value is a variable representing the position of any target value support point in the adaptive support point set Z. The first in the adaptive support point set Z One value support point The interval between adjacent value support points is obtained by evenly dividing the numerical range of the adaptive value support point set. This projection operation ensures that the target value distribution matches the adaptive value support point set.

[0121] It should be noted that the current value distribution output by the policy network is defined by the adaptive support point set and its corresponding probability values, that is,

[0122] For the adaptive support point set For each value support point in the network, the policy network outputs a probability value, and the value distribution is defined as a discrete distribution with probability values ​​at the value support points.

[0123] S43. The soft update formula for the parameters of the target network is: ,in, These are the current parameters of the policy network. These are the parameters of the target network. =0.001~0.01 is the preset update coefficient.

[0124] In one embodiment, step five involves simultaneously calculating the loss function and Shannon entropy based on the current value distribution, and dynamically adjusting the model's learning rate and model parameters accordingly; specifically:

[0125] The current value distribution serves as both a parameter of the dynamic reinforcement learning model and a unified state representation index for the dynamic adjustment of the learning rate.

[0126] S51. Calculate the Shannon entropy value of the value distribution output by the policy network, and dynamically adjust the learning rate of the dynamic reinforcement learning model based on the rate of change of the Shannon entropy value, combined with the Warmup and cosine annealing mechanisms; including steps 511 to S513.

[0127] S511. Calculate the rate of change of Shannon's entropy. ;

[0128] ;

[0129] in, yes Shannon entropy of the value distribution at time intervals The initial Shannon entropy value is the average of the entropy values ​​from the first 10 (optional) training steps of the model. It is used to reflect the changing trend of the current Shannon entropy value relative to the initial Shannon entropy value.

[0130] S512, Set preheating steps =100~500 (adjustable), learning rate during the warm-up phase starts from the initial value. (Can be set) to grow linearly to the target value :

[0131] ;

[0132] in, Preheating stage Learning rate at any given moment This represents the current number of training steps.

[0133] S513. After preheating, dynamically adjust the model's learning rate using the Shannon entropy change rate. :

[0134] ;

[0135] in, This is the target learning rate value during the warm-up phase. =2000~5000 (can be set) is the annealing cycle. =0.5~1.0 (can be set) is the weighting coefficient for the rate of change of Shannon entropy. When the learning rate is >0, the rate of decrease in the learning rate accelerates, enhancing the stability of model exploration; when... When the learning rate is less than 0, the rate of decrease in the learning rate slows down, consolidating the model's learning results.

[0136] This embodiment uses the Shannon entropy change rate of the value distribution, combined with Warmup and cosine annealing to adjust the model's learning rate. In the early stages of training, when the Shannon entropy fluctuates greatly, the learning rate automatically increases linearly to accelerate exploration; in the later stages, when the Shannon entropy converges, the learning rate decreases gradually to stabilize the model. This strategy improves the model's convergence speed while reducing the fluctuation range of diagnostic accuracy, avoiding the shortcomings of insufficient exploration or convergence oscillations caused by a fixed learning rate.

[0137] S52. Using the weighted Wasserstein distance as the loss function, calculate the loss value between the current value distribution and the target value distribution output by the target network; iterate the training according to the convergence condition of no improvement for 10 consecutive (optional) rounds of validation set accuracy to obtain the trained dynamic reinforcement learning model based on the uncertainty-aware Bellman equation, specifically including S521 and S523.

[0138] S521. Construct the weighted Wasserstein distance loss function: , For the first The weights of each value support point are associated with the uncertainty perception and control results in steps S32 and S33, and the fault sample (which corresponds to a frequency domain feature vector) corresponds to... It is 1.5 to 5.0 times that of normal samples (which correspond to frequency domain feature vectors). The first output of the policy network The current value distribution of each value support point The output of the target network The distribution of target value for each value support point.

[0139] S522. The Adam optimizer is used to minimize the loss function. The optimizer parameter update formula is: , , , , ,in , For the first and second moments of the gradient, , This is the moment estimate after bias correction; The gradient of the loss function with respect to the policy network parameters θ; These are the default parameters for the Adam optimizer. The learning rate is dynamically adjusted in step 6. It is a smoothing term to avoid a denominator of 0.

[0140] S523. After each training cycle, evaluate the model's diagnostic accuracy using the validation set. When the diagnostic accuracy of the validation set does not improve for 10 (settable) consecutive training cycles, stop training and save the current policy network parameters as the completed dynamic reinforcement learning model based on the uncertainty-aware Bellman equation.

[0141] For example, the weight of the value support points for faulty samples is set to three times that of normal samples; the Adam optimizer is used. Weight decay coefficient When the hardware supports CUDA, FP16 mixed-precision training is used. The gradient scaling ratio is adjusted using GradScaler, and the scaling ratio is automatically reduced when gradient overflow occurs to ensure training stability. A weighted EMA mechanism is introduced, and an EMA update is performed once for each training batch. The batch size is 128, and the training stops early when the validation accuracy does not improve after 10 consecutive epochs, saving the optimal model weights. The training epochs are 3000, and the loss on the training set and the accuracy on the validation set are monitored in real time during the iteration process. The Shannon entropy change rate and learning rate adjustment curve are visualized.

[0142] This embodiment uses the weighted Wasserstein distance loss function instead of the traditional cross-entropy loss. By amplifying the weight of faulty samples, it alleviates the problem that the loss is dominated by normal samples under imbalanced data.

[0143] After obtaining the trained dynamic reinforcement learning model, the execution process of the dynamic reinforcement learning model in this embodiment is as follows:

[0144] The frequency domain feature vector obtained from the preprocessing of the gear vibration signal to be diagnosed in step one is input into the trained dynamic reinforcement learning model based on the uncertainty-aware Bellman equation. The model outputs value distributions corresponding to 8 (configurable) operating states, resulting in 8 probability values. A confidence threshold of 0.85 is introduced. If the maximum probability value is lower than the preset threshold, it is marked as a low-confidence result requiring further verification; if it is higher than the preset threshold, the action corresponding to the maximum probability value is selected as the fault diagnosis category determination result. =0 indicates good health. =1 indicates gear wear. =2 indicates that the gear broke. =3 indicates pitting on the gear. =4 indicates a missing tooth. =5 indicates a gear tooth crack. =6 indicates that the inner gear is broken. =7 indicates that the outer ring gear is broken; the diagnostic results (i.e., gearbox fault diagnosis categories) are evaluated by indicators such as diagnostic accuracy, confusion matrix, recall rate, and F1-score to ensure the reliability of the model in industrial scenarios.

[0145] To verify the model performance of this embodiment, a comparative experiment was set up: the existing Categorical DQN model and this embodiment were tested on the same test set to verify the model performance.

[0146] The existing Categorical DQN model is cited in the literature as Bellemare MG, Dabney W, Munos, Rémi. A Distributional Perspective on Reinforcement Learning[J]. 2017. DOI:10.48550 / arXiv.1707.06887. The test set is cited in the literature as follows: Chen, Shijin; Liu, Zeyi; He, Xiao; Zou, Dongliang; Zhou, Donghua (2024), “Multi-mode FaultDiagnosis Datasets of Gearbox Under Variable Working Conditions”, MendeleyData, V1, doi: 10.17632 / p92gj2732w.1.

[0147] In the verification process on the test set of this embodiment, the following results were obtained: Figures 4 to 6 As a result, Figure 4 This is a heatmap of the fault diagnosis confusion matrix on the test set in this embodiment. The horizontal axis represents the fault category label predicted by the model, and the vertical axis represents the true fault category label of the sample. The intensity of the heatmap color is positively correlated with the number of correctly predicted samples. It can be seen that the overall diagnostic accuracy is extremely high, the class discrimination is strong, the ability to diagnose rare faults is excellent, there are no large-scale misjudgments or cross-class confusion phenomena, and the model is highly robust.

[0148] Figure 5 and Figure 6 This is a diagnostic cluster diagram for the eight states of the gearbox. Figure 5 and Figure 6 These are two sides of a clustering graph, different perspectives on the same clustering graph. Figure 6 exist Figure 5 It was rotated 270 degrees from the original. Figure 5 and Figure 6Each data point represents a test set sample. Points of different colors correspond to 8 different running results. The sample points corresponding to the 8 states form independent clusters. Each cluster has clear boundaries and no obvious intersections or overlaps, indicating that the features extracted by the model are robust, have high fault feature discrimination, and have good generalization ability. Figure 5 and Figure 6 Clustering effect and Figure 4 The confusion matrices corroborate each other, further verifying the reliability of the gearbox fault diagnosis categories in this embodiment.

[0149] Finally, the existing Categorical DQN model and this embodiment were validated on the same test set. The validation results are shown in Tables 1 and 2. The accuracy is the ratio of total TP to total number of samples, the macro-average recall is the arithmetic mean of the recall rates of all classes, and the macro-average F1 is the arithmetic mean of the recall rates of all classes.

[0150] Understandably, in Tables 1 and 2, TP (True Instances) refers to the number of samples correctly predicted as belonging to that class by the model, i.e., the number of samples whose predicted results match the true labels. The true number of samples refers to the total number of samples in the test set that actually belong to that fault category.

[0151] The existing Categorical DQN model achieved an accuracy of 87.42% (1049 / 1200) on this test set, with a macro-average recall of 87.95% and a macro-average F1 score of 87.78%. This embodiment achieves an accuracy of 95.21% (4570 / 4800) on the same test set, with a macro-average recall of 94.17% and a macro-average F1 score of 94.29%.

[0152] This embodiment improves the overall accuracy of the model from 87.42% to 95.21%, an increase of 7.79%, demonstrating that the method in this embodiment has a significant advantage in overall classification accuracy.

[0153] Compared to the existing Categorical DQN model, this embodiment improves the macro average recall by 6.22% and the precision by 7.79%, reflecting that the model performs better in identifying positive examples and balancing precision and recall.

[0154] Furthermore, in the existing Categorical DQN model, the recall rates for categories that perform poorly (such as "gear health status" and "gear broken tooth") are only 68.00% and 77.33%, respectively. This embodiment significantly improves these rates to 92.00% and 82.17%, indicating that the method in this embodiment also has better recognition capabilities for difficult-to-distinguish states.

[0155] Compared to the existing Categorical DQN model, the recall rate and F1 score of all eight fault states are improved in the verification performance results of this embodiment. In particular, the previously well-identified categories (such as "gear pitting" and "outer ring tooth fracture") remain at a high level (98.00% and 99.50%), which shows that this embodiment improves the overall performance without sacrificing existing advantages.

[0156] The uncertainty-aware Bellman equation and sample sparseness weighting factor introduced in this embodiment enable the dynamic reinforcement learning model to dynamically adjust its learning objective when faced with different deterministic states and imbalanced sample distributions. The significant performance improvement seen in categories such as "gear health status" and "gear missing tooth" where samples are relatively few or features are easily confused demonstrates that the method in this embodiment enhances the dynamic reinforcement learning model's ability to model minority classes and uncertain states.

[0157] This embodiment, based on Categorical DQN, constructs an uncertainty-aware Bellman equation through mechanisms such as adaptive support point generation, uncertainty hierarchical control, and rare class weighting. The data in Table 2 are comprehensively superior to those in Table 1, demonstrating that these innovative mechanisms work together to effectively improve the generalization ability and stability of distributed reinforcement learning in fault diagnosis.

[0158] Table 1. Performance results of existing Categorical DQN models on the test set.

[0159]

[0160] Table 2 Performance results of this embodiment on the test set

[0161]

[0162] This embodiment dynamically adjusts the value support point range through K-means clustering analysis, addressing the low fitting accuracy issue caused by insufficient value support points in small sample scenarios. Furthermore, its periodic update mechanism ensures it remains aligned with dynamic data distribution changes, balancing diagnostic accuracy and computational efficiency. Compared to traditional reinforcement learning models such as Categorical DQN, this embodiment uses a pre-defined fixed range of values ​​for its value support points (i.e., an adaptive set of value support points), which remains unchanged throughout training regardless of data distribution variations.

[0163] Based on the above description of the embodiments, those skilled in the art will understand that the dynamic reinforcement learning-based gear fault diagnosis method, system, and medium described in this embodiment can be implemented in pure software or deployed and run on a general-purpose or dedicated computing hardware platform. Based on this essence, the technical solution of this embodiment can be specifically implemented in the form of a software product containing program instructions. This software product can be stored on various non-volatile storage media or directly deployed as a local or cloud service. The program instructions are used to cause computer devices with processing capabilities—including but not limited to personal computers, server clusters, mobile terminals, or other network devices—to execute the steps described in this embodiment.

[0164] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A gear fault diagnosis method based on dynamic reinforcement learning, characterized in that, The preprocessed gear vibration signal is input into the trained dynamic reinforcement learning model to generate gearbox fault diagnosis categories. The training process of the dynamic reinforcement learning model is as follows: Gear vibration signals from various operating states are acquired and preprocessed to generate frequency domain feature vectors to construct a training set; The training set is processed using a policy network to generate interactive data, which is then stored in a hierarchical experience replay pool. Based on the degree of dispersion of the value distribution corresponding to the interaction data in the hierarchical experience replay pool, an adaptive set of value support points is generated; The uncertainty measure of the state is calculated based on the value distribution represented by the adaptive support point set output by the policy network, and the uncertainty-aware Bellman equation is constructed by combining the preset sample sparseness weight factor. A dual-deep Q-network architecture is adopted. The policy network receives the frequency domain feature vector and outputs the current value distribution and the optimal action. The optimal action is used as the fault diagnosis category. The target network calculates the target value distribution based on the uncertainty-aware Bellman equation and uses a soft update method to progressively optimize the parameters; Based on the current value distribution, the loss function and Shannon entropy are calculated synchronously, and the learning rate and model parameters are dynamically adjusted accordingly. The process of constructing the uncertainty-aware Bellman equation is as follows: Calculate the next state The uncertainty measure, the state Corresponding to the frequency domain eigenvectors; For the value distribution represented by the adaptive support point set output by the policy network, the logarithm of the probability value corresponding to each value support point is added to the zero-prevention factor, and then multiplied by the corresponding probability value. The sum of all the multiplication results is then negative to obtain the Shannon entropy value. Based on the Shannon entropy value and the preset threshold range, the state uncertainty is classified and determined, and a state uncertainty metric is generated accordingly. The uncertainty-perceived Bellman equation is as follows: ； in, This represents the discrete probabilistic target value distribution that matches the target network output with the adaptive value support point. It is the immediate reward value corresponding to the current fault. As a discount factor, It is a dynamic control function based on the level of state uncertainty, triggered by the state uncertainty measure. It is the Shannon entropy value of the value distribution corresponding to the next state. This is a state-action weight adjustment function based on a sample rarity weight factor. Fault category, The optimal action for the next state. For optimal action The expected distribution of the corresponding next-state value distribution. For the target network based on parameters The target value distribution is calculated. For policy networks based on parameters The defined strategy is used to select the optimal action in the current state. for The state at any given moment, for The optimal action to be performed at any given time is the fault category output by the policy network.

2. The method according to claim 1, characterized in that, The process of generating the frequency domain feature vector is as follows: The gear vibration signal is segmented into non-overlapping segments according to a set length to obtain vibration fragments; The frequency domain features of each vibration segment are extracted by performing a fast Fourier transform and then standardized to obtain the frequency domain feature vector.

3. The method according to claim 1, characterized in that, The hierarchical experience replay pool is divided into sub-pools according to fault categories. The number of sub-pools corresponds to the number of fault categories. During sampling, the sampling weight is allocated inversely proportional to the proportion of interactive data in each type of sub-pool. The interactive data includes the current state, actions representing the gear's operating state, immediate rewards corresponding to gearbox fault diagnosis categories, the next state, and termination state markers.

4. The method according to claim 1, characterized in that, The process of generating an adaptive set of value support points based on the dispersion of the value distribution corresponding to the interactive data in the hierarchical experience replay pool is as follows: (1) When the amount of data in the hierarchical experience replay pool reaches the set lower limit capacity threshold, extract the value distribution V of all interaction data in the hierarchical experience replay pool and calculate the variance. ; (2) If Then, the K-means clustering algorithm is used to cluster the value distribution V, and the silhouette coefficient of the value distribution V corresponding to each interaction data is calculated to evaluate the rationality of the clustering results, thereby generating each value support point. The preset variance threshold is used; (3) If Then, a linear interval is used to generate each value support point; (4) Repeat steps (1) to (3) according to the preset cycle to dynamically update the values ​​of each value support point and generate an adaptive value support point set accordingly.

5. The method according to claim 4, characterized in that, Value distribution of the interactive data The calculation formula is as follows: ； in, The current state The corresponding value distribution results, As a discount factor, Indicates the next state Follows the state transition probability distribution Expectation calculation, For policy network prediction Expected value for The corresponding current optimal action.

6. The method according to claim 1, characterized in that, The process of simultaneously calculating the loss function and Shannon entropy based on the current value distribution, and then dynamically adjusting the model's learning rate and parameters, specifically involves: Using the weighted Wasserstein distance as the loss function, the loss value between the current value distribution and the target value distribution output by the target network is calculated, and the model parameters are adjusted accordingly. The learning rate of the model is dynamically adjusted based on the rate of change of Shannon entropy, combined with the Warmup and cosine annealing mechanisms.

7. The method according to claim 6, characterized in that, The method of dynamically adjusting the model's learning rate based on the rate of change of Shannon entropy, combined with Warmup and cosine annealing mechanisms, specifically involves: Calculate the rate of change of Shannon entropy : Set the number of warm-up steps, and during the warm-up phase, the learning rate increases linearly from the initial value to the target value. After preheating, the learning rate of the model is dynamically adjusted using the Shannon entropy change rate. .

8. A computer system comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the method according to any one of claims 1-7.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a plurality of classification programs, which are used by a processor to execute the method as described in any one of claims 1-7.

Images