Deep reinforcement learning based unbalanced intelligent fault quantitative diagnosis method optimized based on reward

Abstract

The application discloses a reward-based optimization deep reinforcement learning unbalanced intelligent fault quantitative diagnosis method, and obtains multi-channel time-frequency representation (TFRs) through synchronous compression wavelet transform (SWT) to avoid non-stationarity of original vibration signals. A ResNet is introduced to construct a performer network for extracting representative deep fault features to improve the accuracy of fault diagnosis. Meanwhile, an AlexNet is used to construct a critic network to guide the performer to train in the correct direction according to an evaluation mechanism. The model is based on an unbalanced ratio to design a reasonable and practical reward function, and the minimum distance between various centers is used as real-time feedback of the reward to further improve the robustness of the model. The optimized state transition function improves the learning frequency of the minority class. Through the verification of the SQ rolling bearing data set, the ResDPG can independently and autonomously realize accurate fault quantitative identification and has high stability.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of quantitative diagnosis of rolling bearing faults, and particularly relates to an unbalanced intelligent fault quantitative diagnosis method based on reward optimization deep reinforcement learning. BACKGROUND

[0002] Accurate and timely identification of faults in the operation of mechanical equipment is of great significance to ensure its safe operation, avoid economic losses and major catastrophic accidents. The successful development of intelligent manufacturing has brought a large number of industrial process data records. The massive data has brought great opportunities and challenges to the field of fault diagnosis, making intelligent data-driven fault diagnosis based on flexibility, ease of use and low cost increasingly widely studied by the engineering and academic communities.

[0003] Generally speaking, existing bearing diagnosis technology research can be divided into two categories: qualitative diagnosis and quantitative diagnosis. Mechanical qualitative diagnosis is an important basis for quantitative diagnosis, and quantitative diagnosis of mechanical faults is an extension of qualitative diagnosis. In recent years, quantitative analysis of the severity of rolling bearing faults has attracted widespread attention from scholars around the world, and more research results have been achieved at home and abroad. However, most of the above results are based on signal processing and fault mechanism, and there is currently less research on quantitative diagnosis through deep learning models.

[0004] Generally speaking, the distribution of industrial data is skewed. Under normal circumstances, it faces very little fault data, especially catastrophic or sudden mechanical failures, which is a great challenge to data-driven fault diagnosis. In recent years, research on unbalanced fault diagnosis has received widespread attention from the academic community, and has great application value and significance for industrial data processing. In order to improve the problem of data deficiency and sample imbalance in the field of fault diagnosis, current research is divided into three categories: data augmentation-based strategies, feature learning-based strategies, and classifier design-based strategies.

[0005] Data augmentation-based strategies expand limited datasets by generating synthetic samples, oversampling existing samples, and reweighting samples to improve diagnostic model performance. Feature learning-based strategies learn fault features directly from limited fault data and use feature adaptation or design regularized neural networks for fault diagnosis. Classifier design-based strategies classify the health state of machines by designing fault classifiers for small-scale and imbalanced data without data expansion or feature extraction models. While these solutions alleviate the problem of data imbalance to some extent, they also have limitations. In data augmentation-based strategies, when only a small number of samples are available for training, the generated fault samples are too low in quality to accurately display the true distribution of the data, which often leads to difficulties in training the generated model. Since the fault information provided by a small amount of fault data is limited, the diagnostic performance improved by feature learning-based learning models is also very limited. Classifier design-based strategies generally require manual design and expert knowledge, so it is difficult to achieve optimal results. SUMMARY

[0006] The technical problem solved by the present application is to provide a reward-based optimization deep reinforcement learning imbalanced intelligent fault quantitative diagnosis method, which solves the problem that the imbalance between fault samples and normal samples in the prior art seriously affects the performance of intelligent fault diagnosis models.

[0007] The present application adopts the following technical solutions to solve the above technical problems:

[0008] The reward-based optimization deep reinforcement learning imbalanced intelligent fault quantitative diagnosis method comprises the following steps:

[0009] S1, signal acquisition and preprocessing, obtaining original vibration data of a rolling bearing under different health conditions, and dividing the vibration data in a non-overlapping manner to construct initial training samples and test samples;

[0010] S2, time-frequency representation generation, performing synchronous compression wavelet transform on the initial training samples and test samples to obtain two-dimensional time-frequency representations, and performing standardization processing thereon, and constructing an imbalanced distribution training set and a test set by randomly deleting samples;

[0011] S3, a reward-based optimization deep reinforcement learning intelligent quantitative diagnosis model is established, the reward-based optimization deep reinforcement learning includes a performer network, a critic network, a reward function and a state transition function; the performer network includes an online performer network and a target performer network, the critic network includes an online critic network and a target critic network, ResNet is introduced to replace the original CNN to construct the performer network, which is used for extracting representative deep fault features; AlexNet is introduced to replace the original CNN to construct the critic network, and the performer network is guided to be correctly trained according to the evaluation mechanism; a reward function based on a task is constructed, and the state transition function randomly gives a state according to the sample category order;

[0012] S4, a game environment is constructed based on the training set, and an intelligent agent and the game environment are cyclically interacted to obtain an intelligent quantitative diagnosis model;

[0013] S5, the intelligent fault quantitative diagnosis model is verified by using a test set.

[0014] In S3, the reward-based optimization deep reinforcement learning intelligent quantitative diagnosis model includes a state space S, an action space A and a reward function R, wherein t is the time step of training the reward-based optimization deep reinforcement learning intelligent quantitative diagnosis model; the state space S is represented as an unbalanced distribution training set constructed by the multi-channel time-frequency representation, each environment state s t corresponds to a training sample; the action space A corresponds to K categories of rolling bearing health states, A={0,1,2,…K-1}, wherein K represents the number of classification categories; the reward function R optimized based on the task is:

[0015]

[0016] wherein, ρ is an unbalanced ratio, |D F | represents the number of all fault samples, |D N | represents the number of all correct samples, D F represents all fault samples, D N represents normal samples, a t is the predicted action of the intelligent agent, y t is the true label of the state s t , and FB is the feedback of the reward function.

[0017] The reward function takes the minimum distance between each class center as real-time feedback, which is calculated by the following formula:

[0018]

[0019] In the formula, FB represents the feedback of the reward function, d minis a vector with the same dimension as the number of classes, containing the shortest distance from the class center of each class to the class center of the rest classes, is the minimum class distance representing the i-th fault class, is the maximum distance among all minimum class distances.

[0020] The S4 comprises the following steps:

[0021] S4.1: Set the maximum number of autonomous training iterations Episode, and the game environment initiates T diagnosis requests each time the autonomous training, each diagnosis request corresponds to an environment state s t ;

[0022] S4.2: The game environment randomly initiates a diagnosis request, and the current state s t ∈S, s t is input into the online performer network, and the agent selects the current action a t ∈A according to the dynamic ε-greedy algorithm, and the game environment returns a quantitative reward r t ∈R to the agent according to the task-based reward function, and randomly initiates the next diagnosis request, i.e., according to the state transition function to the next state s t+1 , the experience data e t ={s t ,a t ,r t ,s t+1} generated above is stored in the replay buffer in the form of a tuple;

[0023] S4.3: Repeat S4.2 until the T diagnosis requests in this round are completed, and output the cumulative reward obtained in this round of iteration;

[0024] S4.4: After the replay buffer is full, randomly sample batches of experience data E={e0,e1,e2,…,e n-1} from the replay buffer, and based on the experience data E, update the online performer network and the online critic network using the gradient descent method, and update the target performer network and the target critic network using a soft update method with an update amplitude τ;

[0025] S4.5: Start the next round of autonomous training process, and repeat S4.2 to S4.4 until the maximum number of autonomous training rounds is reached, and the autonomous training process is completed;

[0026] S4.6: Save the model parameters obtained in each round of diagnosis request in the autonomous training process whose cumulative reward is higher than a certain threshold, as the optimal intelligent quantitative diagnosis model learned by the agent.

[0027] The S4.2 comprises the following steps:

[0028] S4.2.1: initialization of epsilon = 1, decay rate epsilon decay = 0.998, minimum value epsilon min = 0.02;

[0029] S4.2.2: issuing a predicted action a t Before that, the intelligent quantitative diagnosis model randomly generates a random number between 0 and 1, and if the random number belongs to [0, epsilon], a predicted action a is randomly selected from the action space A t ; if the random number belongs to (epsilon, 1], the predicted action corresponding to the maximum Q value output by the current performer network is executed as the action a t ;

[0030] S4.2.3: after each diagnosis request, the value of epsilon is dynamically iteratively updated, if epsilon is less than epsilon min , epsilon = epsilon min , otherwise epsilon = epsilon * epsilon decay .

[0031] In S1, the original vibration signal is segmented to construct initial training samples and test samples, and each sample length contains 2048 data points.

[0032] In order to further solve the problem of fault detection encountered in practical application, the application also provides an unbalanced intelligent fault quantitative diagnosis system, and the specific technical scheme is as follows:

[0033] The unbalanced intelligent fault quantitative diagnosis system comprises a data acquisition unit, a data transmission unit, a data processing unit and a display terminal, wherein the data acquisition unit is used for acquiring original sample data sets, and the original sample data is sent to the data processing unit through the data transmission unit; the data processing unit applies the fault quantitative diagnosis method to process the received original sample data, and obtains fault type, fault degree and fault property information; and outputs to the display terminal.

[0034] The data acquisition unit comprises a vibration data acquisition instrument arranged on the rolling bearing, and the vibration data acquisition instrument acquires vibration data of the rolling bearing and sends the vibration data to the data processing unit.

[0035] The data acquisition unit is a fault simulation experiment platform, and the experiment vibration data of the rolling bearing are acquired through the simulation experiment platform and sent to the data processing unit.

[0036] The computer storage medium stores computer instructions, and the computer instructions are used to execute all or part of the steps of the method when called.

[0037] Compared with the prior art, the application has the following beneficial effects:

[0038] 1、Deep reinforcement learning as the frontier of artificial intelligence, fully integrates the perception ability of deep learning and the decision advantage of reinforcement learning, and has made innovative breakthroughs in computer games, control systems, recommendation systems, autonomous driving, communication, etc.

[0039] 2、The model is based on the unbalanced ratio and designed a reasonable and practical reward function, and the minimum distance between various centers is used as the real-time feedback of the reward, which further improves the robustness of the model.

[0040] 3、ResNet is introduced to construct the performer network, which is used to extract representative deep fault features to improve the accuracy of fault diagnosis; at the same time, AlexNet is used to construct the critic network, which guides the performer to train in the correct direction according to the evaluation mechanism.

[0041] 4、The optimized state transition function improves the learning frequency of the minority class. Through the verification of the SQ (spectral quest, SQ) rolling bearing data set, ResDPG can independently and autonomously realize accurate fault quantitative identification, and has high stability. BRIEF DESCRIPTION OF DRAWINGS

[0042] Figure 1 The model framework structure of the method of the application.

[0043] Figure 2 The fault quantitative diagnosis flowchart of the method of the application.

[0044] Figure 3 The box plot of F1-score of SQ diagnosis data set.

[0045] Figure 4 The normalization confusion matrix of the method in the SQ data set.

[0046] Figure 5 The t-SNE feature visualization result of SQ data set input into ResDPG under unbalanced working condition. DETAILED DESCRIPTION

[0047] The structure and working process of the application will be further described below in combination with the drawings.

[0048] The imbalance between fault samples and normal samples seriously affects the performance of intelligent fault diagnosis model. In order to solve the above problems, an improved deep deterministic policy gradient algorithm based on actor-critic structure (DDPG with ResNet, referred to as ResDPG) is proposed. In ResDPG, multi-channel time-frequency representations (TFRs) are obtained by synchronous squeezed wavelet transform (SWT) to avoid the non-stationarity of the original vibration signal. ResNet is introduced to build the actor network, which is used to extract representative deep fault features to improve the accuracy of fault diagnosis. At the same time, AlexNet is used to build the critic network, which guides the actor to train in the correct direction according to the evaluation mechanism. The model cleverly designs a reasonable and practical reward function based on the imbalance ratio, and takes the minimum distance between each class center as the real-time feedback of the reward, further improving the robustness of the model. The optimized state transition function improves the learning frequency of the minority class. Through the verification of the SQ rolling bearing data set, ResDPG can independently and accurately realize fault quantitative identification, and has high stability.

[0049] The reward-optimized deep reinforcement learning imbalance intelligent fault quantitative diagnosis method comprises the following steps:

[0050] S1, signal acquisition and preprocessing, obtaining the original vibration data of rolling bearings under different health conditions, and dividing the vibration data in a non-overlapping manner to construct initial training samples and test samples;

[0051] S2, time-frequency representation generation, performing synchronous squeezed wavelet transform on the initial training samples and test samples to obtain two-dimensional time-frequency representations, and performing standardization processing, and constructing an imbalanced distribution training set and test set by randomly deleting samples;

[0052] S3, establish a reward-optimized deep reinforcement learning intelligent quantitative diagnosis model, the reward-optimized deep reinforcement learning includes an actor network, a critic network, a reward function and a state transition function; the actor network includes an online actor network and a target actor network, the critic network includes an online critic network and a target critic network, ResNet is introduced to replace the original CNN to build the actor network, which is used to extract representative deep fault features; AlexNet is introduced to replace the original CNN to build the critic network, which guides the actor network to train correctly according to the evaluation mechanism; a task-based reward function is constructed, and the state transition function randomly gives the state according to the sample category order;

[0053] S4, constructing a game environment based on the training set, obtaining an intelligent quantitative diagnosis model by cyclic interaction of the agent and the game environment;

[0054] S5, verifying the intelligent fault quantitative diagnosis model by applying the test set.

[0055] Specific embodiments, as shown in Figures 1 to 5

[0056] The flowchart of the method for intelligent fault quantitative diagnosis based on reward optimization deep reinforcement learning under unbalanced distribution includes the following steps:

[0057] Offline training:

[0058] 1) Data preprocessing

[0059] In order to make the segmented samples conform to the original unbalanced data distribution, the original vibration signal is segmented without overlap. In order to further improve the stationarity of the vibration signal, the original signal is subjected to synchronous compression wavelet transform to obtain high stationary TFRs. Then, the obtained TFRs are standardized to further improve the accuracy of the model. An unbalanced data set is constructed by randomly deleting a certain number of fault samples. Finally, the obtained unbalanced data set is divided into a training set and a test set. The test set is used as an online data set. The original vibration signal is segmented to construct initial training samples and test samples, and each sample length contains 2048 data points.

[0060] 2) Pre-training ResDPG.

[0061] In the pre-training phase, the initial hyperparameters of the ResDPG actor network are determined by a weighted cross-entropy loss function based on the unbalanced training set, and the output of the network is the predicted class of the input sample.

[0062] 3) Autonomous training of the agent.

[0063] ​In self-training, the simulation environment is initialized, and the unbalanced fault diagnosis problem is transformed into a sequential decision problem. The state is all samples in the training set, including TFRs and sample labels. The actor diagnoses all sample classes in the training set. At each step, the actor interacts with the environment, learns through trial and error, the environment generates a state, and the actor generates a corresponding action according to the current state and the strategy. The critic evaluates the state and the action, guiding the agent to develop in the direction of accumulating large rewards. The environment gives timely rewards according to the correctness of the action, and gives the next state according to the state transition function. The state transition function is designed according to the sample order. ResDPG stores the above data into the replay buffer, and learns the data in batches after the replay buffer is full. In this way, the correlation between data can be broken, the data can meet the independent and identically distributed, and the accuracy and stability of the model can be improved. ResDPG learns through experience replay, gradually updates and optimizes network parameters, and updates the target network through soft update. The ResDPG diagnosis model is continuously trained until the network converges, and then the optimal actor network hyperparameters and network structure are saved as the best diagnosis strategy for online testing. Specifically, the following steps are included:

[0064] S4.1: Set the maximum number of autonomous training iterations Episode. Each autonomous training, the game environment will initiate T diagnosis requests, each diagnosis request corresponds to an environment state s t ;

[0065] S4.2: The game environment randomly initiates a diagnosis request, the current state s t ∈S, s t is input into the online actor network, and the agent selects the current action a t ∈A according to the dynamic ε-greedy algorithm. The game environment returns a quantitative reward r t ∈R to the agent according to the task-based reward function, and randomly initiates the next diagnosis request, i.e., according to the state transition function to the next state s t+1 , the experience data e t ={s t ,a t ,r t ,s t+1} generated above is stored in the replay buffer in the form of a tuple;

[0066] S4.3: Repeat S4.2 until the T diagnosis requests in this round are completed, and output the cumulative reward obtained in this round iteration;

[0067] S4.4: After the replay buffer is full, randomly sample experience data E = {e0, e1, e2, …, e n-1} Based on the experience data E, the online performer network and the online critic network are updated by using the gradient descent method, and the target performer network and the target critic network are updated by using a soft update method with an update amplitude τ;

[0068] S4.5: Start the next round of autonomous training process, repeat the execution of S4.2 to S4.4 until the maximum autonomous training round is reached, and the autonomous training process is ended;

[0069] S4.6: Save the model parameters obtained by each round of diagnosis request in the autonomous training process whose cumulative reward is higher than a certain threshold, as the optimal intelligent quantitative diagnosis model learned by the agent.

[0070] The S4.2 includes the following steps:

[0071] S4.2.1: Initialize ε = 1, decay rate ε decay = 0.998, minimum value ε min = 0.02;

[0072] S4.2.2: Before issuing the predicted action a t , the intelligent quantitative diagnosis model randomly generates a random number between 0 and 1, if the random number belongs to [0, ε], a predicted action a t is randomly selected from the action space A; if the random number belongs to (ε, 1], the predicted action a t corresponding to the maximum Q value output by the current performer network is executed.

[0073] S4.2.3: After each diagnosis request, the value of ε is dynamically and iteratively updated, if ε is less than ε min , ε = ε min , otherwise ε = ε × ε decay .

[0074] The purpose of reinforcement learning (RL) is to maximize the correct identification of training samples and achieve its goal G by maximizing the cumulative discounted reward R t . When the agent performs a series of actions, the discount factor γ accumulates the rewards of the present and the future in a decaying manner to quantify the quality of each action.

[0075]

[0076] where k represents the expected reward R obtained by the model after the kth step, and γ determines the importance of future rewards. The following equation represents the Bellman equation, which represents the recursive relationship between the current reward and the future reward, and the expected reward of state s t can be represented by equation (2):

[0077] V(s) = E[G t |s t = s) = E[R t+1 + γR t+2 + γ 2 R t+3 +…]

[0078] = E[R t+1 + γ(R t+2 + γ 2 R t+3 +…)] = E[R t+1 + γV(s t+1 )] (2)

[0079] DDPG is a policy gradient algorithm based on continuous state and continuous action space. The agent selects an action a t according to the deterministic policy μ and the state s t to achieve a higher expected return. Therefore, the Q function is used to achieve the above process as described below.

[0080]

[0081] In autonomous training, given the state s t , the agent agent selects a deterministic action a t according to the actor online network and exploration noise to interact with the environment as shown in equation (4):

[0082]

[0083] Then, the agent agent receives the reward r t and the next state s t+1 returned by the environment. The agent agent stores the above data in the form of a tuple to the replay buffer. After the replay buffer is full, the agent randomly selects a small batch of data from the replay buffer, and then sends it to the actor network and the critic network respectively. The agent agent uses the actor target network to calculate the action of the next state, and sends it to the critic target network for updating in a similar way to DQN. The optimal Q value is calculated using the Bellman equation and the critic target network, which is expressed as equation (5):

[0084]

[0085] The critic online network calculates the evaluation value of the state and action. The loss between the evaluation value and the optimal value is minimized by the gradient descent algorithm, which can be expressed as equation (6) to update the critic online network.

[0086]

[0087] The update of the performer on the online network is performed using a deterministic policy action gradient with stochastic gradient ascent, the result of which is calculated as shown in equation (7):

[0088]

[0089] In DDPG, since the neural network is not easy to adapt to the action value function, a soft update method is adopted for the two target networks to slow down the update speed and improve the stability of the model. The target network does not participate in the actual action decision. The parameter update of the two target networks is realized by setting the update amplitude τ.

[0090] θ Q′ ←τθ Q +(1-τ)θ Q′ (8)

[0091] θ μ′ ←τθ μ +(1-τ)θ μ′ (9)

[0092] 3-1) Reward function design

[0093] In reinforcement learning, the agent learns the best behavior policy through continuous interaction with the environment, and the training goal is achieved by maximizing the cumulative reward. During the learning process, the agent cannot distinguish the learning effect of the sample from the input signal. Such a mechanism forces the agent to find the inherent differences between fault modes, which makes the diagnosis model more robust. In order to better identify the minority class samples, if the agent encounters a minority class, the agent should pay more attention and get more rewards or punishments. The reward function is designed as follows.

[0094] 3-1-1) Since it is difficult to correctly identify the minority class in unbalanced data sets, an reward function based on the imbalance ratio ρ is designed.

[0095]

[0096] In the formula, |D F | represents the number of all fault samples, and |D N | represents the number of all correct samples.

[0097] 3-1-2) In order to further expand the minimum inter-class center distance and improve the classification effect, on the basis of obvious classification effect, the extracted high-dimensional abstract features are rescaled to two-dimensional space using t-SNE. The minimum distance of the class center of each fault class is calculated by the Euclidean metric method. If the minimum distance of the inter-class center of a certain class is small, it means that the classification effect is not good, so negative feedback is given in the next iteration, as shown in equation (11):

[0098]

[0099] where FB represents the feedback of the reward function, d min is a vector with the same dimension as the number of classification categories, containing the shortest distance from the class center of each category to the class center of the remaining categories, represents the minimum class distance of the i-th fault category, is the maximum distance among all minimum class distances.

[0100] The reward-based optimization deep reinforcement learning intelligent quantitative diagnosis model comprises a state space S, an action space A, and a reward function R, wherein t is the time step for training the reward-based optimization deep reinforcement learning intelligent quantitative diagnosis model; the state space S is represented as an unbalanced distribution training set constructed from the multi-channel time-frequency representation, and each environment state s t corresponds to a training sample; the action space A corresponds to K categories of rolling bearing health states, A = {0, 1, 2, … K-1}, wherein K represents the number of classification categories; and the reward function R based on task optimization is set as shown in the following formula (12):

[0101]

[0102] where D F represents all fault samples, D N represents normal samples, a t is the predicted action of the agent proxy, y t is the true label of the state s t , and FB is the feedback of the reward function.

[0103] 3-2) Network Settings

[0104] ResDPG consists of two parts: performer network and critic network. In the original model, the initial networks of the performer and the critic are relatively simple. In order to better learn deep abstract features from unbalanced data, ResNet-18 is introduced, which is easy to train, optimize, and has high accuracy when constructing the performer network. The input of the performer network is the state provided by the environment, and the output is the action selected by the performer network according to the current policy. In order to improve the stability of the critic network, we introduce AlexNet to construct the critic network. The input of the critic network is the state and the action, and the output is the Q value corresponding to the state and the action. The specific settings of the performer network and the critic network are shown in Table 1.

[0105] Table 1 Basic configuration of performer network and critic network

[0106]

[0107] Online testing:

[0108] 4) Obtain online diagnosis model

[0109] The ResDPG optimal diagnosis strategy is used for online fault diagnosis, which is decoupled from the offline fault diagnosis model. The offline fault diagnosis model trained on unbalanced bearing data can be used to identify the fault type of online samples.

[0110] 5) Online fault diagnosis

[0111] When a new sample comes, the online fault diagnosis model can be directly applied to the online diagnosis task to determine the fault category of the new sample.

[0112] To further illustrate the scheme, the following simulation experiment of fault qualitative and quantitative diagnosis of rolling bearings is described in detail.

[0113] Experimental device:

[0114] In order to study the effectiveness and feasibility of intelligent fault quantitative diagnosis under unbalanced distribution, this experiment selects the SQ (Spectral Quest, SQ) rolling bearing vibration data set. The test platform includes a rotor system, an alternating current motor, a speed controller and multiple accelerometers. The digital acquisition system is installed at the driver end to collect the original vibration signal at a sampling frequency of 25.6 kHz. The bearing type of the experiment is NSK 6203. In order to simulate different failure modes of rolling bearings, six single-point defect rolling bearings with different degrees of defect are made. The measured vibration signals include two different fault states: inner raceway fault IF (inner raceway failure, IF) and outer raceway fault OF (outer raceway failure, OF), each fault degree contains three degrees (mild, moderate and severe). In addition, the healthy bearing is tested as the normal state. The above bearing data is labeled as inner raceway failure IF-1, IF-2, IF-3, OF-1, OF-2, OF-3 and Normal.

[0115] II. Data processing and experimental setup:

[0116] To extract the hidden effective abstract features, the original vibration signal is processed by the synchrosqueezed wavelet transform (SWT) and then by the image standardization process to obtain the time frequency representation (TFRs) with a shape of 224x224x3. Compared with the original signal, the TFRs carry more potential fault information and enhance the robustness of the model. These TFRs are used as the game environment of the proposed intelligent fault quantitative diagnosis model. The slower the rotating speed of the rolling bearing is, the less information the fixed-length sample contains. To prove the effectiveness of the method, low-speed bearing data is used in the experiment, and the motor speed is controlled at 523 rpm. In order to conform to the original unbalanced data distribution, the unbalanced rate is divided according to the number of original data points, and the non-overlapping sampling method is used to divide the samples. The original vibration signal collected is divided by 2048 data points. The number of samples in each category with the same damage degree is 187, and 1309 samples are obtained in total with 7 labels. 50 samples are selected from each category as the test set, and the rest are used as the training set.

[0117] To verify the performance of the proposed model under the condition of small sample and unbalanced data, an unbalanced data set is constructed by randomly deleting fault samples, as shown in Table 2. The learning rates of the actor and critic of ResDPG are set to 0.0001 and 0.0001 respectively, the discount factor is set to 0.25, and the capacity of the replay buffer is set to 200. 125 cycles are used to ensure the good accuracy of the model while saving computing resources, because overtraining will cause overfitting of the model.

[0118] Table 2 Setting of SQ unbalanced data set

[0119]

[0120] To objectively test the superiority of the proposed ResDPG method in quantitative diagnosis under imbalanced distribution, we conducted comparative experiments. The comparison strategies are as follows: CLUSTER, ResNet, SMOTE, Focal Loss, and DQN. CLUSTER is an under-sampling method that obtains a balanced dataset by randomly discarding samples of the majority class. SMOTE belongs to the over-sampling method, which obtains a balanced dataset by generating samples of the minority class. ResNet adopts the ResNet-18 network structure. Focal loss solves the class imbalance problem by reshaping the standard cross-entropy loss and reducing the loss weight assigned to well-classified classes. The model parameters are: a = 0.25, g = 2. DQN adopts the original structure and improves the reward function with the imbalance rate. DCNN is used as the base classifier for other strategies. This paper selects F1-score as the evaluation index of the experiment, which can better reflect the classification accuracy of imbalanced data. In addition, in order to ensure a comprehensive evaluation of the model performance, accuracy (ACC) is also used as an auxiliary evaluation index. In order to overcome the randomness caused by too few samples and make the experimental results more convincing, all experiments are repeated 10 times. The training set is verified by stratified three-fold cross-validation.

[0121] III. Experimental results and analysis

[0122] In order to fully verify the superiority of ResDPG under imbalanced data, comparative experiments were conducted. Table 3 shows the average value of 10 independent repeated experiments of all strategies. From Table 3, it can be found that the diagnostic performance of all strategies based methods decreases with the increase of imbalance, especially CLUSTER and ResNet, when the imbalance rate is 1:10, the F1-score decreases to 31.51% and 56.75%, respectively. It can be seen that if a relatively large number of samples are deleted from the majority class, the most valuable information may be deleted, and the distribution of the dataset will also change. This leads to the decrease of the accuracy of the fault quantitative diagnosis model. FocalLoss has some improvement in dealing with such highly imbalanced data. The accuracy of SMOTE is improved more than ResNet, which shows that the generated data can effectively improve the accuracy of the diagnosis model, but the degree of improvement is limited. DQN performs best among all comparison methods, but still not as good as the proposed method. Through comparative experiments, it is found that ResDPG has good fault recognition ability and data adaptability, but it consumes more time than other comparison methods.

[0123] Table 3 Diagnostic performance of different methods on SQ diagnostic dataset

[0124]

[0125] As Figure 3As shown, the box plot is used to compare the distribution of F1-score in 10 comparative experiments. In 10 independent repeated experiments, the data of the proposed ResDPG method is the most concentrated, showing the best stable performance. SMOTE has obvious fluctuations in the case of 1:10, indicating that the quality of the generated data is not high, affecting the stability of the model. ResNet has more outliers in the case of 1:10, which shows that in the case of less sample and extremely uneven data, the algorithm has obvious fluctuations.

[0126] In the case of 0.09 of all fault data accounting for the total data, the classification of each fault category is intuitively shown by the confusion matrix, as shown in Figure 4 The experimental results verify the excellent performance of the proposed method in predicting all categories. The proposed method not only can identify different damage degrees of the same fault category, but also can accurately distinguish different fault categories caused by different damage degrees. This further proves the superiority of ResDPG in intelligent fault quantitative diagnosis under extremely unbalanced data.

[0127] Four, feature visualization:

[0128] In order to more intuitively illustrate that the proposed method has more effective feature classification performance under the condition of extremely unbalanced data (taking the training set C as an example), the t-SNE technology which can capture the complex flow structure of high-dimensional data is used to visualize the high-dimensional features extracted from the last hidden layer of the model in a dimension-reduced manner, as shown in Figure 5 As shown in the figure, ResDPG has a clear decision boundary and no classification error, and is very compact in the same category and relatively dispersed in different categories. This shows that the hidden features extracted by the proposed method have strong distinguishability, which further proves that the method has good feature extraction capability and can distinguish samples with different health conditions.

[0129] The unbalanced intelligent fault quantitative diagnosis system comprises a data acquisition unit, a data transmission unit, a data processing unit, and a display terminal. The data acquisition unit is used to acquire original sample data sets, and the original sample data is sent to the data processing unit through the data transmission unit. The data processing unit applies the fault quantitative diagnosis method to process the received original sample data to obtain fault type, fault degree, and fault property information, and outputs to the display terminal.

[0130] The data acquisition unit comprises a vibration data acquisition instrument arranged on the rolling bearing, which acquires vibration data of the rolling bearing and sends it to the data processing unit.

[0131] The data acquisition unit is a fault simulation experiment platform, which acquires experimental vibration data of the rolling bearing through the simulation experiment platform and sends the data to the data processing unit.

[0132] A computer storage medium stores computer instructions, which are used to execute all or part of the steps of the method when invoked.

[0133] Those skilled in the art should understand that the skilled in the art can realize the variants in combination with the prior art and the above-mentioned embodiments, and such variants do not affect the essential content of the scheme, and are not described here.

[0134] It should be understood that the present scheme is not limited to the above specific embodiments, and the devices and structures not fully described should be understood as being implemented in the ordinary way in the art; any person skilled in the art can make many possible changes and modifications to the present scheme or modify it into equivalent embodiments without departing from the scope of the present scheme, which does not affect the essential content of the present scheme. Therefore, any simple modification, equivalent change and modification made to the above embodiments according to the technical essence of the present scheme without departing from the content of the present scheme still belongs to the protection scope of the present scheme.

Claims

1. A quantitative fault diagnosis method for imbalanced intelligent systems based on reward-optimized deep reinforcement learning, characterized by: Includes the following steps: S1. Signal acquisition and preprocessing: Obtain raw vibration data of rolling bearings under different health conditions, and segment the vibration data in a non-overlapping manner to construct initial training samples and test samples. Each sample contains 2048 data points. S2. Time-frequency representation generation: The initial training samples and test samples are subjected to synchronous compressed wavelet transform to obtain a two-dimensional time-frequency representation, which is then standardized. An imbalanced training set and test set are constructed by randomly deleting samples. S3. Establish a reward-optimized deep reinforcement learning intelligent quantitative diagnostic model. The reward-optimized deep reinforcement learning includes a performer network, a critic network, a reward function, and a state transition function. The performer network includes an online performer network and a target performer network. The critic network includes an online critic network and a target critic network. ResNet is introduced to construct the performer network for extracting representative deep fault features. AlexNet is introduced to construct the critic network, which guides the performer network to train correctly based on the evaluation mechanism. The basic configurations of the performer network and the critic network are shown in the table below: ； Construct a task optimization reward function with minimum distance feedback between class centers, and optimize the state transition function for minority class learning frequency, randomly assigning states according to sample class order; the reward-optimized deep reinforcement learning intelligent quantitative diagnostic model includes a state space. Action space reward function ,in, The time step for training a reward-optimized deep reinforcement learning intelligent quantitative diagnostic model; the state space The unbalanced training set constructed from the multi-channel time-frequency representation is represented as follows: each environmental state... Corresponding to one training sample; the action space Corresponding to the health status of rolling bearings Types, ,in, Indicates the number of classification categories; the task-optimized reward function for: ； in, , For the unbalanced ratio, Represents the number of all fault samples. Represents the number of all correct samples. Represents all fault samples, This represents a normal sample. It is the agent's predictive action. It is a state The true label, The reward function is the minimum distance feedback between class centers, expressed by the following formula: , It is a vector with the same dimension as the number of categories, containing the shortest distance from the class center of each category to the class centers of the other categories. Representing the Minimum class spacing for each fault category It is the maximum distance among all minimum class intervals; S4. Construct a game environment based on the training set, and obtain an intelligent quantitative diagnostic model through cyclical interaction between the agent and the game environment. S5. The intelligent fault quantitative diagnosis model is validated using the application test set.

2. The quantitative diagnosis method for imbalanced intelligent faults based on reward optimization deep reinforcement learning according to claim 1, characterized in that: S4 includes the following steps: S4.1: Set the maximum number of autonomous training iterations (Episode). During each autonomous training iteration, the game environment will initiate... Each diagnostic request corresponds to a batch of environmental statuses. ; S4.2: The game environment randomly initiates a diagnostic request; current state. ,Will Inputting an online performer network, the agent selects the current action based on a dynamic ε-greedy algorithm. The game environment returns a batch of quantified rewards to the agent based on a task-optimized reward function. It then randomly initiates the next diagnostic request, that is, it transitions to the next batch of states according to the state transition function. The resulting empirical data Store the data as a tuple in the replay buffer; S4.3: Repeat S4.2 until this round. The diagnostic request ends, and the cumulative reward obtained in this iteration is output; S4.4: After the replay buffer is full, randomly sample batches of empirical data from the replay buffer. Based on empirical data The gradient descent method is used to update the online performer network and the online critic network, and the soft update method is used to update the target performer network and the target critic network with an update magnitude τ. S4.5: Begin the next round of autonomous training. Repeat steps S4.2 to S4.4 until the maximum number of autonomous training rounds is reached, at which point the autonomous training process ends. S4.6: Save the model parameters that have accumulated rewards exceeding a certain threshold for each round of diagnostic requests during autonomous training, as the optimal intelligent quantitative diagnostic model learned by the agent.

3. The quantitative diagnosis method for imbalanced intelligent faults based on reward optimization deep reinforcement learning according to claim 2, characterized in that: S4.2 includes the following steps: S4.2.1: Initialize ε=1, attenuation rate ε decay =0.998, minimum value ε min =0.02; S4.2.2: Issue the predicted action Previously, the intelligent quantitative diagnostic model randomly generated a random number between [0, 1]. If the random number belongs to the range [0, ε], it starts from the action space. Randomly select a prediction action If the random number belongs to the range (ε, 1], the predicted action corresponding to the maximum Q value output by the current performer network is the action to be executed. ; S4.2.3: After each diagnostic request ends, dynamically iterate and update the ε value. If ε is less than ε min , ε=ε min Otherwise, ε = ε × ε decay .

4. An intelligent quantitative diagnostic system for imbalance faults, characterized in that: The system includes a data acquisition unit, a data transmission unit, a data processing unit, and a display terminal. The data acquisition unit acquires a set of raw sample data and transmits the raw sample data to the data processing unit via the data transmission unit. The data processing unit applies the fault quantitative diagnosis method according to any one of claims 1 to 3 to process the received raw sample data, acquire fault type, fault degree, and fault nature information, and outputs the information to the display terminal.

5. The unbalanced intelligent fault quantitative diagnosis system according to claim 4, characterized in that: The data acquisition unit includes a vibration data acquisition instrument mounted on the rolling bearing, which acquires the vibration data of the rolling bearing and sends it to the data processing unit.

6. The unbalanced intelligent fault quantitative diagnosis system according to claim 5, characterized in that: The data acquisition unit is a fault simulation experimental platform. It acquires experimental vibration data of the rolling bearing through the simulation experimental platform and sends it to the data processing unit.

7. A computer storage medium, characterized in that: The computer storage medium stores computer instructions, which, when invoked, are used to perform all or part of the steps of the method according to any one of claims 1 to 3.

Images