An unbalanced sample fault diagnosis method based on contrastive representation deep reinforcement learning

Abstract

The application discloses an unbalanced sample fault diagnosis method based on contrast representation deep reinforcement learning, and comprises the following steps: a two-stage fault diagnosis model CRIDDQN combining contrast learning and deep reinforcement learning is proposed. In the pre-training stage, a balanced data set based on under-sampling is adopted to train a feature extraction model, and contrast learning is used to optimize the feature representation capability of the model for fault samples. In the fine-tuning stage, the original unbalanced data set is used to fine-tune the fault diagnosis model, so that the fault diagnosis accuracy of the model under the original unbalanced distribution condition is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of aerospace technology, and particularly relates to an unbalanced sample fault diagnosis method based on contrastive representation deep reinforcement learning. BACKGROUND

[0002] Rotating machinery is widely used as a key component of equipment in the fields of aerospace, rail transportation, wind power generation, etc. Traditional fault diagnosis methods mostly use signal processing methods to extract features of vibration signals, and further perform fault identification based on artificial experience. Such methods are severely dependent on artificial experience and are inefficient in processing high-dimensional massive data. In recent years, with the rapid development of artificial intelligence technology, intelligent fault diagnosis technology that does not rely on artificial experience and adaptively mines high-dimensional nonlinear features has become a current research hotspot. However, training an intelligent fault diagnosis model requires sufficient, balanced, and labeled fault samples. However, in actual operation of rotating machinery, the number of normal samples collected is much larger than that of fault samples, showing an unbalanced distribution.

[0003] Sample imbalance refers to a situation in which the number of training samples of different classes in a classification task differs greatly. Typical methods for fault diagnosis of unbalanced data sets can be roughly divided into two categories, including data-level methods and model-level methods. Data-level methods mainly include resampling methods and data augmentation methods, which balance the unbalanced data set to alleviate the influence of unbalanced distribution on classification effect. Model-level methods mainly include methods of changing sample weights and methods of modifying model architecture. The method of changing sample weights balances the attention degree of the model to different types of samples by increasing the weight of minority class samples and reducing the weight of majority class samples. However, these two methods are prone to cause the fault diagnosis model to deviate to the majority class samples when dealing with unbalanced samples, and even there may be a sample class overlap problem, resulting in low fault diagnosis accuracy of the model for minority class samples. Contrastive learning can increase the distance between different class samples and reduce the distance between same class samples, thereby improving the sample overlap problem and improving the fault pattern recognition ability of minority class samples. In order to improve the diagnosis ability of intelligent fault diagnosis model on unbalanced data set, the hyperparameters and architecture of the model need to be set according to the prior knowledge of specific data, resulting in a less robust model and poor generalization ability. Reinforcement learning has strong self-learning ability and special reward mechanism, which can mine original data and reconstruct reward function to make the best decision for unbalanced data classification. SUMMARY

[0004] The present application provides an unbalanced sample fault diagnosis method based on contrastive representation deep reinforcement learning, in order to solve the problem of low fault diagnosis accuracy.

[0005] The embodiment of the application provides a kind of unbalanced sample fault diagnosis method based on contrast representation deep reinforcement learning, comprising:

[0006] Obtain unbalanced sample data set under different fault modes, and obtain balanced sample data set by undersampling processing to the unbalanced sample data set;

[0007] Construct feature extraction model based on contrast learning, and utilize the balanced sample data set to train the feature extraction model based on contrast learning, and obtain the feature extraction model based on contrast learning trained;

[0008] Construct fault diagnosis model based on contrast representation improved deep reinforcement learning, and utilize the unbalanced sample data set to train the fault diagnosis model based on contrast representation improved deep reinforcement learning, and obtain the fault diagnosis model based on contrast representation improved deep reinforcement learning trained;

[0009] Obtain unbalanced sample data of target diagnostic fault equipment, and input the unbalanced sample data of target diagnostic fault equipment into the fault diagnosis model based on contrast representation improved deep reinforcement learning trained, and obtain the fault diagnosis result of the target diagnostic fault equipment.

[0010] Preferably, after obtaining the fault diagnosis model based on contrast representation improved deep reinforcement learning trained, further comprising:

[0011] Input the unbalanced sample data set under different fault modes as test set into the agent of the fault diagnosis model based on contrast representation improved deep reinforcement learning in turn, and obtain the prediction value of fault class label corresponding to fault sample;

[0012] By comparing the prediction value of fault class label corresponding to fault sample and real value, obtain the fault diagnosis accuracy of the fault diagnosis model based on contrast representation improved deep reinforcement learning trained.

[0013] Preferably, the fault diagnosis model based on contrast representation improved deep reinforcement learning comprises:

[0014] Construct double deep Q network fault diagnosis model based on classical deep reinforcement learning method;

[0015] By adaptive improvement processing to reward function in the double deep Q network fault diagnosis model, obtain improved double deep Q network fault diagnosis model;

[0016] By migrating the feature extraction model based on contrast learning trained into the agent of the improved double deep Q network fault diagnosis model, obtain the fault diagnosis model based on contrast representation improved deep reinforcement learning.

[0017] Preferably, the reward function in the improved double-depth Q network fault diagnosis model comprises:

[0018]

[0019] wherein D i is a fault sample set with fault mode category i, and the number of samples is n i The fault mode category with the least number of samples is recorded as min, and the number is n min The corresponding fault sample set is recorded as D min ; the reward value of the fault sample in D min is 1, and the reward value of the fault sample in D i is λ i , and λ i =n min / n i , l t is the category label of the sample corresponding to state s t ; a t represents the action at the tth time step; s t represents the state at the tth time step.

[0020] Preferably, the training of the feature extraction model based on contrastive learning using the balanced sample dataset comprises:

[0021] A positive-negative sample pair is constructed in batches in the balanced sample dataset using a contrastive learning loss function;

[0022] The loss value generated in a single iteration process is calculated using the corresponding deep embedding vector of the positive-negative sample pair;

[0023] The training and updating of the feature extraction model based on contrastive learning are realized by backpropagating the loss value, and the trained feature extraction model based on contrastive learning is obtained when the loss value reaches stability.

[0024] Preferably, the training of the fault diagnosis model based on contrastive representation improved deep reinforcement learning using the unbalanced sample dataset comprises:

[0025] The fault diagnosis model based on contrastive representation improved double-depth reinforcement learning is trained using the unbalanced sample dataset, and the loss value of the model is obtained based on the loss function thereof. The updating of the fault diagnosis model parameters is realized by backpropagating the loss value, and the trained fault diagnosis model based on contrastive representation improved deep reinforcement learning is obtained when the loss value reaches stability.

[0026] The beneficial effects of the present application include the following:

[0027] (1) The present application innovatively proposes a two-stage fault diagnosis model CRIDDQN combining contrastive learning and deep reinforcement learning. In the pre-training stage, a balanced dataset constructed based on undersampling is used to train the feature extraction model, and contrastive learning is used to optimize the feature representation ability of the model for fault samples. In the fine-tuning stage, the original unbalanced dataset is used to fine-tune the fault diagnosis model, and the fault diagnosis accuracy of the model under the original unbalanced distribution condition is improved.

[0028] (2) In the pre-training stage, the contrastive learning loss function is used to construct positive and negative sample pairs in batches, and the feature extraction model based on convolutional neural network is trained to reduce the distance between samples of the same type, increase the distance between samples of different types, increase the discrimination between the feature representations of majority class samples and minority class samples, and reduce the recognition difficulty of minority class samples.

[0029] (3) In the fine-tuning stage, the autonomous learning ability and reward mechanism of reinforcement learning are used to construct a fault diagnosis model based on DDQN, and the fault diagnosis problem is converted into a decision-making problem based on deep reinforcement learning. The general reward function is adaptively improved and designed, so that it can dynamically increase the reward value of the minority class and reduce the reward value of the majority class without relying on expert knowledge according to the sample distribution in the training process. BRIEF DESCRIPTION OF DRAWINGS

[0030] Figure 1 is the flowchart of the unbalanced sample fault diagnosis method based on contrastive representation deep reinforcement learning provided by the present application;

[0031] Figure 2 is the detailed flowchart of the unbalanced sample fault diagnosis method based on contrastive representation deep reinforcement learning provided by the present application;

[0032] Figure 3 is the structure diagram of the feature extraction model based on contrastive learning provided by the present application;

[0033] Figure 4 is the batch sample pair construction process diagram based on contrastive loss provided by the present application;

[0034] Figure 5 is the feature visualization diagram based on t-SNE provided by the present application;

[0035] Figure 6 is the feature visualization diagram based on t-SNE provided by the present application;

[0036] Figure 7A fault diagnosis accuracy mean and standard deviation diagram of different fault diagnosis models on a long tail data set is provided by the present application. DETAILED DESCRIPTION

[0037] It should be understood that the specific embodiments described herein are merely illustrative of the present application and do not limit the present application. In the following description, the suffixes such as "module", "part", or "unit" used for elements are used only for convenience of explanation of the present application, and have no particular meaning by themselves. Therefore, "module", "part", or "unit" can be mixedly used.

[0038] The present application proposes an unbalanced fault diagnosis model CRIDDQN based on contrast representation deep reinforcement learning, and combines the minority class recognition ability of contrast learning and the autonomous learning ability and reward mechanism of reinforcement learning, to provide a solution for the self-learning problem of the fault diagnosis model with high efficiency, accuracy and low energy consumption.

[0039] The present application tests the effectiveness and universality of the unbalanced fault diagnosis model CRIDDQN based on contrast representation deep reinforcement learning based on the fault data of bearings commonly used in engineering. The experimental results show that the method proposed by the present application can effectively realize the self-learning of the fault diagnosis model, and has certain universality and application convenience.

[0040] Figure 1 The present application provides a flowchart of an unbalanced sample fault diagnosis method based on contrast representation deep reinforcement learning, as shown in Figure 1 The method can include the following steps: S101, acquiring an unbalanced sample data set under different fault modes, and obtaining a balanced sample data set by performing undersampling processing on the unbalanced sample data set; S102, constructing a feature extraction model based on contrast learning, and training the feature extraction model based on contrast learning using the balanced sample data set to obtain a trained feature extraction model based on contrast learning; S103, constructing a fault diagnosis model based on contrast representation improved deep reinforcement learning, and training the fault diagnosis model based on contrast representation improved deep reinforcement learning using the unbalanced sample data set to obtain a trained fault diagnosis model based on contrast representation improved deep reinforcement learning; and S104, acquiring unbalanced sample data of a target diagnosis fault equipment, and inputting the unbalanced sample data of the target diagnosis fault equipment into the trained fault diagnosis model based on contrast representation improved deep reinforcement learning to obtain a fault diagnosis result of the target diagnosis fault equipment.

[0041] After obtaining the trained fault diagnosis model based on contrast representation improved deep reinforcement learning, the embodiment of the application further comprises: inputting the unbalanced sample data set under different fault modes as a test set into the agent of the fault diagnosis model based on contrast representation improved deep reinforcement learning in turn to obtain the predicted value of the fault class label corresponding to the fault sample; and obtaining the fault diagnosis accuracy of the trained fault diagnosis model based on contrast representation improved deep reinforcement learning by comparing the predicted value and the real value of the fault class label corresponding to the fault sample.

[0042] The constructing the fault diagnosis model based on contrast representation improved deep reinforcement learning comprises: constructing a double deep Q network fault diagnosis model based on a classical deep reinforcement learning method; obtaining an improved double deep Q network fault diagnosis model by adaptively improving the reward function in the double deep Q network fault diagnosis model; and obtaining the fault diagnosis model based on contrast representation improved deep reinforcement learning by migrating the trained feature extraction model based on contrast learning into the agent of the improved double deep Q network fault diagnosis model.

[0043] Specifically, the reward function in the improved double deep Q network fault diagnosis model comprises:

[0044]

[0045] wherein, D i is a fault sample set of a fault mode category i, and the number of samples is n i The fault mode category with the least number of samples in all fault modes is denoted as min, and the number is n min The corresponding fault sample set is denoted as D min ; the reward value of the fault sample in D min is 1, and the reward value of the fault sample in D i is λ i , and λ i =n min / n i , l t is the category label of the sample corresponding to the state s t ; a t represents the action at the tth time step; and s t represents the state at the tth time step.

[0046] Furthermore, the use of the balanced sample data set to train the feature extraction model based on contrastive learning to obtain a trained feature extraction model based on contrastive learning includes: using a contrastive learning loss function to batch construct positive and negative sample pairs in the balanced sample data set; using the deep embedding vectors corresponding to the positive and negative sample pairs to calculate the loss value generated during a single iteration; and implementing the training update of the feature extraction model parameters based on contrastive learning by backpropagating the loss value until the loss value reaches stability, thereby obtaining a trained feature extraction model based on contrastive learning.

[0047] Furthermore, the method of using the unbalanced sample data set to train the fault diagnosis model based on contrastive representation improved deep reinforcement learning to obtain a trained fault diagnosis model based on contrastive representation improved deep reinforcement learning includes: using the unbalanced sample data set to train the fault diagnosis model based on contrastive representation improved dual deep reinforcement learning, obtaining the loss value of the model based on its loss function, and updating the fault diagnosis model parameters by backpropagating the loss value until the loss value reaches stability, thereby obtaining a trained fault diagnosis model based on contrastive representation improved deep reinforcement learning.

[0048] Figure 2 This is a detailed flow chart of the imbalanced sample fault diagnosis method based on contrastive representation deep reinforcement learning provided by the present invention, such as Figure 2 As shown, including:

[0049] Step 1: Preprocessing of rolling bearing vibration signals

[0050] Under the conditions of certain working conditions, certain loads and certain health status of the rolling bearing, a sensor is used to collect vibration signals. The collected vibration signals are subjected to data preprocessing. Common data preprocessing methods include removing unique attributes, processing missing values, attribute encoding, data standardization and regularization, feature selection, principal component analysis, etc. In order to increase the number of samples, the present invention adopts a sliding window cutting method for data preprocessing, and divides the original vibration signal into multiple segments as an initial training set. Since the initial training set is unbalanced, directly using an unbalanced data set to train a deep learning model will lead to problems such as overfitting and weak generalization ability. Therefore, the random undersampling (RUS) method is further used to undersample the unbalanced data set to obtain a balanced data set as a training set for the feature extraction model.

[0051] Step 2: Constructing a feature extraction model based on contrastive learning

[0052] The present invention adopts convolutional neural network (CNN) as the underlying architecture of the feature extraction model. The main structure of the feature extraction model is as follows Figure 3The model is mainly composed of an input layer, a convolutional layer, a pooling layer, and a fully connected layer. The input layer is used to receive input data, the convolutional layer and the pooling layer are used to learn high-level features from the input data, and the fully connected layer and the output layer act as a classifier.

[0053] 1) Convolutional layer

[0054] The main role of the convolutional layer in the model is feature extraction, and the parameters in the convolutional kernel are calculated using the backpropagation principle. The most important feature of the convolutional layer is weight sharing, which means that when the convolution operation is used within the image range, each convolutional kernel will not change the value of its own parameters. The convolutional layer contains many filters that convolve the input from the previous layer with their kernels. The output of the convolution operation, also known as the feature map, is obtained through an activation function. The mathematical expression of the convolution operation can be described as follows:

[0055]

[0056] where (*) represents the convolution operation; l represents the number of layers in the network; k is the index of the input layer feature; k' is the index of the convolutional layer feature; X represents the input data matrix; is the kernel matrix of the k'th filter; the bias of the l'th layer is represented by ; f is a nonlinear activation function, usually ReLU, tanh, or sigmoid.

[0057] 2) Pooling layer

[0058] The pooling layer is usually followed by the convolutional layer. The main role of the pooling layer in the model is feature selection, and it is generally used in conjunction with the convolutional layer. The pooling layer reduces the number of parameters in the network training, which speeds up the training process and also controls overfitting. The pooling layer reduces the size of the input features and reduces the number of parameters in the network. In addition, the pooling layer has invariance to translation, deformation, rotation, etc. of the input. The down-sampling of the pooling layer can be achieved through maximum pooling, mean pooling, or weighted pooling. Among them, maximum pooling is the most commonly used in CNN and can be described as follows:

[0059] y ijk = max(0, x ijk )

[0060] where x ijk is the (i, j) component of the k'th feature of the convolutional layer.

[0061] 3) Fully connected layer

[0062] Since each node of the full connection layer is connected to all nodes of the upper layer, the full connection layer has such a classification function in the model, and its main function is to integrate the features extracted from the upper layer and input them to the classifier for recognition. The full connection layer is similar to the traditional feedforward neural network, which expands the feature map (matrix) obtained by the last layer of convolution into a one-dimensional vector and provides input for the output layer. The full connection layer is connected with the output layer to serve as a classifier.

[0063] Step three: feature extraction model training based on contrastive learning

[0064] The application adopts a CNN with deep feature mining capability to construct a feature extraction model, and a batch sample training model constructed by means of a contrastive learning loss function, so as to increase the distance between feature representations of different types of samples, especially the distance between majority class samples and minority class samples, and obtain a training model with strong minority class sample fault feature representation capability.

[0065] Step 301: extracting N pairs of samples from N class samples

[0066] Step 302: extracting N+1 samples from the N pairs of samples, denoted as Wherein, x i is a query sample of S i , is a positive sample, is a negative sample, and j≠i.

[0067] Step 303: batch constructing sample pairs by sampling in the training set by means of contrastive loss. In the convolutional neural network, the sample x i corresponds to a deep embedding vector represented as f i , and the batch sample pair construction process based on contrastive loss is as shown in Figure 4 .

[0068] Step 304: substituting the deep embedding vectors corresponding to the sample pairs into the following formula to calculate the contrastive loss value generated in a single iteration process.

[0069]

[0070] Wherein, f represents a feature vector, f i and f i + respectively represent the weight vectors corresponding to the samples x i and .

[0071] Step 305: the training update of the feature extraction model parameter is realized by back propagating the loss value, and the pre-training of the feature extraction model is completed when the loss value reaches stability. The contrast learning feature extraction model completed by training can pull the distance between different category samples away and pull the distance between the same category samples closer, thereby providing a contrast representation feature with enhanced fault category recognition for subsequent fault diagnosis.

[0072] Step four: construction of the fault diagnosis model based on the contrast representation deep reinforcement learning

[0073] Under the condition of unbalanced samples, the fault diagnosis model based on the classic deep reinforcement learning method double deep Q network (DDQN) often ignores the fault features of minority class samples, and the pattern recognition accuracy is not high. In order to improve the recognition accuracy of the DDQN fault diagnosis model for the fault pattern of the minority class sample, the reward function of the DDQN model is adaptively improved to obtain an improved double deep Q network (IDDQN) model. The unbalanced sample fault diagnosis process based on IDDQN contains four main elements, namely state space, action space, reward function and discount factor for calculating cumulative reward, which are described as follows:

[0074] State space S: in the fault diagnosis problem, the state space S is determined by the training set samples. At the beginning of training, the agent receives the first sample x1 as its initial state s1. The environment state s t corresponding to the sample x t When the next training starts, the environment will shuffle the order of the samples in the training set, and the state space will also change.

[0075] Action space A: the action of the fault diagnosis agent corresponds to the category label of the sample one by one, and the agent takes action a t ∈A under the state s t ∈S. For the multi-classification problem studied in the present application, A={0,1,2,…N}, wherein 0 represents the category label of the normal sample, and 1-N represents the category label of the fault sample.

[0076] Since the category label of the sample in the fault diagnosis problem is a discrete space, the action strategy of the agent is a discrete strategy, and the commonly used discrete strategies include ε-greedy strategy and Boltzmann strategy. The present application adopts the ε-greedy strategy to solve the fault diagnosis problem, which can be expressed as:

[0077]

[0078] Reward function R: When the original DDQN is used for fault diagnosis problems, if the agent can correctly identify the class label of the sample, the reward function gives a positive reward, and a negative reward is fed back if the class label of the sample is incorrectly identified. The reward function can be expressed as:

[0079]

[0080] In view of the problem that the DDQN model is difficult to accurately identify the minority class samples in the fault diagnosis problem under the unbalanced data condition, the reward function is adaptively improved and designed, and an IDDQN fault diagnosis model suitable for an unbalanced data set is constructed. In the process of each iteration training of the IDDQN model, according to the sample quantity of each fault mode in the current experience replay buffer area, the reward value corresponding to each fault mode is adaptively generated, and the reward value does not need to be given in advance or manually updated. For the fault mode with a small sample quantity, the IDDQN model gives a higher reward and punishment, and for the sample with a large sample quantity, the model gives a lower reward and punishment.

[0081] The reward function in the IDDQN model is defined as follows:

[0082]

[0083] Where, D i is a fault sample set of fault mode category i, and the sample quantity is n i The fault mode category with the smallest sample quantity in all fault modes is min, and the quantity is n min The corresponding fault sample set is D min According to the design of the reward function of the classic DQN algorithm, the reward value should belong to the range of [0, 1], therefore, the reward value of the fault sample in D min is 1, the reward value of the fault sample in D i is λ i , and λ i =n min / n i , l t is the class label of the sample corresponding to the state s t When the agent correctly or incorrectly diagnoses the sample in D min , the environment gives a reward value of 1 or -1; when the agent correctly or incorrectly diagnoses the sample in D i , the environment adaptively gives a reward value of λ i or -λ i according to the proportion of the sample quantity of each fault mode in the current experience replay buffer area; a t represents the action at the tth time step; s t represents the state at the tth time step.

[0084] Discount factor γ: The value of the discount factor is usually [0, 1], which is used to balance the reward of the current moment and the reward of the future moment. The closer γ is to 0, the more the agent pays attention to the reward value corresponding to the current diagnosis result. In the mechanical fault diagnosis problem studied in the present application, γ is set to 0.1, which to some extent weakens the influence of the diagnosis result at the future moment on the current agent decision.

[0085] The IDDQN model mainly consists of a target network Q target and a prediction network Q eval First, Q eval is used to calculate the Q value corresponding to each action value in the s' state, then the action value a max (s';θ) corresponding to the maximum Q value is selected, and then the TD target value is calculated based on the action value using Q target If the action is taken and the fault diagnosis is completed, i.e. the round is over, the target value y j is the reward generated by this action; if the round is not over, the target value y j will consist of two parts, one is the reward generated by this action, and the other is calculated by the target network, i.e. the Q value corresponding to each action in the next state is calculated, and then the maximum Q value is selected. The optimal target value y j can be represented as:

[0086]

[0087] where s' is the next state value, a' is the action value taken, θ is the parameter of Q eval , and θ - is the parameter of Q target . terminal represents the state of the round, and the value True indicates that the round is terminated, the TD target is the reward value, and the value False indicates that the round has not been terminated, and the TD target value needs to be calculated using Q target .

[0088] Further, the loss function of the IDDQN algorithm can be defined as:

[0089]

[0090] where s represents the current state value and a represents the current action value.

[0091] In the process of training the IDDQN model, the model parameters are updated by backpropagating the loss value, and when the loss value reaches stability, the model is trained and has the best fault diagnosis strategy.

[0092] During this fine-tuning phase, leveraging the autonomous learning capabilities and reward mechanisms of reinforcement learning, we constructed a fault diagnosis model based on the DDQN (Deep Dimensional Queries Network), transforming the fault diagnosis problem into a decision-making problem based on deep reinforcement learning. We also adaptively improved the general reward function, dynamically increasing the reward for the minority class and decreasing the reward for the majority class based on the sample distribution during training, without relying on expert knowledge.

[0093] The pre-trained contrastive representation feature extraction model is transferred to the intelligent agent of the IDDQN model as the feature extraction network part of the intelligent agent. The fully connected layer is then connected as a classifier to complete the construction of the contrastive representation deep reinforcement learning (CRIDDQN) fault diagnosis model.

[0094] Step 5: Fault diagnosis model training based on contrastive representation deep reinforcement learning

[0095] The comparison shows that after the deep reinforcement learning fault diagnosis model is built, it is trained using the rolling bearing vibration signal training unbalanced sample set obtained in step one.

[0096] The fault diagnosis process is transformed into a decision-making process based on deep reinforcement learning. At each time step, samples for training and updating the IDDQN model are randomly extracted from the training set. Each time the agent observes the state of a sample, it outputs an action corresponding to it. π θ is the fault diagnosis agent, T is the maximum time step, and K is the maximum number of iterations.

[0097] Step 501: Disrupt D tr The sample order in, initialize the state sequence s1 = x1, and randomly initialize the prediction network Q eval Parameter θ, initialize the target network Q target Parameter θ - =θ.

[0098] Step 502: a t Input into the fault diagnosis environment to get the current time a t Reward r t , fault diagnosis process terminalt t and the state s at the next moment t+1 .

[0099] Step 503: t ,a t ,r t ,s t+1 ,terminalt t} is stored in the experience replay buffer M until the end of a single fault diagnosis round.

[0100] Step 504: randomly sampling a mini-batch size of training set in the replay buffer as the input of the fault diagnosis agent π θ .

[0101] Step 505: calculating the loss value of the model according to the loss function of the IDDQN algorithm, and updating the network parameters of the CRIDDQN model by backpropagating the loss value.

[0102] Step 506: completing the training of a single CRIDDQN model.

[0103] The above process is repeated multiple times until the model is trained to the set number of iterations K, and the CRIDDQN fault diagnosis model with the best fault diagnosis strategy is obtained.

[0104] Step six: testing of the fault diagnosis model

[0105] The trained CRIDDQN fault diagnosis agent has the best fault diagnosis strategy. The test set data is input into the agent in turn, and the agent gives the prediction value of the fault class label corresponding to the sample. By comparing the true class label of the fault sample, the fault diagnosis accuracy of the model is obtained. Thus, the testing process of the CRIDDQN fault diagnosis model is completed.

[0106] The technical solutions of the present application will be described in detail below with specific embodiments

[0107] Step 1, rolling bearing vibration signal preprocessing

[0108] The bearing fault data set used in the fault diagnosis case study 1 is provided by the University of Padua (PU) and contains 6 groups of healthy bearing data, 12 groups of artificially damaged bearing data, and 14 groups of actually damaged bearing data. All data are collected on a modular test bench, which mainly consists of a drive motor, a torque measurement shaft, a test module, and a load motor. In this study, data from healthy bearings and artificially damaged bearings were collected at a sampling frequency of 64 kHz under the conditions of 900 rpm speed, 0.7 Nm load torque, and 1000 N radial force for unbalance fault diagnosis case verification. The fault data contains three states, including normal state (Normal State, NS), outer ring fault (Outer ring Fault, OF), and inner ring fault (Inner ring Fault, IF). According to different artificial damage methods and fault severity, it can be divided into nine fault modes. The detailed information of the bearings corresponding to each fault mode is shown in Table 1.

[0109] Table 1 Detailed information of bearing fault modes

[0110] Bearing number Failure type Artificial damage method Failure severity Failure label K001 NS - - 0 KA01 OF EDM 1 1 KA03 OF Electric engraving 2 2 KA05 OF Electric engraving 1 3 KA07 OF Drilling 1 4 KA08 OF Drilling 2 5 KI01 IF EDM 1 6 KI03 IF Electric engraving 1 7 KI07 IF Electric engraving 2 8

[0111] To verify the effectiveness of the method proposed in the present application on imbalanced datasets, an imbalanced dataset is constructed based on the vibration signals under nine fault modes, as shown in Table 2. First, the original vibration signal is cut by a sliding window with a width of 2048, and the sliding step of the sliding window is 300, to obtain fault samples under nine fault modes. Then, a certain number of samples are randomly extracted to construct a balanced test set. In the fault diagnosis experiment of each imbalanced dataset, the number of test samples of each fault mode is 4500, so that the evaluation indexes such as diagnosis accuracy, F1-score and its standard deviation can effectively reflect the fault diagnosis performance of the model. In order to fully verify the generalization ability of the method proposed in the present application to imbalanced data, the present application constructs an imbalanced training set with a long-tail distribution. 10000 samples are extracted from normal samples, and 800, 700, 600, 500, 400, 300, 200 and 100 samples are extracted from 8 fault samples according to labels 1-8, respectively, to construct an imbalanced dataset with a long-tail distribution.

[0112] Table 2 Bearing imbalanced dataset

[0113] Label 0 1 2 3 4 5 6 7 8 Sample size 10000 800 700 600 500 400 300 200 100

[0114] Step 2, feature extraction model construction based on contrastive learning

[0115] In the pre-training stage, a one-dimensional convolutional neural network is used to construct a feature extraction model, mainly including a convolutional layer, a pooling layer, a flattening layer, a fully connected layer and an output layer. The activation function used in the convolutional layer, the fully connected layer and the output layer is a rectified linear unit (ReLU) function. In order to avoid overfitting, the Dropout regularization method is introduced in the convolutional layer and the fully connected layer. In addition, in the two fault diagnosis case studies, the optimizer of the feature extraction model is the stochastic gradient descent (SGD) method, and the learning rate is 0.0025. The input of the feature extraction model in the two cases is the fault sample with a size of 1*2048, and the output is the sample feature with a size of 64. The feature extraction model calculates the contrastive loss value by constructing the contrastive features of positive and negative samples, and updates the model parameters by backpropagating the loss value. The maximum number of iterations of the feature extraction model is 5000 times. The setting of the main parameters of the feature extraction model is shown in Table 3.

[0116] Table 3 Parameter setting of feature extraction model

[0117]

[0118] Step 3, training of feature extraction model based on contrastive learning

[0119] In order to more intuitively reflect the feature extraction model based on contrastive learning proposed in the application, the feature representation ability and pattern recognition ability of each type of fault sample in the bearing long-tail data set, the t-SNE method is used to reduce the dimension of the original long-tail data set and the output of the feature extraction model based on contrastive learning, and the obtained two-dimensional features are visualized, as shown in Figure 5 (a) represents the feature of the original long-tail data after dimension reduction, and (b) represents the feature of the feature extraction model output layer after dimension reduction.

[0120] As can be seen from Figure 5 , the original long-tail distribution of the unbalanced data set has a serious inter-class overlap problem, and most of the minority class fault samples overlap with the majority class normal samples. After the feature extraction model based on contrastive learning is trained on the long-tail distribution training set, only a small number of samples of the extracted bearing fault features overlap. This shows that the feature extraction model based on contrastive learning can effectively learn the feature representation of the bearing samples and increase the distance between the features of different classes.

[0121] Step 4, construction of a fault diagnosis model based on contrastive representation deep reinforcement learning

[0122] The pre-trained feature extraction model is migrated to the CRIDDQN fault diagnosis model as the feature extraction network of the fault diagnosis agent. Through the connection of the classifier network composed of full connection layers, the fault diagnosis model is jointly composed. In the two fault diagnosis cases, the optimizer of the fault diagnosis model is consistent with the feature extraction model, and the SGD optimizer is adopted, and the learning rate decay strategy is adopted to increase the stability of the training process. In addition, the capacity of the experience replay buffer in the CRIDDQN model is 2000, that is, at most 2000 steps of model learning can be stored in the experience replay buffer. t ,a t ,r t ,s t+1 ,terminalt t} data group. Before the model starts to iterate and train, the random strategy is used to observe the {s t ,a t ,r t ,s t+1 ,terminalt t} data group and store it in the experience replay buffer. The time step of this observation process is 2000 steps. The CRIDDQN model adopts the ε-greedy strategy to explore the action a t (including the class label of the sample) in the learning process. The exploration rate is linearly decayed from 1.0 to 0.1, and the exploration times are 100000 times. In addition, since the observation of the agent in the fault diagnosis problem is relatively independent, and the model determines the action a tThe reward value of the current sample needs more attention, so the reward discount factor g is set to 0.1. The key parameter settings of the CRIDDQN model are shown in Table 4.

[0123] Table 4 Parameter settings of the CRIDDQN fault diagnosis model

[0124]

[0125] Step 5, training of the fault diagnosis model based on the comparative representation of deep reinforcement learning

[0126] In order to more intuitively reflect the feature representation ability and pattern recognition ability of the CRIDDQN fault diagnosis model proposed in the application to each type of fault sample in the long-tail data set of bearings, the t-SNE method is used to perform dimensionality reduction processing on the original long-tail data set and the output of the CRIDDQN fault diagnosis model, and the obtained two-dimensional features are visualized, as shown in Figure 6 (a) represents the features of the original long-tail data after dimensionality reduction, and (b) represents the features of the output layer of the CRIDDQN fault diagnosis model after dimensionality reduction.

[0127] After the CRIDDQN fault diagnosis model is trained based on the long-tail imbalanced data set, there is basically no class overlap problem in the output of the training set sample features, which indicates that the CRIDDQN fault diagnosis model constructed based on the pre-trained feature extraction model can map samples of different categories to different regions far away, and map samples of the same category to the same region. In addition, the CRIDDQN model increases the attention degree to the minority class samples by means of the adaptive reward function, further improving the feature extraction ability and fault pattern recognition ability of the fault samples.

[0128] Step 6, fault diagnosis model test

[0129] Based on the long-tail distribution of the imbalanced data set, the CRIDDQN fault diagnosis model proposed in the application and six comparative models are trained to carry out comparative experimental research on fault diagnosis, further verifying the generalization ability and applicability of the CRIDDQN fault diagnosis model proposed in the application on the long-tail imbalanced data set. The mean and standard deviation of the fault diagnosis accuracy and F1 score of the CRIDDQN model and the comparative model are shown in Table 5. In order to more intuitively compare the fault diagnosis performance of different fault diagnosis models on the long-tail imbalanced data set, the fault diagnosis accuracy and F1 score are shown by bar charts with error bars, as shown in Figure 7

[0130] Table 5 Mean and standard deviation of fault diagnosis accuracy of different fault diagnosis models

[0131] Failure diagnosis model Acc±Std F1-score±Std CNN 0.7435±0.0095 0.7216±0.0120 CRCNN 0.7790±0.0019 0.7672±0.0026 FLCNN 0.7607±0.0091 0.7500±0.0119 DDQN 0.7422±0.0056 0.7214±0.0076 RUS+DDQN 0.6446±0.0046 0.6440±0.0054 ROS+DDQN 0.7519±0.0030 0.7294±0.0038 CRIDDQN 0.8182±0.0016 0.8175±0.0018 ​

[0132] From Table 5 and Figure 7 It can be seen that the fault diagnosis accuracy and F1 score of the CRIDDQN model on the long-tail data set are significantly better than those of the other 6 comparative models. The fault diagnosis accuracy of CRIDDQN is 3.92% higher than that of the second highest comparative model CRCNN, and the F1 score of the fault diagnosis result is 5.03% higher than that of the second highest CRCNN model. In addition, the stability of the CRIDDQN model is the highest. In multiple independent fault diagnosis experiments, the fluctuations of the fault diagnosis accuracy and F1 score of the CRIDDQN model are small, and the standard deviations of the two types of fault diagnosis performance evaluation indexes are lower than those of other models. The experiment proves that the CRIDDQN model proposed in the application has significant advantages on the long-tail distribution unbalanced data set, and further verifies that the model is more suitable for solving the fault diagnosis problem under the condition of sample imbalance.

[0133] In summary, the application has the following advantages:

[0134] 1. In terms of accuracy and stability of fault diagnosis model self-learning, the imbalance sample fault diagnosis research method based on combination of contrastive representation deep reinforcement learning proposed in the application is superior to single data level method and model level method. This is because the imbalance sample fault diagnosis model CRIDDQN based on combination of contrastive representation deep reinforcement learning proposed in the application, in the pre-training stage, adopts a convolutional neural network with deep feature mining capability to construct a feature extraction model, and uses a batch sample training model constructed by means of a contrastive learning loss function to increase the distance between the feature representations of different types of samples, in the fine-tuning stage, a fault diagnosis agent of the CRIDDQN model is constructed based on the pre-trained model, and the agent is fine-tuned using the original unbalanced data set to enhance the perception ability of the fault diagnosis model to the original distribution and fault features of the unbalanced data set, and improve the fault diagnosis accuracy of the model under the original unbalanced distribution condition.

[0135] 2. The object of the application is to provide a set of self-learning process and method with universality for imbalance sample fault diagnosis model. Traditional fault diagnosis model self-learning methods have certain object and data limitations and cannot meet the demand of generalization application of current fault diagnosis model self-learning method. Therefore, the application proposes an imbalance sample fault diagnosis model based on combination of contrastive representation deep reinforcement learning, proposes an effective fault diagnosis process under unbalanced data, and uses the self-learning ability, strategy optimization ability and special reward mechanism of the model to improve the recognition ability of the model to minority class fault samples and the generalization ability of the fault diagnosis strategy on unbalanced data.

[0136] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, and are not intended to limit the scope of the present application. Any modification, equivalent replacement and improvement made by those skilled in the art without departing from the scope and spirit of the present application shall fall within the scope of the present application.

Claims

1. A method for fault diagnosis of imbalanced samples based on contrastive representation deep reinforcement learning, characterized in that: The method comprises: Obtaining an unbalanced sample data set under different fault modes, and obtaining a balanced sample data set by undersampling the unbalanced sample data set; Constructing a feature extraction model based on contrastive learning, and training the feature extraction model based on contrastive learning using the balanced sample data set to obtain a trained feature extraction model based on contrastive learning, comprising: using a contrastive learning loss function to batch construct positive and negative sample pairs in the balanced sample data set; calculating a loss value generated in a single iteration using the deep embedding vectors corresponding to the positive and negative sample pairs; and updating the training of the feature extraction model parameters based on contrastive learning by backpropagating the loss value until the loss value reaches a stable state, thereby obtaining a trained feature extraction model based on contrastive learning; Constructing a fault diagnosis model based on contrastive representation improved deep reinforcement learning, and training the fault diagnosis model based on contrastive representation improved deep reinforcement learning using the imbalanced sample data set to obtain a trained fault diagnosis model based on contrastive representation improved deep reinforcement learning, which includes: training the fault diagnosis model based on contrastive representation improved deep reinforcement learning using the imbalanced sample data set, obtaining a loss value of the model based on its loss function, and updating the fault diagnosis model parameters by backpropagating the loss value until the loss value reaches stability, thereby obtaining a trained fault diagnosis model based on contrastive representation improved deep reinforcement learning; Obtain unbalanced sample data of a target fault diagnosis device, and input the unbalanced sample data of the target fault diagnosis device into the trained fault diagnosis model based on contrastive representation improved deep reinforcement learning to obtain a fault diagnosis result of the target fault diagnosis device.

2. The method according to claim 1, characterized in that After obtaining the trained fault diagnosis model based on contrastive representation improved deep reinforcement learning, it also includes: Inputting the unbalanced sample data sets under different fault modes as test sets into the intelligent agent of the fault diagnosis model based on contrastive representation improved deep reinforcement learning in sequence to obtain the predicted values ​​of the fault category labels corresponding to the fault samples; By comparing the predicted value and the true value of the fault category label corresponding to the fault sample, the fault diagnosis accuracy of the trained fault diagnosis model based on contrastive representation improved deep reinforcement learning is obtained.

3. The method according to claim 1, characterized in that The construction of a fault diagnosis model based on contrastive representation to improve deep reinforcement learning includes: Construct a dual-deep Q network fault diagnosis model based on the classic deep reinforcement learning method; An improved dual-depth Q network fault diagnosis model is obtained by adaptively improving the reward function in the dual-depth Q network fault diagnosis model; By migrating the trained contrastive learning-based feature extraction model into the intelligent agent of the improved dual-depth Q-network fault diagnosis model, a fault diagnosis model based on contrastive representation-improved deep reinforcement learning is obtained.

4. The method according to claim 3, characterized in that The reward function in the improved dual-depth Q network fault diagnosis model includes: Among them, D i is a fault sample set with fault mode category i, and the number of samples is n i , the failure mode category with the least number of samples among all failure modes is recorded as min, and its number is n min , and its corresponding fault sample set is recorded as D min ;D min The reward value of the fault sample is 1, D i The reward value of the fault sample is λ i , and λ i =n min / n i , l t For state s t The category label of the corresponding sample; a t represents the action at the tth time step; s t represents the state at the tth time step.

Images