Open set identification method based on neural network and hypothesis testing

Abstract

The invention provides an open set identification method based on a neural network and hypothesis testing, and the method comprises the following steps: S1, obtaining fault data, carrying out the preprocessing of the fault data, and dividing the fault data into a training set, a verification set and a test set; s2, establishing a joint loss function, and training a convolutional neural network (CNN) by using the joint loss function; s3, after training is completed, a meta recognition system is established through features extracted by the last but one full-connection layer of the CNN; s4, calibrating the features extracted by the last full-connection layer of the CNN by using the output of the meta-recognition system; and S5, performing hypothesis testing on the calibrated features and making a final decision. The method can better adapt to a real open environment, can better reject unknown faults on the premise of ensuring the known fault recognition accuracy, and reduces the risk that unknown fault samples are classified into known faults by models.

Description

Technical Field

[0001] The invention belongs to the technical field of fault processing, and in particular relates to an open set identification method based on neural network and hypothesis testing. Background Art

[0002] In recent years, with the continuous development of artificial intelligence and big data, data-driven fault diagnosis models have been widely used, especially convolutional neural network models, which have gradually become one of the main methods of fault diagnosis due to their fast and accurate diagnostic performance. However, traditional convolutional neural network models have two obvious limitations when performing fault diagnosis: 1. Following the closed set assumption, that is, assuming that the training set and the test set contain the same categories, there will be no unknown faults that have never been seen. But in the real world, due to the difficulty in collecting all fault category information, the model often faces an open environment where both known faults and unknown faults may occur. In this environment, if the traditional convolutional neural network model is still used, it will usually cause the problem of unknown faults being misclassified as known faults. 2. The diagnosis basis is single, and only the features extracted by the network are used to measure the differences between samples, which makes the network vulnerable to open space risks and inaccurate diagnosis results in open environments.

[0003] Therefore, in view of the above problems, the present invention improves the convolutional neural network. First, the closed set hypothesis is eliminated, and the closed SoftMax layer is replaced by the hypothesis testing method. Secondly, a meta-recognition system based on extreme value theory is added to feed back the distance-based classification information to the features, so that the diagnosis results not only rely on the features extracted by the network, but are also affected by the distance information to a certain extent, avoiding the classification basis being too single. The improved model can better adapt to the real open environment, better reject unknown faults while ensuring the accuracy of known fault recognition, and reduce the risk of unknown fault samples being classified as known faults by the model. Summary of the invention

[0004] In view of the shortcomings of the prior art, the present invention provides an open set identification method based on convolutional neural network and hypothesis testing. This method overcomes the problem that the traditional convolutional neural network model cannot reject unknown faults to a certain extent, and basically realizes the identification of all known faults and the rejection of most unknown faults, thereby achieving the purpose of making the model adapt to the real open environment and being more in line with practical applications.

[0005] The technical solution of the present invention is as follows:

[0006] An open set identification method based on neural network and hypothesis testing comprises the following steps:

[0007] S1 acquires fault data and preprocesses the fault data to divide it into a training set, a validation set, and a test set;

[0008] S2 establishes a joint loss function and uses the joint loss function to train a convolutional neural network (CNN);

[0009] After S3 completes training, the meta-recognition system is established using the features extracted by the penultimate fully connected layer of CNN;

[0010] S4 uses the output of the meta-recognition system to calibrate the features extracted by the last fully connected layer of CNN;

[0011] S5 performs hypothesis testing on the calibrated features and makes a final decision;

[0012] Furthermore, the preprocessing in S1 is specifically as follows:

[0013] Irrelevant information in the fault data is eliminated, the data is simplified, and the fault data is processed into a two-dimensional form.

[0014] Furthermore, the S2 is specifically:

[0015] S21 designs a network structure, wherein the convolutional neural network mainly consists of convolution, activation, batch normalization, pooling, full connection and SoftMax;

[0016] S22 determines the loss function, specifically using the features output by the last fully connected layer to calculate the SoftMax loss, using the features output by the penultimate fully connected layer to calculate the center loss, and then adding the center loss to the SoftMax loss to obtain a joint loss function;

[0017] S23 determines the optimizer and other hyperparameters used when training the network;

[0018] S24 trains the convolutional neural network using the above parameters.

[0019] Furthermore, the joint loss function formula in S22 is:

[0020]

[0021] The first term in the formula is the SoftMax loss function, where Indicates the yth i The i-th deep feature of the class, d is the feature dimension, represents the weights in the last fully connected layer The jth column of is the bias term, the batch size of each iteration is m, the total number of categories is N, and the second term in the formula is the center loss function, where Indicates the yth iThe center of the deep features of the class samples.

[0022] Furthermore, the S3 is specifically:

[0023] S31 sends the validation set to the trained convolutional neural network, saves the features of the correctly classified samples output in the second-to-last fully connected layer, calculates the mean of each category of features, and uses the mean as the center of each category;

[0024] S32 calculates the distance between each correctly classified sample and the center of each class;

[0025] S33 uses the extreme value theory to fit the respective Weibull distribution model using the tail maximum of each type of distance. The formula of Weibull distribution is as follows:

[0026]

[0027] Among them, l is a random variable, is the distance from the sample to the class center, k is the shape parameter, and λ is the scaling factor.

[0028] The Weibull distribution models of all categories of S34 together constitute the meta-recognition system.

[0029] Furthermore, the S4 is specifically:

[0030] S41 Let the output of the meta-recognition system be p i =1-Weibull i (l)

[0031] S42 will p i The features output by the last fully connected layer are multiplied sequentially to achieve calibration.

[0032] Furthermore, the S5 is specifically:

[0033] First, several null hypotheses and alternative hypotheses are established for each category. Then, the samples in the validation set are sent to a convolutional neural network with a meta-recognition system to save the calibrated features of the correctly classified samples, and the α quantile of the set of eigenvalues ​​corresponding to each category is found. The part of the feature that is less than or equal to the quantile is set as the rejection region. If the maximum eigenvalue of the sample falls into the rejection region, the sample belongs to the unknown class, otherwise it belongs to the known class corresponding to the maximum eigenvalue.

[0034] Compared with the prior art, the present invention has the following advantages:

[0035] On the one hand, the traditional closed set hypothesis is eliminated, and a more open hypothesis testing method is used to replace the closed SoftMax layer, so that the model can reasonably reject unknown faults at a certain confidence level. On the other hand, a meta-recognition system based on extreme value theory is added on the basis of the traditional convolutional neural network, and the distance-based classification information obtained by the system can be fed back to the features output by the last fully connected layer, so that the diagnosis results not only rely on the features extracted by the convolutional neural network, but are also affected by the distance information to a certain extent, making full use of the features and information in the feature space, avoiding classification using only the features extracted by the neural network, and improving the stability and accuracy of fault diagnosis in an open environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] The accompanying drawings generally illustrate various embodiments by way of example and not limitation, and together with the description and claims, serve to illustrate the embodiments of the invention. Where appropriate, the same reference numerals are used throughout the drawings to refer to the same or similar parts. Such embodiments are illustrative and are not intended to be exhaustive or exclusive embodiments of the present apparatus or method.

[0037] Figure 1 Sliding window captures fault data;

[0038] Figure 2 Convolutional neural network architecture diagram;

[0039] Figure 3 Loss function calculation method;

[0040] Figure 4 Open set recognition model based on neural network and hypothesis testing. DETAILED DESCRIPTION

[0041] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0042] In this embodiment, the data used are from a certain nuclear reactor fault condition simulation data, which has five categories, four of which are fault condition data and one is normal condition data, and all condition types are listed in Table 1. Each fault condition data includes 11 parameter features, and the sampling interval of each parameter is 0.1s.

[0043] Table 1 Types of fault conditions of space nuclear reactors

[0044]

[0045] An open set recognition method based on an improved convolutional neural network comprises the following steps:

[0046] Step 1: Data preprocessing. First, since different parameters have different units, each parameter needs to be normalized to eliminate the impact of the dimension. Secondly, the training set, validation set, and test set are divided according to the ratio of 7:2:1. Finally, since the data used in this embodiment is time series data, each type of data needs to be captured by a sliding window. The sliding window size is 30×11, the sliding step is 3, and it starts from the time the fault occurs and stops when the fault state stabilizes again. A more intuitive capture process can be seen in Figure 1 As shown, the sample information of the training set and test set after truncation is listed in Table 2.

[0047] Table 2 Training and test sets of nuclear system failure data

[0048]

[0049]

[0050] Step 2: Train the convolutional neural network model.

[0051] 1. Design the network structure. The convolutional neural network used in this embodiment is mainly composed of convolution, activation, batch normalization, pooling, full connection and SoftMax. The designed network structure is as follows: Figure 2 shown.

[0052] 2. Determine the loss function. In order to make the later classification based on distance measurement more accurate, the present invention uses a joint loss function with center loss. Specifically, the features output by the last fully connected layer are used to calculate the SoftMax loss, and the features output by the second to last fully connected layer are used to calculate the center loss. The two are then added together to obtain the joint loss function, and the formula is as follows:

[0053]

[0054] The first term in the formula is the SoftMax loss function, where Indicates the yth i The i-th deep feature of the class, d=5 is the feature dimension. represents the weights in the last fully connected layer The jth column of is the bias term. m=32 is the batch size for each iteration, and N=5 is the total number of categories. The second term in the formula is the center loss function, where Indicates the yth iThe center of the deep features of the class sample. It should be noted here that the dimension d=2 of the features and class centers used in the second center loss in this embodiment is the feature dimension of the output of the second-to-last fully connected layer. A more intuitive way to calculate the loss function is as follows Figure 3 shown.

[0055] 3. Determine the optimizer and other hyperparameters used when training the network. In this embodiment, the network parameters are optimized using the stochastic gradient descent method, where the learning rate is 0.001 and the momentum is 0.9. In addition, the number of iterations is 100, and the batch size fed in each time is 32.

[0056] 4. Start training. During training, the network must continuously optimize parameters and reduce the distance from the extracted features to the class center to reduce the loss function. The specific parameter update method is as follows:

[0057]

[0058]

[0059]

[0060]

[0061]

[0062]

[0063] where θ C Represents the initialization parameters in the convolutional layer, W and c j are the parameters in the loss layer, α and μ t is a hyperparameter and t is the number of iterations.

[0064] Step 3: Establish a meta-recognition system based on extreme value theory. Since the features output by the penultimate fully connected layer are used to calculate the central loss function in step 2, the features of this layer will show smaller intra-class distance and larger inter-class distance after the network is trained. Therefore, performing distance measurement in the feature space extracted from this layer and classifying based on distance can achieve better classification results. The following are the specific steps to establish a meta-recognition system:

[0065] 1. Send the validation set to the trained model, save the features of the correctly classified samples output in the second to last fully connected layer, calculate the mean of each class of features, and use this mean as the center of each class. For example, in the validation set, there are m samples in the i-th class {X1, X2, ..., X m},in is the feature vector output by the penultimate fully connected layer. If there are k samples {X1,X2,……,X k} is correctly classified into the i-th category, then we use these k samples to calculate the center of the i-th category

[0066] 2. Calculate the distance from each correctly classified sample to the center of each class. Taking the i-th class as an example, we need to calculate {X1, X2, ..., X k} to M i The Euclidean distance of D i ={d1, d2, ..., d k}.

[0067] 3. According to the extreme value theory, the tail maximum of each type of distance is used to fit the respective extreme value distribution model. Because the data is bounded, the Weibull distribution is used for fitting. Taking the i-th type as an example, that is, using D i ={d1, d2, ..., d k The maximum value in} is used to establish the Weibull distribution model and obtain the cumulative distribution function of the Weibull distribution, whose formula is as follows:

[0068]

[0069] Among them, l is a random variable, is the distance from the sample to the class center, k is the shape parameter, and λ is the scaling factor.

[0070] 4. The Weibull distributions corresponding to all categories together constitute the meta-recognition system.

[0071] Step 4: Feature calibration.

[0072] 1. Assume that the output of the meta-recognition system is:

[0073] p i =1-Weibull i (l i )

[0074] Among them l i is the distance from the sample to the center of the i-th class, Weibull i is the Weibull distribution model of the i-th type, p i Indicates the probability that the sample belongs to the i-th category. Distance l i The larger the Weibull i The larger (l) is, the smaller the probability of belonging to the i-th class is.

[0075] 2. Let the probability of the output of the meta-recognition system for a certain sample be Prob = {p1, p2, p3, p4, p5}, and let the feature output of the last fully connected layer be vector V = {v1, v2, v3, v4, v5}. Let V·Prob to achieve feature calibration.

[0076] Step 5: Conduct hypothesis testing on the calibrated features and make a final decision. The specific process is as follows:

[0077] 1. Determine the null hypothesis and alternative hypothesis

[0078] Null hypothesis The sample belongs to the alternative hypothesis of class i The sample does not belong to the i-th class, where i=1, 2, ..., 5.

[0079] 2. Constructing the test statistic

[0080] The present invention sets the test statistic as the calibrated feature, which can also be called the score for ease of understanding. The score of the sample in the i-th category is denoted as s i .

[0081] 3. Determine the significance level and rejection region

[0082] Assuming the significance level α = 0.01, the rejection region in is the α quantile of the score of the i-th category, which is obtained from the validation set. We feed the data of the i-th category in the validation set into CNN to obtain the features extracted by the last fully connected layer, and multiply this set of features by the probability value output by the meta-recognition system to obtain the scores of these samples. Save the scores of all correctly classified samples in the i-th category on the i-th category, and find the α quantile from this set of scores, that is,

[0083] 4. Calculate the test statistic

[0084] The five categories correspond to five different hypothesis tests, five null hypotheses and alternative hypotheses. When calculating the test statistic, you only need to get the scores s1, s2, s3, s4, s5 of each category, which are the test statistics corresponding to the five hypothesis tests.

[0085] 5. Make decisions

[0086] In fact, when making a decision, it is not necessary to judge the scores of all categories. It is only necessary to take the maximum value of the five scores to perform hypothesis testing. i Maximum, then when When , it means that the sample has too few scores in the i-th category and has fallen into the rejection region, so the original hypothesis is rejected, and it is believed that there is a 99% probability that the sample does not belong to the i-th category. Considering that the highest score and the most likely category are rejected by the hypothesis test, it is even less likely that the sample belongs to other known categories, so the sample belongs to the unknown category. When When , it means that the sample has a large score in the i-th category and does not fall into the rejection region. Therefore, the original hypothesis is accepted, and it is believed that there is a 99% probability that the sample belongs to the i-th category.

[0087] In summary, through the above operations, a fault diagnosis model that is more suitable for open scenarios can be obtained. The model is used to test the simulation data of nuclear reactor fault conditions. In order to simulate an open environment, one class is extracted as an unknown class in each experiment. The final experimental results are as follows:

[0088] Table 1 The reactor is increased to rated power is unknown

[0089]

[0090] Table 2 Heat sink loss is unknown

[0091]

[0092] Table 3 Load rejection is unknown

[0093]

[0094]

[0095] Table 4 Steady state is unknown

[0096]

[0097] Table 5 Speed ​​reduction is unknown

[0098]

[0099] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. An open set identification method based on neural network and hypothesis testing, characterized in that: The following steps are included S1 acquires fault data and preprocesses the fault data to divide it into a training set, a validation set, and a test set; S2 establishes a joint loss function and uses the joint loss function to train the convolutional neural network; After S3 completes training, the meta-recognition system is established through the features extracted by the penultimate fully connected layer of the convolutional neural network; S4 uses the output of the meta-recognition system to calibrate the features extracted by the last fully connected layer of the convolutional neural network; S5 performs hypothesis testing on the calibrated features and makes the final decision.

2. The open set identification method based on neural network and hypothesis testing according to claim 1, characterized in that: The preprocessing in S1 is specifically as follows: Irrelevant information in the fault data is eliminated, the data is simplified, and the fault data is processed into a two-dimensional form.

3. The open set identification method based on neural network and hypothesis testing according to claim 1, characterized in that: The S2 is specifically: S21 designs a network structure, wherein the convolutional neural network mainly consists of convolution, activation, batch normalization, pooling, full connection and SoftMax; S22 determines the loss function, specifically using the features output by the last fully connected layer to calculate the SoftMax loss, using the features output by the penultimate fully connected layer to calculate the center loss, and then adding the center loss to the SoftMax loss to obtain a joint loss function; S23 determines the optimizer and other hyperparameters used when training the network; S24 trains the convolutional neural network using the above parameters.

4. The open set identification method based on neural network and hypothesis testing according to claim 3 is characterized in that: The formula of the joint loss function in S22 is: The first term in the formula is the SoftMax loss function, where Indicates the yth i The i-th deep feature of the class, d is the feature dimension, represents the weights in the last fully connected layer The jth column of is the bias term, the batch size of each iteration is m, the total number of categories is N, and the second term in the formula is the center loss function, where Indicates the yth i The center of the deep features of the class samples.

5. The open set identification method based on neural network and hypothesis testing according to claim 1, characterized in that: The S3 is specifically: S31 sends the validation set to the trained convolutional neural network, saves the features of the correctly classified samples output in the second-to-last fully connected layer, calculates the mean of each category of features, and uses this mean as the center of each category; S32 calculates the distance between each correctly classified sample and the center of each class; According to the extreme value theory, S33 uses the tail maximum of each type of distance to fit the respective Weibull distribution model. The formula of Weibull distribution is as follows: Among them, l is a random variable, is the distance from the sample to the class center, k is the shape parameter, and λ is the scaling factor. The Weibull distribution models of all categories of S34 together constitute the meta-recognition system.

6. The open set identification method based on neural network and hypothesis testing according to claim 1, characterized in that: The S4 is specifically: S41 Let the output of the meta-recognition system be p i =1-Weibull i (l) S42 will p i The features output by the last fully connected layer are multiplied sequentially to achieve calibration.

7. The open set identification method based on neural network and hypothesis testing according to claim 1, characterized in that: The S5 is specifically: First, several null hypotheses and alternative hypotheses are established for each classification. Then, the samples in the validation set are sent to a convolutional neural network with meta-recognition to save the calibrated features of the correctly classified samples, and the α quantile of the set of eigenvalues ​​corresponding to each class is found. The part of the feature that is less than or equal to the quantile is set as the rejection region. If the maximum eigenvalue of the test sample falls into the rejection region, the sample belongs to the unknown class, otherwise it belongs to the known class corresponding to the maximum eigenvalue.

Images