A boiler liquid level risk warning method based on CNN-Transformer

Abstract

The present invention discloses a boiler level risk warning method based on CNN-Transformer. After the local features obtained by the CNN network are input into the multi-head attention mechanism of the Transformer model, the features extracted from the multi-head attention mechanism are returned to the pooling layer of the CNN network for pooling operation, and the data after pooling is returned to the Transformer model. The present invention has the advantages of being able to extract more important features, reduce sequence length, and improve processing efficiency.

Description

Technical Field

[0001] The present invention relates to the technical field of boilers, and in particular to a CNN-Transformer-based boiler liquid level risk early warning method. Background Art

[0002] Boilers are key equipment in industrial production. When the liquid level of a boiler is too low and it works for a period of time, it will dry burn. In addition to easily burning the boiler body, it will also cause major safety accidents such as explosion when adding water to the dry pot by mistake. Therefore, it is urgent to establish an effective boiler liquid level risk warning method.

[0003] Due to the particularity of the boiler system, boiler fault signs and parameters are fuzzy and random, the status of the boiler liquid level cannot be described simply as "normal" or "faulty", and there is a lack of systematic evaluation.

[0004] Currently, most boiler level fault judgment and diagnosis are implemented locally by humans and machines, which limits the role of expert knowledge and external experience.

[0005] Although the current intelligent level of boiler liquid level risk warning methods is relatively low, a warning method based on CNN-Transformer has emerged among other boiler warning methods. CNN is a convolutional neural network and one of the representative algorithms of deep learning. Transformer is a sequence-to-sequence model based on the attention mechanism and is a deep neural network architecture for processing sequence modeling problems.

[0006] For example, the Chinese invention patent application with application publication number CN116894397A discloses a method for predicting boiler wall temperature increment based on multi-time graph convolution bidirectional Transformer. This method combines CNN network, bidirectional Transformer and incremental learning for predicting boiler wall temperature. It combines the advantages of CNN network in local feature extraction with the advantages of Transformer model in global information modeling to further improve the performance of the model. It has the ability to accurately predict boiler wall temperature, enabling the boiler to operate efficiently and safely.

[0007] However, the above patent is aimed at early warning of boiler wall temperature and cannot be applied to early warning of boiler liquid level risk. Even if it is simply applied to early warning of boiler liquid level, the following technical problems still exist:

[0008] 1. When obtaining local features through the CNN network, only horizontal convolution operations are performed without further extracting more important features;

[0009] 2. Although the parallel computing capability of the Transformer model improves the efficiency of training and reasoning, the length of the sequence to be processed is relatively long, and the processing efficiency still needs to be improved. Summary of the invention

[0010] The purpose of the present invention is to provide a boiler liquid level risk warning method based on CNN-Transformer. The present invention has the advantages of being able to extract more important features, while reducing sequence length and improving processing efficiency.

[0011] The technical solution of the present invention is a boiler liquid level risk early warning method based on CNN-Transformer, the steps are as follows:

[0012] S1. extracting boiler operation data from the boiler operation history database;

[0013] S2, preprocessing the operation data and generating time series features;

[0014] S3, input the time series feature into the convolution layer of the CNN network for a one-dimensional convolution operation to obtain the first feature;

[0015] S4. Position-encode the first feature through the word embedding layer of the Transformer model, and add the position-encoded feature to the first feature to obtain the second feature;

[0016] S5. Input the second feature into the multi-head attention mechanism of the first layer encoder of the Transformer model;

[0017] S6. The output of each attention head in the multi-head attention mechanism of the current layer encoder is passed to the pooling layer of the CNN network for pooling operation, and the pooling results of all attention heads are concatenated and linearly transformed to obtain the features that integrate the information of all attention heads;

[0018] S7, determine whether the current encoder layer is the last layer, if so, enter S8; if not, input the final result of the current encoder layer after processing in step S6 into the multi-head attention mechanism of the next encoder layer as the input feature of the next encoder layer, and then the next encoder layer is used as the current encoder layer to continue to return to step S6;

[0019] S8. Input the final result of the last encoder layer after processing in step S6 into a fully connected layer. The fully connected layer is used for dimensionality reduction and generating a score vector. The generated score vector is then converted into the probabilities of various working status types through a multi-classification activation function to generate a predicted working status classification result.

[0020] The aforementioned CNN-Transformer-based boiler liquid level risk warning method further includes S9 model optimization; the S1 also includes simultaneously acquiring the actual working state type corresponding to each operating data as the actual working state classification result;

[0021] The S9 model optimization is to calculate the corresponding loss degree of the predicted working status classification result and the actual working status classification result through the cross entropy loss function, and the optimizer optimizes the generated model.

[0022] The aforementioned CNN-Transformer-based boiler liquid level risk warning method further includes S10 model verification; the S1 also includes dividing the operating data into a training set and a verification set; the training set is used for model training of S2-S9;

[0023] S10 model verification specifically includes inputting the verification set into the trained model to obtain the predicted working status classification result, and obtaining the prediction accuracy of the model based on the predicted working status classification result and the actual working status classification result.

[0024] In the aforementioned CNN-Transformer-based boiler liquid level risk warning method, the S6 also includes, after obtaining the features of the current encoder layer that fuse all the attention head information, adding the features of the current encoder layer that fuse all the attention head information to the input features of the current encoder layer to perform the first residual connection processing.

[0025] In the aforementioned CNN-Transformer-based boiler liquid level risk warning method, the S6 also includes normalizing the result after the first residual connection processing.

[0026] In the aforementioned CNN-Transformer-based boiler liquid level risk warning method, S6 also includes inputting the normalized result into the feedforward neural network as an input feature of the feedforward neural network, and then outputting the result processed by the feedforward neural network.

[0027] In the aforementioned CNN-Transformer-based boiler liquid level risk warning method, the S6 also includes adding the input features and output features of the feedforward neural network to perform a second residual connection processing.

[0028] In the aforementioned CNN-Transformer-based boiler liquid level risk warning method, the feedforward neural network includes two fully connected layers, and the calculation formula of the feedforward neural network is, FNN(x)=ReLU(xW1+b1)W2+b2, where x: is the input vector of the first linear transformation, W1 is the weight matrix of the first linear transformation, b1 is the bias vector of the first linear transformation, ReLU is a nonlinear activation function and its output is used as the input of the second linear transformation, W2 is the weight matrix of the second linear transformation, and b2 is the bias vector of the second linear transformation.

[0029] In the aforementioned CNN-Transformer-based boiler liquid level risk warning method, the operating data in S1 includes exhaust gas temperature, liquid level and steam pressure; S1 also includes calculating the exhaust gas temperature change value, liquid level change value and steam pressure change value according to the operating data.

[0030] Compared with the prior art, after the local features obtained by the CNN network are input into the multi-head attention mechanism of the Transformer model, the features extracted from the multi-head attention mechanism are not directly used to complete all subsequent operations in the Transformer model, but are returned to the pooling layer of the CNN network for pooling operations, and the data after pooling is returned to the Transformer model.

[0031] Because of the addition of the pooling operation, the dimension of the time series features and the length of the sequence are reduced, while retaining the most significant feature information. Therefore, more important features can be further extracted, which helps the model focus on the most important features and reduces the number of parameters and the amount of calculation to improve processing efficiency.

[0032] Therefore, the present invention has the advantages of being able to extract more important features, while reducing sequence length and improving processing efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 is a flow chart of a boiler liquid level risk early warning method based on CNN-Transformer in Example 1 of the present invention;

[0034] Figure 2 It is a flow chart of the boiler liquid level risk warning method based on CNN-Transformer in Example 2 of the present invention. DETAILED DESCRIPTION

[0035] The present invention is further described below in conjunction with the accompanying drawings and embodiments, but they are not intended to limit the present invention.

[0036] Example 1. A boiler liquid level risk warning method based on CNN-Transformer, such as Figure 1 As shown, the following process is included:

[0037] S1. Extract the boiler operation data from the boiler operation history database. The boiler operation data is collected during the operation of the IoT boiler and stored in the boiler operation history database as the original data for the working state classification. The operation data includes exhaust gas temperature, liquid level, and steam pressure, and the exhaust gas temperature change value, liquid level change value, and steam pressure change value are calculated from them. The calculation formula is:

[0038] v′=v t+1 -v t ,

[0039] v′ is the data change value, v t+1 is the data of the next moment, v t is the data value at this moment;

[0040] S2. Preprocessing the operation data and generating time series features, wherein the preprocessing includes data cleaning and data filling; the data filling specifically includes filling the missing values ​​with data by at least manually filling or filling the average value;

[0041] S3. Input the time series feature into the convolution layer of the CNN network for a one-dimensional convolution operation to obtain the first feature. The calculation formula of the one-dimensional convolution operation is:

[0042]

[0043] y l is the output result of the one-dimensional convolution operation, l is the position where the convolution kernel slides, and x l-n is the lth data point of the input time series, w n is the nth weight of the convolution kernel, n is the index of the convolution kernel, and N is the width of the convolution kernel;

[0044] S4. Positionally encode the first feature through the word embedding layer of the Transformer model, and add the position-encoded feature to the first feature to obtain the second feature. The calculation formula for position encoding is:

[0045]

[0046] PE represents position encoding, pos represents position index, d represents the dimension of input, and i represents the dimension in the embedding vector;

[0047] S5. Input the second feature into the multi-head attention mechanism of the first encoder layer of the Transformer model. The calculation formula for the output of each attention head is,

[0048]

[0049] Attention(Q, K, V) is the output of each attention head, Q is the query vector, K is the T is the transpose of the key vector, V is the value vector, and D K is the dimension of the column vectors in vectors Q and K;

[0050] In order to prevent the inner product from being too large, we divide it by D K The square root of the softmax function here is used to convert the dot product result of the query vector Q and the key vector K into a probability distribution, which will be used to perform a weighted summation on the value vector V;

[0051] S6. The output of each attention head in the multi-head attention mechanism of the current encoder layer is passed to the pooling layer of the CNN network for pooling operation, and the pooling results of all attention heads are concatenated and linearly transformed to obtain the features that integrate the information of all attention heads;

[0052] The calculation formula for the pooling operation is,

[0053] TemporalPool=MaxPool(Conv1(Attention(Q,K,V))),

[0054] TemporalPool is a pooling operation, Conv1 is a one-dimensional convolution operation, MaxPool is a maximum pooling operation, and Attention (Q, K, V) is the output of each attention head in the multi-head attention mechanism;

[0055] The calculation formula for integrating the features of all attention head information is:

[0056] MultiHeadAttention(a)=concat(head1,...,head h )W O ,

[0057] head1,...,head h is the output of each attention head, calculated by Attention(Q,K,V), concat is the concatenation function used to connect the output of each attention head, W O is the weight matrix, MultiHeadAttention (a) is the feature that integrates all attention head information;

[0058] S7, determine whether the current encoder layer is the last layer, if so, proceed to step S8; if not, input the final result of the current encoder layer after processing in step S6 into the multi-head attention mechanism of the next encoder layer as the input feature of the next encoder layer, and then the next encoder layer is used as the current encoder layer to continue to return to step S6;

[0059] S8. Input the final result of the last encoder layer after processing in step S6 into a fully connected layer. The fully connected layer is used for dimensionality reduction and generation of a score vector. The generated score vector is then converted into the probabilities of various working state types through a multi-classification activation function to generate a predicted working state classification result. The score vector contains the scores of various working state types, but these scores are not normalized, that is, they do not constitute a probability distribution, and their values ​​can be any real number, which may be positive or negative, so they need to be normalized by a multi-classification activation function before they can be converted into the probabilities of various working state types. In this embodiment, the working state types include boiler slightly full of water, boiler severely full of water, boiler slightly short of water, boiler severely short of water, boiler dry burning, and normal.

[0060] Example 2. A boiler liquid level risk warning method based on CNN-Transformer, such as Figure 2 As shown, the steps are as follows:

[0061] S1. Extract the boiler operation data from the boiler operation history database, and obtain the actual fault type corresponding to each operation data as the actual fault classification result; divide the operation data into a training set and a validation set; the training set is used for model training, and the validation set is used for model validation; the operation data includes exhaust gas temperature, liquid level and steam pressure, and the exhaust gas temperature change value, liquid level change value and steam pressure change value are calculated from them. The calculation formula of the change value is,

[0062] v′=v t+1 -v t ,

[0063] v′ is the data change value, v t+1 is the data of the next moment, v t is the data value at this moment;

[0064] S2. Preprocess the running data, normalize the preprocessed data, and generate time series features; preprocessing includes data cleaning and data completion, and data completion specifically includes manually filling in missing values ​​or filling in average values; normalization helps to eliminate the adverse effects caused by singular sample data, thereby accelerating the convergence speed of the model. The calculation formula for normalization is as shown in S6;

[0065] S3. Input the time series feature into the convolution layer of the CNN network for a one-dimensional convolution operation to obtain the first feature. The calculation formula of the one-dimensional convolution operation is:

[0066]

[0067] y l is the output result of the one-dimensional convolution operation, l is the position where the convolution kernel slides, and x l-n is the lth data point of the input time series, w n is the nth weight of the convolution kernel, n is the index of the convolution kernel, and N is the width of the convolution kernel;

[0068] S4. Positionally encode the first feature through the word embedding layer of the Transformer model, and add the position-encoded feature to the first feature to obtain the second feature. The calculation formula for position encoding is:

[0069]

[0070] PE represents position encoding, pos represents position index, d represents the dimension of input, and i represents the dimension in the embedding vector;

[0071] S5. Input the second feature into the multi-head attention mechanism of the first encoder layer of the Transformer model. The calculation formula for the output of each attention head is,

[0072]

[0073] Attention(Q, K, V) is the output of each attention head, Q is the query vector, K is the T is the transpose of the key vector, V is the value vector, and D K is the dimension of the column vectors in vectors Q and K; to prevent the inner product from being too large, it is divided by D K The square root of the softmax function here is used to convert the dot product result of the query vector Q and the key vector K into a probability distribution, which will be used to perform a weighted summation on the value vector V;

[0074] S6. The output of each attention head in the multi-head attention mechanism of the current encoder layer is passed to the pooling layer of the CNN network for pooling operation, and the pooling results of all attention heads are concatenated and linearly transformed to obtain the features that integrate the information of all attention heads;

[0075] Add the features of the current encoder layer that fuse all the attention head information to the input features of the current encoder layer to perform the first residual connection processing;

[0076] Normalize the result after the first residual connection processing;

[0077] The normalized result is input into the feedforward neural network as the input feature of the feedforward neural network, and then the result processed by the feedforward neural network is output;

[0078] Then add the input features and output features of the feedforward neural network to perform a second residual connection process;

[0079] Normalize the result after the second residual connection processing;

[0080] The calculation formula for the pooling operation is,

[0081] TemporalPool=MaxPool(Conv1(Attention(Q,K,V))),

[0082] TemporalPool is a pooling operation, Conv1 is a one-dimensional convolution operation, MaxPool is a maximum pooling operation, and Attention (Q, K, V) is the output of each attention head in the multi-head attention mechanism;

[0083] The calculation formula for integrating the features of all attention head information is:

[0084] MultiHeadAttention(a)=concat(head1,...,head h )W O ,

[0085] head1,...,head h is the output of each attention head, calculated by Attention(Q, K, V), concat is the concatenation function used to connect the output of each attention head, W O is the weight matrix, MultiHeadAttention (a) is the feature that integrates all attention head information;

[0086] The calculation formula for the first residual connection processing is:

[0087] z=a+MultiHeadAttention(a),

[0088] a is the input feature of the current encoder layer, MultiHeadAttention(a) is the feature of the current encoder layer that integrates all attention head information, and z is the result after the first residual connection processing;

[0089] The calculation formula for normalizing the result after the first residual connection processing is:

[0090]

[0091] z is the result of the first residual connection processing, μ is the mean of z, σ is the standard deviation of z, γ and β are learnable parameters;

[0092] The feedforward neural network includes two fully connected layers. The calculation formula of the feedforward neural network is:

[0093] FNN(x)=ReLU(xW1+b1)W2+b2,

[0094] x is the input vector of the first linear transformation, W1 is the weight matrix of the first linear transformation, b1 is the bias vector of the first linear transformation, ReLU is a nonlinear activation function and its output is used as the input of the second linear transformation, W2 is the weight matrix of the second linear transformation, and b2 is the bias vector of the second linear transformation;

[0095] Perform a second residual connection process on the output of the feedforward neural network. The calculation formula refers to the calculation formula of the first residual connection process. The result after the second residual connection process is normalized and the calculation formula of the normalization process is also referred to.

[0096] S7, determine whether the current encoder layer is the last layer, if so, enter S8; then input the final result of the current encoder layer after processing in step S6 to the multi-head attention mechanism of the next encoder layer as the input feature of the next encoder layer, at this time, the final result of the current encoder layer after processing in step S6 is the result of the feedforward neural network output of the current encoder layer after the second residual connection processing and normalization processing, and then the next encoder layer is used as the current encoder layer to continue to return to execute step S6;

[0097] S8, inputting the final result of the last encoder layer after processing in step S6 into a fully connected layer, the fully connected layer is used for dimensionality reduction and generating a score vector, the generated score vector is then converted into the probabilities of various working state types through a multi-classification activation function, and a predicted working state classification result is generated;

[0098] At this time, the final result of the last encoder layer after the processing in step S6 is the result of the feedforward neural network output of the last encoder layer after the second residual connection processing and normalization processing;

[0099] The working status types include boiler slightly full of water, boiler severely full of water, boiler slightly short of water, boiler severely short of water, boiler dry burning, and normal; the multi-classification activation function uses the softmax function, and the calculation formula is,

[0100]

[0101] c jis the jth element in the input vector, It is c j The index of It is the sum of the indices of all elements in the input vector. This summation operation ensures that the sum of the output probabilities is 1.

[0102] S9. The predicted working state classification result and the actual working state classification result are calculated by the cross entropy loss function to obtain the corresponding loss degree, and the generated model is optimized by the optimizer; the optimizer adopts the adam optimizer, and the calculation formula of the cross entropy loss function is:

[0103]

[0104] H(p,q) is the loss value, p(g s ) is the probability vector of various actual working status types, g s represents the label of the sth category (0 or 1). In a multi-classification problem, only one is 1 and the rest are 0. s ) is the probability vector of various working status types predicted by the model, and r is the number of working status types;

[0105] S10. The validation set is input into the trained model to obtain the predicted working status classification result, and the prediction accuracy of the model is obtained based on the predicted working status classification result and the actual working status classification result.

[0106] The running data includes the changing values ​​of the corresponding parameters, which can improve the accuracy of model prediction.

[0107] In S1, 80% of the operating data is set as a training set, and 20% is set as a validation set; the operating data can not only be used for model training, but also for model accuracy prediction, thereby further ensuring the accuracy of the prediction results.

[0108] S9 is conducive to generating models with better performance, thus making the prediction results more accurate.

[0109] S6 sets residual connections to retain previous data features during the backward transmission of information, preventing network degradation; setting normalization processing can speed up the convergence of the model; setting a feedforward neural network enables the model to capture more complex and higher-level abstract features, enhancing the model's representation capabilities; adding the input features and output features of the feedforward neural network for a second residual connection process can prevent deep gradients from being difficult to backpropagate to shallow layers during model training.

[0110] Using a feedforward neural network to perform two linear transformations can extract features more deeply.

[0111] Setting up a fully connected layer can be used to map the learned features to the final output, to reduce the dimension, and to make the output of the fully connected layer match the category data.

Claims

1. A boiler liquid level risk warning method based on CNN-Transformer, characterized by , the steps are as follows: S1. extracting boiler operation data from the boiler operation history database; S2, preprocessing the operation data and generating time series features; S3, input the time series feature into the convolution layer of the CNN network for a one-dimensional convolution operation to obtain the first feature; S4. Position-encode the first feature through the word embedding layer of the Transformer model, and add the position-encoded feature to the first feature to obtain the second feature; S5. Input the second feature into the multi-head attention mechanism of the first layer encoder of the Transformer model; S6. The output of each attention head in the multi-head attention mechanism of the current layer encoder is passed to the pooling layer of the CNN network for pooling operation, and the pooling results of all attention heads are concatenated and linearly transformed to obtain the features that integrate the information of all attention heads; S7, determine whether the current encoder layer is the last layer, if so, enter S8; if not, input the final result of the current encoder layer after processing in step S6 into the multi-head attention mechanism of the next encoder layer as the input feature of the next encoder layer, and then the next encoder layer is used as the current encoder layer to continue to return to step S6; S8. Input the final result of the last encoder layer after processing in step S6 into a fully connected layer. The fully connected layer is used for dimensionality reduction and generating a score vector. The generated score vector is then converted into the probabilities of various working status types through a multi-classification activation function to generate a predicted working status classification result.

2. The boiler liquid level risk early warning method based on CNN-Transformer according to claim 1 is characterized by: It also includes S9 model optimization; the S1 also includes simultaneously obtaining the actual working state type corresponding to each operation data as the actual working state classification result; The S9 model optimization is to calculate the corresponding loss degree of the predicted working status classification result and the actual working status classification result through the cross entropy loss function, and the optimizer optimizes the generated model.

3. The boiler liquid level risk early warning method based on CNN-Transformer according to claim 2 is characterized by: It also includes S10 model verification; S1 also includes dividing the operating data into a training set and a verification set; the training set is used for model training in S2-S9; S10 model verification specifically includes inputting the verification set into the trained model to obtain the predicted working status classification result, and obtaining the prediction accuracy of the model based on the predicted working status classification result and the actual working status classification result.

4. The boiler liquid level risk early warning method based on CNN-Transformer according to claim 1 is characterized by: The S6 also includes, after obtaining the feature of the current encoder layer that fuses all the attention head information, adding the feature of the current encoder layer that fuses all the attention head information to the input feature of the current encoder layer to perform the first residual connection processing.

5. The boiler liquid level risk early warning method based on CNN-Transformer according to claim 4 is characterized by: The S6 also includes normalizing the result after the first residual connection processing.

6. The boiler liquid level risk early warning method based on CNN-Transformer according to claim 5 is characterized by: The S6 also includes inputting the normalized result into the feedforward neural network as an input feature of the feedforward neural network, and then outputting the result processed by the feedforward neural network.

7. The boiler liquid level risk early warning method based on CNN-Transformer according to claim 6 is characterized by: The S6 also includes adding the input features and output features of the feedforward neural network to perform a second residual connection process.

8. The boiler liquid level risk early warning method based on CNN-Transformer according to claim 6 is characterized by: The feedforward neural network includes two fully connected layers, and the calculation formula of the feedforward neural network is, FNN(x)=ReLU(xW1+b1)W2+b2, wherein x is the input vector of the first linear transformation, W1 is the weight matrix of the first linear transformation, b1 is the bias vector of the first linear transformation, ReLU is a nonlinear activation function and its output is used as the input of the second linear transformation, W2 is the weight matrix of the second linear transformation, and b2 is the bias vector of the second linear transformation.

9. The boiler liquid level risk early warning method based on CNN-Transformer according to claim 1 is characterized by: The operation data in S1 include exhaust gas temperature, liquid level and steam pressure; S1 also includes calculating exhaust gas temperature change value, liquid level change value and steam pressure change value according to the operation data.

Images