Deep learning method for pumping unit fault diagnosis based on improved adaptive activation function

Abstract

The present application relates to the technical field of oil and gas exploration and development, in particular to a deep learning pumping unit fault diagnosis method based on an improved adaptive activation function, the designed activation function combines learnable parameters and attention mechanisms, updates the learnable parameters through chain derivation, and forms global optimal search; the attention mechanism improves the learning ability of input features, forming local optimal search; combining local optimal search with global optimal search, the nonlinearity of the activation function is improved, and the defects of traditional activation functions are solved; the activation function of the present application is more flexible, has good generalization performance and search ability, and improves the accuracy of fault diagnosis.

Description

Technical Field

[0001] This invention relates to the field of oil and gas exploration and development technology, and specifically to a deep learning-based method for diagnosing pumping unit faults based on an improved adaptive activation function. Background Technology

[0002] Fault diagnosis during oil extraction using pumping units has always been an important research topic. Due to the complex downhole environment and the many unknown factors involved in the reciprocating motion of the sucker rod, pumping units are prone to malfunction, leading to safety hazards. Load (P) and displacement (S) are parameters generated during the up-and-down movement of the pumping unit's sucker rod; the closed curve formed by these parameters is the dynamometer diagram, which reflects in real time the impact of factors such as gas, oil, water, sand, and wax on the pumping unit's operating conditions. If the pump remains in a faulty state for an extended period, pump wear will increase, further reducing the equipment's service life.

[0003] Due to the complexity of working conditions, the shape of the dynamometer card varies under different operating states. Dynamometer cards of different types of faults share some similarities, resulting in samples that are difficult to distinguish. This necessitates specialized engineers to identify the specific operating state from the shape and outline of the dynamometer card. However, with the increasing complexity and cost of industrial systems, the tolerance for performance degradation, productivity decline, and safety hazards is decreasing. This urgently requires the early detection and identification of various potential anomalies and faults. Therefore, it is necessary to replace manual fault diagnosis of oil pumping units with computer-aided diagnosis.

[0004] With the rapid development of deep learning technology, deep learning models with better nonlinear approximation capabilities can automatically learn the potential features of faults, achieving better accuracy than traditional machine learning methods. Backpropagation (BP) neural networks, Fuzzy neural networks, VGG-16, AlexNet, and self-organizing neural networks have shown good performance in oil pumping unit fault diagnosis. However, as the number of network layers increases, model training becomes difficult due to vanishing and exploding gradients. To address these issues, and given the crucial role of activation functions in model training, researchers are increasingly reconstructing activation functions. For example, ReLU (Rectified Linear Unit) has low computational complexity, fast convergence, and can solve the problems of vanishing and saturation gradients. In recent years, many improved versions of ReLU have been developed. To address the DeadReLU phenomenon, LeakyReLU was proposed, replacing the negative portion of ReLU with a non-zero slope, making LeakyReLU more inclined to activate in the negative region.

[0005] In deep learning, the choice of activation function is generally determined based on the specific situation, and there is no fixed choice. Adaptive activation functions have seen widespread development because they can automatically adjust according to the network structure and the actual problem. The Parametrically Corrected Linear Unit (PReLU) is also used to address dead flow. The slope of the negative part can be learned from the data, rather than from a defined fixed value. Therefore, PReLU theoretically possesses all the advantages of ReLU and is more flexible than LeakyReLU. The Swish activation function was proposed in 2017, which is lower bounded, unbounded at the upper limit, and non-monotonic. Its first derivative is very smooth, and it outperforms ReLU in many aspects. In 2021, researchers proposed a new method that adds only a small number of parameters to traditional activation functions such as Sigmoid, Tanh, and ReLU to explore optimal activation functions with greater flexibility and adaptability. This method avoids local minima by introducing a small number of parameters into a fixed activation function.

[0006] For pumping unit fault diagnosis, the main approach relies on the contour features of the dynamometer card. However, since the dynamometer card consists of a large amount of point-to-point displacement and load data, which serves as input to the neural network model, the mapping between different layers in the model becomes complex, reducing the model's diagnostic accuracy. Furthermore, the sparse contour lines on the dynamometer card, composed of displacement and load data, further degrade diagnostic accuracy. Therefore, selecting an appropriate activation function is crucial for improving the accuracy and performance of pumping unit fault diagnosis. Summary of the Invention

[0007] The purpose of this invention is to overcome the shortcomings of existing technologies and propose a deep learning-based pumping unit fault diagnosis method based on an improved adaptive activation function. This method can better extract features from the contour lines of the indicator diagram, improve the accuracy of fault diagnosis, and has good search capabilities and generalization performance.

[0008] The technical solution adopted by this invention to solve its technical problem is:

[0009] A deep learning-based method for diagnosing pumping unit faults, based on an improved adaptive activation function, uses the pumping unit fault indicator diagram as the dataset and inputs it into the following model. The specific steps include:

[0010] S1. Design of Adaptive Activation Function

[0011] S101. Perform internal linearization, introducing several learnable parameters into the input. The internal linearization equation is as follows:

[0012] ψ=A i z+B i ,

[0013] Where ψ is the internal linearization equation, and Ai B i (i = 1, 2, ..., n) are the learnable parameters of the i-th layer, n is the number of layers in the neural network, and z is the weighted sum of the inputs to each neuron;

[0014] S102. In the positive region, the activation function formula is as follows:

[0015] C i * Max(ψ,0)+D i , ψ≥0,

[0016] Where C i and D i These are the learnable parameters of the i-th layer. The learnable parameters are added to the positive region of the activation function and externally linearized.

[0017] S103. In the negative region, the slope of the proposed adaptive activation function is generated by the attention mechanism, as shown in the following formula:

[0018] Scales*Min(ψ,0),ψ<0,

[0019] Among them, Scales is the slope obtained through an attention mechanism;

[0020] The activation function is as follows:

[0021]

[0022] S2. Update pattern of learnable parameters

[0023] The output of a neural network with a single activation function is as follows:

[0024] =f(z),

[0025] Where y is the output of the neural network, and the formula for the loss function L(y, d) is as follows:

[0026]

[0027] Where, d j Let j be a real number, j = 1, 2, ..., m, where m is the sample size;

[0028] The chain derivation process of the loss function is as follows:

[0029]

[0030]

[0031] The weights are updated as follows:

[0032]

[0033] Where η is the learning rate, which affects the update speed of the loss function; substituting the chain derivation formula of the loss function into the above equation, we get:

[0034]

[0035] Based on the chain-like derivation of the loss function and the overall formula of the activation function, the update rules of the parameters in the adaptive activation function are derived as follows:

[0036] y = f(ψ)

[0037]

[0038]

[0039]

[0040] From the above formula, we can derive:

[0041]

[0042] S3, Attention Mechanism Structure

[0043] ReLU is chosen as the activation function for the first fully connected layer, as shown in the following formula:

[0044]

[0045] Then a BN layer is added; normalization is described by the following formula:

[0046]

[0047]

[0048]

[0049]

[0050] Where x i yi and yi are each N batch The observed inputs and outputs of a batch, where μ represents the average value of the inputs, and σ represents the average value of the outputs. 2 ε represents the variance of the input, θ is a constant close to zero, γ and θ are learnable parameters that control scaling and shifting the distribution, and i represents the number of layers.

[0051] The above methods and steps can effectively represent the mapping relationship between displacement and load in the dynamometer diagram, extract the features of the dynamometer diagram, correctly classify difficult-to-distinguish samples, and complete the fault diagnosis of the oil pumping unit.

[0052] Furthermore, in step S101, the weighted sum z of the inputs to each neuron is defined as follows:

[0053] z = wx + b

[0054] Where w is the chain weight matrix and b is the deviation term.

[0055] Furthermore, the gradient descent method employs backpropagation, the backpropagation process of which is as follows:

[0056] After each batch of training data is fed into the model, the predicted value is output through forward computation, and the loss function is calculated based on the predicted value and the true value.

[0057] Then, the parameters are updated through chain derivation and backpropagation to reduce the value of the loss function until the predicted value generated by the model is close to the true value.

[0058] Furthermore, the activation function of the second-layer FC is Sigmoid.

[0059] Furthermore, in the structure of the attention mechanism, the input of the subnet is a one-dimensional vector obtained by global average pooling of three inputs; these three inputs are the input features, the separated positive features, and the separated negative features.

[0060] The activation function of this invention combines learnable parameters with an attention mechanism, updating the learnable parameters through chain-like differentiation to form a global optimal search. The attention mechanism improves the learning ability of input features, forming a local optimal search. Combining local and global optimal searches improves the nonlinearity of the activation function and solves the defects of traditional activation functions. The activation function of this invention is more flexible and versatile, with better generalization performance and search ability, thus improving the accuracy of fault diagnosis.

[0061] Technical effects of the present invention:

[0062] Compared with existing technologies, the deep learning-based oil pumping unit fault diagnosis method of the present invention, based on an improved adaptive activation function, has the following advantages:

[0063] (1) This invention proposes an improved adaptive activation function that can achieve local search optimization and global search optimization, and has strong adaptability to different deep learning models;

[0064] (2) This invention applies the proposed activation function to pumping unit fault diagnosis to better extract features from the contour lines of the indicator diagram. This activation function improves the accuracy of fault diagnosis and has good search capability and generalization performance. The proposed activation function is extended to the public datasets CIFAR10 and CIFAR100 to verify its applicability and universality. Attached Figure Description

[0065] Figure 1 This is a diagram showing the overall structure of the adaptive activation function of this invention;

[0066] Figure 2 This is a structural diagram of the five networks in this invention;

[0067] Figure 3 This is a confusion matrix diagram of five models in the oil pumping unit fault dataset of this invention;

[0068] Figure 4 This is a graph of the loss function of the present invention;

[0069] Figure 5 This is a graph showing the variation of learnable parameters in the ResNet layer of this invention. Detailed Implementation

[0070] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings.

[0071] Example:

[0072] This embodiment relates to a deep learning-based oil pumping unit fault diagnosis method based on an improved adaptive activation function, including...

[0073] S1. Design of Adaptive Activation Function

[0074] S101. Perform internal linearization, introducing several learnable parameters into the input. The internal linearization equation is as follows:

[0075] ψ=A i z+B i ,

[0076] Where ψ is the internal linearization equation, and A i B i (i = 1, 2, ..., n) are the learnable parameters of the i-th layer, n is the number of layers in the neural network, and z is the weighted sum of the inputs to each neuron; the above settings have been internally linearized, which expands the generalization space of the activation function.

[0077] S102. In the positive region, the activation function formula is as follows:

[0078] C i * Max(ψ,0)+D i , ψ≥0,

[0079] Where C i and D iThese are the learnable parameters of the i-th layer. The learnable parameters are added to the positive region of the activation function and then externally linearized.

[0080] S103. In the negative region, the slope of the proposed adaptive activation function is generated by the attention mechanism. The formula is as follows:

[0081] Scales*Min(ψ,0),ψ<0,

[0082] Scales are slopes obtained through an attention mechanism. They can be adapted to different tasks based on different input data.

[0083] The activation function is as follows:

[0084]

[0085] Overall structure as Figure 1 As shown.

[0086] Furthermore, in step S101, the weighted sum z of the inputs to each neuron is defined as follows:

[0087] z = wx + b

[0088] Where w is the chain weight matrix and b is the deviation term.

[0089] Using the chain weight matrix w and the learnable parameters A i B i C i D i The mutual adjustment of the inner and outer linear equations can improve the learning speed and classification accuracy of the neural network. Furthermore, the inner linear equations have their own intercept, allowing for adjustments to the parameters in the vertical direction during training, thus avoiding local minima. Similarly, the outer linear equations have the same effect on improving performance.

[0090] Furthermore, gradient descent can perform backpropagation to continuously update parameters, thereby optimizing the network's performance during the training phase. The backpropagation process of this invention is as follows:

[0091] After each batch of training data is fed into the model, the predicted value is output through forward computation, and the loss function is calculated based on the predicted value and the true value.

[0092] Then, the parameters are updated through chain derivation and backpropagation to reduce the value of the loss function until the predicted value generated by the model is close to the true value.

[0093] Furthermore, this embodiment takes a single-layer activation function as an example. The output of the neural network using the traditional activation function is as follows:

[0094] v = f(z),

[0095] Where y is the output of the neural network, and the formula for the loss function L(y,d) is as follows:

[0096]

[0097] Where d j Let j be a real number, j = 1, 2, ..., m, where m is the number of samples.

[0098] The chain derivation process of the loss function is as follows:

[0099]

[0100]

[0101] The weights are updated as follows:

[0102]

[0103] Where η is the learning rate, which affects the update speed of the loss function. Substituting the chain derivation formula of the loss function into the above equation, we get:

[0104]

[0105] Based on the chain-like derivation of the loss function and the overall formula of the activation function, the update rules of the parameters in the adaptive activation function are derived as follows:

[0106] y = f(ψ)

[0107]

[0108]

[0109]

[0110] From the above formula, we can derive:

[0111]

[0112] Similar to A i The learnable parameter B can be derived. i C i D i The update pattern is determined. Therefore, this embodiment adjusts the chain weight matrix w, learnable parameters, and scale, thereby avoiding local minima. Neural networks can better adapt to sample data, extract key features of samples, and improve classification accuracy.

[0113] Furthermore, such as Figure 1As shown, the input of the subnet in this invention is a concatenation of three one-dimensional vectors obtained by global average pooling (GAP). These three inputs are the input features, the separated positive features, and the separated negative features. The separation of positive and negative features highlights key features, while the input features ensure that key information is not lost.

[0114] GAP can replace fully connected (FC) layers to achieve dimensionality reduction. In particular, GAP preserves the spatial information extracted by previous convolutional and pooling layers and strengthens the relationship between categories and feature maps. The computation path is as follows: FC → Batch nomoation (BN) → ReLU → FC → BN → Sigmoid → Scales.

[0115] To reduce computational burden and improve training speed, the number of neurons in the fully connected (FC) layer is equal to 1 / 4 of the number of channels in the input feature map. Simultaneously, ReLU is chosen as the activation function for the first FC layer to reduce computational complexity and keep gradient values ​​within a reasonable range for feature extraction. The formula is as follows:

[0116]

[0117] Then, a Batch Normalization (BN) layer is added. This is a method for unifying distributed data, similar to regular data normalization. It's also a method for optimizing neural networks. Normalized data makes it easier for deep learning models to learn rules within the data and also solves the gradient vanishing problem. Normalization is described by the following formula:

[0118]

[0119]

[0120]

[0121]

[0122] Where x i and y i Each N batch The observed inputs and outputs of a batch, where μ represents the average value of the inputs, and σ represents the average value of the outputs. 2 ε represents the variance of the input, θ is a constant close to zero, γ and θ are learnable parameters that control scaling and shifting the distribution, and i represents the number of layers.

[0123] The activation function of the second fully connected layer is Sigmoid, which can limit the output value to the interval (0, 1) to prevent excessive slope from affecting the performance of the activation function.

[0124] The adaptive activation function proposed in this invention has the ability to automatically learn complex features and performs different nonlinear transformations on different inputs, thereby improving the generalization performance of deep learning models and effectively solving the problems of feature contour extraction and sparsity of indicator maps in pumping unit fault diagnosis.

[0125] Experimental example:

[0126] The effectiveness of the designed adaptive activation function is verified through experimental simulation.

[0127] This embodiment was tested on AlexNet, VGG-16, GoogleNet, ResNet, and DenseNet. The architecture diagrams of these five networks are shown below. Figure 2 As shown, it is compared with traditional activation functions such as ReLU, Sigmoid, Tanh, LReLU and PReLU.

[0128] The experiment mainly simulates the fault diagnosis dataset of the pumping unit to prove that the proposed adaptive activation function can extract the features of the indicator diagram and solve the sparsity problem of the indicator diagram; the improvement in fault diagnosis accuracy indicates that the samples that are difficult to distinguish were correctly classified.

[0129] Oil pumping unit dataset

[0130] This pumping unit fault dataset originates from real data generated during the operation of pumping units in an oilfield in Northeast China. The dataset consists of indicator diagrams for nine types of faults: insufficient fluid supply, sand production, piston jamming, gas interference, pump bulge, dual-valve leakage, normal operation, pump bulge, and rod failure. This embodiment employs the stochastic gradient descent (SGD) method with an initial learning rate of 0.001. Momentum is used to accelerate the optimization process, with a momentum ratio of 0.9. Training is performed for at least 200 epochs. The average accuracy of each model is shown in Table 1. The ResNet model outperforms other models in the pumping unit fault diagnosis dataset. Compared to traditional activation functions ReLU, Tanh, Sigmoid, LeakReLU, and PReLU, the ResNet model using this activation function achieves average accuracy improvements of 12.81%, 5.09%, 5.72%, 0.05%, and 11.6%, respectively.

[0131] A confusion matrix is ​​a commonly used metric and visualization tool for evaluating the results of classification models. It can determine the strengths and weaknesses of a classifier. The rows of the matrix represent the true values, and the columns represent the predicted values. The confusion matrix can calculate the number of misclassified and correctly classified cases separately, and then display the results in a single matrix. Figure 3The confusion matrices for the five models in the pumping unit fault dataset show that the designed adaptive activation function effectively represents the mapping relationship between displacement and load in the dynamometer diagram, extracts features from the dynamometer diagram, and thus correctly classifies samples that are difficult to distinguish. Table 2 shows the accuracy of various fault types for the five models, proving that the proposed adaptive activation function can be well applied to the five models and has strong adaptability to the models.

[0132] Table 1. Classification accuracy of various activation functions for different models in oil pumping unit fault diagnosis.

[0133] Methods AlexNet (%) VGG-16 (%) GoogleNet (%) ResNet(%) DenseNet (%) Ours 97.14±0.7737 96.55±0.4176 94.81±0.6254 95.29±0.3292 96.41±0.4706 ReLU 96.21±0.3944 96.41±0.5183 94.66±0.9948 82.48±0.9877 95.68±0.3220 Sigmoid 95.24±0.9030 96.26±0.3633 96.17±0.2831 89.56±0.8547 95.29±0.2475 Tanh 9626±0.4998 96.89±0.2378 95.39±0.1535 90.19±0.7917 95.10±0.3567 LReLU 96.55±0.3883 96.60±0.4342 94.66±0.8267 95.24±0.4502 95.44±0.3883 PReLU 96.26±0.2475 96.55±0.3220 94.47±0.6933 84.22±0.1189 95.97±0.2475

[0134] Table 2. Diagnostic accuracy of five models for each type of fault in the oil pumping unit.

[0135]

[0136] The loss function curve is as follows Figure 4 As shown, (A) AlexNet, (B) VGG-16, and (C) GoogleNet converge faster. (D) ResNet and (E) DenseNet have relatively slower loss function decreases, but eventually converge to the optimal value. In the oil pumping unit fault dataset, the proposed adaptive activation function has better accuracy than the traditional five-network activation function. In the designed adaptive activation function, as the loss function decreases, the learnable parameters are continuously updated, and they adjust with the parameters in the network structure to make the loss function tend to the optimal solution. For example, the parameter change curve of the ResNet layer is shown in the figure. Figure 5 As shown, the learnable parameters have different value ranges and tend to certain constants as training time increases. The results indicate that the ResNet model using this adaptive activation function has strong input adaptability and can extract input features.

[0137] This invention designs a novel adaptive activation function and applies it to five neural network models. Specifically, each layer of the neural network embeds an activation function with learnable parameters and an attention mechanism, thus giving each layer a unique nonlinear transformation in its input signal. Compared to traditional fixed activation functions, the activation function of this invention has better nonlinear transformation capabilities and can be well embedded into these five neural network models.

[0138] The activation function of this invention can be demonstrated using a fault diagnosis dataset of an oil pumping unit to effectively display the mapping relationship between displacement and load in the indicator map, thereby extracting features from the indicator map, solving the sparsity problem of the indicator map, and correctly classifying indistinguishable samples. The adaptive activation function proposed in this invention improves the accuracy of fault diagnosis and has good generalization performance and search capability. In addition, the proposed adaptive activation function can also be well embedded into other neural network models.

[0139] The above-described specific embodiments are merely specific examples of the present invention. The patent protection scope of the present invention includes, but is not limited to, the above-described specific embodiments. Any appropriate changes or modifications made by a person skilled in the art that conform to the claims of the present invention should fall within the patent protection scope of the present invention.

Claims

1. A deep learning-based fault diagnosis method for oil pumping units based on an improved adaptive activation function, characterized in that, Using the pumping unit fault indicator diagram as the dataset, input it into the following model. The specific steps include: S1. Design of Adaptive Activation Function S101. Perform internal linearization, introducing several learnable parameters into the input. The internal linearization equation is as follows: ψ=A i z+B i , Where ψ is the internal linearization equation, and A i B i (i = 1, 2, ..., n) are the learnable parameters of the i-th layer, n is the number of layers in the neural network, and z is the weighted sum of the inputs to each neuron; S102. In the positive region, the activation function formula is as follows: C i *Max(ψ,0)+D i ,ψ≥0, Where C i and D i These are the learnable parameters of the i-th layer. The learnable parameters are added to the positive region of the activation function and externally linearized. S103. In the negative region, the slope of the proposed adaptive activation function is generated by the attention mechanism, as shown in the following formula: Scales*Min(ψ,0),ψ<0, Among them, Scales is the slope obtained through an attention mechanism; The activation function is as follows: S2. Update pattern of learnable parameters The output of a neural network with a single activation function is as follows: y = f(z), Where y is the output of the neural network, and the formula for the loss function L(y,d) is as follows: Where, d j Let j be a real number, j = 1, 2, ..., m, where m is the sample size; The chain derivation process of the loss function is as follows: The weights are updated as follows: Where η is the learning rate, which affects the update speed of the loss function; substituting the chain derivation formula of the loss function into the above equation, we get: Based on the chain-like derivation of the loss function and the overall formula of the activation function, the update rules of the parameters in the adaptive activation function are derived as follows: y = f(ψ) From the above formula, we can derive: S3, Attention Mechanism Structure ReLU is chosen as the activation function for the first fully connected layer, as shown in the following formula: Then a BN layer is added; normalization is described by the following formula: Where x i and y i Each N batch The observed inputs and outputs of a batch, where μ represents the average value of the inputs, and σ represents the average value of the outputs. 2 ε represents the variance of the input, θ is a constant close to zero, γ and θ are learnable parameters that control scaling and shifting the distribution, and i represents the number of layers. The above methods and steps can effectively represent the mapping relationship between displacement and load in the dynamometer diagram, extract the features of the dynamometer diagram, correctly classify difficult-to-distinguish samples, and complete the fault diagnosis of the oil pumping unit.

2. The deep learning-based oil pumping unit fault diagnosis method based on an improved adaptive activation function according to claim 1, characterized in that: In step S101, the weighted sum z of the inputs to each neuron is defined as follows: z = wx + b Where w is the chain weight matrix and b is the deviation term.

3. The deep learning-based oil pumping unit fault diagnosis method based on an improved adaptive activation function according to claim 1, characterized in that: The backpropagation process is as follows: After each batch of training data is fed into the model, the predicted value is output through forward computation, and the loss function is calculated based on the predicted value and the true value. Then, the parameters are updated through chain derivation and backpropagation to reduce the value of the loss function until the predicted value generated by the model is close to the true value.

4. The deep learning-based oil pumping unit fault diagnosis method based on an improved adaptive activation function according to claim 1, characterized in that: The activation function for the second fully connected layer is Sigmoid.

5. The deep learning-based oil pumping unit fault diagnosis method based on an improved adaptive activation function according to claim 1, characterized in that: Note that in the structure of the mechanism, the input of the subnet is a one-dimensional vector obtained by global average pooling of three inputs; these three inputs are the input features, the separated positive features, and the separated negative features.

Images