Electric energy metering device fault diagnosis method based on improved deep reinforcement learning

Abstract

The invention relates to the technical field of power system fault processing, in particular to an improved deep reinforcement learning-based electric energy metering device fault diagnosis method, which comprises the following steps of: acquiring corresponding operation data under various fault types, taking the operation data and an operation state as a sample set, and dividing the sample set into a test sample and a training sample; establishing an interaction strategy based on a classification Markov decision process, and establishing an agent model based on an attention mechanism, a one-dimensional convolutional neural network and a bidirectional conversion gating long-short term memory network; inputting a training sample into the agent model, wherein the agent model performs model training according to the interaction strategy; and inputting a test sample into the trained agent model to complete fault diagnosis of the electric energy metering device. According to the method, the fault diagnosis accuracy can be improved, the optimal diagnosis strategy is autonomously learned, and the network training and parameter updating process is optimized.

Description

Technical Field

[0001] The present invention relates to the technical field of power system fault processing, and particularly to a fault diagnosis method for an electric energy metering device based on improved deep reinforcement learning. Background Art

[0002] In recent years, due to the development of social economy, the construction of smart grids, and the continuous advancement of the social informatization process, the domestic electricity consumption has been increasing, and the requirements for the measurement accuracy and reliability of electric energy metering devices have also become higher and higher. The National Energy Administration clearly requires the promotion and application of new technologies such as communication technology, big data technology, and modern detection technology in electric energy metering devices, the establishment of an electric energy metering management information system, the improvement of the operation database of electric energy metering devices, and the online monitoring and dynamic analysis of the operation conditions through electric energy metering devices to achieve informatization and intelligent management.

[0003] In order to improve the operation management level of electric energy metering devices and ensure the safe and reliable operation of electric energy metering devices, metering professionals must conduct on-site fault inspections and tests on the electric energy metering system at regular intervals. However, there are several problems with traditional maintenance methods: (1) For the periodic inspection method, faults occurring within the period cannot be detected in time, and hidden fault hazards cannot be accurately detected, resulting in a series of metering errors; (2) For the on-site detection method of fault handling, only the error and the operation status of the metering device at that moment can be obtained each time, and the operation status and reliability inspection of the metering device under all working conditions cannot be comprehensively analyzed, and the detection result error is relatively large under low load conditions. Due to the complexity of the maintenance work environment, the limitations of monitoring devices, and the instability of manual maintenance, the maintenance results are also uncertain. Therefore, analyzing the real-time operation data of electric energy metering devices and accurately diagnosing their fault states is the future development trend.

[0004] In the process of scientific and technological development, fault detection and diagnosis based on power data have become the mainstream technology compared with the past. Nowadays, the monitoring system is becoming increasingly perfect, and machine learning has increasingly become the mainstream and efficient data processing algorithm in the data era. Machine learning does not require additional data information collection. By mining and processing the database information of the remote monitoring platform of the electric energy metering system, relevant information data can be extracted to achieve the detection of fault data and the diagnosis of fault types of metering devices. However, when there are many fault types and the data is relatively complex, the ability of the machine learning diagnosis method will decline, resulting in insufficient accuracy. Therefore, a fault diagnosis method for metering devices based on improved deep reinforcement learning is proposed. Summary of the Invention

[0005] The present invention provides a fault diagnosis method for an electric energy metering device based on improved deep reinforcement learning, which solves the problems such as low fault diagnosis accuracy in the prior art.

[0006] The technical solution of the present invention is as follows:

[0007] A fault diagnosis method for an electric energy metering device based on improved deep reinforcement learning, comprising:

[0008] S10: Collect the corresponding operation data under various fault types, use the operation data and operation status as a sample set, and divide the sample set into test samples and training samples;

[0009] S20: Establish an interaction strategy based on a classification Markov decision process, and establish an agent model based on an attention mechanism, a one-dimensional convolutional neural network, and a bidirectional transformational gated long short-term memory network;

[0010] S30: Input the training samples into the agent model, and the agent model performs model training according to the interaction strategy;

[0011] S40: Input the test samples into the trained agent model to complete the fault diagnosis of the electric energy metering device.

[0012] Furthermore, step S10 further includes:

[0013] wherein, the fault types include: metering device faults and communication faults;

[0014] The operation data includes: the phase voltage between phases A and B of the circuit, the phase current of phase A of the circuit, the voltage and current of the metering device, and the detection signal of the current transformer.

[0015] Furthermore, in step S20,

[0016] The classification Markov decision process includes: {S, A, R, P, γ}, and {S, A, R, P, γ} represents a sequential decision-making task;

[0017] When the training sample is D = {(x1, l1), (x2, l2),..., (x n , l n )}, wherein, x i represents the i-th sample, and l i represents the label corresponding to x i , and n represents the total number of samples;

[0018] Define {S, A, R, P, γ} as follows:

[0019] S represents the environmental state s t , which is determined by the sample x t in the training sample D, and the initial environmental state is x1;

[0020] A represents the action a of the agent model predicting the sample category t, associated with the labels in the training set D;

[0021] R represents the reward r t , when the agent model correctly predicts the sample category, it obtains a positive reward, otherwise it gets a penalty, which is specifically expressed as follows:

[0022]

[0023] P represents the transition from the current state s according to the sample order in D t to the next state s t+1 ;

[0024] γ represents the balance value, which is used to balance the current and future rewards;

[0025] Set the episode of the classification Markov decision process, and the episode contains one round of diagnosis process Episode = {s1, a1, r1, s2, a2, r2, …, s t , a t , r t}, t ≤ n, when the agent model makes a diagnostic error or t = n, the episode ends;

[0026] Set the diagnostic policy π with parameter θ θ , guiding the agent model to select the corresponding action a t when in state s t to complete the diagnostic behavior, and the agent model can autonomously learn the optimal diagnostic policy

[0027] Furthermore, the step S30 includes:

[0028] The agent model determines the current environmental state s from the training samples t ;

[0029] The agent model, based on the greedy algorithm, according to the current environmental state s t , executes the action a of predicting the sample category t , and obtains the reward r t ;

[0030] Complete the prediction of the current environmental state s t When the environment reaches the next state S t+1 , store the previous state reward r t , the action a t , the environmental state s t and the next state S t+1 into the experience pool;

[0031] The agent model calculates the target loss function and updates the parameters of the agent model using the gradient descent method.

[0032] Further, in step S20,

[0033] The agent model includes: an input hidden layer and an output hidden layer. Among them, the input hidden layer includes a one-dimensional convolutional layer, a pooling layer, and an attention layer connected in sequence. The input end of the first input hidden layer is connected to the input layer, the output end is connected to the input end of the next input hidden layer, and the output end of the last input hidden layer is connected to the output hidden layer;

[0034] The output hidden layer includes a bidirectional transform gated long short-term memory network layer, an attention layer, a flattening layer, and a fully connected layer connected in sequence. The output end of the output hidden layer is connected to the output layer.

[0035] Further, the one-dimensional convolutional layer is used for extracting local features of the running data, and the max pooling layer combines all the local features extracted by the one-dimensional convolutional layer to obtain depth features;

[0036] The one-dimensional convolutional layer u i has the following formula:

[0037]

[0038] where b is the bias value, σ represents the non-linear activation function of the convolutional calculation, the rectified linear unit, κ represents the filter, x i is the input quantity of the convolutional layer, represents the cross product operation.

[0039] Further, the final feature value is input into the output hidden layer, and the bidirectional transform gated long short-term memory network layer of the output hidden layer learns the sequence association pattern across long distances between the depth features;

[0040] The formula of the bidirectional transform gated long short-term memory network layer of the output hidden layer is as follows:

[0041]

[0042] φ t = 1 - tanh(f t )

[0043] c t = φ t ⊙ tanh(c t-1 ) + tanh(i t ) ⊙ g t

[0044] h t = o t ⊙ tanh(c t )

[0045] In the formula, it , f t and o t represent the input gate, forget gate, and output gate respectively; ω is the weight; b is the bias; x t , c t and h t-1 are the input value, memory cell state, and hidden state at time t-1 respectively; ⊙ is the element-wise multiplication; g t , φ t represent intermediate quantities; tanh is the activation function.

[0046] Furthermore, the agent model extracts the fault features of the learning metering device through a one-dimensional convolutional layer and a bidirectional transform gated long short-term memory network layer, and highlights the key information through an attention layer;

[0047] The formula of the attention layer is as follows:

[0048] e t = u tanh(ω t h t + b)

[0049]

[0050] In the formula, β t is the weight assigned by the attention layer; is the high-level data feature obtained by weighted summation of h t ; u is the output of the one-dimensional convolutional layer.

[0051] The beneficial effects of the present invention are as follows:

[0052] First, improve the accuracy of fault diagnosis

[0053] By dividing the operation data of the electric energy metering device into a training set and a test set, and using the improved deep reinforcement learning technology, especially building and optimizing the state space and action space of the CMDP, the constructed agent can achieve accurate classification of fault features. Compared with the traditional fault diagnosis method of metering devices, this feature classification method based on deep learning can more carefully mine and distinguish different types of fault features, reduce misjudgment and missed judgment, significantly improve the accuracy of fault diagnosis results, and provide a more reliable guarantee for the stable operation of the distribution network.

[0054] Second, achieve autonomous learning of the best diagnosis strategy

[0055] The interaction mechanism between the agent and the CMDP model environment is innovative. During the interaction process, the agent can autonomously learn the best diagnosis strategy based on the experience of the interaction between the constructed agent and the diagnostic simulation environment. In the classification Markov decision process, the agent receives an environmental state s t, it will select action a with probability P t , and obtain the maximum cumulative reward under the guidance of the policy . The agent makes decisions on actions to the environment through the policy, and obtains rewards and the next state in the environment. This means that the system does not need to rely on a large number of artificial rules and prior knowledge, but can continuously adjust and optimize its own diagnostic strategy according to the information feedback from the actual environment. This autonomous learning ability can better adapt to various complex situations and new fault modes that may occur during the operation of the distribution network, making the diagnostic strategy more adaptable and robust, and capable of dynamically adjusting with changes in the environment and equipment status, providing support for continuous and stable fault diagnosis.

[0056] III. Optimize the network training and parameter update process

[0057] Calculate the loss function using the Q values output by the main network and the target network in the agent, and update the network parameters using the gradient descent method. This process helps to improve the training efficiency and performance of the network model. Compared with some traditional training methods, this method can more effectively find the optimal parameters of the network, avoid falling into local optimal solutions, enable the network to better fit the data, and thus improve the overall performance of the metering device fault diagnosis.

[0058] IV. Promote the intelligent management of the distribution network

[0059] Input the test data into the trained network model to complete the fault diagnosis of the metering device, which helps to achieve the intelligent management of the distribution network. Accurate fault diagnosis results can provide timely and accurate information for the operation and maintenance personnel of the distribution network, enabling them to make advance equipment maintenance and repair plans, reduce the power outage time and power metering errors caused by metering device failures, improve the power supply reliability of the distribution network and the accuracy of power metering, and thus enhance the operation efficiency and management level of the entire distribution network system.

[0060] V. Provide data-driven decision-making basis

[0061] The present invention provides a scientific and objective decision-making basis for the fault diagnosis of distribution network metering devices based on a data-driven method. It is based on the analysis of operation data and the training of deep learning models, reducing the subjectivity of manual judgment, making the fault diagnosis decision more rational and scientific, helping to optimize the reasonable allocation and scheduling of distribution network resources, and improving the comprehensive management efficiency of the distribution network.

[0062] VI. Reduce labor costs and time costs

[0063] Since the agent can autonomously learn the optimal diagnosis strategy, it reduces the dependence on a large amount of manual experience and the need for professional technicians to observe and analyze faults for a long time. Therefore, it can effectively save labor costs. At the same time, through efficient network training and a rapid fault diagnosis process, the time for fault detection and handling is shortened, the timeliness of fault diagnosis is improved, and the economic losses caused by faults are reduced. In the long-term operation, it also helps to reduce the operation and maintenance costs of the distribution network.

[0064] These beneficial effects further highlight the advantages and application value of the present invention in the fault diagnosis of distribution network metering devices, providing strong support for its popularization and use in the actual distribution network technology field. You can adjust and refine the above beneficial effects according to specific needs to better reflect the characteristics and advantages of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] Figure 1 is a flowchart of the present invention.

[0066] Figure 2 is a schematic structural diagram of the agent model in the present invention.

[0067] Figure 3 is a schematic diagram of the bidirectional transformational gated long short-term memory network layer of the output hidden layer in the present invention. [[ID=‘19]]

[0068] Figure 4 is a flowchart of the training of the agent model in the present invention.

[0069] Figure 5 is a schematic diagram of the simulation of the present invention. [[ID=‘27]]DETAILED DESCRIPTION OF THE EMBODIMENTS

[0070] In order to enable those skilled in the art to better understand the solution of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. The described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0071] In the technical solution of the present invention, Figure 1 is a flowchart provided according to the specific process of a fault diagnosis method for an electric energy metering device based on improved deep reinforcement learning. As Figure 1 shown, the present invention includes:

[0072] S10: Collect the operation data corresponding to various fault types, use the operation data and operation status as a sample set, and divide the sample set into test samples and training samples.

[0073] Among them, the fault types include: metering device faults and communication faults;

[0074] The operating data includes: the phase voltage between phases A and B of the circuit, the phase current of phase A of the circuit, the voltage and current of the metering device, and the detection signal of the current transformer.

[0075] The electric energy metering device can be divided into parts such as electric energy meters, current transformers (CTs), voltage transformers (PTs), electric energy acquisition terminals, etc. Eight operating parameters closely related to the faults of the electric energy metering device are selected from numerous operating data, which are U AB 、I A 、I a 、I b 、U a 、U b 、u′ a 、u′ b . Among them, U AB represents the phase voltage between phases A and B, I A represents the phase current of phase A, I a 、I b 、U a 、U b represent the current and voltage of metering units 1 and 2 respectively, and u′ a 、u′ b represent the detection signals of the current transformers in the metering system respectively. The electric energy metering fault diagnosis is to analyze the above parameters, summarize different fault characteristics, and then obtain the potential laws between faults and data.

[0076] The reasons for the faults of the electric energy metering device are various. Combining with the actual situation, the fault types are roughly divided into two categories: metering devices and communication acquisition: 1) Metering device faults: ① Electric energy meter faults include: a) The meter is old and out of tolerance, the components are damaged, the sampling is inaccurate, the fault, and the stop; b) The voltage is interrupted and the voltage is lost; c) The current is lost, shunted, or short-circuited; d) The meter configuration is unreasonable or configured incorrectly (such as voltage 57.7V, 100V, 220V, 380V; current 0.3A, 1.5A, 5A, 10A). ② Secondary circuit faults: a) The secondary wiring is incorrect; b) The secondary wiring has poor contact, is loose, or is disconnected; c) The voltage drop is too large, the voltage wire is too long, the secondary load is too large, and the wire diameter is too thin; d) The secondary wiring polarity is reversely connected (national grid unified standard: based on the substation busbar and the user's power consumption as the reference); e) Relay faults, circuit breaker tripping, etc. cause phase loss. ③ Transformer faults: a) One-phase (multi-phase) faults or two-phase metering of CT and PT; b) The voltage comes from PT of other voltage levels.

[0077] Communication failure: ① Failure of the electric energy device: a) Freezing (long running time or poor quality causes the device to freeze, and a hardware restart or software restart is required, usually a hardware restart); b) Self-failure (failure of the power supply board, communication port, central processing unit, storage unit, etc.) requires replacing the device, or the version model is old and cannot meet the requirements, so the device needs to be replaced; c) Incorrect order or parameters of the point table, such as protocol, address, slot number, baud rate, parity bit, etc. ② Communication failure: a) Downlink communication failure, the 485 line is broken, misconnected or short-circuited; b) Uplink communication failure, the dispatching data network is not connected or the master station parameters are set incorrectly, and the encryption device fails.

[0078] S20: Establish an interaction strategy based on the classification Markov decision process, and establish an agent model based on the attention mechanism, one-dimensional convolutional neural network, and bidirectional transformational gated long short-term memory network.

[0079] The method of the present invention regards the fault diagnosis of the electric energy metering device as an identification game, and adopts a classification Markov decision process composed of elements {S, A, R, P, γ}.

[0080] The classification Markov decision process includes: {S, A, R, P, γ}, and {S, A, R, P, γ} represents a sequential decision-making task;

[0081] When the training samples are D = {(x1, l1), (x2, l2), …, (x n , l n )}, where x i represents the i-th sample, and l i represents the label corresponding to x i , and n represents the total number of samples;

[0082] Define {S, A, R, P, γ} as follows:

[0083] S represents the environmental state s t , which is determined by the sample x t in the training sample D. The initial environmental state is x1. Each time a new episode starts, the training set D is randomly shuffled.

[0084] A represents the action a t of the agent model to predict the sample category, which is associated with the label in the training set D. For a K-classification task, A = {0, 1, …, K - 1}. The classification task is the type of fault to be diagnosed, and the K classification tasks correspond to K types, which constitute the action set.

[0085] R represents the reward r t , when the agent model correctly predicts the sample category, it obtains a positive reward, otherwise it gets a penalty. The specific expression is as follows:

[0086]

[0087] Reward r t From environmental feedback, used to measure the performance of the agent model when in state s t when performing action a t and guide the agent to learn the best recognition strategy

[0088] P represents the transition from the current state s to the next state s t according to the sample order in D t+1 ;

[0089] γ represents the balance value, used to balance the current and future rewards. Given the small correlation between states, the agent pays more attention to the current immediate reward and is set to 0.1.

[0090] Set the episode of the classification Markov decision process, and the episode contains a round of diagnostic process Episode = {s1, a1, r1, s2, a2, r2, …, s t , a t , r t}, t ≤ n. When the agent model makes a diagnostic error or t = n, the episode ends;

[0091] Set the diagnostic policy π with network parameters θ θ to guide the agent model to select the corresponding action a t when in state s t to complete the diagnostic behavior, and the agent model can autonomously learn the best diagnostic strategy

[0092]

[0093] As Figure 2 shown, the agent model in the present invention includes: an input hidden layer and an output hidden layer. Among them, the input hidden layer includes a one-dimensional convolutional layer, a pooling layer, and an attention layer connected in sequence. The input end of the first input hidden layer is connected to the input layer, and the output end is connected to the input end of the next input hidden layer. The output end of the last input hidden layer is connected to the output hidden layer.

[0094] The output hidden layer includes a bidirectional transform gate-controlled long short-term memory network layer, an attention layer, a flattening layer, and a fully connected layer connected in sequence. The output end of the output hidden layer is connected to the output layer.

[0095] After inputting the data into the agent model through the input layer, the analysis result is calculated using the input hidden layer and the output hidden layer and output by the output layer.

[0096] In the present invention, the one-dimensional convolutional neural network is a CNN that obtains sequence data in the form of a one-dimensional grid for feature recognition. Although the one-dimensional convolutional neural network has only one dimension, it also has the advantage of translational invariance of the two-dimensional convolutional neural network in terms of feature recognition. Based on this, the proposed method constructs the metering feature data into sequence data with normal and fault labels, and first applies the one-dimensional convolutional neural network to achieve local feature extraction of the metering data. The one-dimensional convolutional neural network model realizes the function of local spatial feature extraction by stacking one-dimensional convolutional layers and pooling layers, and solves the problem of local feature loss.

[0097] The one-dimensional convolutional layer is the key to feature extraction. By training the metering data, a set of optimal convolutional kernels with the minimum loss is obtained, and the convolutional kernels (filters) are used to automatically extract the features of complex metering devices. The one-dimensional convolutional layer only performs convolution in the vertical direction of the metering feature data sequence. Therefore, the width of its convolutional kernel is the dimension of the metering data feature. A feature map is constructed by applying a convolution operation to the input data using the filter κ to achieve local spatial feature extraction.

[0098] The pooling layer further aggregates and retains the short-term features extracted by the convolutional layer to obtain the most important features. The commonly used pooling methods are max pooling and average pooling. The proposed method uses the max pooling layer to merge the maximum values of the feature vectors of each convolutional layer as the final feature value. After operating on the one-dimensional convolutional layer and the pooling layer, a 1×n-dimensional data feature is obtained, which can well analyze and retain the local features of the data sequence.

[0099] In summary, the one-dimensional convolutional layer is used for local feature extraction of the running data, and the max pooling layer merges all the local features extracted by the one-dimensional convolutional layer to obtain the deep features;

[0100] The one-dimensional convolutional layer u i has the following formula:

[0101]

[0102] where b is the bias value, σ represents the non-linear activation function of the convolution calculation, the rectified linear unit function, κ represents the filter, and x i is the input quantity of the convolutional layer, represents the cross product operation.

[0103] The one-dimensional convolutional neural network has limited ability in long-distance learning modeling, while the bidirectional long short-term memory network is mainly used to achieve long-distance sequence feature learning. Therefore, the deep features extracted by the one-dimensional convolutional neural network are input into the bidirectional long short-term memory network to further learn the sequence association patterns across long distances between the deep feature vectors. Since the long short-term memory network model is difficult to accurately capture the features of mutation information, the proposed method uses a conversion gated unit to improve it, obtaining a conversion gated long short-term memory network (CGLSTM) model, which can adaptively adjust the data weights to better learn the features of various samples.

[0104] The final feature values are input into the output hidden layer, and the bidirectional conversion gated long short-term memory network layer of the output hidden layer learns the sequence association patterns across long distances between the deep features;

[0105] As Figure 3 shown, the formula of the bidirectional conversion gated long short-term memory network layer of the output hidden layer is as follows:

[0106]

[0107] φ t = 1 - tanh(f t )

[0108] c t = φ t ⊙ tanh(c t-1 ) + tanh(i t ) ⊙ g t

[0109] h t = o t ⊙ tanh(c t )

[0110] In the formula, i t , f t and o t respectively represent the input gate, forget gate, and output gate; ω is the weight; b is the bias; x t , c t and h t-1 are respectively the input value, storage unit state, and hidden state at time t - 1; ⊙ is the element-wise multiplication; g t , φ t represent intermediate quantities; tanh is the activation function.

[0111] The present invention adds an effective attention layer to the model, which is attached to the end of each pooling layer of the one-dimensional convolutional neural network and the end of the bidirectional transform gate long short-term memory network. During the process of learning features, the weight of features related to categories is increased, making the model tend to pay attention to features that are more important for fault data detection.

[0112] For the one-dimensional convolutional neural network, the attention layer is attached to the end of the convolutional block to improve the situation where the convolutional neural network only focuses on local features and leads to inaccurate learning of global features. For the bidirectional transform gate long short-term memory network layer, the attention mechanism performs weighted summation on the output expression of its hidden layer vector, and the analysis effect is better. The attention mechanism replaces the original random weight assignment by assigning probabilities. The hidden layer vector h obtained from the convolutional block or the bidirectional transform gate long short-term memory network t is used as the input to the attention layer.

[0113] The intelligent agent model extracts and learns the fault features of the metering device through the one-dimensional convolutional layer and the bidirectional transform gate long short-term memory network layer, and highlights the key information through the attention layer.

[0114] The formula of the attention layer e t is as follows:

[0115] e t = utanh(ω t h t + b)

[0116]

[0117] In the formula, β t is the weight assigned by the attention layer; is the high-level data feature obtained by weighted summation of h t ; u is the output of the one-dimensional convolutional layer.

[0118] S30: Input the training samples into the intelligent agent model, and the intelligent agent model trains the model according to the interaction strategy.

[0119] Specifically, as Figure 4 shown,

[0120] The intelligent agent model determines the current environmental state s t from the training samples;

[0121] The intelligent agent model, based on the greedy algorithm, according to the current environmental state s t , executes the action a t for predicting the category of the sample, and obtains the reward r t ;

[0122] After completing the current environmental state s tWhen predicting that the environment reaches the next state S t+1 the reward r of the previous state t action a t environmental state s t and the next state S t+1 are stored in the experience pool;

[0123] The agent model calculates the target loss function and updates the parameters of the agent model using the gradient descent method.

[0124] The target network is a deep Q-network with fixed parameters for a period of time, which is used to generate stable Q-values to reduce overfitting during the learning process. The improved deep reinforcement learning network uses a dual-network structure, namely the main network and the target network, and optimizes the parameters by calculating the loss function. The target loss function is defined as the gap between the predicted Q-value and the target Q-value.

[0125] Network update: The main network and the target network in the agent adopt a joint attention mechanism and a 1DCNN-BiCGLSTM model, calculate the loss function through the Q-values output by the two networks, and update the network parameters using the gradient descent method to obtain the network corresponding to the optimal Q-value.

[0126] The improved deep reinforcement learning network uses a dual-network structure, namely the main network and the target network, and optimizes the parameters by calculating the loss function.

[0127] S40: Input the test samples into the trained agent model to complete the fault diagnosis of the electric energy metering device.

[0128] The verification of the present invention is as follows:

[0129] The classification results of 1120 samples are presented in the form of a confusion matrix, where normal samples are marked as 0, and the results are as Figure 5 shown. The fault types of the metering devices diagnosed by the proposed method are basically consistent with the actual types, and the accuracy rate reaches 98.66%. Since there are many normal samples and the features are obvious, the classification accuracy rate is 100%. At the same time, due to the certain similarity between different fault types, misdiagnosis may occur. However, overall, for different types of metering device faults, the proposed method can ensure relatively ideal diagnostic accuracy.

[0130] The agent constructed by the present invention can achieve accurate classification of fault features; it can autonomously learn the best diagnostic strategy based on the experience of the interaction between the constructed agent and the diagnostic simulation environment, so as to obtain accurate fault diagnosis results of the metering device.

[0131] Finally, it should be noted that the above specific embodiments are only used to illustrate the technical solutions of the present invention rather than to limit them. Although the present invention has been described in detail with reference to the examples, those of ordinary skill in the art should understand that the technical solutions of the present invention can be modified or equivalently replaced without departing from the spirit and scope of the technical solutions of the present invention, and they should all be covered within the scope of the claims of the present invention.

Claims

1. A fault diagnosis method for an electric energy metering device based on improved deep reinforcement learning, characterized in that, Including: S10: Collect the operation data corresponding to various fault types, use the operation data and operation status as a sample set, and divide the sample set into test samples and training samples; S20: Establish an interaction strategy based on the classification Markov decision process, and establish an agent model based on the attention mechanism, one-dimensional convolutional neural network, and bidirectional transform gated long short-term memory network; S30: Input the training samples into the agent model, and the agent model trains the model according to the interaction strategy; S40: Input the test samples into the trained agent model to complete the fault diagnosis of the electric energy metering device.

2. The fault diagnosis method for the electric energy metering device based on improved deep reinforcement learning according to claim 1, characterized in that, The step S10 also includes: Among them, the fault types include: metering device faults and communication faults; The operation data includes: the phase voltage between phases AB of the circuit, the phase current of phase A of the circuit, the voltage and current of the metering device, and the detection signal of the current transformer.

3. The fault diagnosis method for a metering device based on improved deep reinforcement learning according to claim 2, wherein, In the step S20, The classification Markov decision process includes: {S, A, R, P, γ}, and {S, A, R, P, γ} represents a sequential decision-making task; When the training samples are D = {(x1, l1), (x2, l2), …, (x n , l n )}, where x i represents the i-th sample, and l i represents the label corresponding to x i , and n represents the total number of samples; Define {S, A, R, P, γ} as follows: S represents the environmental state s t , which is determined by the sample x t in the training sample D. The initial environmental state is x1; A represents the action a of the predicted sample category of the agent model t , which is associated with the label in the training set D; R represents the reward r t , when the agent model correctly predicts the sample category, it obtains a positive reward, otherwise it gets a penalty, which is specifically expressed as follows: P represents a transition from the current state s according to the sample order in D t to the next state s t+1 ; γ represents the balance value, which is used to balance the current and future rewards; Set the episode of the categorical Markov decision process, where the episode contains one round of the diagnostic process Episode = {s1, a1, r1, s2, a2, r2, …, s t , a t , r t}, t ≤ n, and when the agent model makes a diagnostic error or t = n, the episode ends; Set the diagnostic strategy π with network parameters θ θ , guiding the agent model to select the corresponding action a t when in state s t to complete the diagnostic behavior, and the agent model can autonomously learn the optimal diagnostic strategy 4. The fault diagnosis method for a metering device based on improved deep reinforcement learning according to claim 3, wherein, The step S30 includes: The agent model determines the current environmental state s from the training samples t ; The agent model is based on the greedy algorithm and executes the action a for predicting the sample category according to the current environmental state s t , and obtains the reward r t ; t ; Complete the prediction of the current environmental state s t When the environment reaches the next state St+1, store the previous state reward r t , action a t , environmental state s t and the next state St+1 into the experience pool; The agent model calculates the target loss function and updates the parameters of the agent model using the gradient descent method.

5. The fault diagnosis method of the metering device based on improved deep reinforcement learning according to claim 1, characterized in that In the step S20, The agent model includes an input hidden layer and an output hidden layer. Among them, the input hidden layer includes a one-dimensional convolutional layer, a pooling layer, and an attention layer connected in sequence. The input end of the first input hidden layer is connected to the input layer, and the output end is connected to the input end of the next input hidden layer. The output end of the last input hidden layer is connected to the output hidden layer; The output hidden layer includes a bidirectional transform gated long short-term memory network layer, an attention layer, a flattening layer, and a fully connected layer connected in sequence. The output end of the output hidden layer is connected to the output layer.

6. The fault diagnosis method for a metering device based on improved deep reinforcement learning according to claim 5, characterized in that, The one-dimensional convolutional layer is used to extract the local features of the operation data, and the max pooling layer combines all the local features extracted by the one-dimensional convolutional layer to obtain the depth features; The one-dimensional convolutional layer u i has the following formula: where b is the bias value, σ represents the non-linear activation function of the convolution calculation, the rectified linear unit function, κ represents the filter, and x i is the input of the convolutional layer, represents the cross product operation.

7. The fault diagnosis method for a metering device based on improved deep reinforcement learning according to claim 6, characterized in that Input the final feature value into the output hidden layer, and the bidirectional transform gated long short-term memory network layer of the output hidden layer learns the sequence association pattern across long distances between the depth features; The formula of the bidirectional transform gated long short-term memory network layer of the output hidden layer is as follows: φ t = 1 - tanh(f t ) c t = φ t ⊙ tanh(c t-1 ) + tanh(i t ) ⊙ g t h t = o t ⊙tanh(c t ) where, i t , f t and o t represent the input gate, forget gate, and output gate respectively; ω is the weight; b is the bias; x t , c t and h t-1 are the input value, memory cell state, and hidden state at time t-1 respectively; ⊙ represents element-wise multiplication; g t , φ t represents intermediate quantities; tanh is the activation function.

8. The fault diagnosis method for a metering device based on improved deep reinforcement learning according to claim 7, characterized in that The agent model extracts and learns the fault features of the metering device through the one-dimensional convolutional layer and the bidirectional transform gated long short-term memory network layer, and highlights the key information through the attention layer; The attention layer e t is publicly announced as follows: e t = utanh(ω t h t + b) where β t assigns weights to the attention layer; is the high-level data feature obtained by weighted summation of h t ; u is the output of the one-dimensional convolutional layer.

Images