Power grid fault diagnosis method, system and device based on prior knowledge reinforcement learning

Abstract

The present invention relates to the technical fields of reinforcement learning and power grid fault diagnosis, and specifically to a power grid fault diagnosis method, system and device based on prior knowledge reinforcement learning, which obtains alarm data texts of power grid faults under different environmental states, firstly performs text matching of key features on the alarm data texts and key feature rules in prior knowledge, and directly outputs actions if the match is successful, and converts the alarm data texts that fail to match into embedded vectors after embedding processing, and further processes the embedded vectors to obtain expected values ​​of different actions under the current environmental state, and then obtains the action with the highest estimated expected value and the corresponding power grid fault type under the current environmental state based on switch features, switch state features and estimated expected values ​​of different actions, and introduces experience playback to update the estimated expected value, thereby effectively improving the effectiveness and accuracy of power grid fault diagnosis.

Description

Technical Field

[0001] The present invention relates to the technical field of power grid fault diagnosis, and in particular to a power grid fault diagnosis method, system and device based on prior knowledge reinforcement learning. Background Art

[0002] As the scale of power grids gradually expands and the characteristics of power grid faults become increasingly complex, higher requirements are placed on power grid fault diagnosis and safe and stable operation of power grids. Once a power grid fault occurs in a certain area, if the fault type cannot be determined in time and corresponding isolation and remedial measures cannot be taken, it is likely to cause a large-scale power outage, endangering the safe operation of the entire power grid system and causing significant losses to the national economy.

[0003] Power grid fault diagnosis relies on analyzing the electrical and switch fault information obtained by the monitoring system to determine the area where the fault occurs. As the basis for diagnosis, power grid text alarm information has obvious advantages in power grid fault diagnosis. The fault diagnosis process can be converted into a model solving problem, such as using a Bayesian network model, but it is impossible to perform fault diagnosis on massive alarm information.

[0004] With the rapid development of deep learning technology, more options are provided for power grid fault diagnosis, which can process large amounts of data. Among them, deep reinforcement learning can achieve end-to-end learning from raw data to target output, and is an effective method for intelligent fault diagnosis. Although some research works on the application of deep reinforcement learning to power grid fault diagnosis have achieved good results, there are still some shortcomings.

[0005] At present, most of the research on power grid fault diagnosis using deep reinforcement learning only focuses on the information of each fault, but ignores the semantic and logical relationship between the text of power grid fault alarm information, which often makes it difficult to extract complete and key feature information; at the same time, when faced with unbalanced sample data, it still faces problems such as low feature extraction efficiency, slow model convergence and low diagnostic accuracy. Summary of the invention

[0006] The present invention provides a power grid fault diagnosis method based on prior knowledge reinforcement learning.

[0007] The technical solution of the present invention is as follows:

[0008] A power grid fault diagnosis method based on prior knowledge reinforcement learning includes the following steps:

[0009] S1. Obtain alarm data texts of power grid faults under different environmental states, construct a data set, extract switch features and key features based on the alarm data texts, and match the key features of the texts with key feature rules in prior knowledge, wherein the prior knowledge includes a set number of switch state features and a set number of key feature rules.

[0010] If the matching probability value with any key feature rule is greater than the probability threshold, the match is successful, and the corresponding action and power grid fault type are obtained; if the match is unsuccessful, S2 is executed;

[0011] S2, embedding the alarm data text that failed to match, obtaining the embedding vector under the current environment state, sequentially performing multi-head attention mechanism processing, residual connection processing with the embedding vector, normalization processing and nonlinear processing on the embedding vector to obtain deep features, performing full connection processing on the deep features to obtain the estimated expected values ​​of different actions under the environment state;

[0012] S3. Based on the switch characteristics, switch state characteristics, and estimated expected values ​​of different actions, the action with the highest estimated expected value and the corresponding power grid fault type under the current environmental state are obtained, specifically:

[0013] Construct an action space based on all actions, and construct different action subspaces based on different switch state characteristics. Each action subspace includes a specific number of actions as an action exploration space.

[0014] The extracted switch feature is matched with any switch state feature. If there is a successfully matched switch state feature, an action is randomly selected for exploration in its corresponding action exploration space based on the exploration probability, or the action with the largest estimated expected value is selected for use based on 1-exploration probability to obtain the corresponding power grid fault type;

[0015] Otherwise, explore or exploit in the action space.

[0016] Furthermore, S3 also includes obtaining feedback rewards for the actions obtained after exploration or utilization based on the set reward function, transferring to the next environment state, building an experience group based on the current environment state, each action obtained under the current environment state and the corresponding feedback rewards, and the next environment state and storing it in the experience buffer pool, setting sampling probabilities for several experience groups, and performing experience playback processing based on the sampling probabilities, and updating the estimated expected value until the set round is reached;

[0017] Sampling probabilities are set for several experience groups, and experience playback is performed based on the sampling probabilities to update the estimated expected value, specifically:

[0018] Based on the feedback reward, discount factor, and the maximum expected value of the next environmental state, the target expected value is obtained; based on the difference between the estimated expected value and the target expected value and the set priority, the sampling probability of each experience group is obtained, and based on the sampling probability, several experience groups are randomly selected from the experience buffer pool for experience playback processing to update the estimated expected value.

[0019] Based on the difference between the estimated expected value and the target expected value and the set priority, the sampling probability of each experience group is obtained, and the formula is as follows:

[0020] ,

[0021] in, is the difference between the estimated expected value and the target expected value, is the target expected value, To estimate the expected value, is the current environment status, For the current action, To update the parameters of the estimated expected value; For feedback rewards, is the discount factor, is the expected value of the next environmental state, is the next environment state, For the next action, is the parameter to update the target expected value;

[0022] ,

[0023] in, is the sampling probability for each experience group, For the i The probability of an experience group, The priority set.

[0024] The reward function for the described setting Specifically:

[0025] ,

[0026] in, t is the time step, For the time step t The current state of the environment, For the time step t The current action, For the time step t The fault sample label under the current environment state, For most fault samples, ρ is the ratio of majority class fault samples to minority class fault samples, DF It is a minority class fault sample;

[0027] When the majority class fault samples are classified correctly and incorrectly, the reward values ​​are 1 and -1 respectively; when the minority class fault samples are classified correctly and incorrectly, the reward values ​​are ρ 、- ρ .

[0028] The method randomly extracts several experience groups from the experience buffer pool based on the sampling probability to perform experience playback processing, and calculates the loss function based on the estimated expected value and the target expected value for the randomly extracted several experience groups. , the formula is as follows:

[0029] ,

[0030] in, N is the number of randomly selected experience groups, i is the serial number, is the target expected value, To estimate the expected value, is the current environment status, For the current action, Parameters for updating the estimated expected value.

[0031] The parameters of the target expected value are updated using the gradient descent method based on the loss function, and the parameters of the estimated expected value are updated to the parameters of the target expected value at every specific time step.

[0032] The priority is set based on the amount of fault data in the alarm data text in the data set, and the setting priority is inversely proportional to the amount of fault data in the alarm data text in the data set.

[0033] The exploration probability in S3 is set differently based on the amount of data occurring in different fault types in the data set. The exploration probability is set proportional to the amount of data occurring in different fault types in the data set. At the same time, the exploration probability is set to decrease at a specific decay rate in each round until it decreases to a minimum exploration probability value.

[0034] In the S2, the alarm data text that fails to match is embedded to obtain an embedding vector under the current environment state. Specifically, the alarm data text is subjected to word embedding processing, sentence embedding processing and position embedding processing respectively to obtain a word embedding vector, a sentence embedding vector and a position embedding vector respectively. The word embedding vector, sentence embedding vector and position embedding vector are concatenated to obtain an embedding vector.

[0035] The present invention also provides a power grid fault diagnosis system based on prior knowledge reinforcement learning, comprising:

[0036] Key feature matching module: used to obtain the alarm data text of power grid faults under different environmental states, build a data set, extract switch features and key features based on the alarm data text, and match the key features of the text with the key feature rules in the prior knowledge. The prior knowledge includes a set number of switch state features and a set number of key feature rules. If the matching probability value with any key feature rule is greater than the probability threshold, the match is successful, and the corresponding action and power grid fault type are obtained; otherwise, the match is unsuccessful;

[0037] Feature extraction module: used to embed the alarm data text that fails to match, obtain the embedding vector under the current environment state, and sequentially perform multi-head attention mechanism processing, residual connection processing with the embedding vector, normalization processing and nonlinear processing on the embedded vector to obtain deep features, and perform full connection processing on the deep features to obtain the estimated expected values ​​of different actions under the environment state;

[0038] Fault diagnosis module: used to obtain the action with the highest estimated expected value and the corresponding power grid fault type under the current environmental state based on switch features, switch state features, and estimated expected values ​​of different actions. Specifically, an action space is constructed based on all actions, and different action subspaces are constructed based on different switch state features. Each action subspace includes a specific number of actions as an action exploration space. The extracted switch features are matched with any switch state features. If there is a successfully matched switch state feature, an action is randomly selected in the corresponding action exploration space based on the exploration probability for exploration, or the action with the largest estimated expected value is selected based on 1-exploration probability for utilization to obtain the corresponding power grid fault type. Otherwise, exploration or utilization is performed in the action space.

[0039] In addition, the present invention provides a power grid fault diagnosis device based on prior knowledge reinforcement learning, including a processor and a memory, wherein the processor implements the power grid fault diagnosis method based on prior knowledge reinforcement learning as described above when executing the computer program stored in the memory.

[0040] The beneficial effects of the present invention are:

[0041] 1. The power grid fault diagnosis method based on prior knowledge reinforcement learning proposed in the present invention intervenes in the action selection process by introducing prior knowledge. Prior knowledge is introduced in the process of selecting actions, which can make full use of valuable prior knowledge accumulated in advance, reduce the randomness of action selection in the exploration process, effectively avoid invalid exploration, save training costs, and improve training efficiency. When dealing with the problem of unbalanced sample data, by introducing prior knowledge, the problem of insufficient training cycles due to the small number of samples is avoided, and the dependence on a large amount of data is reduced, so that learning and inference can be carried out more effectively, while improving the accuracy of power grid fault diagnosis.

[0042] 2. The present invention first performs text matching of key features in the alarm data text with key feature rules in prior knowledge. If a key feature rule matching the current environmental state is found, the recommended action corresponding to the key feature rule is directly output, which can significantly speed up the decision-making process of power grid fault diagnosis without the need to learn the action through trial and error.

[0043] 3. The present invention converts the unsuccessfully matched alarm data text into an embedded vector after embedding processing, and further processes the embedded vector, paying more attention to the meaning and association between words in the alarm data text, so as to more effectively extract and utilize the key features in the alarm information. Finally, through experience playback, the priority of experience is dynamically adjusted so that important and rare experiences are selected more frequently, while continuously interacting with the environment to make decisions on the maximum reward and output the optimal fault diagnosis results. By utilizing the perception ability of deep learning and the decision-making ability of reinforcement learning, there is no need for manual feature extraction, and end-to-end intelligent autonomous decision-making from text alarm information to fault categories is achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In the attached picture:

[0045] Figure 1 A schematic diagram of a power grid fault diagnosis process based on prior knowledge reinforcement learning in an embodiment;

[0046] Figure 2 The experimental results on the total reward value;

[0047] Figure 3 Experimental results on loss value;

[0048] Figure 4 The experimental results on accuracy are as follows;

[0049] Figure 5 Experimental results on F1 score;

[0050] Figure 6 The comparative experimental results on accuracy are as follows;

[0051] Figure 7 The comparative experimental results on F1 score;

[0052] Figure 8 This is the result of ablation experiment;

[0053] Fig. 9 Comparative experimental results on F1 scores for small sample fault datasets;

[0054] Fig.10 Comparative experimental results on F1 scores for small sample fault datasets. DETAILED DESCRIPTION

[0055] Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings.

[0056] Example

[0057] This embodiment provides a power grid fault diagnosis method based on prior knowledge reinforcement learning, see Figure 1 , including the following steps:

[0058] S1. Obtain alarm data texts of power grid faults under different environmental states, construct a data set, extract switch features and key features based on the alarm data texts, and match the key features of the texts with key feature rules in prior knowledge, wherein the prior knowledge includes a set number of switch state features and a set number of key feature rules.

[0059] If the matching probability value with any key feature rule is greater than the probability threshold, the match is successful, and the corresponding action and power grid fault type are obtained; if the match is unsuccessful, S2 is executed.

[0060] First of all, it should be noted that the present invention introduces prior knowledge and diagnoses power grid faults based on prior knowledge. In the field of power grid fault diagnosis, prior knowledge refers to the knowledge and experience that the diagnostic personnel or system already has about the power grid system, power equipment, fault type, fault manifestation characteristics and its processing methods before performing a specific fault diagnosis process. This knowledge and experience may come from professional training, actual work experience, historical cases and other aspects, and its effectiveness has been verified in many cases. This knowledge is more targeted and reliable, and they are usually based on strict fault mechanism analysis, so they are also explainable and can be understood by diagnostic personnel. When faced with a new fault, diagnostic personnel with rich prior knowledge can quickly identify the fault phenomenon, reduce the scope of the fault, and reduce unnecessary inspection and testing steps, thereby significantly improving the diagnostic efficiency.

[0061] In this step, the prior knowledge includes several switch state features and several key feature rules, which are all set in advance. The key feature text in the alarm data text is first matched with the key feature rules in the prior knowledge. If a key feature rule matching the current environmental state is found, the recommended action corresponding to the key feature rule is directly output without learning the action through trial and error.

[0062] S2. Embed the unsuccessfully matched alarm data text to obtain the embedding vector under the current environmental state. The embedded vector is processed by multi-head attention mechanism, residual connection with the embedded vector, normalization and nonlinear processing in sequence to obtain deep features. The deep features are fully connected to obtain the estimated expected values ​​of different actions under the environmental state.

[0063] Specifically, the alarm information text of each fault sample in the data set contains three categories of features, which are converted into word embedding vectors, sentence embedding vectors and position embedding vectors after embedding processing, corresponding to semantic features, sentence features and position features respectively. The word embedding vector represents the semantic features of each word in the alarm information text, the sentence embedding vector represents the positional features of each alarm signal in the alarm information text, and the position embedding vector represents the positional features of each word in the alarm information text.

[0064] The specific process of embedding the unsuccessfully matched alarm data text to obtain the embedding vector under the current environment state is as follows: respectively perform word embedding processing, sentence embedding processing and position embedding processing on the alarm data text to obtain a word embedding vector, a sentence embedding vector and a position embedding vector respectively, and concatenate the word embedding vector, sentence embedding vector and position embedding vector to obtain an embedding vector.

[0065] The present invention mines the fault features contained in the alarm information text and uses the alarm information for fault diagnosis, because it has good deep knowledge extraction and data mining capabilities for natural language text. The embedded vector obtained after the embedding process is input into the Transformer encoder for subsequent feature extraction, such as using the Transformer encoder for feature extraction.

[0066] Specifically, in this step, the input embedding vector is used Indicates that the query is defined as , defining the key as , defining the value as ,in , , and are weight matrices, d is the matrix dimension, is a matrix.

[0067] Then the attention is calculated by the following formula:

[0068] ,

[0069] in, The dimension of the key.

[0070] Since the multi-head attention mechanism has multiple attention heads at the same time, it can capture information from different subspaces. Therefore, in order to learn the expression of multiple semantics in multiple channels, the multi-head attention mechanism is used to extract information in different semantic spaces of each channel.

[0071] The process of multi-head attention mechanism processing is as follows:

[0072] ,

[0073] in, is the extracted feature, is a weight matrix, h is the number of heads. i indivual Calculated using the following formula:

[0074] ,

[0075] in, , and The query, key and value are i A weight matrix.

[0076] The extracted features are further connected and normalized with the residual of the embedded vector to obtain the feature , the formula is as follows:

[0077] ,

[0078] In order to further process and transform the output of the attention layer, the input vector is nonlinearly transformed by the following formula:

[0079] ,

[0080] in, W 1 , W 2 are the weight matrices of the feedforward neural network, b 1 , b 2 are the corresponding bias vectors, is the ReLU activation function, which is used to introduce nonlinearity.

[0081] The output after the current nonlinear processing is processed by residual connection and normalization to obtain the deep feature, that is, the output of the current Transformer encoder. The deep feature is input into the fully connected layer for full connection processing to obtain the estimated expected value of different actions under the environment state.

[0082] S3. Based on the switch characteristics, switch state characteristics, and estimated expected values ​​of different actions, the action with the highest estimated expected value and the corresponding power grid fault type under the current environmental state are obtained, specifically:

[0083] Construct an action space based on all actions, and construct different action subspaces based on different switch state characteristics. Each action subspace includes a specific number of actions as an action exploration space.

[0084] The extracted switch feature is matched with any switch state feature. If there is a successfully matched switch state feature, an action is randomly selected for exploration in its corresponding action exploration space based on the exploration probability, or the action with the largest estimated expected value is selected for use based on 1-exploration probability to obtain the corresponding power grid fault type;

[0085] Otherwise, explore or exploit in the action space.

[0086] This step is mainly to guide the intelligent agent in the early stage of training based on the introduced prior knowledge, avoid too much ineffective exploration, and help the intelligent agent make decisions quickly. It is a training process.

[0087] The prior knowledge has been explained in step S1. Furthermore, in this step, fault exploration based on the prior knowledge is specifically introduced.

[0088] In this embodiment, taking a 10KV line as an example, the following diagnostic rules can be formulated according to the switch state changes:

[0089] Rule 1: When the monitoring system receives the "switch disconnected" alarm information and the switch is not seen to be reclosed within a short period of time, it is judged as a "line fault, no reclosing" type;

[0090] Rule 2: When the switch is disconnected, and the alarm information of the switch closing is received again within a short period of time, and there is no disconnection signal afterwards, it means that it is a short-term fault, which has been successfully cleared by the reclosing device and the system has returned to normal, then it is judged as a "line instantaneous fault" type;

[0091] Rule 3: When the switch is disconnected, the reclosing action fails, and the switch is disconnected again, it means that the fault has not been eliminated and the reclosing device fails to restore power supply, then it is judged as "line fault, reclosing unsuccessful" type.

[0092] These rules based on switch state changes provide an effective basis for power grid fault diagnosis.

[0093] In addition to the above three rules, the prior knowledge also contains several rules with key features, namely key feature rules. For example, when the monitoring system receives the alarm information of "switch disconnection" and the fault occurs on the bus, it can be judged as a "bus fault". These rules are diverse in form, covering different fault scenarios, and are designed to provide accurate fault judgments for various situations. With the continuous development of power grid scale and technology, the prior knowledge base of the diagnostic system is also constantly improving. By combining historical data, expert experience and advanced algorithms, the rule set can be continuously optimized, thereby continuously improving the accuracy and response speed of diagnosis.

[0094] In this embodiment, the prior knowledge is defined as follows:

[0095] Assume that the environmental state sequence is S, and the formula is: S=[s 1 ,s 2 ,s 3 ,…,s i ], i=1,2,3,…, where s i is the i-th environmental state in the environmental state sequence;

[0096] Assume the action space is A, the formula is: A=[a 1 ,a 2 ,a 3 ,…,a i ], i=1,2,3,…, where a i is the i-th action in the action space;

[0097] Assume that the switch state characteristic sequence in the prior knowledge is F, and the formula is expressed as: F=[f 1 ,f 2 ,f 3 ,…,f i ],i=1,2,3,…,where f i is the i-th switch state feature in the switch state feature sequence.

[0098] Different action subspaces are further constructed based on different switch state features. Each action subspace includes a specific number of actions as an action exploration space. For example, when the switch state feature f i When the action exploration space is reduced to a 1 ,a 2 ,…,a n , the number of actions it includes is smaller than the number of actions in the action space.

[0099] Then, the switch feature extracted in step S1 is matched with any switch state feature in the prior knowledge. This is also a text matching process. If there is a successfully matched switch state feature, an action is randomly selected for exploration based on the exploration probability in the action exploration space corresponding to the switch state feature, or an action with the largest estimated expected value is selected for use based on 1-exploration probability to obtain the corresponding power grid fault type.

[0100] Otherwise, explore or exploit in the action space.

[0101] The exploration and exploitation process described in this embodiment is implemented based on the introduced ε-greedy strategy. The formula is as follows:

[0102] ,

[0103] Among them, ε is the exploration probability corresponding to the greedy factor, ε∈(0,1], To randomly select an action to explore, To select the action with the largest estimated expected value for use.

[0104] The exploration probability is set differently based on the amount of data occurring in different fault types in the data set. The exploration probability is set proportional to the amount of data occurring in different fault types in the data set. In this embodiment, the exploration probability is also set to decrease at a specific decay rate in each round until it decreases to a minimum exploration probability value.

[0105] In the present invention, the action exploration space and the corresponding probability are different for different switch characteristics. The probability of fault types with a large number of data sets is higher, and the probability of fault types with a small number is lower. This design effectively promotes the tendency exploration of effective strategies in power grid fault diagnosis. In the early exploration, the action with the largest current estimated expected value is selected with a probability of 1-ε, and the action is selected according to the switch state characteristics of prior knowledge with a probability of ε; in the later stage of training, ε gradually decreases to the minimum exploration probability value ε min This process of transitioning from exploration to exploitation focuses more on executing those verified high-return actions, thereby accelerating the learning process and improving the final performance. This has the following three advantages: First, by introducing prior knowledge, when the switch state characteristics are detected, the action selection process can be influenced according to the prior knowledge rules, reducing unnecessary exploration; second, the introduction of prior knowledge does not affect the training process; third, when the switch state characteristics do not appear, the action selection can also be judged to fully explore the action space.

[0106] By introducing prior knowledge and intervening in the action selection process, valuable information can be fully utilized, and prior knowledge such as expert knowledge, characteristic signals, and classification features can be added to the training, so that the improved model can effectively avoid invalid exploration. There are significant improvements in convergence speed and accuracy, which can effectively improve training efficiency and save training costs.

[0107] The actions obtained after exploration or utilization are rewarded with feedback based on the set reward function, and then transferred to the next environment state. An experience group is constructed based on the current environment state, each action obtained under the current environment state and the corresponding feedback reward, and the next environment state and stored in the experience buffer pool. The sampling probability is set for several experience groups, and the experience playback is performed based on the sampling probability to update the estimated expected value until the set round is reached.

[0108] Sampling probabilities are set for several experience groups, and experience replay is performed based on the sampling probabilities to update the estimated expected value. Specifically, the target expected value is obtained based on the feedback reward, discount factor, and the maximum expected value of the next environmental state. The sampling probability of each experience group is obtained based on the difference between the estimated expected value and the target expected value and the set priority. Several experience groups are randomly selected from the experience buffer pool based on the sampling probability for experience replay and the estimated expected value is updated.

[0109] In this step, the actions obtained after exploration or utilization are rewarded based on the set reward function. Because the minority class fault samples in the data set are difficult to be correctly identified in an unbalanced data set, in order to better identify the minority class fault samples, more attention should be paid to them. When encountering minority class fault samples, a larger reward or penalty is given. Set reward function Specifically:

[0110] ,

[0111] in, t is the time step, For the time step t The current state of the environment, For the time step t The current action, For the time step t The fault sample label under the current environment state, For most fault samples, ρ is the ratio of majority class fault samples to minority class fault samples, D F It is a minority class fault sample;

[0112] When the majority class fault samples are classified correctly and incorrectly, the reward values ​​are 1 and -1 respectively; when the minority class fault samples are classified correctly and incorrectly, the reward values ​​are ρ 、- ρ .

[0113] use Output estimated expected value , the formula is as follows:

[0114] ,

[0115] in To select an action in the current environment state s a Estimated expected value processing when .

[0116] Based on the feedback reward, discount factor, and the maximum expected value of the next environmental state, the target expected value is obtained. The formula is as follows:

[0117] ,

[0118] in, y is the target expected value, r To select an action in the current environment state s a The rewards received afterwards; γ is the discount factor, with a value range of [0,1], which is used to measure the importance of current rewards and future rewards; s ' is the next state, a 'For all possible actions in the next state, θ ' is the parameter for updating the target expected value.

[0119] It should be noted that the parameters for updating the target expected value θ 'Not updated every iteration, but every certain number of steps N Will θ The value of θ ', thereby reducing the fluctuation of the estimated target value.

[0120] Further, based on the difference between the estimated expected value and the target expected value and the set priority, the sampling probability of each experience group is obtained, and based on the sampling probability, several experience groups are randomly selected from the experience buffer pool for experience playback processing to update the estimated expected value.

[0121] The priority is set based on the amount of fault data in the alarm data text in the data set, and the setting priority is inversely proportional to the amount of fault data in the alarm data text in the data set.

[0122] Specifically, based on the difference between the estimated expected value and the target expected value and the set priority, the sampling probability of each experience group is obtained, and the formula is as follows:

[0123] ,

[0124] in, is the difference between the estimated expected value and the target expected value, is the target expected value, To estimate the expected value, is the current environment status, For the current action, To update the parameters of the estimated expected value; For feedback rewards, is the discount factor, is the expected value of the next environmental state, is the next environment state, For the next action, is the parameter to update the target expected value;

[0125] ,

[0126] in, is the sampling probability for each experience group, For the i The probability of an experience group, The priority set.

[0127] Based on the sampling probability, several experience groups are randomly selected from the experience buffer pool for experience playback processing, and the loss function is calculated based on the estimated expected value and the target expected value for the randomly selected experience groups. , the formula is as follows:

[0128] ,

[0129] in, N is the number of randomly selected experience groups, i is the serial number, is the target expected value, To estimate the expected value, is the current environment status, For the current action, To estimate the Q network parameters.

[0130] The estimated expected value is updated using the gradient descent method based on the loss function, and the parameters of the estimated expected value are updated to the parameters of the target expected value at every specific time step.

[0131] The present invention also provides a power grid fault diagnosis system based on prior knowledge reinforcement learning, comprising:

[0132] Key feature matching module: used to obtain the alarm data text of power grid faults under different environmental states, build a data set, extract switch features and key features based on the alarm data text, and match the key features of the text with the key feature rules in the prior knowledge. The prior knowledge includes a set number of switch state features and a set number of key feature rules. If the matching probability value with any key feature rule is greater than the probability threshold, the match is successful, and the corresponding action and power grid fault type are obtained; otherwise, the match is unsuccessful;

[0133] Feature extraction module: used to embed the alarm data text that fails to match, obtain the embedding vector under the current environment state, perform multi-head attention mechanism processing, residual connection processing with the embedding vector, normalization processing and nonlinear processing on the embedded vector in turn to obtain deep features, perform full connection processing on the deep features, and obtain the estimated expected values ​​of different actions under the environment state;

[0134] Fault diagnosis module: used to obtain the action with the highest estimated expected value and the corresponding power grid fault type under the current environmental state based on switch features, switch state features, and estimated expected values ​​of different actions. Specifically, the action space is constructed based on all actions, and different action subspaces are constructed based on different switch state features. Each action subspace includes a specific number of actions as an action exploration space. The extracted switch features are matched with any switch state feature. If there is a successfully matched switch state feature, an action is randomly selected in the corresponding action exploration space based on the exploration probability for exploration, or the action with the largest estimated expected value is selected based on 1-exploration probability for utilization to obtain the corresponding power grid fault type. Otherwise, exploration or utilization is performed in the action space.

[0135] In addition, the present invention also provides a power grid fault diagnosis device based on prior knowledge reinforcement learning, including a processor and a memory, wherein the processor implements the power grid fault diagnosis method based on prior knowledge reinforcement learning as described above when executing the computer program stored in the memory.

[0136] Experimental content

[0137] In order to verify the performance of the power grid fault diagnosis method, system and device based on prior knowledge reinforcement learning proposed in the present invention, the present invention conducted experimental verification, using the data acquisition and monitoring control system (SCADA system) to collect the alarm data text of the power grid fault, and constructed a data set for extensive experiments. The experimental part of the present invention mainly includes fault sample data, parameter settings, comparative experiments, comparative experimental results, ablation experimental results and small sample fault data experimental results. The specific introduction is as follows:

[0138] Fault sample data

[0139] In order to verify the effectiveness of the power grid fault diagnosis method based on prior knowledge reinforcement learning proposed in the present invention, the present invention uses a historical alarm information data set collected by an electronics company for experiments. Samples corresponding to 10 important fault events are extracted from it, and a total of 13,411 fault event samples are obtained. Each fault sample is divided into a training set and a test set in a ratio of 7:3. The number of training sets and test sets for each fault sample is shown in Table 1. It can be seen from the table that the amount of data for the fault categories "0, 1, 2" is relatively large, and the amount of data for other fault categories is relatively small, which fully reflects the problem of data imbalance.

[0140] Table 1 Statistics of the number of fault event samples

[0141]

[0142] Parameter settings

[0143] The experiment of the present invention was conducted on a Windows PC equipped with an NVIDIA GeForceGTX 3090 GPU processor and an Intel Core i910900 3.70 GHz CPU processor. The parameters specifically set in the experiment are shown in Table 2.

[0144] Table 2 Model parameters

[0145]

[0146] Comparative experiment

[0147] In order to prove the effectiveness of the model proposed in this invention, the present invention selected the existing detection methods of CNN, LSTM, Bert+Transformer and DQN as the benchmark, and carried out experimental comparison with the power grid fault diagnosis method based on prior knowledge reinforcement learning proposed in this invention. The description of the four comparative detection methods used is as follows:

[0148] (1) CNN: It can automatically extract text features and realize classification and prediction of warning information text. The process is further simplified, and the accuracy of diagnosis is not limited by the inherent model, which improves the adaptive ability.

[0149] (2) LSTM: It can more fully mine the time series features in the data and use a simple sliding time window to accurately capture fault events. It is simple to implement and has high accuracy.

[0150] (3) Bert+Transformer: Automatically focuses on the key content in the alarm information text, autonomously extracts the text description features and naming rules of electrical equipment in the alarm information text, quickly infers the fault-related situation, and promptly removes the faulty equipment.

[0151] (4) DQN: It has a unique exploration-exploitation mechanism. By exploring unknown environments, it can reduce the disadvantage of poor diagnostic effect caused by unbalanced sample data.

[0152] Comparative test results

[0153] After the fault samples in the dataset of the present invention are divided into a training set and a test set at a ratio of 7:3, the training set is used for training. Through continuous experiments, it is found that when the training round is set to 10 times, the training accuracy is already extremely high. The experimental results are as follows Figure 2-5 shown.

[0154] pass Figure 2 It can be seen that the total reward value obtained in each round is constantly increasing, which means that it is constantly learning and taking better actions. Figure 3The loss value represents the training error, and the training error decreases steadily with the increase of training rounds. The correctness of the parameter setting and training method of the power grid fault diagnosis method based on prior knowledge reinforcement learning proposed in the present invention is verified. Figure 4 The accuracy and Figure 5 The F1 scores can represent the accuracy of diagnosis in each round. After ten rounds of training, the accuracy of diagnosis can reach 100%, which proves the effectiveness of the power grid fault diagnosis method based on prior knowledge reinforcement learning proposed in the present invention.

[0155] Figure 6 , Figure 7 The performance results of the present invention and four comparative methods evaluated by accuracy and F1 score are listed respectively. It can be seen that the results of the present invention in terms of accuracy and F1 score are the best. In the experimental results, CNN has the worst effect because the method is relatively simple, lacks the ability to capture long-distance dependencies, and has limited learning ability for complex patterns. LSTM can capture the relationship between contexts in sentences, but compared with Bert+Transformer, Bert+Transforemr can process long sequence texts through the self-attention mechanism, capture complex semantic relationships in the context, and a large number of data sets can train more parameters of Bert+Transformer. DQN optimizes strategies using reinforcement learning methods, can dynamically adjust decisions, and can still maintain high accuracy and F1 scores when the task complexity is high. The method proposed in the present invention combines the semantic understanding ability of Transformer and the dynamic decision-making of reinforcement learning, and introduces prior knowledge in reinforcement learning, so as to perform well in the task.

[0156] Ablation experiment

[0157] like Figure 8As shown, the present invention designs 4 experiments for verification, and gives specific ablation experimental results. The baseline in Experiment 1 is the existing DQN method, and the invention is encoded using the Word2Vec method; Experiment 2 is a DQN in which the encoding method is changed to a Transformer encoder; Experiment 3 is a DQN with prior knowledge added on the basis of Experiment 1; Experiment 4 is a complete power grid fault diagnosis method based on prior knowledge reinforcement learning proposed in this paper. The results of Experiments 1, 2, and 3 show that all evaluation indicators have decreased, which illustrates the effectiveness of each method. Experiment 2 introduces the Transformer encoder on the basis of Experiment 1 to enhance the learning of key information. The experimental results show that its effect is significantly improved, which shows that the Transformer encoder can play its role, pay more attention to key information, and help improve the fault diagnosis effect. Experiment 3 adds prior knowledge on the basis of Experiment 1, and the experimental effect is significantly higher than that of Experiments 1 and 2, which shows that the combination of prior knowledge and DQN model can significantly enhance the fault diagnosis effect and also speed up the training speed of the model.

[0158] Small sample fault data experiment

[0159] In order to verify that the present invention can effectively improve the fault diagnosis accuracy under a small sample fault data set, the present invention selects a small number of fault categories in the data set to form a small sample fault data set, including: "3: station transformer fault", "6: busbar / section fault", "8: reactor fault" and "9: busbar fault". The number of each type of training set and test set is shown in Table 3.

[0160] Table 3. Small sample fault data volume

[0161]

[0162] The small sample fault dataset is compared with various baselines and the results are as follows: Fig. 9 and Fig.10As shown. CNN is usually suitable for processing local features in images and space, and performs poorly for text or sequence tasks. It requires a large amount of data to capture complex patterns. In the case of small sample fault data, due to insufficient training data, CNN cannot effectively learn the features of small sample fault data. LSTM is suitable for processing sequence problems and has certain advantages in memorizing and capturing long-distance dependencies, so that it can rely on contextual information under small sample fault data and learn the intrinsic structure of features. Although Bert+Transformer performs well on big data, it performs poorly on small sample fault data because the method has a highly complex parameter structure. When the amount of data is insufficient, it is difficult to fully train, resulting in overfitting. DQN has exploration capabilities in the early stage of training, and can explore more state-action pairs during training, increasing the diversity of learning. The present invention proposes a power grid fault diagnosis method based on prior knowledge reinforcement learning, which uses prior knowledge to limit the exploration space, improve the exploration accuracy, and thus speed up the training speed.

[0163] In summary, the power grid fault diagnosis method based on prior knowledge reinforcement learning proposed in the present invention utilizes the perception ability of deep learning and the decision-making ability of reinforcement learning, without the need for manual feature extraction, to achieve end-to-end intelligent autonomous decision-making from text alarm information to fault categories. The experimental results show that the method proposed in the present invention significantly improves the accuracy of fault diagnosis when dealing with the problem of sample data imbalance. The present invention can map the text description of the alarm information into a vector representation of state semantics, pay more attention to the meaning and association between words, and thus more effectively extract and utilize the key features in the alarm information. In addition, prior knowledge is introduced in the process of selecting actions, which can reduce the randomness of action selection in the exploration process, so that learning and inference can be carried out more effectively, thereby accelerating the convergence speed and reducing the dependence on a large amount of data. Finally, priority experience playback is used in the experience buffer pool to dynamically adjust the priority of experience, so that important and rare experience is selected more frequently, and the maximum reward is continuously interacted with the environment to output the optimal fault diagnosis result. Through extensive experiments and analysis, the experimental results prove the effectiveness of the power grid fault diagnosis method based on prior knowledge reinforcement learning of the present invention.

Claims

1. A power grid fault diagnosis method based on prior knowledge reinforcement learning, characterized in that: The following steps are involved: S1. Obtain alarm data texts of power grid faults under different environmental states, construct a data set, extract switch features and key features based on the alarm data texts, and match the key features of the texts with key feature rules in prior knowledge, wherein the prior knowledge includes a set number of switch state features and a set number of key feature rules. If the matching probability value with any key feature rule is greater than the probability threshold, the match is successful, and the corresponding action and power grid fault type are obtained; if the match is unsuccessful, S2 is executed; S2, embedding the alarm data text that failed to match, obtaining the embedding vector under the current environment state, sequentially performing multi-head attention mechanism processing, residual connection processing with the embedding vector, normalization processing and nonlinear processing on the embedding vector to obtain deep features, performing full connection processing on the deep features to obtain the estimated expected values ​​of different actions under the environment state; S3. Based on the switch characteristics, switch state characteristics, and estimated expected values ​​of different actions, the action with the highest estimated expected value and the corresponding power grid fault type under the current environmental state are obtained, specifically: Construct an action space based on all actions, and construct different action subspaces based on different switch state characteristics. Each action subspace includes a specific number of actions as an action exploration space. The extracted switch feature is matched with any switch state feature. If there is a successfully matched switch state feature, an action is randomly selected for exploration in its corresponding action exploration space based on the exploration probability, or the action with the largest estimated expected value is selected for use based on 1-exploration probability to obtain the corresponding power grid fault type; Otherwise, explore or exploit in the action space.

2. The power grid fault diagnosis method based on prior knowledge reinforcement learning according to claim 1 is characterized in that: The S3 also includes obtaining feedback rewards for the actions obtained after exploration or utilization based on the set reward function, transferring to the next environment state, building an experience group based on the current environment state, each action obtained under the current environment state and the corresponding feedback rewards, and the next environment state and storing it in the experience buffer pool, setting sampling probabilities for several experience groups, and performing experience playback processing based on the sampling probabilities, and updating the estimated expected value until the set round is reached; Sampling probabilities are set for several experience groups, and experience playback is performed based on the sampling probabilities to update the estimated expected value, specifically: Based on the feedback reward, discount factor, and the maximum expected value of the next environmental state, the target expected value is obtained; based on the difference between the estimated expected value and the target expected value and the set priority, the sampling probability of each experience group is obtained, and based on the sampling probability, several experience groups are randomly selected from the experience buffer pool for experience playback processing to update the estimated expected value.

3. The power grid fault diagnosis method based on prior knowledge reinforcement learning according to claim 2 is characterized in that: Based on the difference between the estimated expected value and the target expected value and the set priority, the sampling probability of each experience group is obtained, and the formula is as follows: ， in, is the difference between the estimated expected value and the target expected value, is the target expected value, To estimate the expected value, is the current environment status, For the current action, To update the parameters of the estimated expected value; For feedback rewards, is the discount factor, is the expected value of the next environmental state, is the next environment state, For the next action, is the parameter to update the target expected value; ， in, is the sampling probability for each experience group, For the i The probability of an experience group, The priority set.

4. The power grid fault diagnosis method based on prior knowledge reinforcement learning according to claim 2 is characterized in that: The reward function for the described setting Specifically: ， in, t is the time step, For the time step t The current state of the environment, For the time step t The current action, For the time step t The fault sample label under the current environment state, For most fault samples, ρ is the ratio of majority class fault samples to minority class fault samples, D F It is a minority class fault sample; When the majority class fault samples are classified correctly and incorrectly, the reward values ​​are 1 and -1 respectively; when the minority class fault samples are classified correctly and incorrectly, the reward values ​​are ρ 、- ρ .

5. The power grid fault diagnosis method based on prior knowledge reinforcement learning according to claim 2 is characterized in that: The method randomly extracts several experience groups from the experience buffer pool based on the sampling probability to perform experience playback processing, and calculates the loss function based on the estimated expected value and the target expected value for the randomly extracted several experience groups. , the formula is as follows: ， in, N is the number of randomly selected experience groups, i is the serial number, is the target expected value, To estimate the expected value, is the current environment status, For the current action, To update the parameters of the estimated expected value; The parameters of the target expected value are updated using the gradient descent method based on the loss function, and the parameters of the estimated expected value are updated to the parameters of the target expected value at every specific time step.

6. The power grid fault diagnosis method based on prior knowledge reinforcement learning according to claim 2 is characterized in that: The priority is set based on the amount of fault data in the alarm data text in the data set, and the setting priority is inversely proportional to the amount of fault data in the alarm data text in the data set.

7. The power grid fault diagnosis method based on prior knowledge reinforcement learning according to claim 1 is characterized in that: The exploration probability in S3 is set differently based on the amount of data occurring in different fault types in the data set. The exploration probability is set proportional to the amount of data occurring in different fault types in the data set. At the same time, the exploration probability is set to decrease at a specific decay rate in each round until it decreases to a minimum exploration probability value.

8. The power grid fault diagnosis method based on prior knowledge reinforcement learning according to claim 1 is characterized in that: In the S2, the alarm data text that fails to match is embedded to obtain an embedding vector under the current environment state. Specifically, the alarm data text is subjected to word embedding processing, sentence embedding processing and position embedding processing respectively to obtain a word embedding vector, a sentence embedding vector and a position embedding vector respectively. The word embedding vector, sentence embedding vector and position embedding vector are concatenated to obtain an embedding vector.

9. A power grid fault diagnosis system based on prior knowledge reinforcement learning, characterized in that: include: Key feature matching module: used to obtain the alarm data text of power grid faults under different environmental states, build a data set, extract switch features and key features based on the alarm data text, and match the key features of the text with the key feature rules in the prior knowledge. The prior knowledge includes a set number of switch state features and a set number of key feature rules. If the matching probability value with any key feature rule is greater than the probability threshold, the match is successful, and the corresponding action and power grid fault type are obtained; otherwise, the match is unsuccessful; Feature extraction module: used to embed the alarm data text that fails to match, obtain the embedding vector under the current environment state, and sequentially perform multi-head attention mechanism processing, residual connection processing with the embedding vector, normalization processing and nonlinear processing on the embedded vector to obtain deep features, and perform full connection processing on the deep features to obtain the estimated expected values ​​of different actions under the environment state; Fault diagnosis module: used to obtain the action with the highest estimated expected value and the corresponding power grid fault type under the current environmental state based on switch features, switch state features, and estimated expected values ​​of different actions. Specifically, an action space is constructed based on all actions, and different action subspaces are constructed based on different switch state features. Each action subspace includes a specific number of actions as an action exploration space. The extracted switch features are matched with any switch state features. If there is a successfully matched switch state feature, an action is randomly selected in the corresponding action exploration space based on the exploration probability for exploration, or the action with the largest estimated expected value is selected based on 1-exploration probability for utilization to obtain the corresponding power grid fault type. Otherwise, exploration or utilization is performed in the action space.

10. A power grid fault diagnosis device based on prior knowledge reinforcement learning, characterized in that: The method comprises a processor and a memory, wherein when the processor executes the computer program stored in the memory, the method for diagnosing power grid faults based on prior knowledge reinforcement learning as described in any one of claims 1 to 7 is implemented.

Images