Broadband oscillation classification method based on deep reinforcement learning and pattern mining

Abstract

The invention relates to the technical field of power transmission, in particular to a broadband oscillation classification method based on deep reinforcement learning and pattern mining. According to the broadband oscillation classification method based on deep reinforcement learning and pattern mining, multivariate time series data such as voltage and current in a power grid line are coded into a univariate clustering sequence, and then a candidate pattern extracted from the clustering sequence is utilized to learn a multi-pattern time series. A discrimination mode is trained through a reinforcement learning module, and a classification model is trained at the same time. According to the method, the category of the sample is mapped from the multi-mode time sequence by using the neural network, and the multi-mode time sequence is updated by using the reinforcement learning method, so that the classification method is ensured to have a certain degree of interpretability while the advantage of high fitting of the neural network is utilized, and the classification efficiency is improved. Meanwhile, the time sequence relation between data existing in the broadband oscillation phenomenon can be effectively explored while the classification algorithm is achieved; the problem that an existing oscillation classification method is low in accuracy and efficiency is solved.

Description

Technical Field

[0001] The present application relates to the field of power transmission technology, and in particular to a broadband oscillation classification method based on deep reinforcement learning and pattern mining. Background Art

[0002] In the "double-high" power system, the interaction between power generation equipment, transmission network, power load, etc. will cause unstable oscillations in the frequency range of several hertz to several thousand hertz. Oscillation refers to the periodic fluctuation of electrical quantities (such as voltage, current, power, etc.) over time caused by the interaction between power electronic equipment and between the power grid due to the influence of the power system itself or external factors. The dynamic process in which the oscillation frequency changes in a wide range is called power system broadband oscillation. At present, due to the complex internal mechanism of broadband oscillation, the existing control-based analysis and ordinary artificial intelligence algorithms are difficult to capture its internal complex characteristics while efficiently classifying it for further suppression.

[0003] The broadband oscillation problem in the power system seriously affects equipment safety and power quality, restricts the efficient consumption of new energy, and threatens the safety and stability of the power grid, which has attracted widespread attention from academia and industry. However, people have not yet formed a unified understanding of the broadband oscillation problem, and cannot accurately reveal its physical mechanism. How to classify broadband oscillations can often only rely on traditional control analysis or ordinary artificial intelligence methods. It is difficult to accurately capture the internal laws of broadband oscillation phenomena, which brings certain difficulties to the subsequent suppression. In view of this, it is necessary to provide a method that can accurately capture the internal laws of broadband oscillations and efficiently classify broadband oscillation phenomena that occur in power grids. Summary of the invention

[0004] In order to overcome the defects of the lack of broadband oscillation classification, low accuracy and low efficiency in the above-mentioned prior art, the present application proposes a broadband oscillation classification method based on deep reinforcement learning and pattern mining, and the broadband oscillation classification of power grid transmission lines based on deep reinforcement learning and pattern mining improves the accuracy and efficiency of broadband oscillation classification.

[0005] The present invention proposes a broadband oscillation classification method based on deep reinforcement learning and pattern mining. First, the oscillation classification model is trained by the Markov algorithm. The input of the oscillation classification model is a multi-mode time series obtained by encoding multivariate time series data representing the working conditions of the power system, and the output is the predicted oscillation category. The method of encoding multivariate time series data to obtain multimodal time series is as follows: first, the multivariate time series data collected in a continuous time period is discretized into single time point data, and then the single time point data is clustered, and each single time point data in the multivariate time series data is replaced with the corresponding cluster cluster number to form a cluster sequence; the common subsequence of the cluster sequence corresponding to the multivariate time series data number under different oscillation categories is obtained, and the common subsequence in the cluster sequence corresponding to the multivariate time series data is extracted to form a multimodal time series; Acquire multivariate time series data as the test object, combine it with the known clustering results, obtain the clustering sequence of the test object, and extract the common subsequence to form the multi-mode time series of the test object; input the multi-mode time series of the test object into the oscillation classification model to obtain the oscillation category prediction value of the test object.

[0006] Preferably, the training method of the oscillation classification model comprises the following steps: St1. Obtain multivariate time series data from the historical operation data of the power system and convert it into a multi-mode time series as a learning sample and store it in the experience pool. The learning sample is marked with the oscillation category; build a basic model; the basic model includes a feature extraction network and a SAC network; the SAC network includes an actor network, a first state evaluation network, a second state evaluation network, a first action evaluation network and a second action evaluation network; The feature extraction network generates a hidden state h and a predicted category y' for the multimodal time series; the actor network generates an adjustment strategy based on the multimodal time series and h; the multimodal time series forms a new multimodal time series after executing the adjustment strategy; the inputs of the first state evaluation network, the second state evaluation network, the first action evaluation network, and the second action evaluation network are all the multimodal time series, h, and the reward r, and the outputs are the first state evaluation value, the second state evaluation value, the first action evaluation value, and the second action evaluation value, respectively; The calculation formula of reward r is: ; ; Among them, m is the number of shock categories, f i is the representation of sample s in the i-th oscillation category; is the value of the ith oscillation in the predicted category obtained by the feature extraction module for sample s; y i is the value of the ith shock in the shock category marked by sample s; ε is the set smoothing parameter; St2, randomly select multiple samples from the experience pool and input them into the basic model, calculate the loss function and update the feature extraction network according to the predicted category and the real shock category; calculate the reward r as well as the action loss and state loss; update the first action evaluation network and the second action evaluation network according to the action loss, update the first state evaluation network according to the state loss, and copy the first state evaluation network to the second state evaluation network; St3, calculate the strategy loss based on the updated first action evaluation network, and update the Actor network based on the strategy loss; St4, input sample s into the updated basic model, the Actor network obtains the action strategy and the new multi-mode time series, and associates the new multi-mode time series with the original oscillation category as a sample and puts it into the experience pool; St5, determine whether the number of basic model updates reaches the set number; if not, return to step St2; if yes, fix the feature extraction network as the oscillation classification model, and the oscillation classification model predicts the oscillation category according to the input multi-modal time series.

[0007] Preferably, the action loss function is: Lq(i')=∑ s∈B [q(s,a;wi')-U(q)] 2 / |B|, i'=1 or 2; U(q)=r+γv(s') Where s represents the multimodal time series of the basic model as sample input; B is the training batch, |B| is the training batch size; wi' represents the approximate parameters of the i'th action evaluation network, a represents the policy action generated by the actor network when the basic model input sample is s; q(s,a;wi') represents the action evaluation value output by the i'th action evaluation network when the basic model input sample is s and the action strategy is a; U(q) represents the true value estimate of the sample; s' is the new multimodal time series formed by adjusting s in combination with the action strategy a; v(s') is the first state evaluation value output by the first state evaluation network when the basic model input is s'; γ is the set reward discount coefficient; In step St3, the first action evaluation network is updated according to the first action loss Lq(1), and the second action evaluation network is updated according to the second action loss Lq(2).

[0008] Preferably, the state loss function is: Lv=∑ s∈B [q(s,a)-U(v)] 2 / |B|; U(v)=E a~π(a|s;θ) [ (s,a;wi')-αlnπ(a|s;θ)] Where s represents the multimodal time series of the basic model as sample input; B is the training batch, |B| represents the training batch size; π(a|s;θ) is the probability set of the actor network output; θ is the approximate parameter of the first state value network; q(s,a) is the probability value corresponding to the action strategy a in π(a|s;θ), and U(v) is the true value estimate of sample s; E a~π(a|s;θ) represents the expectation of the distribution of π(a|s;θ); (s,a;wi') means taking the smaller value between q(s,a;w1) and q(s,a;w2), q(s,a;w1) and q(s,a;w2) respectively represent the action evaluation values ​​output by the first action evaluation network and the second action evaluation network when the basic model input sample is s and the action strategy is a; α is the entropy parameter in the maximum entropy exploration in the SAC algorithm.

[0009] Preferably, the strategy loss function is: La=∑ s∈B E a~π(a|s;θ) [q 0 (s,a)-αlnπ(a|s;θ)] / |B| Among them, q 0 (s, a) represents the first action evaluation value generated by the updated first action evaluation network when the base model input is s.

[0010] Preferably, the acquisition of learning samples in St1 includes the following steps: S1. Obtain a set of multivariate time series data marked with shock categories, and discretize the multivariate time series data in the set in units of time points to obtain a set of single time point data; S2, clustering the data in the set of single time point data to obtain the mapping relationship between each data in the multivariate time series data and the clustering clusters; S3, replacing the data in the multivariate time series data with the serial numbers of the corresponding clusters to form a cluster sequence; S4. Extract common subsequences from cluster sequences under the same oscillation category to form continuous subsequences, and use the continuous subsequences as pattern candidates of the corresponding category; obtain the union L of pattern candidates of all oscillation categories; S5. For the clustering sequence of the multivariate time series data, replace the continuous cluster number array with a continuous subsequence, and extract the continuous subsequence to form a multi-mode time series output of the multivariate time series data.

[0011] Preferably, Toeplitz inverse covariance clustering method is used for clustering in S2; Step S5 specifically includes the following sub-steps: S51, constructing a feature candidate set for multivariate time series data, where the feature candidate set is initially an empty set; S52, searching for a continuous numerical combination constituting any continuous subsequence in the union L in the cluster sequence of the multivariate time series data; S53, combining continuous numerical values ​​to form a continuous subsequence and storing it in a feature candidate set, and then deleting the continuous numerical combination from the clustering sequence; S54, judging whether there is a continuous value combination in the cluster sequence that can constitute any continuous subsequence in the union set L; If yes, return to step S53; If not, the continuous subsequences in the feature candidate set are arranged in order to form a multi-modal time series output.

[0012] Preferably, the feature extraction network adopts an RNN network.

[0013] The present application proposes a broadband oscillation classification system based on deep reinforcement learning and pattern mining, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is connected to the memory, and the processor is used to execute the computer program to implement the broadband oscillation classification method based on deep reinforcement learning and pattern mining.

[0014] The present application proposes a storage medium storing a computer program, which, when executed, is used to implement the broadband oscillation classification method based on deep reinforcement learning and pattern mining.

[0015] The advantages of this application are: (1) The broadband oscillation classification method based on deep reinforcement learning and pattern mining proposed in this application encodes multivariate time series data such as voltage and current in the power grid line into a single variable cluster sequence, and then uses the candidate patterns extracted from the cluster sequence to learn the multi-modal time series, and trains the discriminant pattern through the reinforcement learning module, and at the same time trains the classification model.

[0016] (2) In this application, a clustering method is used to classify the discretized time series data to obtain a univariate cluster sequence of the time series data, from which a multimodal time series of each sample is extracted. A neural network is used to map the category of the sample from the multimodal time series, and a reinforcement learning method is used to update the multimodal time series. While taking advantage of the high fitting of the neural network, the classification method is ensured to have a certain degree of interpretability. At the same time, it can effectively explore the time series relationship between data in the broadband oscillation phenomenon while implementing the classification algorithm.

[0017] (3) This application adopts a multi-modal time series method to extract discriminant patterns and adopts a reinforcement learning optimization framework. Therefore, the oscillation classification model proposed in this application has good reliability, scalability and generalizability, and can discriminate time series data more quickly when processing similar data or processing additional tasks.

[0018] (4) This application adopts the Soft Actor-Critic method to establish a reinforcement learning framework. This method adopts the idea of ​​offline reinforcement learning and the maximum entropy action selection strategy, which can ensure the stability of network training and the exploratory nature of action selection, making it difficult for network training to fall into local optimality, thereby improving the speed and accuracy of network training.

[0019] (5) The reinforcement learning method adopted in this application uses the SAC algorithm as the input for updating the Critic network (i.e., the state evaluation network and the action evaluation network) as the multimodal time series of the current sample and the hidden state of its mapping space, so that the update of the Critic network and the deep neural network are synchronized, which speeds up the fitting process and improves the efficiency of training. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 It is a flow chart of a feature extraction method for multivariate sequence data; Figure 2 Training flow chart for the shock classification model; Figure 3 is a deep reinforcement learning framework diagram; where M is a multimodal time series, h is a hidden state extracted by the feature extraction module, S represents a Critic network including a first state evaluation network, a second state evaluation network, a first action evaluation network, and a second action evaluation network, r represents a reward, π θ Represents an actor network; is the predicted shock category vector, y is the real shock category vector; Figure 4 This is a flow chart of a broadband oscillation classification method based on deep reinforcement learning and pattern mining; Figure 5 ROC curves of different models. DETAILED DESCRIPTION

[0021] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0022] Reference Figure 1, a feature extraction method for multivariate sequence data proposed in this embodiment includes the following steps: S1. Obtain a sample set X of multivariate time series data Xn marked with shock categories, and transform X into a multivariate sequence X', where the essence of X' is a set of single time point data; X={X1,X2,…,Xn,…,XN} Xn={x(n,1),x(n,2),…,x(n,j),…,x(n,t n )} X'={x(n,j)|1≤j≤t n ,1≤n≤N} Among them, X1, X2, Xn, and XN represent the 1st, 2nd, nth, and Nth multivariate time series data samples respectively; x(n,1),x(n,2),x(n,j),x(n,t n ) represent the time series data samples Xn at the 1st, 2nd, jth, and tth n The collected data is at time steps; N is the number of samples, t n is the number of time slots contained in Xn; 1≤j≤t n , 1≤n≤N; It is worth noting that each sample Xn may have one or more oscillations. The oscillation category of the sample can be represented by a vector of length m, where m is the number of oscillation categories. For example, when m=8, the oscillation category vector is denoted as G. G={G1;G2;G3;G4;G5;G6;G7;G8} G1 indicates oscillation caused by poor matching between generator and load, G2 indicates oscillation caused by dynamic characteristics of power electronic equipment, G3 indicates oscillation caused by delayed response of generator control system, G4 indicates oscillation caused by load fluctuation, G5 indicates oscillation caused by high-frequency response of power electronic equipment, G6 indicates nonlinear oscillation caused by nonlinear characteristics of power electronic device, G7 indicates oscillation caused by harmonics, G8 indicates oscillation caused by system short circuit or system disturbance; When the sample has oscillation Gg, Gg is expressed as the amplitude of this type of oscillation, in KHz; when the sample does not have oscillation Gg, Gg is expressed as 0.

[0023] For example, assuming that the oscillation category vector of sample X1 is recorded as (10;5;0;0;0;0;0;0), the amplitude of oscillation G1 caused by the poor matching between the generator and the load of X1 is 10KHz, the amplitude of oscillation G2 caused by the dynamic characteristics of power electronic equipment is 5, and X1 does not have other types of oscillations.

[0024] S2. Cluster the data x(n, j) in the multivariate sequence X' to obtain C clusters, and obtain the mapping relationship between the data x(n, j) and the clusters; define K(n, j) as the cluster to which the data x(n, j) belongs, that is, K(n, j)∈K; K={K1,K2,…,Kc,…,KC} Among them, K represents the cluster set, K1, K2, Kc, and KC represent the 1st, 2nd, cth, and Cth clusters obtained by clustering respectively, and C is the total number of clusters.

[0025] In specific implementation, the Toeplitz inverse covariance clustering method can be used to cluster the data x(n, j) in X' to obtain a cluster set K.

[0026] S3, obtain the clustering sequence corresponding to Xn, denoted as U(n); U(n)={K(n,1),K(n,2),…,K(n,j),…,K(n,t n )} S4. For the cluster sequence U(n) corresponding to the sample Xn under the same oscillation category, extract the common subsequence to form a continuous subsequence, and use the continuous subsequence as the pattern candidate of the corresponding category; obtain the union of the pattern candidates of all oscillation categories and record it as L; L={L1,L2,…,Lp,…,LP} Where P is the number of continuous subsequences in the union L, L1, L2, Lp, LP are the 1st, 2nd, pth, Pth continuous subsequences in the union L respectively; obviously, each continuous subsequence Lp is the numbered set of at least two clusters in the cluster set K; S5. For U(n)={K(n,1),K(n,2),…,K(n,j),…,K(n,t n )}, replace the continuous cluster number arrays that constitute the continuous subsequences in the union L with the continuous subsequences, extract the continuous subsequences to form the multi-mode time series Xn and output them.

[0027] Specifically, step S5 specifically includes the following sub-steps: S51, constructing and clearing a feature candidate set of Xn; S52, searching in U(n) for a continuous combination of values ​​that can form any continuous subsequence in the union L; S53, storing the continuous subsequence formed by the continuous numerical combination into the feature candidate set, and then deleting the continuous numerical combination from U(n); In specific implementation, in step S53, the continuous value combination that can constitute a continuous subsequence is always searched from the first value.

[0028] S54, determining whether there is a continuous numerical combination in U(n) that can constitute any continuous subsequence in the union set L; If yes, return to step S53; If not, the continuous subsequences in the feature candidate set are arranged in order to form a multi-modal time series output.

[0029] Reference Figure 2 , Figure 3 , the present application also proposes a shock classification model training method, comprising the following steps: St1, using the above steps S1-S5, the acquired multivariate time series data Xn is converted into a multi-mode time series as a learning sample and stored in the experience pool, and the learning sample is marked with the shock category; construct a basic model; the basic model includes a feature extraction network and a SAC network; the SAC network includes an actor network, a first state evaluation network, a second state evaluation network, a first action evaluation network and a second action evaluation network; The feature extraction network generates a hidden state h and a predicted category y' for a multimodal time series; the actor network generates an adjustment strategy based on the multimodal time series and h; the multimodal time series forms a new multimodal time series after executing the adjustment strategy; the first state evaluation network generates a first state evaluation value based on the multimodal time series, h and a reward r; the second state evaluation network generates a second state evaluation value based on the multimodal time series, h and a reward r; the first action evaluation network generates a first action evaluation value based on the multimodal time series, h and a reward r; the second action evaluation network generates a second action evaluation value based on the multimodal time series, h and a reward r; St2, randomly select multiple samples from the experience pool and input them into the basic model, calculate the loss function and update the feature extraction network according to the predicted category and the real oscillation category, calculate the reward r, action loss Lq(1), Lq(2) and state loss Lv, update the first action evaluation network according to Lq(1), update the second action evaluation network according to Lq(2), update the first state evaluation network according to Lv, and copy the first state evaluation network to the second state evaluation network; ; ; Among them, m is the number of shock categories, f i is the representation of sample s in the i-th oscillation category; is the value of the ith oscillation in the predicted category obtained by the feature extraction module for sample s; y i is the value of the i-th shock in the shock category marked by sample s; ε is the smoothing parameter; Lq(1)=∑ s∈B [q(s,a;w1)-U(q)] 2 / |B| Lq(2)=∑ s∈B [q(s,a;w2)-U(q)]2 / |B| Where s represents the multimodal time series of the basic model as sample input; B is the training batch, that is, a batch of samples input to the basic model; |B| is the training batch size, that is, the number of samples in a batch; w1 represents the approximate parameters of the first action evaluation network, a represents the policy action generated by the actor network when the basic model input sample is s; q(s,a;w1) represents the first action evaluation value output by the first action evaluation network when the basic model input sample is s and the action strategy is a; w2 represents the approximate parameters of the second action evaluation network, q(s,a;w2) represents the second action evaluation value output by the second action evaluation network when the basic model input sample is s and the action strategy is a; U(q) represents the true value estimate of the sample; U(q)=r+γv(s') s' is the new multi-modal time series formed by adjusting s in combination with action strategy a when the basic model input is s; v(s') is the first state evaluation value output by the first state evaluation network when the basic model input is s'; γ is the set reward discount coefficient, which can be specifically set to 0.05; r is the reward function corresponding to s; Lv=∑ s∈B [q(s,a)-U(v)] 2 / |B| U(v)=E a~π(a|s;θ) [ (s,a;wi')-αlnπ(a|s;θ)] Where π(a|s;θ) is the probability set of the actor network output; θ is the approximate parameter of the first state value network; q(s,a) is the probability value corresponding to action strategy a in π(a|s;θ), U(v) is the true value estimate of sample s; E a~π(a|s;θ) represents the expectation of the distribution of π(a|s;θ); (s,a;wi') means taking the smaller value between q(s,a;w1) and q(s,a;w2), q(s,a;w1) and q(s,a;w2) respectively represent the action evaluation values ​​output by the first action evaluation network and the second action evaluation network when the basic model input sample is s and the action strategy is a; α is the entropy parameter in the maximum entropy exploration in the SAC algorithm, and its size indicates the enthusiasm for exploring non-optimal paths; ln represents the natural logarithm; St3, combine the updated first action evaluation network to calculate the strategy loss La, and update the Actor network based on La; La=∑ s∈B E a~π(a|s;θ) [q 0 (s,a)-αlnπ(a|s;θ)] / |B| Among them, q0 (s, a) represents the first action evaluation value generated by the updated first action evaluation network when the basic model input is s; St4, input s into the updated basic model, the Actor network obtains the action strategy and the new multi-mode time series, and associates the new multi-mode time series with the original oscillation category as a sample and puts it into the experience pool; St5, determine whether the number of basic model updates reaches the set number; if not, return to step St2; if yes, fix the feature extraction network as the oscillation classification model, and the oscillation classification model predicts the oscillation category according to the input multi-modal time series.

[0030] Reference Figure 4 , a broadband oscillation classification method based on deep reinforcement learning and pattern mining proposed in this application comprises the following steps: SA1, collect power system operating data, denoted as X0={x(0,1),x(0,2),…,x(0,z),…,x(0,t 0 )};x(0,1), x(0,2), x(0,z), x(0,t 0 ) represent the 1st, 2nd, zth, and tth in X0 respectively. 0 The collected data at time steps, 1≤z≤t 0 ; SA2, obtain the clustering result obtained in step S2 of the feature extraction method for multivariate sequence data, and then perform clustering on x(0,z)(1≤z≤t 0 ) to classify, obtain the cluster number of each collected data x(0,z), and construct a cluster sequence; then execute steps S3-S5 to obtain the multi-mode time series corresponding to X0 as observation data; Specifically, the method for classifying x(0,z) is as follows: calculate the distance between x(0,z) and the centroid of each cluster in S2, and select the cluster corresponding to the minimum distance as the cluster to which x(0,z) belongs.

[0031] SA3. Input the observed data into the shock classification model to obtain the predicted category.

[0032] The broadband oscillation classification method and oscillation classification model based on deep reinforcement learning and pattern mining are explained and verified in combination with specific embodiments below.

[0033] In this embodiment, about 15,000 multivariate time series data Xn are extracted from the historical data of the power system, each of which contains 20,000 time slots, and the time slot length is set to 0.02 seconds; the data elements of the multivariate time series data Xn include three-phase voltage, three-phase current and power; In this embodiment, the oscillation category is set in combination with the cause of the oscillation, and specifically includes eight categories, namely: oscillation caused by poor matching between the generator and the load, oscillation caused by the dynamic characteristics of the power electronic equipment, oscillation caused by the reaction delay of the generator control system, oscillation caused by load fluctuation, oscillation caused by the high-frequency response of the power electronic equipment, nonlinear oscillation caused by the nonlinear characteristics of the power electronic device, oscillation caused by harmonics, and oscillation caused by system short circuit or system disturbance.

[0034] In the calculation process of this embodiment, the smoothing parameter ε is set to 0.1.

[0035] In this embodiment, the extracted multivariate time series data Xn is divided into a training set and a test set.

[0036] When implementing the broadband oscillation classification method based on deep reinforcement learning and pattern mining proposed in this application, the above steps St1-St5 are used to construct the oscillation classification model proposed in this application, the feature extraction network adopts the RNN network, and the above steps S1-S5 are used to process the data in the training set into training samples for training the oscillation classification model.

[0037] After the oscillation classification model converges, it is used to oscillate the data in the test set to verify the model performance. First, the data in the test set is clustered based on the clustering clusters of the training samples, and the multi-modal time series of each data in the test set is constructed based on the clustering results as the test sample. Then the test sample is input into the trained oscillation classification model to predict the oscillation category.

[0038] In order to verify the effectiveness of the oscillation classification model proposed in this application, the oscillation classification model UMR proposed in this application is simulated and compared with the comparison model, including the traditional convolutional neural network RNN, the classification model Transformer with multi-head attention mechanism, and the global feature algorithm DTWD. The input of the comparison model is multivariate time series data Xn, and the output is the oscillation category.

[0039] In this implementation, the three comparison models are all trained using machine learning algorithms on the training set, and then tested on the test set. Table 1 is a comparison of the accuracy of the four models on the test set. It can be seen that the UMR model has a leading accuracy compared to other models, and its AUC (Area Under Curve, the area under the ROC curve and the coordinate axis) leads by a large margin.

[0040] Table 1 Experimental results of each algorithm ; Figure 5 This is the ROC diagram of the four algorithms. Combined with Table 1, Figure 5It can be seen that the UMR model provided in this application not only has better accuracy and precision than other deep learning models, but its algorithm design mechanism also ensures that it has interpretability that other deep learning models lack. This is because the UMR model provided in this application not only completes the classification, but also refines the timing relationship in the data when broadband oscillations occur, which is something that general deep learning models cannot do.

[0041] Of course, for those skilled in the art, the present application is not limited to the details of the above exemplary embodiments, but also includes the same or similar structures that can be implemented in other specific forms without departing from the spirit or basic characteristics of the present application. Therefore, no matter from which point of view, the embodiments should be regarded as exemplary and non-restrictive, and the scope of the present application is defined by the attached claims rather than the above description, so it is intended to include all changes that fall within the meaning and scope of the equivalent elements of the claims. Any figure mark in the claims should not be regarded as limiting the claim involved.

[0042] In addition, it should be understood that although the present specification is described according to implementation modes, not every implementation mode contains only one independent technical solution. This description of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment may also be appropriately combined to form other implementation modes that can be understood by those skilled in the art.

[0043] The technologies, shapes, and structural parts not described in detail in this application are all well-known technologies.

Claims

1. A broadband oscillation classification method based on deep reinforcement learning and pattern mining, characterized in that: Firstly, the oscillation classification model is trained by using the Markov algorithm. The input of the oscillation classification model is the multi-mode time series obtained by encoding the multivariate time series data representing the working conditions of the power system, and the output is the predicted oscillation category. The method of encoding multivariate time series data to obtain multimodal time series is as follows: first, the multivariate time series data collected in a continuous time period is discretized into single time point data, and then the single time point data is clustered, and each single time point data in the multivariate time series data is replaced with the corresponding cluster cluster number to form a cluster sequence; the common subsequence of the cluster sequence corresponding to the multivariate time series data number under different oscillation categories is obtained, and the common subsequence in the cluster sequence corresponding to the multivariate time series data is extracted to form a multimodal time series; Acquire multivariate time series data as the test object, combine it with the known clustering results, obtain the clustering sequence of the test object, and extract the common subsequence to form the multi-mode time series of the test object; input the multi-mode time series of the test object into the oscillation classification model to obtain the oscillation category prediction value of the test object.

2. The broadband oscillation classification method based on deep reinforcement learning and pattern mining according to claim 1, characterized in that: The training method of the oscillation classification model includes the following steps: St1. Obtain multivariate time series data from the historical operation data of the power system and convert it into a multi-mode time series as a learning sample and store it in the experience pool. The learning sample is marked with the oscillation category; build a basic model; the basic model includes a feature extraction network and a SAC network; the SAC network includes an actor network, a first state evaluation network, a second state evaluation network, a first action evaluation network and a second action evaluation network; The feature extraction network generates a hidden state h and a predicted category y' for the multimodal time series; the actor network generates an adjustment strategy based on the multimodal time series and h; the multimodal time series forms a new multimodal time series after executing the adjustment strategy; the inputs of the first state evaluation network, the second state evaluation network, the first action evaluation network, and the second action evaluation network are all the multimodal time series, h, and the reward r, and the outputs are the first state evaluation value, the second state evaluation value, the first action evaluation value, and the second action evaluation value, respectively; The calculation formula of reward r is: ； ； Among them, m is the number of shock categories, f i is the representation of sample s in the i-th oscillation category; is the value of the ith oscillation in the predicted category obtained by the feature extraction module for sample s; y i is the value of the ith shock in the shock category marked by sample s; ε is the set smoothing parameter; St2, randomly select multiple samples from the experience pool and input them into the basic model, calculate the loss function and update the feature extraction network according to the predicted category and the real shock category; calculate the reward r as well as the action loss and state loss; update the first action evaluation network and the second action evaluation network according to the action loss, update the first state evaluation network according to the state loss, and copy the first state evaluation network to the second state evaluation network; St3, calculate the strategy loss based on the updated first action evaluation network, and update the Actor network based on the strategy loss; St4, input sample s into the updated basic model, the Actor network obtains the action strategy and the new multi-mode time series, and associates the new multi-mode time series with the original oscillation category as a sample and puts it into the experience pool; St5, determine whether the number of basic model updates reaches the set number; if not, return to step St2; if yes, fix the feature extraction network as the oscillation classification model, and the oscillation classification model predicts the oscillation category according to the input multi-modal time series.

3. The broadband oscillation classification method based on deep reinforcement learning and pattern mining as claimed in claim 2, characterized in that: The action loss function is: Lq(i') = ∑ s∈B [q(s,a;wi') - U(q)] 2 / |B|, i' = 1 or 2; U(q)=r+γv(s') Where s represents the multimodal time series of the basic model as sample input; B is the training batch, |B| is the training batch size; wi' represents the approximate parameters of the i'th action evaluation network, a represents the policy action generated by the actor network when the basic model input sample is s; q(s,a;wi') represents the action evaluation value output by the i'th action evaluation network when the basic model input sample is s and the action strategy is a; U(q) represents the true value estimate of the sample; s' is the new multimodal time series formed by adjusting s in combination with the action strategy a; v(s') is the first state evaluation value output by the first state evaluation network when the basic model input is s'; γ is the set reward discount coefficient; In step St3, the first action evaluation network is updated according to the first action loss Lq(1), and the second action evaluation network is updated according to the second action loss Lq(2).

4. The broadband oscillation classification method based on deep reinforcement learning and pattern mining as claimed in claim 2, characterized in that: The state loss function is: Lv=∑ s∈B [q(s,a)-U(v)] 2 / |B| U(v)=E a~π(a|s;θ) [ (s,a;wi')-αlnπ(a|s;θ)] Where s represents the multimodal time series of the basic model as sample input; B is the training batch, |B| is the training batch size; π(a|s;θ) is the probability set of the actor network output; θ is the approximate parameter of the first state value network; q(s,a) is the probability value corresponding to the action strategy a in π(a|s;θ), and U(v) is the true value estimate of sample s; E a~π(a|s;θ) represents the expectation of the distribution of π(a|s;θ); (s,a;wi') means taking the smaller value between q(s,a;w1) and q(s,a;w2), q(s,a;w1) and q(s,a;w2) respectively represent the action evaluation values ​​output by the first action evaluation network and the second action evaluation network when the basic model input sample is s and the action strategy is a; α is the entropy parameter in the maximum entropy exploration in the SAC algorithm.

5. The broadband oscillation classification method based on deep reinforcement learning and pattern mining as claimed in claim 4, characterized in that: The policy loss function is: La=∑ s∈B E a~π(a|s;θ) [q0(s,a)-αlnπ(a|s;θ)] / |B| Among them, q0(s,a) represents the first action evaluation value generated by the updated first action evaluation network when the basic model input is s.

6. The broadband oscillation classification method based on deep reinforcement learning and pattern mining as claimed in claim 2, characterized in that: The acquisition of learning samples in St1 includes the following steps: S1. Obtain a set of multivariate time series data marked with shock categories, and discretize the multivariate time series data in the set in units of time points to obtain a set of single time point data; S2, clustering the data in the set of single time point data to obtain the mapping relationship between each data in the multivariate time series data and the clustering clusters; S3, replacing the data in the multivariate time series data with the serial numbers of the corresponding clusters to form a cluster sequence; S4, extracting common subsequences from clustered sequences under the same oscillation category to form continuous subsequences, and using the continuous subsequences as pattern candidates for the corresponding category; Get the union L of pattern candidates of all oscillation categories; S5. For the clustering sequence of the multivariate time series data, replace the continuous cluster number array with a continuous subsequence, and extract the continuous subsequence to form a multi-mode time series output of the multivariate time series data.

7. The broadband oscillation classification method based on deep reinforcement learning and pattern mining according to claim 6, characterized in that: In S2, the Toeplitz inverse covariance clustering method is used for clustering; step S5 specifically includes the following sub-steps: S51, constructing a feature candidate set for multivariate time series data, where the feature candidate set is initially an empty set; S52, searching for a continuous numerical combination constituting any continuous subsequence in the union L in the cluster sequence of the multivariate time series data; S53, combining continuous numerical values ​​to form a continuous subsequence and storing it in a feature candidate set, and then deleting the continuous numerical combination from the clustering sequence; S54, judging whether there is a continuous value combination in the cluster sequence that can constitute any continuous subsequence in the union set L; If yes, return to step S53; If not, the continuous subsequences in the feature candidate set are arranged in order to form a multi-modal time series output.

8. The broadband oscillation classification method based on deep reinforcement learning and pattern mining as claimed in claim 2, characterized in that: The feature extraction network uses RNN network.

9. A broadband oscillation classification system based on deep reinforcement learning and pattern mining, characterized in that: It includes a memory and a processor, the memory stores a computer program, the processor is connected to the memory, and the processor is used to execute the computer program to implement the broadband oscillation classification method based on deep reinforcement learning and pattern mining as described in any one of claims 1 to 8.

10. A storage medium, characterized in that: A computer program is stored, and when the computer program is executed, it is used to implement the broadband oscillation classification method based on deep reinforcement learning and pattern mining as described in any one of claims 1 to 8.

Images