Power plant anti-misoperation monitoring optimization method based on deep reinforcement learning and related system

Abstract

The invention discloses a power plant anti-misoperation monitoring optimization method based on deep reinforcement learning and a related system. The system comprises a data preprocessing module, an environment simulation module and a detection agent module. Firstly, sensor data of the state of secondary power transformation equipment such as a disconnecting link in a power grid is obtained in a data preprocessing module, standardized processing is carried out on the data, the data is transmitted to a next module, and a data set for model training is formed. The environment simulation module and the detection agent module jointly form the core of deep reinforcement learning. The environment simulation module provides a simulated environment for interaction with an intelligent agent, reads a series of equipment state change events formed by related sensor readings, and extracts a feature set. The detection intelligent body internally comprises a power plant secondary power transformation equipment state anti-error detection model which is mainly responsible for judging the equipment state of an input sensor and adjusting the model according to a result fed back by a simulation environment. According to the method, the model is continuously updated through continuous interactive circulation of the detection intelligent agent and environment simulation, so that power plant anti-error monitoring which is updated in real time can be optimized.

Description

Technical Field

[0001] The present invention belongs to the field of intelligent monitoring of power plants, and mainly relates to an anti-error monitoring optimization method for power plants based on deep reinforcement learning and related systems. Background Art

[0002] In recent years, with the rapid development of information technology, the continuous expansion of the network scale and the continuous development of various network applications, the safe operation of power plants has become particularly important. For example, during the process of a power plant restoring its system, due to the operator mistakenly calling out the circulating water screen of another unit and blindly shutting down the running circulating pump, the low-vacuum protection of the unit was triggered, the automatic main steam valve was closed, and finally the unit could not be started. Also, due to reasons such as data transmission delay or inaccurate processing algorithms, the monitoring data cannot timely and accurately reflect the true state of the equipment or system. Such situations are common, not only because of insufficient manual monitoring and anti-error monitoring means, but also because of deficiencies in the monitoring algorithms. In order to further improve the anti-error monitoring ability of power plants, reduce the risk of misoperation, and be able to effectively carry out fault diagnosis and early warning, therefore, improving the accuracy and stability of the anti-error monitoring algorithm of power plants and realizing real-time monitoring and intelligent optimization of the power plant operation process is a comprehensive technical challenge.

[0003] Currently, in response to the above problems in power plant anti-error monitoring, the following two solutions have been proposed. One is a real-time power dispatching anti-error control method based on a finite state machine. This method establishes a dispatching anti-error model and generates anti-error rules. In real-time power dispatching, a finite state machine can be used to describe various states of the power grid and the transition relationships between states. By defining various states of the power grid (such as normal operation state, fault state, maintenance state, etc.) and the transition conditions and operations after the transition, the finite state machine can achieve precise control and anti-error of the power dispatching process. The other is a topological analysis method for power plant anti-error operation based on a fault tree. This method is a top-down system reliability analysis method. By constructing a fault tree model of the system, analyzing the causes and paths of system failures, and establishing connections between various data and the priority nodes in the fault tree, potential misoperation nodes can be discovered, and the diagnosis of anti-error operation can be achieved.

[0004] However, these technologies have the following problems and disadvantages:

[0005] When the real-time power dispatching anti-error control method based on a finite state machine faces some current large-scale power systems, the power data and network traffic are relatively large, the response speed is slow, the accuracy of anti-error monitoring will also change, and it is usually designed for specific system structures and fault conditions, and may not be able to effectively respond to unseen system states or fault conditions. The anti-error operation topology analysis method for power plants based on a fault tree has limitations when dealing with dynamically changing systems. Various emergencies may occur during the actual operation of the system, and it is often difficult for the fault tree to be updated in real time to reflect these changes. Summary of the Invention

[0006] Invention Objective: To overcome the deficiencies in the prior art, the present invention provides an optimized method for anti-error monitoring of power plants based on deep reinforcement learning.

[0007] Technical Solution: To achieve the above objective, the technical solution adopted by the present invention is as follows:

[0008] An optimized method for anti-error monitoring of power plants based on deep reinforcement learning, comprising the following steps:

[0009] Step S1, read power data, extract and process the features of abnormal data, complete the event construction of the data, and obtain a data set.

[0010] Step S2, in deep reinforcement learning, the detection agent module extracts the state from the data set, and obtains the batch and state, and realizes loading the data set in the environment simulation module and performing data interaction with the detection agent module.

[0011] The returned data enters the detection agent module. The E-Greedy algorithm is selected as the random policy, the loss function is selected as categorical_crossentropy, and the actual network and the target network adopt CNN. The input part receives the state and reward data input from the environment simulation module and passes them to the experience replay pool and the actual network. The output part is used to return the action value selected by the E-Greedy algorithm to the environment simulation module, thereby starting the interactive training of the next round of the detection agent module and the environment simulation module.

[0012] Step S4, put the predicted Q value generated in the actual network and the target Q value generated in the target network into the Adam optimizer to calculate the loss value of the model on each sample.

[0013] Step S5: The output part is used to return the action value selected by the E-Greedy algorithm to the environment simulation module. The environment simulation module compares it with the actual action value to set the reward. At the same time, the parameter values are continuously adjusted in the Adam optimizer, exploring the possibility of obtaining benefits from other choices while making the known optimal choice in the current state, continuously optimizing the detection agent module, and finally realizing the continuous update of the model.

[0014] Step S6: Use the optimized detection agent module to achieve anti-error warning for the power plant.

[0015] Preferably: The method of loading the data set in the environment simulation module and performing data interaction with the detection agent module in Step S2:

[0016] Step S201: Read the standard data set saved after data processing, perform state extraction, and load it in memory in the form of a DataFrame.

[0017] Step S202: Obtain a batch. Read out the data set in the form of a DataFrame saved in memory according to the defined batch_size, and save the read data in the class variable tempbatch.

[0018] Step S203: Obtain the state. Read the batch data by calling the function to obtain the Batch, read out the data saved in tempbatch, separate the data for training, and return it to the intelligent detector in the form of a feature vector.

[0019] Step S204: Judge the action, which is used to judge the action value received from the detection agent module. The action corresponds to the label of the power anomaly data. If the input action matches the actual action value, set the reward value to 1; otherwise, set it to 0. After the action judgment is completed, continue to call the function to obtain the Batch, and return the obtained state and reward to the detection agent module together.

[0020] Preferably: The method of interactive training between the detection agent module and the environment simulation module in Step S3 is as follows:

[0021] Step S301: The experience replay pool collects the action values from the random policy, as well as the state and reward input by the environment simulation module.

[0022] Step S302. Then, through random batch sampling, data of the size of sample_batch sampled is respectively passed to the actual network and the target network. The data passed to the actual network is state and action, and the data passed to the target network is next state and reward. Reward refers to the reward given for triggering the anti-error warning. The experience replay pool is implemented by a buffer.

[0023] Step S303. In the actual network, the anti-error operation recognition model is constructed using a 1D convolutional neural network for anti-error operation recognition.

[0024] Step S304. The detection agent module, through continuous interaction with the environment simulation module, pursues the action that maximizes the reward under different states, so as to better adapt to the environment. The detection agent module passes the predicted Q value generated in the convolutional neural network to the random policy module.

[0025] Step S305. The action generated by the random policy is then passed to the environment simulation module to start the interactive training of the next round of the detection agent module and the environment simulation module.

[0026] Preferably: In step S302, the buffer of the experience replay pool stores quadruples in the format of (state, action, reward, next state). When obtaining data from the buffer, a batch of quadruples is randomly returned.

[0027] Preferably: The specific steps in step S303 are as follows:

[0028] Step S3031. The first two 1D convolutional neural networks are set to contain 64 filters and the kernel_size is 3.

[0029] Step S3032. The last two 1D convolutional neural networks are set to contain 128 filters and the kernel_size is 3.

[0030] Step S3033. In the convolutional layer, ReLU is selected as the activation function. ReLU introduces non-linearity, which helps the neural network capture complex patterns and relationships.

[0031] Step S3034. Use 1D maxpooling to concatenate the two 1D convolutional neural networks, and after passing through the flattening layer, input it to the fully connected layer MLP, and use Softmax to achieve classification.

[0032] Preferably, the anti-misoperation recognition model includes an input layer, a first convolutional layer, a second convolutional layer, a first max pooling layer, a third convolutional layer, a fourth convolutional layer, a second max pooling layer, a flattening layer, a first fully connected layer, a second fully connected layer, and an output layer that are connected in sequence. The activation functions used in the first convolutional layer, the second convolutional layer, the third convolutional layer, and the fourth convolutional layer are ReLU activation functions. The activation function used in the first fully connected layer is the ReLU activation function, and the activation function used in the second fully connected layer is the Softmax activation function.

[0033] Preferably, the method by which the detection agent module in step S304 transfers the predicted Q value generated in the convolutional neural network to the random policy module:

[0034] Step S3041: The random policy used is the E-Greedy policy.

[0035] Step S3042, in the E-Greedy policy, a random number between 0 and 1 is generated each time. It is judged whether the random number is greater than or equal to the threshold ∈ of the probability boundary. If the condition is met, an action with the largest Q value is selected from the existing actions. If the condition is not met, an action is randomly selected from all actions.

[0036] Preferably, the E-Greedy policy formula in step S3041 is as follows:

[0037]

[0038] Among them, a represents the action value that the detection agent module can take, s represents the current state, p represents a random number between 0 and 1 randomly generated, and ∈ represents the threshold of the probability boundary, with a value between 0 and 1.

[0039] Preferably, step 4 includes the following steps:

[0040] S401, calculate the categorical_crossentropy loss value. The calculation method of categorical_crossentropy is based on the cross-entropy principle.

[0041] Step S402, by minimizing the categorical_crossentropy loss value, adjust the parameters of the model to make the prediction result of the model closer to the true label. Continuously update the parameters of the neural network, and gradually learn the optimal Q value function, so as to guide the agent to select the optimal action in different states.

[0042] Step S403, after the model training is completed, use the categorical_crossentropy loss value to evaluate the performance of the model. The smaller the loss value, the smaller the difference between the prediction result of the model and the true label, and the better the performance of the model.

[0043] Preferably, the calculation formula of the categorical_crossentropy loss value in S401 is as follows:

[0044]

[0045] where Loss represents the categorical_crossentropy loss value, C is the total number of categories, that is, the Q value, is the predicted label output by the classifier for the nth Q value, y n is the true label of the nth Q value.

[0046] Preferably, the method for constructing events for the data in step S1 includes the following steps:

[0047] Step S101, use the set R to represent the sensor readings, and use the pandas library of python to read useful data from the captured data packets, including the timestamp, device ID, sensor type, and reading value. Among them, the timestamp represents the time point when the reading is recorded, the device ID uniquely identifies the device that generates the reading, the sensor type indicates the type of the sensor that records the reading, and the reading value is the specific value measured by the sensor at the current timestamp.

[0048] R = {r1, r2, r3…r i}, r i = (t i , deviceID i , sensorType i , value i ), R represents the sensor readings, i represents the number of sensors, r i represents the i-th sensor reading, t i represents the timestamp, deviceID i represents the device ID, sensorType i represents the sensor type, value i represents the reading value.

[0049] Step S102, based on the sensor data of the secondary power transformation equipment status of the power plant, perform data cleaning, feature extraction, event construction, and data standardization on the data.

[0050] Step S1023: Use testFS to represent the extraction of statistical features from sensor readings and device events. Consider the data stream of current flowing from the power supply to the load as the forward flow based on the read values, and consider the possible reverse current as the reverse flow, forming a set containing the forward flow and reverse flow for feature extraction.

[0051] textFS = {fs1, fs2, fs3…fs i}

[0052] Among them, textFS represents the extraction of statistical features from sensor readings and device events.

[0053] Step S1024: According to the device ID and sensor type, combine the relevant sensor readings into events, and each event represents a specific state change of the device. Use the set E to represent the device state change events composed of a series of relevant sensor readings, and each event is associated with one or more devices.

[0054] E = {e1, e2, e3…e M}

[0055] Among them, E represents the device state change events composed of a series of relevant sensor readings.

[0056] Step S1025: After completing the event construction of the data, use StandardScaler to standardize the feature data and standardize all feature values to the same scale.

[0057] Preferably: The steps of data cleaning, feature extraction, event construction, and data standardization for the data in Step S102 are as follows:

[0058] Step S1021: Clean the sensor device data to remove its missing values, outliers, and duplicate data.

[0059] Step S1022: Use the set D to represent the secondary equipment in the power grid, and N is the total number of devices.

[0060] D = {d1, d2, d3…d N}. d N represents the state of the Nth secondary equipment in the power grid.

[0061] Another object of the present invention is to provide an anti-misoperation monitoring optimization system for power plants based on deep reinforcement learning, including a data preprocessing module, an environment simulation module, and a detection agent module, where:

[0062] The data preprocessing module is used to read power data, extract and process features of abnormal data, complete the event construction of the data, and obtain a data set.

[0063] The detection agent module is used to extract states from a dataset, and obtain batches and states in deep reinforcement learning, and to load the dataset in the environment simulation module and perform data interaction with the detection agent module. The returned data enters the detection agent module. The E-Greedy algorithm is selected as the random policy, the loss function is categorical_crossentropy, and the actual network and the target network adopt CNN. The input part receives the state and reward data input from the environment simulation module and passes them to the experience replay pool and the actual network. The output part is used to return the action value selected by the E-Greedy algorithm to the environment simulation module. The parameter values are continuously adjusted in the Adam optimizer to explore the possibility of obtaining benefits from other choices while making the known optimal choice in the current state. The detection agent module is continuously optimized, and finally the continuous update of the model is achieved.

[0064] The environment simulation module is used to receive the action value output by the output part of the detection agent module, and set the reward by comparing the action value output by the output part with the actual action value. The obtained state and the set reward are returned to the detection agent module.

[0065] Preferably: The detection agent module includes an experience recycling pool, an actual network, a target network, a random policy unit, and an Adam optimizer, where:

[0066] The experience recycling pool is used to collect the action values from the random policy unit and the state and reward input from the environment simulation module. Through random batch sampling, the obtained random batch sampling data are respectively passed to the actual network and the target network. The data passed to the actual network are the state and the action, and the data passed to the target network are the next state and the reward.

[0067] The actual network is used to receive the state and reward input from the environment simulation module and the state and action transmitted by the experience recycling pool. And it generates predicted Q values according to the state and reward input from the environment simulation module and the state and action received from the experience recycling pool, and transmits the generated predicted Q values to the random policy unit and the Adam optimizer respectively.

[0068] The random policy unit is used to select an action value according to the predicted Q value generated by the actual network and return the selected action value to the environment simulation module and the experience recycling pool.

[0069] The target network is used to generate target Q-values based on the received next state and reward of the experience replay pool.

[0070] The Adam optimizer optimizes based on the predicted Q-values and target Q-values, making the prediction results of the model closer to the true labels.

[0071] Preferably: The environment simulation module includes a state extraction unit, a get batch unit, a get state unit, and an action judgment unit, where:

[0072] The state extraction unit is used to read the standard data set saved after data processing, perform state extraction, and load it in memory in the form of a DataFrame.

[0073] The get batch unit is used to read out the data set in the form of a DataFrame saved in memory according to the defined batch_size, and save the read data in the class variable tempbatch.

[0074] The get state unit is used to read batch data by calling the get Batch function, read out the data saved in tempbatch, separate the data for training, and convert it into the form of feature vectors.

[0075] The action judgment unit is used to judge the action value received from the detection agent module. The action corresponds to the label of the power anomaly data. If the input action matches the actual action value, the value of the reward is set to 1. If not, the value of the reward is set to 0. After the action judgment is completed, continue to call the get Batch function, and return the obtained state and reward to the detection agent module together.

[0076] Preferably: The data preprocessing module includes a collection unit, a data cleaning unit, a feature extraction unit, an event construction unit, and a data normalization unit, where:

[0077] The collection unit is used to collect sensor readings.

[0078] The data cleaning unit is used to clean the collected sensor readings, removing their missing values, outliers, and duplicate data.

[0079] The feature extraction unit, denoted as testFS, extracts statistical features from sensor readings and device events. The data stream of current flowing from the power supply to the load with the read value is regarded as the forward flow, while the possible reverse current is regarded as the reverse flow, forming a set containing the forward flow and the reverse flow for feature extraction.

[0080] The event construction unit is used to combine relevant sensor readings into events according to the device ID and sensor type. Each event represents a specific state change of the device. The set E represents the device state change events composed of a series of relevant sensor readings, and each event is associated with one or more devices.

[0081] The data normalization unit is used to normalize the feature data using StandardScaler after the event construction of the data is completed, standardize all feature values to the same scale, and obtain the standard data set.

[0082] Another object of the present invention is to provide an electronic device, including: at least one processor, at least one memory, and a communication interface. The processor, the memory, and the communication interface communicate with each other. The memory stores program instructions executable by the processor, and the processor calls the program instructions to execute the power plant anti-error monitoring optimization method based on deep reinforcement learning.

[0083] Another object of the present invention is a non-transitory computer-readable storage medium. The non-transitory computer-readable storage medium stores computer instructions, and the computer instructions cause the computer to execute the power plant anti-error monitoring optimization method based on deep reinforcement learning.

[0084] Compared with the prior art, the present invention has the following beneficial effects:

[0085] Apply the deep reinforcement learning technology to the field of power plant anti-error monitoring, utilize the ability of the intelligent agent to interact and learn with the environment, continuously optimize the anti-error early warning model, and improve the accuracy and real-time performance of monitoring.

[0086] In the decision-making process of the intelligent agent, the E-Greedy strategy is adopted, that is, on the premise of ensuring the existing rewards, random exploration is used to discover the actions that may obtain greater rewards. This strategy enables the intelligent agent to make optimal choices under different states while maintaining a certain degree of exploration and avoiding falling into local optimal solutions.

[0087] Construct an anti-error operation recognition model based on a 1D convolutional neural network, and extract event features through a multi-layer convolution extraction module. This model structure can effectively process the temporal information and spatial features in power data and improve the accuracy of anti-error recognition. Description of the Drawings

[0088] Figure 1 It is an architecture diagram of an anti-error monitoring optimization method for power plants based on deep reinforcement learning.

[0089] Figure 2 It is a schematic diagram of a model for abnormal state recognition based on a one-dimensional convolutional neural network. Specific implementation manners

[0090] The following further clarifies the present invention in conjunction with the accompanying drawings and specific embodiments. It should be understood that these embodiments are only used to illustrate the present invention and not to limit the scope of the present invention. After reading the present invention, various equivalent forms of modification by those skilled in the art fall within the scope defined by the appended claims of this application.

[0091] As Figure 1-2 shown, the present invention designs an anti-error monitoring optimization method for power plants based on deep reinforcement learning, including the following steps:

[0092] Step S1: Read power data, extract and process the features of abnormal data, complete the event construction of the data, and obtain a data set.

[0093] In another embodiment, the method for reading and processing data in step S1 includes the following steps:

[0094] Step S101: Represent the sensor readings with the set R, and use the pandas library in Python to read useful data from the captured data packets, including the timestamp, device ID, sensor type, and reading value. Among them, the timestamp represents the time point when the reading is recorded, the device ID uniquely identifies the device that generates the reading, the sensor type indicates the type of the sensor that records the reading, and the reading value is the specific value measured by the sensor at the current timestamp.

[0095] R = {r1, r2, r3…r i}, r i = (t i , deviceID i , sensorType i , value i ), R represents the sensor readings, i represents the number of sensors, r i represents the i-th sensor reading, t i represents the timestamp, deviceID i represents the device ID, sensorType i represents the sensor type, and value i represents the reading value.

[0096] Step S102: Based on the sensor data of the secondary power transformation equipment status in the power plant, perform data cleaning, feature extraction, event construction, and data standardization on the data.

[0097] In another embodiment, the steps of step S102 for data cleaning, feature extraction, event construction, and data standardization are as follows:

[0098] Step S1021: Perform data cleaning on the sensor device data to remove its missing values, outliers (such as readings beyond the physical range), and duplicate data.

[0099] Step S1022: Use set D to represent the secondary equipment in the power grid, and N is the total number of devices.

[0100] D = {d1, d2, d3…d N}. d N represents the status of the Nth secondary equipment in the power grid.

[0101] Step S1023: Use testFS to represent the statistical features extracted from the sensor readings and device events. Regarding the data flow of current flowing from the power source to the load as the forward flow based on the read values, and regarding the possible reverse current as the reverse flow, form a set containing the forward flow and reverse flow for feature extraction.

[0102] textFS = {fs1, fs2, fs3…fs i}

[0103] where textFS represents the statistical features extracted from the sensor readings and device events.

[0104] Step S1024: According to the device ID and sensor type, combine the relevant sensor readings into events, and each event represents a specific status change of the device. Use set E to represent the device status change events composed of a series of relevant sensor readings, and each event is associated with one or more devices.

[0105] E = {e1, e2, e3…e M}

[0106] where E represents the device status change events composed of a series of relevant sensor readings.

[0107] Step S1025: After completing the event construction of the data, use StandardScaler to perform standardization processing on the feature data, and standardize all feature values to the same scale for subsequent model training.

[0108] Step S2. In reinforcement learning, the detection agent module needs to continuously interact and train with the environment to adapt to the environment. Since the dataset cannot directly interact with the detection agent, it is necessary to extract the state from the dataset and obtain the batch and state, that is, the dataset needs to be loaded in the environment simulation module and data interaction with the detection agent module needs to be carried out.

[0109] In another embodiment, the method for loading the dataset in the environment simulation module and performing data interaction with the detection agent module in step S2 is as follows:

[0110] Step S201. Read the standard dataset saved after data processing, extract the state, and load it in memory in the form of a DataFrame.

[0111] Step S202. Obtain the batch. Read out the dataset in the form of a DataFrame saved in memory according to the defined batch_size, and save the read data in the class variable tempbatch.

[0112] Step S203. Obtain the state. Read the batch data by calling the function to obtain the Batch, read out the data saved in tempbatch, separate the data for training, and return it to the intelligent detector in the form of a feature vector.

[0113] Step S204. Judge the action. It is used to judge the action value received from the detection agent module. The action corresponds to the label of the power anomaly data. If the input action matches the actual action value, the value of the reward is set to 1, and if it does not match, it is set to 0. After the action judgment is completed, continue to call the function to obtain the Batch, and return the obtained state and reward to the detection agent module together.

[0114] Step S3. The returned data enters the detection agent module. The E-Greedy algorithm is selected as the random policy, the loss function is selected as categorical_crossentropy, and the actual network and the target network adopt CNN. The input part receives the state and reward data input from the environment simulation module and passes them to the experience replay pool and the actual network. The output part is used to return the action value selected by the E-Greedy algorithm to the environment simulation module, thereby starting the interactive training of the detection agent module and the environment simulation module in the next round.

[0115] In another embodiment, the method for the interactive training of the detection agent module and the environment simulation module in step S3 is as follows:

[0116] Step S301, the experience replay pool collects the action values from the random policy, as well as the state and reward input by the environment.

[0117] Step S302, then through random batch sampling, the data of the sampled sample_batch size is respectively passed to the actual network and the target network. The data passed to the actual network is the state and action, and the data passed to the target network is the next state and reward. The reward refers to the reward given for triggering the anti-error warning. The experience replay mechanism is implemented by a buffer, and the buffer stores quadruples in the format of (state, action, reward, next state). When obtaining data from the buffer, a batch of quadruples is randomly returned.

[0118] Step S303, in the actual network, the anti-error operation recognition model is constructed using a 1D convolutional neural network for anti-error operation recognition. The anti-error operation recognition model includes 4 layers of 1D convolutional extraction modules for extracting event features. Its network structure is as Figure 2 shown. The anti-error operation recognition model includes an input layer, a first convolutional layer, a second convolutional layer, a first max pooling layer, a third convolutional layer, a fourth convolutional layer, a second max pooling layer, a flattening layer, a first fully connected layer, a second fully connected layer, and an output layer connected in sequence. The activation functions used in the first convolutional layer, the second convolutional layer, the third convolutional layer, and the fourth convolutional layer are ReLU activation functions. The activation function used in the first fully connected layer is the ReLU activation function, and the activation function used in the second fully connected layer is the Softmax activation function. This model contains 4 layers of 1D convolutional extraction modules. The first two 1D convolutional neural networks are set to contain 64 filters and the kernel_size is 3. The last two 1D convolutional neural networks are set to contain 128 filters and the kernel_size is 3. In the convolutional layer, ReLU (Rectified Linear Unit) is selected as the activation function. ReLU introduces non-linearity, which helps the neural network capture complex patterns and relationships. And it can effectively alleviate the vanishing gradient problem, ensuring that the gradient can be smoothly transmitted during training. 1D maxpooling is used to concatenate the two 1D convolutional neural networks, and after passing through the flattening layer, it is input to the fully connected layer (MLP), and classification is achieved using Softmax.

[0119] In another embodiment, step S303 includes the following steps:

[0120] Step S3031, the first two 1D convolutional neural networks are set to contain 64 filters and the kernel_size is 3.

[0121] Step S3032, the latter two 1D convolutional neural networks are set to have 128 filters and a kernel_size of 3.

[0122] Step S3033, in the convolutional layer, ReLU (Rectified Linear Unit) is selected as the activation function. ReLU introduces non-linearity, which helps the neural network capture complex patterns and relationships. And it can effectively alleviate the vanishing gradient problem, ensuring that the gradient can be smoothly transmitted during training.

[0123] Step S3034, use 1D maxpooling to concatenate the two 1D convolutional neural networks, and after passing through the flattening layer, input it to the fully connected layer MLP, and use Softmax for classification.

[0124] Step S304, the detection agent module, through continuous interaction with the environment, pursues the action that maximizes the reward in different states, so as to better adapt to the environment. However, the agent does not explicitly set what action to take in different states. And different from the supervised learning model, in the face of the same state, in order to explore the possible actions that maximize the reward, the agent cannot completely generate a definite action based on the Q value generated by the neural network. Instead, it needs to explore other possibilities more while ensuring the existing rewards, that is, the detection agent module passes the predicted Q value generated in the convolutional neural network to the random policy module.

[0125] The method in which the detection agent module in Step S304 passes the predicted Q value generated in the convolutional neural network to the random policy module:

[0126] In another embodiment, Step S3041: The random policy used is the E-Greedy policy.

[0127]

[0128] Among them, a represents the action value that the detection agent module can take, s represents the current state, p represents a random number randomly generated between 0 and 1, and G represents the threshold of probability demarcation, with a value between 0 and 1, which belongs to a hyperparameter.

[0129] Step S3044, each time a random number between 0 and 1 is generated, determine whether the random number is greater than or equal to the threshold ∈ of probability demarcation. If the condition is met, select an action with the largest Q value from the existing actions. If the condition is not met, randomly select an action from all actions.

[0130] In the decision-making process of the agent, the E-Greedy strategy is adopted. That is, on the premise of ensuring the existing rewards, random exploration is carried out to discover actions that may obtain greater rewards. This strategy enables the agent to make optimal choices under different states while maintaining a certain degree of exploration to avoid falling into local optimal solutions.

[0131] Step S305: The action generated by the random strategy will be passed to the environment simulation module to start the next round of interactive training for detecting the interaction between the agent module and the environment simulation module.

[0132] An anti-misoperation recognition model based on a one-dimensional convolutional neural network is constructed, and event features are extracted through a multi-layer convolution extraction module. This model structure can effectively process the temporal information and spatial features in power data and improve the accuracy of anti-misoperation recognition.

[0133] Through continuous interaction with the environment, the agent pursues actions that maximize the reward under different states, so as to better adapt to the environment. However, the agent does not explicitly set what actions should be taken under different states. Different from the supervised learning model, in the face of the same state, in order to explore possible actions with the maximum reward, the agent cannot completely generate a definite action based on the Q value generated by the neural network. Instead, it needs to explore other possibilities more on the premise of ensuring the existing rewards, that is, passing the predicted Q value generated in the convolutional neural network to the random strategy module.

[0134] Step S4: Put the predicted Q value generated in the actual network and the target Q value generated in the target network into the Adam optimizer. That is, after measuring the probability distribution predicted by the categorical_crossentropy loss function model and the probability distribution of the true label, calculate the loss value of the model on each sample to improve the detection performance and model evaluation of the model.

[0135] In another embodiment, step 4 includes the following steps:

[0136] S401: Calculate the categorical_crossentropy loss value. The calculation method of categorical_crossentropy is based on the principle of cross entropy, and the specific formula is as follows:

[0137]

[0138] Among them, Loss represents the categorical_crossentropy loss value, C is the total number of categories, that is, the Q value. is the predicted label output by the classifier for the nth Q value, y n is the true label of the nth Q value.

[0139] Step S402: By minimizing the categorical_crossentropy loss value, adjust the parameters of the model to make the prediction results of the model closer to the true labels, thereby improving the classification performance of the model. Continuously update the parameters of the neural network, and gradually learn the optimal Q-value function, so as to guide the agent to select the optimal action in different states.

[0140] Step S403: After the model training is completed, use the categorical_crossentropy loss value to evaluate the performance of the model. The smaller the loss value, the smaller the difference between the prediction result of the model and the true label, and the better the performance of the model.

[0141] Use the categorical_crossentropy loss function to make the prediction results of the model closer to the true labels, thereby improving the classification performance of the model. And by calculating the loss value for model evaluation and adjusting the parameters of the model, the optimal Q-value function can be gradually learned.

[0142] Step S5: The output part is used to return the action value selected by the E-Greedy algorithm to the environment simulation module. The environment simulation module compares with the actual action value to set the reward. At the same time, the parameter values are continuously adjusted in the Adam optimizer, exploring the possibility of obtaining benefits from other choices while making the known optimal choice in the current state, continuously optimizing the detection agent module, and finally realizing the continuous update of the model, making the anti-error warning for the power plant more accurate.

[0143] Step S6: Use the optimized detection agent module to realize anti-error warning for the power plant.

[0144] Another embodiment of the present invention provides a power plant anti-error monitoring optimization system based on deep reinforcement learning, including a data preprocessing module, an environment simulation module, and a detection agent module, wherein:

[0145] The data preprocessing module is used to read power data, extract and process the features of abnormal data, complete the event construction of the data, and obtain a data set.

[0146] In another embodiment, the data preprocessing module includes a collection unit, a data cleaning unit, a feature extraction unit, an event construction unit, and a data standardization unit, wherein:

[0147] The collection unit is used to collect sensor readings.

[0148] The data cleaning unit is used to clean the collected sensor readings, removing their missing values, outliers, and duplicate data.

[0149] The feature extraction unit, denoted as testFS, extracts statistical features from sensor readings and device events. Considering the data stream where current flows from the power supply to the load as a positive flow based on the read values, and treating possible reverse current as a reverse flow, a set containing positive and reverse flows is formed for feature extraction.

[0150] The event construction unit is used to combine relevant sensor readings into events according to the device ID and sensor type. Each event represents a specific state change of the device. The set E represents the device state change events composed of a series of relevant sensor readings, and each event is associated with one or more devices.

[0151] The data standardization unit is used to standardize the feature data using StandardScaler after the event construction of the data is completed, standardizing all feature values to the same scale to obtain a standard data set.

[0152] The detection agent module is used in deep reinforcement learning to extract the state from the data set, and obtain the batch and state, realizing loading the data set in the environment simulation module and performing data interaction with the detection agent module. The returned data enters the detection agent module. The E-Greedy algorithm is selected as the random policy, the loss function is selected as categorical_crossentropy, and the actual network and the target network adopt CNN. The input part receives the state and reward data input from the environment simulation module and passes them to the experience replay pool and the actual network. The output part is used to return the action value selected by the E-Greedy algorithm to the environment simulation module. The parameter values are continuously adjusted in the Adam optimizer, exploring the possibility of obtaining benefits from other choices while making the known optimal choice in the current state. The detection agent module is continuously optimized, and finally the continuous update of the model is realized.

[0153] In another embodiment: The detection agent module includes an experience recycling pool, an actual network, a target network, a random policy unit, and an Adam optimizer, where:

[0154] The experience replay pool is used to collect the action values from the random policy unit and the state and reward input by the environment simulation module. Through random batch sampling, the obtained random batch sampling data is respectively transmitted to the actual network and the target network. The data transmitted to the actual network is the state and action, and the data transmitted to the target network is the next state and reward.

[0155] The actual network is used to receive the state and reward input by the environment simulation module, and to receive the state and action transmitted by the experience replay pool. And it generates a predicted Q value based on the state and reward input by the environment simulation module, and the state and action received from the experience replay pool, and transmits the generated predicted Q value to the random policy unit and the Adam optimizer respectively.

[0156] The random policy unit is used to select an action value according to the predicted Q value generated by the actual network, and return the selected action value to the environment simulation module and the experience replay pool.

[0157] The target network is used to generate a target Q value based on the next state and reward received from the experience replay pool.

[0158] The Adam optimizer optimizes according to the predicted Q value and the target Q value, making the prediction result of the model closer to the true label.

[0159] The environment simulation module is used to receive the action value output by the output part of the detection agent module, and set the reward by comparing the action value output by the output part with the actual action value. It returns the obtained state and the set reward to the detection agent module.

[0160] In another embodiment, the environment simulation module includes a state extraction unit, a get batch unit, a get state unit, and an action judgment unit, where:

[0161] The state extraction unit is used to read the standard data set saved after data processing, perform state extraction, and load it into the memory in the DataFrame format.

[0162] The get batch unit is used to read out the data set in the DataFrame format saved in the memory according to the defined batch_size, and save the read data in the class variable tempbatch.

[0163] The acquisition state unit is used to read batch data by calling the acquisition Batch function, read out the data saved in tempbatch, separate the data for training, and convert it into the form of feature vectors.

[0164] The action judgment unit is used to judge the action value received from the detection agent module. The action corresponds to the label of the power anomaly data. If the input action matches the actual action value, the value of the reward is set to 1; if not, the value of the reward is set to 0. After the action judgment is completed, the acquisition Batch function is called continuously, and the acquired state and reward are returned to the detection agent module together.

[0165] First, in the data preprocessing module, sensor data of the states of secondary power transformation equipment such as disconnectors in the power grid is obtained, the data is standardized, and then passed to the next module to form a data set for model training. The environment simulation module and the detection agent module together constitute the core of deep reinforcement learning. The environment simulation module provides a simulated environment for interacting with the agent. This module reads a series of device state change events composed of relevant sensor readings and extracts a feature set. The agent can obtain a batch of training data through environment simulation, and then return the predicted result to the environment simulation. The environment simulation then returns corresponding rewards to the agent according to whether the prediction is correct. This loop is repeated multiple times until the training is completed. The detection agent internally contains a misoperation prevention detection model for the states of secondary power transformation equipment in the power plant, which is mainly responsible for judging the device state of the input sensor and adjusting the model according to the results feedback by the simulated environment. In the continuous interaction loop, the agent is continuously optimized, and finally the continuous update of the model is realized.

[0166] Another embodiment of the present invention provides an electronic device, including: at least one processor, at least one memory, and a communication interface. The processor, the memory, and the communication interface communicate with each other. The memory stores program instructions executable by the processor, and the processor calls the program instructions to execute the power plant misoperation prevention monitoring optimization method based on deep reinforcement learning.

[0167] Another embodiment of the present invention provides a non-transitory computer-readable storage medium. The non-transitory computer-readable storage medium stores computer instructions, and the computer instructions cause the computer to execute the power plant misoperation prevention monitoring optimization method based on deep reinforcement learning.

[0168] The present invention continuously updates the model through the continuous interaction loop between the detection agent and the environment simulation, so as to optimize the real-time updated power plant misoperation prevention monitoring.

[0169] The above are only the preferred embodiments of the present invention. It should be noted that for those of ordinary skill in the art, without departing from the principle of the present invention, several improvements and refinements can be made, and these improvements and refinements should also be regarded as the protection scope of the present invention.

Claims

1. An optimization method for preventing misoperation monitoring in a power plant based on deep reinforcement learning, characterized in that, Including the following steps: Step S1, read power data, extract and process the features of abnormal data, complete the event construction of the data, and obtain the data set; Step S2, in deep reinforcement learning, the detection agent module extracts the state from the data set, and obtains the batch and state, and realizes loading the data set in the environment simulation module and performing data interaction with the detection agent module; The returned data enters the detection agent module. The E-Greedy algorithm is selected as the random policy, the loss function is selected as categorical_crossentropy, and the actual network and the target network adopt CNN; the input part receives the state and reward data input from the environment simulation module and passes them to the experience replay pool and the actual network; the output part is used to return the action value selected by the E-Greedy algorithm to the environment simulation module, thus starting the interactive training of the next round of the detection agent module and the environment simulation module; Step S4, put the predicted Q value generated in the actual network and the target Q value generated in the target network into the Adam optimizer to calculate the loss value of the model on each sample; The output part is used to return the action value selected by the E-Greedy algorithm to the environment simulation module. The environment simulation module compares with the actual action value to set the reward. At the same time, the parameter value is continuously adjusted in the Adam optimizer. While making the known optimal choice in the current state, it can also explore the possibility of obtaining benefits from other choices. The detection agent module is continuously optimized, and finally the continuous update of the model is realized; Step S6, use the optimized detection agent module to realize the anti-error warning for the power plant.

2. The method for optimizing anti-error monitoring of a power plant based on deep reinforcement learning according to claim 1, wherein: The method of loading the data set in the environment simulation module and performing data interaction with the detection agent module in Step S2: Step S201, read the standard data set saved after data processing, extract the state, and load it in the memory in the DataFrame format; Step S202, obtain the batch, read out the data set in the DataFrame format saved in the memory according to the defined batch_size, and save the read data in the class variable tempbatch; Step S203, obtain the state, read the batch data by calling the GetBatch function, read out the data saved in tempbatch, separate the data for training, and return it to the intelligent detector in the form of feature vectors; Step S204: Determine the action, which is used to determine the action value received from the detection agent module. The action corresponds to the label of the power anomaly data. If the input action matches the actual action value, set the value of the reward to 1; otherwise, set it to 0. After the action is determined, continue to call the GetBatch function and return the obtained state and reward to the detection agent module together.

3. The method for optimizing the anti-misoperation monitoring of a power plant based on deep reinforcement learning according to claim 2, wherein: The interactive training method of the detection agent module and the environment simulation module in step S3 is as follows: Step S301: The experience replay pool collects the action values from the random policy, as well as the state and reward input by the environment simulation module. Step S302: Then, through random batch sampling, data with a sample batch size are respectively passed to the actual network and the target network. The data passed to the actual network are the state and action, and the data passed to the target network are the next state and reward; the reward refers to the reward given for triggering the anti-error warning; the experience replay pool is implemented by a buffer; the buffer of the experience replay pool stores quadruples in the format of (state, action, reward, next state). When obtaining data from the buffer, a batch of quadruples are randomly returned. Step S303: In the actual network, the anti-error operation recognition model is constructed using a 1D convolutional neural network for anti-error operation recognition. The specific steps are as follows: Step S3031: The first two 1D convolutional neural networks are set to contain 64 filters with a kernel_size of 3. Step S3032: The last two 1D convolutional neural networks are set to contain 128 filters with a kernel_size of 3. Step S3033: In the convolutional layer, ReLU is selected as the activation function. ReLU introduces non-linearity, which helps the neural network capture complex patterns and relationships. Step S3034: Use 1D maxpooling to concatenate the two 1D convolutional neural networks. After passing through the flattening layer, it is input to the fully connected layer MLP, and classification is achieved using Softmax. Step S304: The detection agent module, through continuous interaction with the environment simulation module, pursues the action that maximizes the reward under different states, so as to better adapt to the environment; the detection agent module passes the predicted Q value generated in the convolutional neural network to the random policy module. Step S305: The action generated by the random policy is then passed to the environment simulation module to start the next round of interactive training of the detection agent module and the environment simulation module.

4. The method for optimizing the anti-misoperation monitoring of a power plant based on deep reinforcement learning according to claim 3, wherein: The anti-misoperation recognition model includes an input layer, a first convolutional layer, a second convolutional layer, a first max pooling layer, a third convolutional layer, a fourth convolutional layer, a second max pooling layer, a flattening layer, a first fully connected layer, a second fully connected layer, and an output layer connected in sequence. The activation functions used in the first convolutional layer, the second convolutional layer, the third convolutional layer, and the fourth convolutional layer are ReLU activation functions. The activation function used in the first fully connected layer is the ReLU activation function, and the activation function used in the second fully connected layer is the Softmax activation function.

5. The method for optimizing the anti-misoperation monitoring of a power plant based on deep reinforcement learning according to claim 4, characterized in that: The method by which the detection agent module in step S304 transfers the predicted Q value generated in the convolutional neural network to the random policy module: Step S3041: The random policy used is the E-Greedy policy; The E-Greedy policy formula is as follows: Among them, a represents the action value that the detection agent module can take, s represents the current state, p represents a random number generated randomly between 0 and 1, ∈ represents the threshold for probability demarcation, and the value ranges from 0 to 1. Step S3042, in the E-Greedy policy, each time a random number between 0 and 1 is generated, it is judged whether the random number is greater than or equal to the probability demarcation threshold ∈. If the condition is met, an action with the largest Q value is selected from the existing actions. If the condition is not met, an action is randomly selected from all actions.

6. The power plant anti-error monitoring optimization method based on deep reinforcement learning according to claim 5, characterized in that: Step 4 includes the following steps: S401, calculate the categorical_crossentropy loss value. The calculation method of categorical_crossentropy is based on the principle of cross entropy; The formula for calculating the categorical_crossentropy loss value is as follows: Among them, Loss represents the categorical_crossentropy loss value, C is the total number of categories, that is, the Q value, is the predicted label output by the classifier for the nth Q value, y n is the true label of the nth Q value; Step S402, by minimizing the categorical_crossentropy loss value, adjust the parameters of the model to make the prediction result of the model closer to the true label. Continuously update the parameters of the neural network, and gradually learn the optimal Q value function, so as to guide the agent to select the optimal action in different states; Step S403, after the model training is completed, use the categorical_crossentropy loss value to evaluate the performance of the model; the smaller the loss value, the smaller the difference between the prediction result of the model and the true label, and the better the performance of the model.

7. The method for optimizing the anti-misoperation monitoring of a power plant based on deep reinforcement learning according to claim 6, characterized in that: The method for constructing events for data in step S1 includes the following steps: Step S101, use the set R to represent the sensor readings. Use the pandas library of python to read useful data from the captured data packets, including the timestamp, device ID, sensor type, and reading value. Among them, the timestamp represents the time point when the reading is recorded, the device ID uniquely identifies the device that generates the reading, the sensor type indicates the type of the sensor that records the reading, and the reading value is the specific value measured by the sensor at the current timestamp; R = {r1, r2, r3…r i}, r i = (t i , deviceID i , sensorType i , value i ), R represents sensor readings, i represents the number of sensors, r i represents the i-th sensor reading, t i represents the timestamp, deviceID i represents the device ID, sensorType i represents the sensor type, value i represents the read value; Step S102: Based on the sensor data of the secondary power transformation equipment in the power plant, perform data cleaning, feature extraction, event construction, and data standardization on the data; The steps of step S102 for data cleaning, feature extraction, event construction, and data standardization are as follows: Step S1021: Clean the sensor device data to remove its missing values, outliers, and duplicate data; Step S1022: Use set D to represent the secondary equipment in the power grid, and N is the total number of equipment; D = {d1, d2, d3…d N}; d N represents the status of the Nth secondary device in the power grid. Step S1023: Use testFS to represent the extraction of statistical features from sensor readings and device events. Consider the data stream with the current flowing from the power source to the load as the forward flow based on the read values, and consider the possible reverse current as the reverse flow, forming a set containing forward and reverse flows for feature extraction; textFS = {fs1, fs2, fs3…fs i} Among them, textFS represents the extraction of statistical features from sensor readings and device events; Step S1024: According to the device ID and sensor type, combine the relevant sensor readings into events, where each event represents a specific state change of the device; use set E to represent the device state change events composed of a series of relevant sensor readings, and each event is associated with one or more devices; E = {e1, e3, e3…e M} Among them, E represents the device state change events composed of a series of relevant sensor readings; Step S1025: After completing the event construction of the data, use StandardScaler to perform standardization processing on the feature data to standardize all feature values to the same scale.

8. An optimization system for the power plant anti-error monitoring optimization method based on deep reinforcement learning according to claim 1, characterized in that It includes a data preprocessing module, an environment simulation module, and a detection agent module, where: The data preprocessing module is used to read power data, extract and process features of abnormal data, complete the event construction of the data, and obtain a data set; The detection agent module is used in deep reinforcement learning to extract the state of the data set, and obtain batch and state, so as to load the data set in the environment simulation module and perform data interaction with the detection agent module; the returned data enters the detection agent module, selects the E-Greedy algorithm as the random policy, selects categorical_crossentropy as the loss function, and uses CNN for the actual network and the target network; the input part receives the state and reward data input from the environment simulation module and passes them to the experience replay pool and the actual network; the output part is used to return the action value selected by the E-Greedy algorithm to the environment simulation module; the parameter values are continuously adjusted in the Adam optimizer, exploring the possibility of obtaining benefits from other choices while making the known optimal choice in the current state, and the detection agent module is continuously optimized, finally realizing the continuous update of the model; The environment simulation module is used to receive the action value output by the output part of the detection agent module, and set the reward by comparing the action value output by the output part with the actual action value; return the obtained state and the set reward to the detection agent module.

9. The optimized system according to claim 8, wherein: The detection agent module includes an experience replay buffer, an actual network, a target network, a random policy unit, and an Adam optimizer, where: The experience replay buffer is used to collect the action values from the random policy unit and the state and reward input by the environment simulation module. Through random batch sampling, the obtained random batch sampling data is respectively transmitted to the actual network and the target network. The data transmitted to the actual network is the state and action, and the data transmitted to the target network is the next state and reward; The actual network is used to receive the state and reward input by the environment simulation module and the state and action transmitted by the experience replay buffer; and generate a predicted Q value based on the state and reward input by the environment simulation module and the state and action received from the experience replay buffer, and transmit the generated predicted Q value to the random policy unit and the Adam optimizer respectively; The random policy unit is used to select an action value based on the predicted Q value generated by the actual network and return the selected action value to the environment simulation module and the experience replay buffer; The target network is used to generate a target Q value based on the next state and reward received from the experience replay buffer; The Adam optimizer optimizes based on the predicted Q value and the target Q value to make the prediction result of the model closer to the true label; The environment simulation module includes a state extraction unit, a batch acquisition unit, a state acquisition unit, and an action judgment unit, where: The state extraction unit is used to read the standard data set saved after data processing, perform state extraction, and load it in memory in the form of a DataFrame; The batch acquisition unit is used to read out the data set in the form of a DataFrame saved in memory according to the defined batch size and save the read data in the class variable tempbatch; The state acquisition unit is used to read batch data by calling the Batch acquisition function, read out the data saved in tempbatch, separate the data for training, and convert it into the form of a feature vector; The action judgment unit is used to judge the action value received from the detection agent module. The action corresponds to the label of the power anomaly data. If the input action matches the actual action value, the reward value is set to 1, and if it does not match, the reward value is set to 0; after the action judgment is completed, the Batch acquisition function is called again, and the acquired state and reward are returned to the detection agent module together; The data preprocessing module includes a collection unit, a data cleaning unit, a feature extraction unit, an event construction unit, and a data normalization unit, where: The collection unit is used to collect sensor readings; The data cleaning unit is used to clean the sensor readings collected, removing their missing values, outliers and duplicate data; The feature extraction unit, denoted as testFS, extracts statistical features from sensor readings and device events. Regarding the data stream where the current flows from the power supply to the load as a forward flow based on the read values, and regarding the possible reverse current as a reverse flow, a set containing forward and reverse flows is formed for feature extraction; The event construction unit is used to combine relevant sensor readings into events according to the device ID and sensor type, and each event represents a specific state change of the device. The set E represents the device state change events composed of a series of relevant sensor readings, and each event is associated with one or more devices; The data standardization unit is used to perform standardization processing on the feature data using StandardScaler after the event construction of the data is completed, standardizing all feature values to the same scale to obtain a standard data set.

10. An electronic device, characterized in that: Including: At least one processor, at least one memory and a communication interface. The processor, memory and communication interface communicate with each other. The memory stores program instructions executable by the processor, and the processor calls the program instructions to execute the power plant anti-error monitoring optimization method based on deep reinforcement learning according to any one of claims 1-7.

Images