Data stream abnormal state detection system based on electric power informatization

Abstract

The invention discloses a data flow abnormal state detection system based on electric power informatization, and the system comprises the steps: firstly deploying an electric power data detection tool to collect data, such as data flow and protocol compliance, and setting a threshold value and a detection rule; when an abnormal data stream is detected, the system searches for an abnormal timestamp and reasons, such as personnel operation, weather change, equipment fault and the like, and establishes an abnormal data stream database for correspondence. And then, the system establishes a multi-factor anomaly analysis model by utilizing the corresponding relationships, so that the anomaly reason can be directly analyzed in the subsequent detection. The system further comprises a plurality of steps of data cleaning, feature extraction, standardized processing, log data cleaning and formatting, visual analysis and the like, so that the accuracy and reliability of anomaly detection are improved. Through the system, the abnormal state in the electric power informatization data stream can be effectively detected and analyzed, and powerful support is provided for stable operation of an electric power system.

Description

Technical Field

[0001] The present invention relates to the technical field of power information detection, and specifically to an abnormal state detection system based on power information data flow. Background Art

[0002] Power informatization refers to the application of information technology means such as communications, automatic control, computers, networks, sensors, etc. to carry out informatization transformation and upgrading of the entire process of production, transmission, sales, etc. of power enterprises, so as to drive the transformation of the power industry from a traditional industry to a highly knowledge and technology intensive industry, and provide support and lead the process of transformation for the stable operation of power enterprises' production and improvement of management level.

[0003] With the rapid development of information technology, data flow is increasingly used in many fields, including power monitoring, network security, financial transactions, etc. In these fields, the real-time and accuracy of data flow are crucial, and any abnormal data flow may have a serious impact on system stability, security and business continuity.

[0004] However, in practical applications, data stream anomaly detection faces many challenges. First, the massive, high-speed, and dynamically changing nature of data streams makes it difficult for traditional anomaly detection methods to effectively cope with them. Traditional anomaly detection methods are usually based on fixed rules and thresholds, which are difficult to adapt to the variability and uncertainty of data streams.

[0005] Secondly, the causes of data flow anomalies are often complex and diverse, and may involve multiple aspects such as human operation, equipment failure, and changes in the external environment. Therefore, accurately identifying the causes of abnormal data flow requires comprehensive consideration of multiple factors, which increases the complexity and difficulty of anomaly detection.

[0006] In addition, existing anomaly detection methods also have shortcomings in terms of efficiency and accuracy. Although some methods can detect abnormal data streams, they cannot accurately identify the causes of the anomalies, resulting in poor treatment measures and poor results. Other methods are too complex and computationally intensive to process data streams in real time in practical applications.

[0007] To this end, we propose an abnormal state detection system based on power information data flow Summary of the invention

[0008] The purpose of the present invention is to provide an abnormal state detection system based on power information data flow to solve the above-mentioned deficiencies in the prior art.

[0009] In order to achieve the above object, the present invention provides the following technical solution: A power information-based data flow abnormal state detection system comprises the following steps:

[0010] Step 1: Deploy a power data detection tool and use it to collect power data flow data. The collected data includes but is not limited to data flow volume and protocol compliance data. At the same time, set data flow volume thresholds and protocol compliance check rules.

[0011] Step 2: According to the preset data flow flow threshold and protocol compliance check rules, if one abnormality occurs, the data flow is determined to be abnormal, and the abnormal data flow is transferred to step 3. If all checks pass, the data flow is determined to be normal;

[0012] Step 3: Find the timestamp of the abnormal data flow, determine the time point when the abnormal data flow occurs, and find the cause of the abnormal data flow at that time point, where the cause includes but is not limited to human operation, weather changes, equipment failure, and large-scale enterprise activities. Find the abnormal cause that actually occurred at that time point, record the abnormal cause that actually occurred, and correspond the abnormal cause to the abnormal data flow type by establishing an abnormal data flow database;

[0013] Step 4: During the detection, if the same abnormal data flow data appears, a cause verification is performed each time the abnormal data flow appears. For each cause that may cause the abnormal data flow to appear, check whether it actually occurs at the time when the abnormal data flow occurs. Set a threshold for the number of cause verifications. If an abnormal cause corresponding to the abnormal data flow does not appear in multiple verifications, delete the cause from the corresponding relationship with the abnormal data flow. If after multiple verifications, there are still multiple abnormal causes that cannot be deleted, it means that these causes jointly cause the appearance of the abnormal data flow.

[0014] Step 5: Establish a multi-factor anomaly analysis model based on the correspondence between the abnormal data flow and the abnormal cause in the abnormal data flow database in step 4;

[0015] Step 6: When the power data flow is subsequently detected, the multi-factor anomaly analysis model is used to directly analyze the cause of the abnormal data flow.

[0016] As a further description of the above technical solution: the specific operation of collecting the store data stream data in step one is divided into the following steps:

[0017] A: Clean the power data stream data collected by the power data detection tool to remove invalid data, duplicate data and noise data to ensure the accuracy and consistency of the data;

[0018] B: Extract key features from the preprocessed data, including the changing trend of data flow and specific violations of protocol compliance, to provide a basis for subsequent anomaly detection;

[0019] C: Standardize the extracted feature data to eliminate the dimensional differences between different features and improve the accuracy of subsequent model building.

[0020] As a further description of the above technical solution: the specific method of searching for the timestamp of the abnormal data stream in step 3 is divided into the following steps:

[0021] D: Deploy Wireshark (network traffic analysis tool) on the network node to capture the data packets passing through the node;

[0022] E: Set filters to capture only packets related to power informationization data flows, analyze the captured packets, and find timestamps of abnormal data flows.

[0023] The abnormal state detection system based on power information data flow according to claim 1 is characterized in that the specific steps of establishing the abnormal data flow database in step 3 are:

[0024] F: Collect log files of the power information system, including but not limited to equipment logs, application logs and weather status logs.

[0025] As a further description of the above technical solution: the specific steps of establishing the abnormal data flow database in step 3 are:

[0026] F: Collect log files of the power information system, including but not limited to equipment logs, application logs and weather status logs;

[0027] G: Use Logstash to clean and format the log data, and then store it in Elasticsearch. Then use Kibana (log processing tool) to visualize the log data in Elasticsearch, find the log entries related to the timestamp of the abnormal data flow, and analyze the log entries to determine the cause of the abnormal data flow.

[0028] H: Design the abnormal data flow table and the abnormal reason table, and connect the abnormal data flow and the abnormal reason through foreign keys, and then import the abnormal data flow and abnormal reason data into the database through phpMyAdmin (database management tool).

[0029] As a further description of the above technical solution: 5. The specific steps of establishing the multi-factor analysis model in step 5 are:

[0030] I: Use SQL query tools to extract abnormal data flows and abnormal cause data from the database that has already imported data, and divide the extracted data into training data sets and verification data sets;

[0031] J: Use Python (programming language) to build the model and import the training dataset into the built model. Then, validate the trained model by using the validation dataset to evaluate its accuracy and reliability.

[0032] K: Use Kibana (visualization tool) to visualize the model results and generate prediction reports to further verify the reliability of the established model.

[0033] In the above technical solution, the abnormal state detection system based on power information data flow provided by the present invention has the following beneficial effects:

[0034] The system collects data such as data flow volume and protocol compliance by deploying power data detection tools, and sets corresponding thresholds and inspection rules. When an abnormal data flow is detected, the system will look for the abnormal timestamp and analyze possible causes, such as human operation, weather changes, equipment failure, etc. These abnormal causes will be recorded and an abnormal data flow database will be established. In order to improve accuracy, the system will verify the abnormal cause corresponding to each abnormal data flow multiple times. If a cause does not appear in multiple verifications, it will be deleted. Based on these corresponding relationships, the system will establish a multi-factor anomaly analysis model for direct analysis of abnormal causes in subsequent detections. The entire system also includes data cleaning, feature extraction, standardization processing, and cleaning, formatting, and visual analysis of log data to further improve the accuracy and reliability of anomaly detection. DETAILED DESCRIPTION

[0035] In order to enable those skilled in the art to better understand the technical solution of the present invention, the present invention is further described in detail below.

[0036] The embodiment of the present invention provides a technical solution: a power information-based data flow abnormal state detection system includes the following steps:

[0037] Step 1: Deploy power data detection tools. These power data detection tools include sensors and various monitoring devices. Then, the power data flow data is collected through the deployed power data detection tools. The collected data includes but is not limited to data flow volume and protocol compliance data. The collected data includes all aspects of the power system, such as power generation, transmission, and distribution. At the same time, data flow volume thresholds and protocol compliance inspection rules are set. When setting the threshold, it can be based on the average flow plus a certain standard deviation, or based on a certain proportion of the flow peak. For example, historical data of power data flow in a certain area shows that the average flow under normal circumstances is 100Mbps and the standard deviation is 20Mbps. Then, a flow threshold can be set to 120Mbps (average flow + standard deviation) or 80Mbps;

[0038] It should be noted that in order to ensure the accuracy and consistency of the collected data, the specific operation of collecting store data flow data in step 1 is divided into the following steps:

[0039] A: Clean the power data stream data collected by the power data detection tool to remove invalid data, duplicate data and noise data to ensure the accuracy and consistency of the data;

[0040] B: Extract key features from the preprocessed data, including the changing trend of data flow and specific violations of protocol compliance, to provide a basis for subsequent anomaly detection;

[0041] C: Standardize the extracted feature data to eliminate the dimensional differences between different features and improve the accuracy of subsequent model building;

[0042] Step 2: According to the preset data flow threshold and protocol compliance check rules, if the data flow exceeds or falls below the data flow threshold or does not comply with the protocol, if an abnormality occurs, the data flow is considered abnormal, and the abnormal data flow is transferred to step 3. If all checks pass, the data flow is considered normal;

[0043] Step 3: Find the timestamp of the abnormal data flow, determine the time point when the abnormal data flow occurs, and find the cause of the abnormal data flow at that time point, where the cause includes but is not limited to human operation, weather changes, equipment failure, and large-scale enterprise activities. Find the abnormal cause that actually occurred at that time point, record the abnormal cause that actually occurred, and correspond the abnormal cause to the abnormal data flow type by establishing an abnormal data flow database;

[0044] The specific method of finding the timestamp of abnormal data flow in step 3 is divided into the following steps:

[0045] D: Deploy Wireshark (network traffic analysis tool) on the network node to capture the data packets passing through the node;

[0046] E: Set filters to capture only packets related to power information data flows, analyze captured packets, and find timestamps of abnormal data flows;

[0047] It should be noted that in order to further improve the verification and detection efficiency of abnormal data flows, in various abnormal conditions, since the cause of the abnormal data flow corresponding to the protocol compliance verification is most likely to be a human factor, therefore, in the power data flow with abnormal protocol compliance verification, the cause of the abnormality is first found in the human factors to improve the detection efficiency;

[0048] It should also be noted that the specific steps for establishing the abnormal data flow database in step 3 are:

[0049] F: Collect log files of the power information system, including but not limited to equipment logs, application logs and weather status logs;

[0050] G: Use Logstash to clean and format the log data, and then store it in Elasticsearch. Then use Kibana (log processing tool) to visualize the log data in Elasticsearch, find log entries related to the timestamp of abnormal data flow, and analyze the log entries to determine the cause of the abnormal data flow.

[0051] H: Design the abnormal data flow table and abnormal reason table, and connect the abnormal data flow and abnormal reason through foreign keys, and then import the abnormal data flow and abnormal reason data into the database through phpMyAdmin (database management tool);

[0052] Step 4: During the actual operation of the power information data flow abnormal state detection system, when the system detects the same abnormal data flow data, the verification process will be triggered. This process aims to deeply analyze the potential causes behind the abnormal data flow and ensure that the identified causes are both accurate and reliable. Specifically, whenever the system captures an abnormal data flow, it will immediately start the cause verification program. The core of this program is that for each factor that may cause the abnormal data flow, the system will check whether these factors actually exist or occur at the time when the abnormality occurs. In order to further improve the rigor and reliability of the verification, the system also sets a threshold for the number of cause verifications. This means that for each potential cause associated with the abnormal data flow, the system will make multiple verification attempts. If a cause fails to be confirmed at the time when the abnormality occurs during multiple verifications, the system will automatically remove this cause from the corresponding relationship with the abnormal data flow. This mechanism ensures that the abnormal causes retained by the system are strictly verified and do exist. However, in actual operation, there may be a situation where multiple abnormal causes cannot be deleted after multiple verifications. At this time, the system will make a comprehensive judgment based on these remaining reasons and believe that they may have jointly caused the abnormal data flow. Through such a verification process, the power information data flow abnormal state detection system can more accurately identify and analyze the reasons behind the abnormal data flow, providing more powerful support for the stable operation of the power system. At the same time, this process also reflects the professionalism and rigor of the system in anomaly detection and analysis, providing a solid guarantee for the safe operation of the power system;

[0053] It should be noted that even if after multiple verifications, a high-probability cause of a type of data flow anomaly is obtained, it is still necessary to set the number of intermittent verifications. For example, after multiple verifications, only one abnormal cause is retained to correspond to the abnormal data flow, but in the subsequent detection process, after the abnormal data flow appears five times, the abnormal cause verification program for the abnormal data flow will be started when it appears for the sixth time to prevent special circumstances and improve the accuracy of abnormal cause detection of abnormal data flow;

[0054] Step 5: Establish a multi-factor anomaly analysis model based on the correspondence between the abnormal data flow and the abnormal cause in the abnormal data flow database in step 4, wherein the types of the established model include but are not limited to decision trees, random forests, support vector machines or neural networks, etc.;

[0055] Among them, the specific steps of establishing the multi-factor analysis model in step 5 are:

[0056] I: Use SQL query tools to extract abnormal data flows and abnormal cause data from the database that has already imported data, and divide the extracted data into training data sets and verification data sets;

[0057] J: Use Python (programming language) to build the model and import the training dataset into the built model. Then, validate the trained model by using the validation dataset to evaluate its accuracy and reliability.

[0058] K: Use Kibana (visualization tool) to visualize the model results and generate prediction reports to further verify the reliability of the established model. You can also use cross-validation, accuracy, recall and other indicators to evaluate the performance of the model and optimize the model based on the evaluation results;

[0059] Step 6: When the power data flow is subsequently detected, the multi-factor anomaly analysis model is used to directly analyze the cause of the abnormal data flow.

[0060] The overall detection process of the present invention is: based on the abnormal state detection system of power information data flow, its complete process is: first, deploy power data detection tools to collect power data flow data, which includes data flow flow, protocol compliance data, etc., and then clean, extract features and standardize the data to ensure the accuracy and consistency of the data. Then, use Wireshark to capture data packets at network nodes, and set filters to capture only data packets related to power information data flows, so as to find the timestamp of abnormal data flows. After finding the abnormal data flow, an abnormal data flow database will be established, which cleans and formats the log data through Logstash, and then stores it in Elasticsearch, and uses Kibana for visual analysis, combined with security event management tools such as Siemens Site Security Center, to further analyze security events and verify the abnormal causes. In this process, the timestamp and corresponding abnormal cause of each abnormal data flow will be recorded, and the system will perform multiple verifications. For each abnormal data flow, the system will check whether its corresponding abnormal cause appears in multiple verifications. If a certain abnormal cause does not appear in multiple verifications, the system will delete the cause from the corresponding relationship with the abnormal data flow. If after multiple verifications, there are still multiple abnormal reasons that cannot be deleted, it means that these reasons may have jointly caused the appearance of the abnormal data flow. Then use phpMyAdmin to import the verified abnormal data flow and abnormal cause data into the database. After that, extract the data from the database and divide it into a training data set and a verification data set. Use the Python programming language to establish a multi-factor abnormal analysis model, and import the training data set into the model. Then, use the verification data set to verify the trained model to evaluate its accuracy and reliability. In the subsequent power data flow detection, the system will directly use the multi-factor abnormal analysis model to analyze the abnormal data flow that appears, and visualize the model results through Kibana to generate a prediction report to further verify and demonstrate the reliability of the model. In this way, the system can effectively detect and analyze the abnormal state in the power information data flow, and provide strong support for the stable operation of the power system.

[0061] The above description is only by way of illustration of certain exemplary embodiments of the present invention. It is undoubted that, for those skilled in the art, the described embodiments can be modified in various ways without departing from the spirit and scope of the present invention. Therefore, the above description is illustrative in nature and should not be construed as limiting the scope of protection of the claims of the present invention.

Claims

1. Based on the abnormal state detection system of power information data flow, it is characterized by: The abnormal state detection system based on power information data flow includes the following steps: Step 1: Deploy a power data detection tool and use it to collect power data flow data. The collected data includes but is not limited to data flow volume and protocol compliance data. At the same time, set data flow volume thresholds and protocol compliance check rules. Step 2: According to the preset data flow flow threshold and protocol compliance check rules, if one abnormality occurs, the data flow is determined to be abnormal, and the abnormal data flow is transferred to step 3. If all checks pass, the data flow is determined to be normal; Step 3: Find the timestamp of the abnormal data flow, determine the time point when the abnormal data flow occurs, and find the cause of the abnormal data flow at that time point, where the cause includes but is not limited to human operation, weather changes, equipment failure, and large-scale enterprise activities. Find the abnormal cause that actually occurred at that time point, record the abnormal cause that actually occurred, and correspond the abnormal cause to the abnormal data flow type by establishing an abnormal data flow database; Step 4: During the detection, if the same abnormal data flow data appears, a cause verification is performed each time the abnormal data flow appears. For each cause that may cause the abnormal data flow to appear, check whether it actually occurs at the time when the abnormal data flow occurs. Set a threshold for the number of cause verifications. If an abnormal cause corresponding to the abnormal data flow does not appear in multiple verifications, delete the cause from the corresponding relationship with the abnormal data flow. If after multiple verifications, there are still multiple abnormal causes that cannot be deleted, it means that these causes jointly cause the appearance of the abnormal data flow. Step 5: Establish a multi-factor anomaly analysis model based on the correspondence between the abnormal data flow and the abnormal cause in the abnormal data flow database in step 4; Step 6: When the power data flow is subsequently detected, the multi-factor anomaly analysis model is used to directly analyze the cause of the abnormal data flow.

2. The abnormal state detection system based on power information data flow according to claim 1 is characterized in that: The specific operation of collecting the store data stream data in step 1 is divided into the following steps: A: Clean the power data stream data collected by the power data detection tool to remove invalid data, duplicate data and noise data to ensure the accuracy and consistency of the data; B: Extract key features from the preprocessed data, including the changing trend of data flow and specific violations of protocol compliance, to provide a basis for subsequent anomaly detection; C: Standardize the extracted feature data to eliminate the dimensional differences between different features and improve the accuracy of subsequent model building.

3. The abnormal state detection system based on power information data flow according to claim 1 is characterized in that: The specific method of finding the timestamp of the abnormal data flow in step 3 is divided into the following steps: D: Deploy Wireshark (network traffic analysis tool) on the network node to capture the data packets passing through the node; E: Set filters to capture only packets related to power informationization data flows, analyze the captured packets, and find timestamps of abnormal data flows.

4. The abnormal state detection system based on power information data flow according to claim 1 is characterized in that: The specific steps of establishing the abnormal data flow database in step 3 are: F: Collect log files of the power information system, including but not limited to equipment logs, application logs and weather status logs; G: Use Logstash to clean and format the log data, and then store it in Elasticsearch. Then use Kibana (log processing tool) to visualize the log data in Elasticsearch, find log entries related to the timestamp of abnormal data flow, and analyze the log entries to determine the cause of the abnormal data flow. H: Design the abnormal data flow table and the abnormal reason table, and connect the abnormal data flow and the abnormal reason through foreign keys, and then import the abnormal data flow and abnormal reason data into the database through phpMyAdmin (database management tool).

5. The abnormal state detection system based on power information data flow according to claim 4 is characterized in that: The specific steps of establishing the multi-factor analysis model in step 5 are: I: Use SQL query tools to extract abnormal data flows and abnormal cause data from the database that has already imported data, and divide the extracted data into training data sets and verification data sets; J: Use Python (programming language) to build the model and import the training dataset into the built model. Then, validate the trained model by using the validation dataset to evaluate its accuracy and reliability. K: Use Kibana (visualization tool) to visualize the model results and generate prediction reports to further verify the reliability of the established model.