Method and device for quickly identifying and labeling power grid log data

Abstract

The invention discloses a rapid identification and labeling method and device for power grid log data, and the method comprises the steps: collecting terminal log data through a plurality of modes, and carrying out the preprocessing of the log data; the data scale is compressed through a multi-feature fusion mechanism, and the correlation of the data is enhanced; matching and searching the extracted fusion features through a security rule base; and marking the matched and searched data on the basis of a predefined marking rule, performing data automatic marking through a depth prediction model for the data which is not covered by the marking rule, and improving the training effect and generalization ability of the prediction model by using a robust learning processing method oriented to power data. The method can effectively solve the problem of rapid data identification in multi-source heterogeneous data acquisition in a digital power grid and the problem of robust learning processing of a depth prediction model in an automatic labeling process.

Description

Technical Field

[0001] The present invention relates to the technical field of power grid data security, and in particular to a method and device for quickly identifying and marking power grid log data. Background Art

[0002] With the construction of digital power grid, the traffic of power grid business has surged, and the log data of power grid has shown explosive growth. These data contain rich information about power grid operation, but also bring huge challenges to data collection, storage, processing and analysis. Power grid log data may contain sensitive information, such as system operation status, abnormal data flow, abnormal data operation behavior, etc. These state changes and operations that may cause power grid instability or failure are also called dangerous events or events, and sometimes reversely called security events. The efficient and safe handling of these log information is directly related to the stable operation of the power grid information system and the privacy protection of users. Therefore, rapid identification and accurate labeling of power grid log data can effectively support the smooth operation of power grid digital services.

[0003] With the continuous improvement of laws and regulations on data security protection, power grid companies need to strictly abide by relevant laws and regulations to ensure the legality and compliance of data processing. The large scale, variety and high value of power data have led to the continuous expansion of the risk exposure of power grid data. In particular, there are many uncontrollable factors in the use of power grid data. How to effectively improve the reliability of data processing has become particularly important. At present, with the rapid development of intelligent algorithms such as machine learning and deep learning, prediction models have been widely used to improve engineering tasks related to data processing, such as automatic labeling, defect prediction and problem classification, and have achieved remarkable success. However, machine learning and deep learning are mostly trained in a supervised manner, which depends to a large extent on high-quality data sets. It is often necessary to collect data through multiple strategies to ensure the sample quality of the data set, but the collection of power grid log data through multiple strategies will suffer from problems such as sample mislabeling and low data labeling efficiency, which will affect the training effect and generalization ability of the model. Summary of the invention

[0004] Purpose of the invention: In order to solve the above-mentioned technical problems, the purpose of the present invention is to provide a method and device for rapid identification and labeling of power grid log data, which can effectively solve the problems of slow identification and labeling speed and low labeling accuracy in the existing power grid log data processing technology, and effectively support digital power grid data security risk monitoring, threat processing analysis and other services.

[0005] Technical solution: In the first aspect, a method for quickly identifying and marking power grid log data includes the following steps:

[0006] Data collection steps: Collect log data from multi-source heterogeneous information systems in the power grid through log file collection, database and API collection;

[0007] Data preprocessing step: preprocess the collected log data, including data deduplication, redundant data filtering, and data format standardization;

[0008] Rapid identification step: extract key features from the preprocessed log data, use a multi-feature fusion mechanism to compress the data size and strengthen the data relevance, and match and search the extracted fusion features through a security rule library that covers known knowledge of power grid data security detection rules;

[0009] Data labeling step: label the matched and searched data based on predefined labeling rules. For data not covered by these rules, use a deep prediction model to automatically label the data. The deep prediction model is trained using a robust learning processing method for power log data. The robust learning processing method for power log data constructs a clean training subset by identifying incorrectly labeled samples from the training data set, and trains a classifier with a balanced decision boundary by using class-balanced sampling, so that the prediction model is robust to both incorrectly labeled samples and class imbalance.

[0010] The data collection steps include:

[0011] Determine the data source: List all data sources that may contain useful information, evaluate each data source, and select the most relevant data source based on the purpose and needs of data collection;

[0012] Capture data: Select a suitable log capture tool based on the characteristics and requirements of the data source, configure it, run the configured tool, and start the data collection process.

[0013] In the data preprocessing step, the specific processing methods include:

[0014] Missing value processing: Check the completeness of log data attributes. If two or more attributes are missing, the log will be filtered out; otherwise, the missing values ​​will be filled with "Null" characters to complete the information;

[0015] Redundant data filtering: Sort the log data by the keyword "occurrence time" and set a fixed-size data sliding window. Slide it continuously on the sorted data set. By detecting the records in the window, find and delete the identical records, and only keep the different log information.

[0016] Format standardization: Normalize the log data, determine the selected log-related attributes, and use the extended IDMEF alarm format for format definition.

[0017] In the fast recognition step, a multi-feature fusion mechanism is used to compress the data size and enhance the data relevance, including:

[0018] By calculating the similarity between two log data, when the similarity is equal to 1, one of the log data is deleted to eliminate duplicate logs;

[0019] For data whose similarity is not equal to 1, firstly, the data is divided into multiple data blocks with a fixed size. For each data block, a sliding window is used to use rolling hashing byte by byte to obtain a series of hash values ​​to form a hash value set. The N transformation method is used to extract features. The hash value of each sliding window is linearly transformed, and the minimum value of each linear transformation result is recorded as a feature value. The minimum values ​​generated by different linear transformations have different distributions, and finally the extracted features correspond to different positions of the original data block; then continue to perform the above operations from different positions to obtain N features Feature[N], and group the obtained N features in order with a fixed number of features to form a super feature SF x ; Then, data block similarity detection is effectively performed based on the fusion features. If similar data blocks are found, differential compression method is used for compression.

[0020] Among them, the security rule base is designed and configured based on the specified requirements and security policies according to the data security development of the power grid. The security rules are stored in the HDFS distributed file system using the Spark framework;

[0021] The extracted fusion features are matched and searched through the security rule base, including:

[0022] For the fusion features of the extracted log data, the Spark SQL statement is used to find the data that meets the rules in the security rule library. Spark SQL establishes the corresponding indexes of the partitioned data sets at different times based on the local temporal index model, and sets the upper and lower partition boundaries of the data set as (B l ,B r ), median value of selected time period mid The data set of the time period corresponding to the root node value of the segment tree index that is less than the median value is called the upper boundary set S l , the data set of the time period greater than the median is called the lower boundary set S r , the third type is the intermediate set S mid; After entering the SQL temporal data query statement, the SQL statement is converted into an abstract syntax tree through the SQL antlr parser, and then the binary segment tree index model is used to compare the segment tree node value and the median time mid The data set of time periods smaller than the median value enters the left subtree, and the data set of time periods larger than the median value enters the right subtree. In the binary segment tree index model, the data in the left subtree and the right subtree represent data sets in different time periods respectively.

[0023] The robust learning processing method for power log data identifies mislabeled samples from a training data set to construct a clean training subset, including:

[0024] Use a set of class prototypes to represent each class category and compare the samples x i Label y i Features of z i With class y i The class prototype is used to judge the sample x i Label y i Whether it is mislabeled;

[0025] In order to obtain the class prototype of class c, k-means clustering is performed on the representation of all samples belonging to c, and then the centroids of k clusters are selected as the class prototype of class c. Based on the class prototype of each class, the sample x is calculated according to the following formula: i With its label y i The similarity score between:

[0026]

[0027] where z i For x i The characteristics of z yi Represents the class prototype set of class y;

[0028] Based on the similarity score of the training samples from the training dataset D t Identify the wrongly labeled samples in and get a clean subset D C , including: using a two-component Gaussian mixture model GMM to model the similarity score distribution of samples in a class, and considering the samples with a mean value greater than a specified threshold in the Gaussian component as clean samples, and selecting clean samples of each category c by iteratively applying GMM to the class.

[0029] The robust learning processing method for power log data adopts class-balanced sampling to train a classifier with a balanced decision boundary, including:

[0030] The architecture of the depth prediction model is decomposed into two components: i Mapping to a high-dimensional representation vi The feature extractor g, and the receiver v i As x i The input and output classifier head f of the deep prediction model is used as the feature extractor g, the last fully connected layer in the deep prediction model is used as the classifier head f, and the projection head h is used to transform v i Mapped to a low-dimensional representation z i to help identify mislabeled samples during the representation learning phase, and discard this projection head after the representation learning phase;

[0031] After the representation learning stage, the learned feature representations are used to train a classifier.

[0032] In a second aspect, a device for quickly identifying and marking power grid log data includes:

[0033] Data collection module: collects log data of multi-source heterogeneous information systems in the power grid through log file collection, database and API collection;

[0034] Data preprocessing module: preprocesses the collected log data, including data deduplication, redundant data filtering, and data format standardization;

[0035] Rapid identification module: extract key features from pre-processed log data, use multi-feature fusion mechanism to compress data size and strengthen data relevance, and match and search the extracted fusion features through the security rule library, which covers known knowledge of power grid data security detection rules;

[0036] Data labeling module: Label the matched and searched data based on predefined labeling rules. For data not covered by these rules, a deep prediction model is used to automatically label the data. The deep prediction model is trained using a robust learning processing method for power log data. The robust learning processing method for power log data constructs a clean training subset by identifying incorrectly labeled samples from the training data set, and trains a classifier with a balanced decision boundary by adopting class-balanced sampling, so that the prediction model is robust to both incorrectly labeled samples and class imbalance.

[0037] In a third aspect, the present invention also provides a computer device, comprising: one or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and when the programs are executed by the processors, the steps of the method for rapid identification and labeling of power grid log data as described above are implemented.

[0038] In a fourth aspect, the present invention further provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the method for rapid identification and labeling of power grid log data as described above.

[0039] Beneficial effects: The present invention uses a multi-source heterogeneous data acquisition tool to obtain power grid log data, performs data preprocessing on the collected power grid log data, and then compresses the data size and strengthens the data correlation through a multi-feature fusion mechanism; at the same time, a security rule library is constructed to cover the known massive detection rule knowledge, and the extracted fusion features are matched and searched through the rule library, and the found data is labeled based on the predefined labeling rules; for the data not covered by this part of the rules, the deep prediction model is constructed to effectively perform automatic labeling, and the labeling results can be used for subsequent data analysis, model training, etc., effectively supporting complex engineering practice requirements such as power grid data security risk monitoring and threat processing analysis. In this process, a robust learning processing method for power data is used to effectively improve the training effect and generalization ability of the prediction model. The rapid identification and labeling method for power grid log data proposed by the present invention effectively improves the matching efficiency through the multi-feature fusion security rule matching method, and at the same time effectively improves the reliability of data automatic labeling processing through the robust learning processing method for power data, solves the problems of slow recognition speed and low reliability of automatic labeling in the existing power grid log data processing, and effectively provides strong support for power grid data security monitoring and analysis, threat processing analysis, etc. by improving data processing technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 is a flow chart of a method for rapid identification and marking of power grid log data in an embodiment of the present invention;

[0041] Figure 2 It is a structural block diagram of a device for rapid identification and labeling of power grid log data in an embodiment of the present invention. DETAILED DESCRIPTION

[0042] The present invention provides a method and device for rapid identification and labeling of power grid log data, aiming to effectively solve the problems of slow identification speed, low labeling efficiency and accuracy in the existing power grid log data processing technology through efficient and reliable data processing measures, lay a foundation for subsequent power grid data security monitoring and detection analysis, and improve the efficiency and accuracy of data-driven power grid monitoring analysis, fault diagnosis analysis and maintenance. The basic idea of ​​the present invention is to improve the identification speed and labeling efficiency of massive power grid log data by designing a processing method for power grid data. On the one hand, in view of the problem of rapid data identification in multi-source heterogeneous data collection, a multi-feature fusion mechanism is constructed to effectively compress data, reduce the data processing volume and improve the identification efficiency under the premise of ensuring data validity; on the other hand, in view of the problem of robust learning processing of deep prediction models in the process of automatic data labeling, a robust learning processing method for power data is designed to improve the quality of data, effectively improve the training effect and generalization ability of the prediction model, and support the data security risk analysis capability of the digital power grid.

[0043] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention is further described in detail below through specific embodiments in conjunction with the accompanying drawings.

[0044] Reference Figure 1 , a method for quickly identifying and marking power grid log data provided by an embodiment of the present invention comprises the following steps:

[0045] Step S1: Multi-source heterogeneous data collection: Collect multi-source heterogeneous log data through various methods such as log file collection, database and API collection;

[0046] According to an embodiment of the present invention, multi-source heterogeneous data collection includes the following steps:

[0047] S11, determine the data source: clarify where to obtain data, such as server logs, network monitoring tools, application logs, etc.;

[0048] First, identify potential data sources: List all data sources that may contain useful information. For example: server logs: record server operation and status information; network monitoring tools: provide network traffic and performance data; application logs: record application operation status and events; databases: store power grid equipment status and historical data; API interfaces: provide real-time equipment status information.

[0049] Then evaluate data sources: Evaluate each data source, considering its data volume, data format, update frequency, and availability.

[0050] Finally, select the data source: Based on the purpose and needs of data collection, select the most relevant data source.

[0051] S12, data capture: Use appropriate log capture tools such as Logstash, Fluentd, Splunk, etc. to capture data from data sources.

[0052] First, choose the tool: Choose the appropriate log capture tool based on the characteristics and requirements of the data source. For example: Logstash: Suitable for processing large numbers of log files, supports multiple data sources and outputs. Fluentd: Lightweight data collector that supports flexible data source configuration. Splunk: Powerful log analysis tool that can process large data sets.

[0053] Then configure the crawler, including: Input configuration: specify the location and type of data source; Filter configuration (if necessary): perform preliminary preprocessing on the data, such as filtering, format conversion, etc.; Output configuration: define the location and format of data storage. Finally, start crawling: run the configured tool to start the data collection process.

[0054] Step S2, data preprocessing: preprocessing the log data, including data cleaning, redundant data filtering, data standardization, etc.;

[0055] According to an embodiment of the present invention, data preprocessing specifically includes the following steps:

[0056] S21, missing value processing: Check the completeness of log data attributes. Log data usually contains multiple attributes, and the specific attributes depend on the type of log and the system that generates the log. For example, for network logs, its attributes include: timestamp, recording the date and time of the event; event type, such as HTTP request, login attempt, file transfer, etc.; log level, such as INFO, WARNING, ERROR, DEBUG, etc.; user / client ID, the user or client ID that initiates the request; source / destination IP address, the IP address where the request is initiated and received; source / destination port, the port number where the request is initiated and received; protocol, such as TCP, UDP, HTTP, HTTPS, etc.; request method, such as GET, POST, PUT, DELETE, etc. (for HTTP requests); transmission bytes, the amount of data transmitted. For system logs, its attributes may include: host name, the name of the server that records the log; process ID, the process ID that generates the log; thread ID, the thread ID that generates the log; module / component, the software module or component that generates the log; error code, the code of the system error or exception; error message, the text information describing the error, etc. For security logs, the attributes may include: security event ID, the unique identifier of the security event; event description, a detailed description of the security event; attack type, such as DDoS, SQL injection, cross-site scripting, etc.; security level, such as low, medium, high, severe, etc. For application logs, possible attributes include: function module, the function module in the application; operation type, the type of operation performed by the user; operation result, the success or failure status of the operation; user input, the data entered by the user; execution time, the duration of the operation execution. For each type of log, you can preset attribute completion standards, such as specifying that certain attributes must be included, and check the attribute completeness of the log data according to the preset standards. If two or more specified attributes are missing in the acquired data, the log will be filtered out; otherwise, the missing values ​​will be filled with the "Null" character to complete the information.

[0057] S22, redundant data filtering: Sort the log data by the keyword "occurrence time". Then, set a fixed-size data sliding window, slide it continuously on the sorted data set, detect the records in the window, find and delete the identical records, and only keep the different log information.

[0058] S23, format standardization: normalize the log data, determine the selected log-related attributes, and use the extended IDMEF (Intrusion Detection Message Exchange Format) alarm format for format definition, including uppercase and lowercase letters, symbols, time format, etc.

[0059] Step S3, data compression: Data compression is based on deduplication, mainly deduplication, but when the data is not repeated but highly similar, a multi-feature fusion detection mechanism is designed to compress the data size and enhance the relevance of the data.

[0060] According to an embodiment of the present invention, data compression specifically includes the following steps:

[0061] S31, when performing log data analysis, effectively integrate different similar log data through multi-source similar data merging processing method. Cosine similarity is an indicator to measure the similarity between two log data, and its calculation formula is as shown in formula (1):

[0062]

[0063] Among them, a and b represent two different data, M represents the dimension of the feature, and i represents the i-th data feature. When the cosine similarity = 1, it can be considered that the two log data are almost the same, and one of the duplicate data is deleted.

[0064] S32, if the data similarity does not reach 1 but exceeds a preset value such as 0.7, it can be considered highly similar, but if it is directly deleted at this time, it will obviously destroy the validity of the data. The present invention designs a multi-feature fusion detection mechanism to compress and maintain the relevance of data.

[0065] The multi-feature fusion detection mechanism mainly includes data block division, feature extraction, and feature detection. After multi-feature fusion detection, data with high similarity is eliminated. First, for data block division, the original non-repetitive data set is mainly divided into multiple data blocks using a fixed size. The data is usually divided into 8KB levels; secondly, for each data block, a sliding window is used to use rolling hashing byte by byte to obtain a series of hash values ​​to form a hash value set, and the N-transformation method is used to extract features. The hash value of each sliding window is linearly transformed, and the minimum value of each linear transformation result is recorded as a feature value. The minimum value distribution generated by different linear transformations is different, and finally the extracted features correspond to different positions of the original data block. Specifically: first, according to the rolling hash algorithm, the digital information of a data block of length L at position j is obtained, and then a pair of random values ​​m is used to extract the digital information of the data block. i and a i The digital information is converted to obtain a feature of the data block, namely:

[0066]

[0067] Among them, Rabin j is the Rabin of the sliding window at position j j feature.

[0068] Then continue to perform the above operations from different positions to obtain N features Feature[N]. The fusion features are constructed by grouping and integrating these features; the super-feature method is introduced to group the obtained N features in order with a fixed number of features to form a super-feature, that is:

[0069] SF x =Rabin(Feature x·k ,…,Feature x·k+k-1 )

[0070] For similar data blocks, only a very small part of them is different, so their features and SF x should be the same, thus reducing the size of the feature value set; then the data block similarity detection is effectively performed based on the fused features, and if similar data blocks are found, the Delta Compression method is used for compression.

[0071] Step S4, build a security rule library, cover the known knowledge of power grid data security rules, and match, search and quickly mark the compressed data through the rule library.

[0072] S41, the power grid data security rules in the power grid data security rule base are mainly customized based on the "Network Data Security Management Regulations", "Power Industry Network Security Management Measures" and "Power Secondary System Security Protection Regulations". These rules are intended to regulate the processing and use of power grid data and ensure data security. The rules in the security rule base are mainly set for the following data:

[0073] Sensitive data mainly refers to sensitive information data including personal identity information, basic organizational information, organizational account information, organizational credit information, financial information, etc.

[0074] System operation and maintenance data mainly refers to the detection and monitoring data of the power grid digital system, system operation data, etc.

[0075] Business data mainly refers to test control, data security, network security and other related business data;

[0076] Fault alarm data mainly refers to fault occurrence time, equipment type, location information, fault information, etc.

[0077] Operation records mainly refer to the detailed recording of relevant information during various operations, including the specific steps, time, results, and related parameters of the operation. These records are very important for subsequent tracking, analysis, and problem solving, and can help quickly locate the source of the problem and take timely measures to repair it.

[0078] The security rule base contains a series of rules for identifying different types of data, which are usually based on the content, format, timestamp or other characteristics of the log. Here are some specific examples to illustrate these rules:

[0079] 1. Rules based on log content

[0080] Sensitive information identification: The rule base may contain rules for identifying sensitive information in logs, such as phone numbers, ID numbers, email addresses, etc. These rules usually use regular expressions or pattern matching to detect whether such sensitive information exists in the logs.

[0081] Event type identification: The rule base can also contain rules for identifying different event types in the logs, such as login events, logout events, transaction events, etc. These rules may identify event types based on specific keywords or phrases in the logs.

[0082] 2. Rules based on log format

[0083] Log structure validation: The rule base may contain rules for validating whether the log structure is correct. For example, the rule base can check whether the log contains expected fields such as timestamp, log level, message content, etc., and whether the format of these fields is as expected.

[0084] Log type identification: Based on the format of the log, the rule base can also identify different types of logs, such as system logs, application logs, security logs, etc. These rules may identify the log type based on the extension of the log file, the specific format of the log content, or specific tags in the log.

[0085] 3. Timestamp-based rules

[0086] Time range check: A rule base may contain rules that check whether a log timestamp is within a certain range.

[0087] Temporal pattern recognition: The rule base can also contain rules for recognizing temporal patterns in logs, such as periodic events (such as backup tasks that occur at a fixed time every day) or abnormal time events (such as login attempts that occur during non-working hours).

[0088] 4. Other features

[0089] IP address identification: A rule base may contain rules for identifying IP addresses in logs, which can be used to track the source or destination of network activity.

[0090] User behavior pattern recognition: The rule base can also contain rules for recognizing user behavior patterns, such as frequent failed login attempts, abnormal transaction patterns, etc.

[0091] In actual applications, security rule bases are usually designed and configured according to specific needs and security policies for the data security development of the power grid. These rule bases can be continuously updated and optimized to cope with new security threats and changes in log data.

[0092] S42, HDFS is a distributed file system of Hadoop, which has the characteristics of high fault tolerance and high throughput. It can store data in multiple nodes in a dispersed manner, realizing distributed storage and parallel processing of data. The present invention stores rule knowledge through HDFS to ensure the security and reliability of data. At the same time, when Spark can be used to write data to HDFS, the data can be uploaded to the specified directory of HDFS through Spark's API or command line tool.

[0093] S43, matching the log data with the rules in the security rule library, then searching for data that meets the rules through Spark SQL, and marking the successfully matched data, such as marking as "sensitive data", "system operation and maintenance data", "business data", "fault alarm data", "operation record", etc.

[0094] Predefined annotation rules are guiding principles in the data annotation process, and they are set according to the characteristics of the data and the analysis objectives. For example, if the log contains personal information such as name, ID number, phone number, email address, etc., it is labeled as "sensitive data". If the log records requests to access sensitive systems or perform sensitive operations, such as database modifications, system configuration changes, etc., it is labeled as "sensitive operations". For another example, if the log type is "system" and the operation involves system configuration changes, it is marked as "system operation and maintenance data". If the log level is "ERROR" or contains the keywords "fail" or "crash", it is marked as "failure alarm data".

[0095] The security rule base needs to be converted into a data structure that Spark can process, such as DataFrame or RDD (Resilient Distributed Dataset). Then, Spark SQL or Spark's DataFrame API can be used to write query statements to match log data with the rules in the security rule base. Spark SQL establishes indexes corresponding to partitioned data sets at different times based on the local temporal index model, sets the upper and lower partition boundaries of the data set to (Bl, Br), and the median timemid of a time period corresponds to the root node value of the segment tree index. The data set of the time period less than the median is called the upper boundary set Sl, the data set of the time period greater than the median is called the lower boundary set Sr, and the third category is the middle set Smid. After entering the SQL temporal data query statement, the SQL statement is converted into an abstract syntax tree through the SQL antlr parser, and then the segment tree node value and the median timemid are compared using the binary segment tree index model. The data set of the time period less than the median enters the left subtree, and the data set of the time period greater than the median enters the right subtree. In the binary segment tree index model, the data in the left subtree and the right subtree represent data sets in different time periods. This data structure is usually used to optimize the query of time series data.

[0096] Left subtree: In a binary segment tree, if the median value (timemid) of a node divides the time range into two parts, then the data set in the left subtree will contain all data with timestamps less than the median value. In other words, the left subtree represents the earlier part of the time range. Right subtree: Correspondingly, the data set in the right subtree will contain all data with timestamps greater than or equal to the median value. The right subtree represents the later part of the time range. During the query process, by comparing the query time range with the median value of the segment tree node, it can be decided whether to enter the left subtree, the right subtree, or visit both at the same time. This recursive search process can efficiently locate the data set containing the query results.

[0097] For the successfully matched data, use Spark SQL's CASE WHEN statement to add a new column to mark the data type (such as "sensitive data", "system operation and maintenance data", etc.), and save the marked data to a distributed file system (such as HDFS) or a database (such as Hive, Cassandra, etc.). If necessary, the marked data can also be exported to files in CSV, JSON and other formats.

[0098] S44, for the Spark mentioned above, there are many parameters that affect it. The present invention covers the CPU, Memory, Disk, etc. in the Spark cluster by selecting 9 active parameters, mainly including spark.driver.cores (the number of CPU cores used for the driver program, which determines the computing power for job scheduling and task scheduling), spark.driver.memory (the amount of memory used by the driver, usually related to storing the execution code and task output of the application), spark.executor.cores (the number of CPU cores used by each executor, which determines the parallel processing capability of the executor), spark.reducer.maxSizeInFlight (the maximum size of data pulled in parallel from each reduce task during the shuffle process, in bytes), spark.shuffle.manager (the shuffle manager used for shuffle data. The default is sort, which can also be set to tungsten-sort or hash), spark.shuffle.compress (whether to compress map output files, compression can reduce disk I / O and network transmission), spark.shuffle.spill.compress (whether to compress shuffle data overflowed to disk, which also helps to reduce disk space usage and I / O), Spark.broadcast.compress (whether to compress broadcast variables, compression can reduce the amount of data transmitted over the network), spark.broadcast.blockSize (the size of the broadcast variable block (in bytes). Larger blocks can reduce the number of task starts, but may increase the overhead of a single task), etc. In view of the complex interaction between different configuration parameters, the present invention constructs a performance model based on Adaboost, and then selects ACRO-PSO (adaptive covariance matrix PSO) to search for the optimal configuration parameters for the Spark program.

[0099] Step S5, for data that cannot be feature-matched and labeled through the rule base, a deep prediction model is constructed to effectively label the data.

[0100] For data that cannot be marked by feature matching through the rule base, the present invention effectively marks the data by constructing a deep prediction model. The initial deep learning prediction model is trained using a small amount of labeled data S, and the samples of the dataset T that cannot be marked are extracted through the deep learning model to represent the features. Samples are selected according to the query strategy for manual marking to form a new labeled dataset S', and S' is used to continue training the deep learning model and update T at the same time. The above process is repeated until the predetermined termination condition is reached.

[0101] In the process of labeling power grid log data through deep learning prediction models, the data is very likely to present an unbalanced class distribution, where a small number of classes have a large number of samples, while other classes are only related to a small number of samples. For data subsets with sample mislabeling and class imbalance, the present invention effectively improves the training effect and generalization ability of the prediction model by designing a two-stage data robust learning processing method.

[0102] Specifically, in the deep prediction model, mislabeled samples will destroy the learned representation, and class imbalance will affect the classifier part. Iteratively perform the two steps of mislabeled sample detection and robust feature learning. Better clean samples will produce better feature representations, and better feature representations will identify better clean samples. In this process, the negative impact of mislabeled samples can be gradually reduced, and robust feature representations can be generated for the deep prediction model.

[0103] The goal of detecting mislabeled samples is to construct a clean subset. , which has as few mislabeled samples as possible, thus ensuring C The feature representation learned in D is less affected by mislabeled samples. T is the original data set, D C is a data set with error samples removed. The present invention measures the sample x i and belongs to class y i The similarity between samples is used to judge the sample (x i ,y i )∈D T Is it mislabeled? i With class y i The samples of x have a high similarity, which means that i may be correctly labeled. The first step in mislabeled sample detection is to i ,y i )∈D T Generate a feature representation. A naive approach is to directly use the representation v produced by the feature extractor g. However, the representation v is usually high-dimensional, resulting in high computational complexity, especially for large-scale datasets. To solve this problem, a projection head h can be used to map the representation v to a low-dimensional space. The projection head h consists of a 3-layer multi-layer perceptron (MLP) with a hidden size of 2048. That is, through z i =g(v i )=g(h(x i )) is the sample x i Generate feature representation z i .

[0104] According to an embodiment of the present invention, in order to effectively detect mislabeled samples, a set of class prototypes is used to represent each class category, and by comparing the sample x i Label y i Features of z i With class y i The class prototype is used to judge the sample x i Label y i Is it mislabeled? In order to obtain the class prototype of class c, perform k-means clustering on the representation of all samples belonging to c, and then select the centroids of k clusters as the class prototype of class c. Set the value of k to Where ρ is the training dataset D t The average number of instances of each class in . With the class prototype of each class, we can then measure the sample x i With its label y i The similarity score between:

[0105]

[0106] where z i For x i Features, Represents the set of class prototypes for class y.

[0107] Based on the similarity score of the training samples from the training dataset D t Identify mislabeled samples in , to construct a clean subset D C .

[0108] Furthermore, it also includes detecting mislabeled samples: by sorting all samples of class c in ascending order of similarity scores, and then selecting the first m samples with lower scores as mislabeled samples. A two-component Gaussian mixture model (GMM) is used to model the similarity score distribution of samples in a class, and samples with larger mean in the Gaussian component are regarded as clean samples. Clean samples of each class c are selected by iteratively applying GMM. Samples belonging to class c are selected by D c ={x i |y i =c} means fitting a two-component GMM to maximize the log-likelihood:

[0109]

[0110] where φ j represents the weight of the jth component, and Indicates that the sample x in GMM i The probability of belonging to the jth component.

[0111] After GMM training is completed, the model can be used to judge the sample x i ∈Dc Is it clean? Without losing generality, assume that the means u1 and u2 of the two components in GMM satisfy u1>u2. If and only if P(s i |u1,σ1)>P(s i |u2,σ2) when the sample x i ∈D c Marked as a clean sample.

[0112] In this way, feature representations extracted from all types of samples can be utilized instead of the original samples themselves.

[0113] The update of the network to learn a robust representation mainly builds a balanced classifier based on the features learned in the representation learning phase. After the representation learning phase, the classifier learning phase further trains the classifier using the learned feature representation to mitigate the impact of class imbalance. Specifically, class-balanced sampling is used in the classifier learning phase to train a classifier with a balanced decision boundary. Once trained, the prediction model F is robust to both mislabeled samples and class imbalance.

[0114] The architecture of the depth prediction model F in the present invention is decomposed into two components: i Mapping to a high-dimensional representation v i The feature extractor g, and the receiver v i As x i The network layer function before the logit layer in F is used as the feature extractor g, and the last fully connected layer in F is used as the classifier head f. In addition, a projection head h is introduced to convert v i Mapped to a low-dimensional representation z i to help identify mislabeled samples during the representation learning phase, and discard this projection head after the representation learning phase.

[0115] The two objective functions for learning representations are as follows:

[0116] The prototype contrast loss forces each sample to be pulled towards the prototype of its corresponding class and pushed away from the prototypes of all other classes. The prototype contrast loss is calculated as:

[0117]

[0118] The classification loss is mainly based on the common training strategy of classification tasks, using cross entropy loss as the classification loss to stabilize convergence and obtain better representation in the robust trainer. The classification loss calculation formula is:

[0119]

[0120] The overall training loss is as follows:

[0121] L total =w1L cl +w2L pcl (6)

[0122] The hyperparameters w1 and w2 are trade-off parameters. In the present invention, w1=w2=1 is set.

[0123] Enter the overall training of the representation learning phase. The input of the algorithm is a training data set D t , which may be mislabeled, may have class imbalance, the maximum (warm-up) epochs T(T0), the batch size B, and the network to be trained, including the feature extractor G, the classifier head f, and the projection head h. First, the dataset D is trained using the loss defined in Formula 6 t Train to warm up the network’s T0 epoch. When the network is not trained, feature representation is not available, so D t All samples in D are considered clean samples. t Such warm-up training on D will only slightly affect the quality of the learned representation. After that, in each training epoch, we first start from D t Select clean samples from the mislabeled samples to construct a clean subset D C Then, the training loss defined in Formula 6 is used to train the dataset D C Mini batch gradient descent is performed to update the network, resulting in cleaner subsets and better feature representations. Finally, after the learning phase, the algorithm returns the trained feature extractor G and the clean subset D selected in the last epoch for training the classifier in the next classifier learning phase. C .

[0124] The present invention designs a robust learning processing method for power data to effectively improve the training effect and generalization ability of the prediction model. In addition, although automated processing can improve efficiency, manual review and optimization are still required. Through manual intervention and adjustment, the recognition accuracy can be further improved and the false alarm rate can be reduced.

[0125] Based on the same inventive concept as the method embodiment, refer to Figure 2The embodiment of the present invention also provides a rapid identification and labeling device for power grid log data, which mainly includes a data acquisition module, a data preprocessing module, a rapid identification module, and a data labeling module. The data acquisition module is used to collect real log data and effectively collect and aggregate data through a variety of acquisition devices; the data preprocessing module is used to perform preprocessing work such as data deduplication, redundant data filtering, and format standardization to reduce the amount of data for subsequent processing and improve processing efficiency; the rapid identification module combines the characteristics of power grid flow and log data, and uses the Spark framework to realize the identification of different types of log events (such as fault alarms, operation records, etc.), that is, data type identification; the data labeling module, on the one hand, labels the identified data based on predefined labeling rules, and on the other hand, for some data that cannot be labeled, a robust learning processing method for power data is used according to the already labeled results to perform effective automatic labeling, and the labeling results can be used for subsequent data analysis, model training, etc., effectively supporting complex engineering practice needs such as power grid data security risk monitoring and threat processing analysis.

[0126] According to an embodiment of the present invention, a data acquisition module collects log data of multi-source heterogeneous information systems in a power grid through log file collection, database and API collection in a variety of ways; a data preprocessing module preprocesses the collected log data, including data deduplication, redundant data filtering, and data format standardization; a fast identification module extracts key features of the preprocessed log data, compresses the data size and strengthens the data relevance using a multi-feature fusion mechanism, and matches and searches the extracted fusion features through a security rule library, wherein the security rule library covers known knowledge of power grid data security detection rules; a data labeling module labels the matched and searched data based on predefined labeling rules, and automatically labels the data using a deep prediction model for data not covered by the rules, wherein the deep prediction model is obtained through training using a robust learning processing method for power log data, wherein the robust learning processing method for power log data constructs a clean training subset by identifying incorrectly labeled samples from a training data set, and trains a classifier with a balanced decision boundary by using class-balanced sampling, thereby realizing that the prediction model is robust to both incorrectly labeled samples and class imbalance.

[0127] It should be understood that the rapid identification and marking device for power grid log data in the embodiment of the present invention can implement all the technical solutions in the above-mentioned method embodiment, and the functions of its various functional modules can be specifically implemented according to the method in the above-mentioned method embodiment. The specific implementation process can refer to the relevant description in the above-mentioned embodiment, which will not be repeated here.

[0128] The present invention also provides a computer device, comprising: one or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and when the programs are executed by the processors, the steps of the method for rapid identification and labeling of power grid flow and log data as described above are implemented.

[0129] The present invention also provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the method for rapid identification and marking of power grid flow and log data as described above are implemented.

[0130] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, devices (systems), computer equipment or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0131] The present invention is described with reference to a flowchart of a method according to an embodiment of the present invention. It should be understood that each process in the flowchart and a combination of processes in the flowchart can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the process in the flowchart. Figure 1 A device that specifies functions in a process or multiple processes.

[0132] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A function specified in a process or multiple processes.

[0133] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 The steps of a specified function in a process or multiple processes.

Claims

1. A method for rapid identification and labeling of power grid log data, characterized in that: The following steps are involved: Data collection steps: Collect log data from multi-source heterogeneous information systems in the power grid through log file collection, database and API collection; Data preprocessing step: preprocess the collected log data, including data deduplication, redundant data filtering, and data format standardization; Rapid identification step: extract key features from the preprocessed log data, use a multi-feature fusion mechanism to compress the data size and strengthen the data relevance, and match and search the extracted fusion features through a security rule library that covers known knowledge of power grid data security detection rules; Data labeling step: label the matched and searched data based on predefined labeling rules. For data not covered by these rules, use a deep prediction model to automatically label the data. The deep prediction model is trained using a robust learning processing method for power log data. The robust learning processing method for power log data constructs a clean training subset by identifying incorrectly labeled samples from the training data set, and trains a classifier with a balanced decision boundary by using class-balanced sampling, so that the prediction model is robust to both incorrectly labeled samples and class imbalance.

2. The method according to claim 1, characterized in that The data collection steps include: Determine the data source: List all data sources that may contain useful information, evaluate each data source, and select the most relevant data source based on the purpose and needs of data collection; Capture data: Select a suitable log capture tool based on the characteristics and requirements of the data source, configure it, run the configured tool, and start the data collection process.

3. The method according to claim 1, characterized in that In the data preprocessing step, the specific processing methods include: Missing value processing: Check the completeness of log data attributes. If two or more attributes are missing, the log will be filtered out; otherwise, the missing values ​​will be filled with "Null" characters to complete the information; Redundant data filtering: Sort the log data by the "occurrence time" keyword, set a fixed-size data sliding window, and slide it continuously on the sorted data set. By detecting the records in the window, find and delete the identical records, and only keep the different log information; Format standardization: Normalize the log data, determine the selected log-related attributes, and use the extended IDMEF alarm format for format definition.

4. The method according to claim 1, characterized in that: In the fast recognition step, a multi-feature fusion mechanism is used to compress the data size and enhance the data relevance, including: By calculating the similarity between two log data, when the similarity is equal to 1, one of the log data is deleted to eliminate duplicate logs; For data whose similarity is not equal to 1, firstly, the data is divided into multiple data blocks with a fixed size. For each data block, a sliding window is used to use rolling hashing byte by byte to obtain a series of hash values ​​to form a hash value set. The N transformation method is used to extract features. The hash value of each sliding window is linearly transformed, and the minimum value of each linear transformation result is recorded as a feature value. The minimum values ​​generated by different linear transformations have different distributions, and finally the extracted features correspond to different positions of the original data block; then continue to perform the above operations from different positions to obtain N features Feature[N], and group the obtained N features in order with a fixed number of features to form a super feature SF x ; Then, data block similarity detection is effectively performed based on the fusion features. If similar data blocks are found, they are compressed using the differential compression method.

5. The method according to claim 1, characterized in that The security rule base is designed and configured based on the specified requirements and security policies according to the data security development of the power grid. The security rules are stored in the HDFS distributed file system using the Spark framework; The extracted fusion features are matched and searched through the security rule library, including: For the fusion features of the extracted log data, the Spark SQL statement is used to find the data that meets the rules in the security rule library. Spark SQL establishes the corresponding indexes of the partitioned data sets at different times based on the local temporal index model, and sets the upper and lower partition boundaries of the data set as (B l ,B r ), median value of selected time period mid The data set of the time period corresponding to the root node value of the segment tree index that is less than the median value is called the upper boundary set S l , the data set of the time period greater than the median is called the lower boundary set S r , the third type is the intermediate set S mid ; After entering the SQL temporal data query statement, the SQL statement is converted into an abstract syntax tree through the SQL antlr parser, and then the binary segment tree index model is used to compare the segment tree node value and the median time mid The data set of time periods smaller than the median value enters the left subtree, and the data set of time periods larger than the median value enters the right subtree. In the binary segment tree index model, the data in the left subtree and the right subtree represent data sets in different time periods respectively.

6. The method according to claim 1, characterized in that The robust learning processing method for power log data identifies mislabeled samples from a training data set to construct a clean training subset, including: Use a set of class prototypes to represent each class category and compare the samples x i Label y i Features of z i With class y i The class prototype is used to judge the sample x i Label y i Whether it is mislabeled; In order to obtain the class prototype of class c, k-means clustering is performed on the representation of all samples belonging to c, and then the centroids of k clusters are selected as the class prototype of class c. Based on the class prototype of each class, the sample x is calculated according to the following formula: i With its label y i The similarity score between: where z i For x i Features, Represents the class prototype set of class y; Based on the similarity score of the training samples from the training dataset D t Identify the wrongly labeled samples in and get a clean subset D C , including: using a two-component Gaussian mixture model GMM to model the similarity score distribution of samples in a class, and considering the samples with a mean value greater than a specified threshold in the Gaussian component as clean samples, and selecting clean samples of each category c by iteratively applying GMM to the class.

7. The method according to claim 1, characterized in that The robust learning processing method for power log data adopts class-balanced sampling to train a classifier with a balanced decision boundary, including: The architecture of the depth prediction model is decomposed into two components: i Mapping to a high-dimensional representation v i The feature extractor g, and the receiver v i As x i The input and output classifier head f of the deep prediction model is used as the feature extractor g, the last fully connected layer in the deep prediction model is used as the classifier head f, and the projection head h is used to transform v i Mapped to a low-dimensional representation z i to help identify mislabeled samples during the representation learning phase, and discard this projection head after the representation learning phase; After the representation learning stage, the learned feature representations are used to train a classifier.

8. A device for rapid identification and labeling of power grid log data, characterized in that: include: Data collection module: collects log data of multi-source heterogeneous information systems in the power grid through log file collection, database and API collection; Data preprocessing module: preprocesses the collected log data, including data deduplication, redundant data filtering, and data format standardization; Fast identification module: extract key features from pre-processed log data, use multi-feature fusion mechanism to compress data size and strengthen data relevance, and match and search the extracted fusion features through the security rule library, which covers known knowledge of power grid data security detection rules; Data labeling module: Label the matched and searched data based on predefined labeling rules. For data not covered by these rules, a deep prediction model is used to automatically label the data. The deep prediction model is trained using a robust learning processing method for power log data. The robust learning processing method for power log data constructs a clean training subset by identifying incorrectly labeled samples from the training data set, and trains a classifier with a balanced decision boundary by adopting class-balanced sampling, so that the prediction model is robust to both incorrectly labeled samples and class imbalance.

9. A computer device, characterized in that: include: one or more processors; Memory; And one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and when the programs are executed by the processors, the steps of the method for rapid identification and labeling of power grid log data as described in any one of claims 1-7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of rapid identification and labeling of power grid log data according to any one of claims 1 to 7 are implemented.

Images