A Power Grid Data Processing Method and System Based on Hadoop

Abstract

The present invention provides a power grid data processing method and system based on Hadoop. The method includes: Step 1: Collect power grid big data based on the power grid data mining and analysis technology of Hadoop. The power grid big data includes real-time data information of the power grid, equipment parameter data, power generation and load data; Step 2: Store and manage the collected power grid big data on the power grid big data storage platform based on the MapReduce technology for data security situation storage; Step 3: Establish a zero-trust framework to achieve power terminal security protection. The present invention can improve data quality, enhance data storage security, enhance terminal security protection, reduce the probability of data being tampered with, damaged, or leaked, promote data circulation, give full play to the value of power grid data, meet the requirements of the country for data sharing and exchange, and provide support for using data innovation, mining data dividends, and promoting data economy.

Description

Technical Field

[0001] The present invention relates to the technical field of smart grids, and specifically to a power grid data processing method and system based on Hadoop. Background Technique

[0002] With the continuous increase in the social demand for electric power energy, the intellectualization of the power grid is also continuously developing in depth; the energy consumption control system on the customer side in the smart grid, as a link connecting customers and the intelligent energy service platform, is an important means to support the ubiquitous power Internet of Things on the customer side, and is also an execution unit for implementing various comprehensive energy services such as demand response and energy efficiency improvement.

[0003] With the continuous expansion of the scale of the power grid, the complexity of the power grid operation has continuously increased, and the problem of data security risks has become increasingly prominent. In view of the deficiencies in the common needs of the power grid for data acquisition and monitoring, demand response, etc., high development costs, poor portability and reusability, etc., it is necessary to use relevant technologies based on big data analysis to achieve the storage of the power grid data security situation, the mining and analysis of power grid data, and the security protection of power terminals. Summary of the Invention

[0004] The purpose of the present invention is to provide a power grid data processing method and system based on Hadoop, which can achieve the storage of the power grid data security situation, the mining and analysis of power grid data, and the security protection of power terminals, and at the same time solve the problems of high development costs, poor portability and reusability in the prior art.

[0005] A power grid data processing method based on Hadoop includes the following steps:

[0006] Step 1: Collect power grid big data based on the power grid data mining and analysis technology of Hadoop, and the power grid big data includes the real-time data information of the power grid, equipment parameter data, power generation and load data;

[0007] Step 2: Store and manage the collected power grid big data on the power grid big data storage platform based on the MapReduce technology for data security situation storage;

[0008] Step 3: Establish a zero-trust framework to achieve the security protection of power terminals.

[0009] Furthermore, the power grid data mining and analysis technology based on Hadoop is implemented by a data collection layer, a data storage layer, a business application layer and a user layer;

[0010] The data collection layer uses a distributed directional collection architecture and takes the terminal sites in different networks as a basic task unit for network data collection to collect the original network data, and converges and transmits it to the data storage layer. Among them, each basic task unit uses independent collection rules and strategies;

[0011] The data storage layer is used to complete the aggregation, storage, and original processing of raw data, and provide different types of function call services. The data storage layer is implemented using the Hadoop framework;

[0012] The business application layer is used to retrieve the network data processed by the data storage layer and perform analysis to separate public components from personalized business application components, and transmit the results of the network data analysis to the user layer for real-time display;

[0013] The user layer is used to transmit and display the data information of the business application layer.

[0014] Furthermore, the basic task unit includes a data acquisition unit, which is used to collect data through dynamic web page collection methods and web page information extraction methods, extract information using a method based on a row block distribution function, and then obtain the data.

[0015] Furthermore, the data acquisition unit obtains Feed addresses by breadth-first traversing websites, collects information corresponding to each Feed address in real time, tracks and updates the information, and collects information in an incremental update manner.

[0016] Furthermore, the acquisition rules and strategies include vertical search template semi-automatic generation technology, dynamic page optimized access technology, and intelligent crawling process scheduling strategies.

[0017] Furthermore, the processing of raw data in the data storage layer includes using window technology to divide the data to be processed into blocks, using a sliding window model to describe the changes in streaming data, and using the sliding window model to preserve the patterns in the original data.

[0018] Furthermore, using the sliding window model to preserve the patterns in the original data is specifically as follows:

[0019] Divide the data according to the changes in the data, store the patterns of the unchanged part of the data in the sliding window; calculate the patterns of the added and deleted parts of the data respectively; update the patterns stored in the sliding window according to the patterns of the changed part of the data;

[0020] Use the multi-window method to support users' online mining requests; the multi-window method divides the data stream into multiple segments of fixed length, and each segment forms a window. When the number of windows in memory reaches a certain number, these multiple windows are merged to form a window with a higher summary level. As the data stream flows in, multiple windows with different summary levels form a hierarchical structure. At this time, each window is equivalent to a snapshot of the data between two predefined timestamps on the data stream.

[0021] Further, the data security situation storage based on MapReduce technology includes the following steps:

[0022] Step 2.1: Retrieve the data information at the user layer and input it into the user program;

[0023] Step 2.2: The MapReduce library divides the input file of the user program into M parts, where M is defined by the user;

[0024] Step 2.3: The worker assigned the Map job reads the input data of the corresponding shard. The Map job extracts key-value pairs from the input data, and each key-value pair is passed as a parameter to the map function. The intermediate key-value pairs generated by the map function are cached in memory;

[0025] Step 2.3: The cached intermediate key values are periodically written to the local disk and divided into R regions. The size of R is defined by the user, and each region will correspond to a Reduce job in the future; The positions of the intermediate key-value pairs will be reported to the master, and the master is responsible for forwarding the information to the Reduce worker;

[0026] Step 2.5: The master notifies the worker assigned the Reduce job of the specific location of the partition it is responsible for. When the Reduce worker reads all the intermediate key values it is responsible for, it sorts the intermediate key values so that the key-value pairs with the same key are grouped together;

[0027] Step 2.5: The reduce worker traverses the sorted intermediate key-value pairs. For each unique key, it passes the key and the associated value to the reduce function, and the output generated by the reduce function is added to the output file of this partition;

[0028] Step 2.7: When all Map and Reduce jobs are completed, the master wakes up the user program, and the MapReduce function call returns the code of the user program.

[0029] Further, the establishment of a zero-trust framework to achieve power terminal security protection includes the following steps:

[0030] Step 3.1: Build a zero-trust module to collect the device information of power terminal devices, perform a trust score based on the collected device information, give a trust value, evaluate the power terminal devices according to the trust value, and classify the power terminal devices into trusted devices and abnormal devices;

[0031] Step 3.2: Collect data from the trusted devices in Step 3.1 to obtain the collected data;

[0032] Step 3.3: Construct a security situation awareness module to perform situation awareness on the collected data in Step 3.2. When the awareness is qualified, convert the collected data into awareness data;

[0033] Step 3.4: Construct a real-time control module to control the awareness data in Step 3.3 and generate a security instruction;

[0034] Step 3.5: Send the security instruction in Step 3.4 to the power terminal device to perform security protection and security reinforcement on the power terminal device.

[0035] A power grid data processing system based on Hadoop includes: a computer-readable storage medium and a processor;

[0036] The computer-readable storage medium is used to store executable instructions;

[0037] The processor is used to read the executable instructions stored in the computer-readable storage medium and execute the power grid data processing method based on Hadoop.

[0038] The present invention aims at improving the security protection ability for power grid data security services, improving the recognition accuracy of data security incidents, the timeliness of traceability, reducing the probability of data being tampered with, damaged, or leaked, promoting data circulation, giving full play to the value of power grid data, meeting the requirements of the country for data sharing and exchange, and providing support for using data innovation, exploring data dividends, and promoting the data economy. Brief Description of the Drawings

[0039] Figure 1 It is a flowchart of a power grid data processing method based on Hadoop according to an embodiment of the present invention;

[0040] Figure 2 It is a flowchart of data security situation storage based on MapReduce technology in an embodiment of the present invention. Detailed Embodiments

[0041] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Apparently, the described embodiments are some, but not all, of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0042] Please refer to Figure 1 , the first aspect of the present invention provides a power grid data processing method based on Hadoop, including the following steps:

[0043] Step 1: Collect power grid big data based on the power grid data mining and analysis technology of Hadoop. The power grid big data includes real-time data information of the power grid, equipment parameter data, power generation and load data.

[0044] Step 2: Store and manage the collected power grid big data on the power grid big data storage platform based on MapReduce technology for data security situation storage.

[0045] Step 3: Establish a zero-trust framework to achieve power terminal security protection.

[0046] The power grid data mining and analysis technology of Hadoop described in this invention is implemented by a data collection layer, a data storage layer, a business application layer, and a user layer.

[0047] The data collection layer adopts a distributed directional collection architecture and uses terminal sites in different networks as a basic task unit for power grid data collection to collect real-time data information, equipment parameter data, power generation and load data, and converge and transmit them to the data storage layer. The basic task unit includes a data collection unit, which is used to collect data through dynamic web page collection methods and web page information extraction methods, extract information using a method based on a row block distribution function, and then obtain data. Specifically, the data collection unit obtains Feed addresses by breadth-first traversing sites, collects information corresponding to each Feed address in real time, tracks and updates the information, and collects information in an incremental update manner. Among them, each basic task unit adopts independent collection rules and strategies. The collection rules and strategies include vertical search template semi-automatic generation technology, dynamic page optimized access technology, and intelligent crawling process scheduling strategy.

[0048] The data storage layer is used to complete the convergence, storage, and original processing of raw data, and provide different types of function call services. The data storage layer is implemented using the Hadoop framework.

[0049] The business application layer is used to retrieve the data processed by the data storage layer and perform analysis to achieve the separation of public components and personalized business application components, and transmit the results of network data analysis to the user layer for real-time display.

[0050] The user layer is used to transmit and display the data information of the business application layer.

[0051] The storage of data security situation based on MapReduce technology is implemented using the distributed file system HDFS and MapReduce. The distributed file system HDFS is the file system of Hadoop and is used to store ultra-large files. MapReduce is the parallel programming model of Hadoop and is used to deeply analyze the data stored on the distributed file system HDFS.

[0052] Figure 2 As shown, the storage of data security situation based on MapReduce technology includes the following steps:

[0053] Step 2.1: Retrieve the data information at the user layer and input it into the user program;

[0054] Step 2.2: The MapReduce library divides the input file of the user program into M parts, where M is defined by the user;

[0055] Step 2.3: The worker assigned the Map task starts to read the input data of the corresponding shard. The number of Map tasks is determined by M and corresponds to the split one by one. The Map task extracts key-value pairs from the input data, and each key-value pair is passed as a parameter to the map function. The intermediate key-value pairs generated by the map function are cached in memory;

[0056] Step 2.3: The cached intermediate key-values are periodically written to the local disk and are divided into R regions. The size of R is defined by the user, and each region will correspond to a Reduce task in the future. The locations of these intermediate key-value pairs will be reported to the master, and the master is responsible for forwarding the information to the Reduce worker;

[0057] Step 2.5: The master notifies the worker assigned the Reduce task of the specific location of the partition it is responsible for. After the Reduce worker reads all the intermediate key-value pairs it is responsible for, it first sorts them so that the key-value pairs with the same key are grouped together;

[0058] Step 2.5: The reduce worker traverses the sorted intermediate key-value pairs. For each unique key, the key and the associated value are passed to the reduce function, and the output generated by the reduce function is added to the output file of this partition;

[0059] Step 2.7: When all Map and Reduce tasks are completed, the master wakes up the user program, and the MapReduce function call returns the code of the user program.

[0060] The implementation of zero-trust framework for power terminal security protection includes the following steps:

[0061] Step 3.1: Construct a zero-trust module to collect the device information of power terminal devices, and conduct trust scoring based on the collected device information to give a trust value. Evaluate the power terminal devices according to the trust value, and classify the power terminal devices into trusted devices and abnormal devices;

[0062] The process of the zero-trust module collecting device information is as follows: reading device data, reading rule files, parsing rule libraries, and collecting device information. At the same time, the zero-trust module conducts continuous dynamic device identity verification on power terminal devices to block false device information. The trust value is an index for identity verification and is obtained through comprehensive scoring based on the basic attributes and access latency of the device. The maintenance of the trust value includes the following contents:

[0063] (1) The maximum trust value is M, and the minimum is N; M > N

[0064] (2) The trust value threshold is H. Users with a trust value higher than or equal to H are legal users, and those lower than H are illegal users;

[0065] (3) Each successful verification adds T to the trust value;

[0066] (4) Each failed verification subtracts T from the trust value;

[0067] The trust value includes a direct trust value, a latency evaluation trust value, and an abnormal behavior evaluation trust value, and its calculation formula is as follows:

[0068] T = T d +T t +T a

[0069] T is the trust value, T d is the direct trust value, T t is the latency evaluation trust value, T a is the abnormal behavior evaluation trust value;

[0070] The direct trust value is an S-shaped function, and its calculation formula is:

[0071]

[0072] where T d is the direct trust value, and f is the direct trust value constraint coefficient for different devices; the latency evaluation trust value and the abnormal behavior evaluation trust value constitute an indirect trust value;

[0073] The latency evaluation trust value is evaluated according to the device response time, and its calculation formula is:

[0074]

[0075] Among which, T t is the trust value for delay evaluation, τ is the maximum allowable delay for device response, and D is the amount of information transmission delay;

[0076] The trust value for abnormal behavior evaluation is evaluated according to the proportion of abnormal behavior and normal behavior of the device. The calculation formula is:

[0077] Among which, T a is the trust value for abnormal behavior evaluation,

[0078] A u is the amount of abnormal behavior, and A n is the amount of normal behavior;

[0079] Step 3.2: Collect data from the trustworthy devices in Step 3.1 to obtain the collected data;

[0080] Step 3.3: Construct a security situation awareness module to perform situation awareness on the collected data in Step 3.2; when the awareness is qualified, convert the collected data into awareness data;

[0081] The situation awareness includes intrusion detection, vulnerability awareness, file integrity detection, and log monitoring operations;

[0082] Step 3.4: Construct a real-time control module to control the awareness data in Step 3.3 and generate security instructions;

[0083] Step 3.5: Send the security instructions in Step 3.4 to the power terminal devices to perform security protection and security reinforcement on the power terminal devices;

[0084] The present invention can improve data quality, enhance data storage security, enhance terminal security protection, reduce the probability of data being tampered with, damaged, and leaked, promote data circulation, give full play to the value of power grid data, meet the national requirements for data sharing and exchange, and provide support for using data innovation, mining data dividends, and promoting data economy.

[0085] On the other hand, the present invention provides a power grid data processing system based on Hadoop, including: a computer-readable storage medium and a processor;

[0086] The computer-readable storage medium is used to store executable instructions;

[0087] The processor is used to read the executable instructions stored in the computer-readable storage medium and execute the power grid data processing method according to the first aspect.

[0088] On the other hand, the present invention provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it implements the Hadoop-based power grid data processing method described in the first aspect.

[0089] Those skilled in the art should understand that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0090] The present application is described with reference to the flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or block in the flowchart and / or block diagram can be implemented by computer program instructions, and the combination of processes and / or blocks in the flowchart and / or block diagram can also be implemented. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing devices generate a device for implementing the functions specified in Figure 1 one process or multiple processes and / or blocks Figure 1 one block or multiple blocks.

[0091] These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory generate a manufactured product including an instruction device, and the instruction device implements the functions specified in Figure 1 one process or multiple processes and / or blocks Figure 1 one block or multiple blocks.

[0092] These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to generate a computer-implemented process. Therefore, the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in Figure 1 one process or multiple processes and / or blocks Figure 1 one block or multiple blocks.

[0093] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to the above embodiments, those of ordinary skill in the art should understand that: modifications or equivalent substitutions can still be made to the specific implementation manners of the present invention, and any modification or equivalent substitution that does not depart from the spirit and scope of the present invention shall be covered by the protection scope of the claims of the present invention.

Claims

1. A power grid data processing method based on Hadoop, characterized in that: The following steps are involved: Step 1: Hadoop-based power grid data mining and analysis technology collects power grid big data, including real-time data information of the power grid, equipment parameter data, power generation and load data; Step 2: Based on MapReduce technology, the collected power grid big data is stored and managed in the power grid big data storage platform for data security status storage; Step 3: Establish a zero-trust framework to implement power terminal security protection; The data security situation storage based on MapReduce technology includes the following steps: Step 2.1: Retrieve user layer data and input it into the user program; Step 2.2: The MapReduce library divides the user program's input file into M parts, where M is user-defined. Step 2.3: The worker assigned the Map job reads the input data for the corresponding shard. The Map job extracts key-value pairs from the input data. Each key-value pair is passed as a parameter to the map function. The intermediate key-value pairs generated by the map function are cached in memory. Step 2.3: The cached intermediate key values are periodically written to the local disk and divided into R partitions. The size of R is user-defined, and each partition will correspond to a Reduce job. The location of the intermediate key-value pairs is communicated to the master, which is responsible for forwarding this information to the Reduce workers. Step 2.5: The master notifies the worker assigned to the Reduce job of the specific location of the partition it is responsible for. After the Reduceworker reads all the intermediate key values it is responsible for, it sorts the intermediate key values so that key-value pairs with the same key are clustered together. Step 2.5: The reduce worker traverses the sorted intermediate key-value pairs. For each unique key, it passes the key and the associated value to the reduce function. The output generated by the reduce function is added to the output file of this partition. Step 2.7: When all Map and Reduce jobs are completed, the master wakes up the user program, and the MapReduce function call returns the user program code; The establishment of a zero-trust framework to achieve power terminal security protection includes the following steps: Step 3.1: Build a zero-trust module to collect device information of power terminal devices, assign trust scores based on the collected device information, and evaluate the power terminal devices based on the trust values, classifying them as trustworthy devices and abnormal devices. Step 3.2: collect data from the trusted device in step 3.1 to obtain collected data; Step 3.3: Build a security situation awareness module to perform situation awareness on the data collected in step 3.

2. When the awareness is qualified, the collected data is converted into awareness data. Step 3.4: Build a real-time control module to control the perception data in step 3.3 and generate security instructions; Step 3.5: Send the security instruction of step 3.4 to the power terminal equipment to perform security protection and security reinforcement on the power terminal equipment.

2. The Hadoop-based power grid data processing method according to claim 1, wherein: The Hadoop-based power grid data mining and analysis technology is implemented using a data acquisition layer, a data storage layer, a business application layer, and a user layer; The data collection layer adopts a distributed directional collection system architecture and uses terminal stations in different networks as a basic task unit for network data collection to collect raw network data and aggregate and transmit it to the data storage layer. Each basic task unit adopts independent collection rules and strategies. The data storage layer is used to complete the aggregation, storage and original processing of raw data and provide different types of function call services. The data storage layer is implemented using the Hadoop framework; The business application layer is used to retrieve and analyze the network data processed by the data storage layer to separate the public components from the personalized business application components, and transmit the results of the network data analysis to the user layer for real-time display; The user layer is used to transmit and display data information of the business application layer.

3. The Hadoop-based power grid data processing method according to claim 2, wherein: The basic task unit includes a data collection unit, which is used to collect data through a dynamic web page collection method and a web page information extraction method, and extract information using a method based on a row block distribution function to obtain data.

4. The Hadoop-based power grid data processing method according to claim 3, wherein: The data collection unit obtains feed addresses by breadth traversing the site, collects information corresponding to each feed address in real time, tracks updated information, and collects information in an incremental update manner.

5. The Hadoop-based power grid data processing method according to claim 2, wherein: The acquisition rules and strategies include vertical search template semi-automatic generation technology, dynamic page optimization access technology and intelligent crawling process scheduling strategy.

6. The Hadoop-based power grid data processing method according to claim 2, wherein: The processing of the original data in the data storage layer includes using window technology to block the data to be processed, using a sliding window model to describe the changes in the stream data, and using the sliding window model to save the patterns in the original data.

7. The Hadoop-based power grid data processing method according to claim 6, wherein: Use the sliding window model to preserve the patterns in the original data, specifically: Divide the data into blocks according to the changes in the data, and store the pattern of the unchanged part of the data in the sliding window; calculate the pattern of the added and deleted parts of the data respectively; update the pattern saved in the sliding window according to the pattern of the changed part of the data; A multi-window approach is used to support users' online mining requests. The multi-window approach divides the data stream into multiple fixed-length segments, each of which forms a window. When the number of windows in memory reaches a certain number, these multiple windows are merged to form a window with a higher summary level. As the data stream flows in, multiple windows with different summary levels form a hierarchical structure. At this time, each window is equivalent to a snapshot of the data between two predefined timestamps on the data stream.

8. A Hadoop-based power grid data processing system comprising: Computer-readable storage medium and processor; The computer-readable storage medium is used to store executable instructions; The processor is configured to read the executable instructions stored in the computer-readable storage medium and execute the Hadoop-based power grid data processing method according to any one of claims 1 to 7.

Images