A storage and analysis method for massive power network big data

Abstract

The present invention discloses a storage and analysis method for massive power network big data, comprising the following steps: step one, analyzing and classifying the big data; step two, displaying and reminding abnormal indicator data; step three, establishing a first buffer layer and storing structured data; step four, establishing a second buffer layer and storing semi-structured and unstructured data; step five, building a panoramic database based on Neo4j; step six, establishing a data partition management module; and step seven, establishing an efficient indexing method. The present invention establishes a first buffer layer and a second buffer layer to perform buffer storage on the massive big data, utilizes the characteristics of a distributed storage model, uses multiple servers to share the storage load, improves storage performance, uses a distributed file system HDFS and a data partition management module, performs distributed storage and multi-directional partitioning effective management on the massive big data, and establishes an efficient indexing method, which greatly shortens the response time of storage and later queries and reduces the difficulty of data management.

Description

Technical Field

[0001] The present invention relates to the technical field of power network big data, and in particular to a storage and analysis method for massive power network big data. Background Art

[0002] The rapid development and widespread application of information technology have enabled power production companies, trading departments and users to accumulate a large amount of data using the Internet of Things and the Internet. As the scale and scope of database applications continue to expand, power management departments and related companies have enhanced their ability to manage affairs using computers, resulting in huge large-scale data sets. It is very complicated to collect and store such a large data set on the server. Originally, many data collection algorithms could achieve good collection results when the data set was small, but for large-scale data sets, the calculation amount is very complicated, and collection and storage are very troublesome;

[0003] The existing traditional technology data processing capabilities store the power big data in a unified manner, which not only increases the load on the memory, but also fails to reasonably plan and organize a large amount of power data. At the same time, simple indexing will cause great trouble for subsequent management queries. Therefore, the present invention proposes a storage and analysis method for massive power network big data to solve the problems existing in the prior art. Summary of the invention

[0004] In view of the above problems, the purpose of the present invention is to propose a storage and analysis method for massive power network big data. The storage and analysis method for massive power network big data establishes a first buffer layer and a second buffer layer to buffer and store massive big data, utilizes the characteristics of a distributed storage model, and uses multiple servers to share the storage load, thereby improving storage performance. The distributed file system HDFS and the data partition management module effectively manage the distributed storage and multi-directional partitions of massive big data, and establishes an efficient indexing method to establish a global index again on the basis of indexing local data, thereby greatly shortening the response time of storage and subsequent queries and reducing the difficulty of data management.

[0005] To achieve the purpose of the present invention, the present invention is implemented by the following technical solution: a storage and analysis method for massive power network big data, comprising the following steps:

[0006] Step 1: Use a distributed storage model to temporarily store the massive power network big data collected from the client, and encrypt the big data after preliminary analysis and classification into structured data, semi-structured data, and unstructured data;

[0007] Step 2: After preliminary analysis of the collected data, some abnormal indicator data are extracted and exported separately for comparison, and the abnormal indicator data are output and displayed;

[0008] Step 3: Establish a first buffer layer based on the distributed file system HDFS, and store the encrypted structured data in the Hive data warehouse based on HDFS;

[0009] Step 4: Establish a second buffer layer based on the distributed file system HDFS, and store the encrypted loose semi-structured data and unstructured data in the distributed database HBase based on HDFS;

[0010] Step 5: Build a panoramic database of the power grid based on Neo4j, establish a device mapping table according to the power network topology, and orderly integrate the massive structured data, semi-structured data, and unstructured data that are scattered and isolated in the distributed file system HDFS;

[0011] Step 6: Establish a data partition management module in the panoramic database, which is divided into horizontal partition management and vertical partition management for internal organization;

[0012] Step 7: Establish an efficient indexing method based on a double-layer indexing model, and provide storage query access services based on the Internet cloud platform.

[0013] A further improvement is that the distributed storage model in step 1 uses multiple servers to share the storage load, and the storage performance of the distributed storage model is further improved through expansion in the later stage.

[0014] A further improvement is that in step 2, a warning is issued in a timely manner while displaying the abnormal indicator data, reminding relevant personnel to pay close attention, and if necessary, track the abnormal indicator data and make relevant processing in a timely manner.

[0015] A further improvement is that the distributed file system HDFS in step 3 and step 4 is a highly fault-tolerant distributed file system, which is deployed on cheap machines to provide high-throughput data access and is suitable for application on large-scale data sets.

[0016] Further improvement lies in: the first buffer layer in step three is a diversified storage medium cache structure based on DRAM and solid-state hard disk, and the access data is divided into read data and write data according to the access mode. The DRAM memory stores both read data and write data, and the solid-state hard disk only stores write data. The write data is stored in the DRAM memory and also in the solid-state hard disk at the same time.

[0017] A further improvement is that the second buffer layer in step 4 is based on the storage cache structure of the parallel execution engine, which caches loose semi-structured data and unstructured data, reducing the storage load of the distributed database HBase.

[0018] A further improvement is that the parallel execution engine divides a storage or reading operation into multiple independent storage or reading operations and executes them, and summarizes the results after the execution is completed.

[0019] A further improvement is that in step 5, Neo4j models the power network big data around the graph, using "node space" to express the big data in the power field. It will traverse nodes and edges at the same speed as traditional relational databases.

[0020] A further improvement is that in step six, horizontal partition management and vertical partition management of the data partition management module are established to adapt to operations of storing and accessing the same record or storing and accessing the same attribute of different records, thereby shortening the operation response time.

[0021] Further improvements are as follows: the method for establishing an efficient index based on a double-layer index model in step seven is to establish a global index for the actual storage range of local data on the basis of indexing local data, thereby improving the time performance of data query and insertion operations; the storage query access service in step six is ​​to receive storage query requests for power big data from the system and users through the Internet cloud platform, and to call related statements to execute storage query operations through SQL statements and complex storage query interfaces.

[0022] The beneficial effects of the present invention are:

[0023] (1) The present invention establishes a first buffer layer and a second buffer layer to perform buffer storage on massive big data, effectively reducing the storage load caused by unified storage of big data, utilizing the characteristics of a distributed storage model, using multiple servers to share the storage load, and improving storage performance;

[0024] (2) The distributed file system HDFS and data partition management module of the present invention can effectively manage the distributed storage and multi-directional partitioning of massive big data, greatly shortening the response time of storage and subsequent queries;

[0025] (3) The present invention establishes an efficient indexing method based on a double-layer indexing model. On the basis of indexing local data, a global index is established again. The double indexes are effective at the same time, which greatly improves the efficiency of data storage and query and reduces the difficulty of data management. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 This is a flow chart of Example 1 of the present invention.

[0027] Figure 2 This is a flow chart of Example 2 of the present invention.

[0028] Figure 3 This is a data classification architecture diagram for step one of the present invention.

[0029] Figure 4 This is a storage architecture diagram of step three and step four of embodiment 1 of the present invention.

[0030] Figure 5 This is a storage architecture diagram of step three of embodiment 2 of the present invention.

[0031] Figure 6 This is a diagram of the data partition management module structure in step five of the present invention. DETAILED DESCRIPTION

[0032] In order to deepen the understanding of the present invention, the present invention will be further described in detail below in conjunction with examples. The examples are only used to explain the present invention and do not constitute a limitation on the protection scope of the present invention.

[0033] Example 1

[0034] according to Figure 1 , 3 As shown in , 4, and 6, this embodiment provides a storage and analysis method for massive power network big data, including the following steps:

[0035] Step 1: Use a distributed storage model to temporarily store the massive power network big data collected from the client, and encrypt the big data after preliminary analysis and classification into structured data, semi-structured data, and unstructured data;

[0036] Step 2: After preliminary analysis of the collected data, some abnormal indicator data are extracted and exported separately for comparison, and the abnormal indicator data are output and displayed;

[0037] Step 3: Establish a first buffer layer based on the distributed file system HDFS, and store the encrypted structured data in the Hive data warehouse based on HDFS;

[0038] Step 4: Establish a second buffer layer based on the distributed file system HDFS, and store the encrypted loose semi-structured data and unstructured data in the distributed database HBase based on HDFS;

[0039] Step 5: Build a panoramic database of the power grid based on Neo4j, establish a device mapping table according to the power network topology, and orderly integrate the massive structured data, semi-structured data, and unstructured data that are scattered and isolated in the distributed file system HDFS;

[0040] Step 6: Establish a data partition management module in the panoramic database, which is divided into horizontal partition management and vertical partition management for internal organization;

[0041] Step 7: Establish an efficient indexing method based on a double-layer indexing model, and provide storage query access services based on the Internet cloud platform.

[0042] The distributed storage model in step 1 uses multiple servers to share the storage load, and the storage performance of the distributed storage model is further improved through expansion in the later stage.

[0043] In step 2, when abnormal indicator data is displayed, a warning is issued in a timely manner to remind relevant personnel to pay close attention and, if necessary, track the abnormal indicator data and make relevant processing in a timely manner.

[0044] The distributed file system HDFS in step 3 and step 4 is a highly fault-tolerant distributed file system, which is deployed on cheap machines to provide high-throughput data access and is suitable for application on large-scale data sets.

[0045] The first buffer layer in step three is a diversified storage medium cache structure based on DRAM and solid-state hard disks. Access data is divided into read data and write data according to the access mode. DRAM memory stores both read data and write data, and solid-state hard disks only store write data. Write data is stored in the solid-state hard disk at the same time as it is stored in the DRAM memory.

[0046] The second buffer layer in step 4 is a storage cache structure based on a parallel execution engine, which caches loose semi-structured data and unstructured data to reduce the storage load of the distributed database HBase. The parallel execution engine divides a storage or read operation into multiple independent storage or read operations and executes them, and summarizes the results after the execution is completed.

[0047] In step 5, Neo4j models the power network big data around the graph, using "node space" to express the big data in the power field. It traverses nodes and edges at the same speed as traditional relational databases.

[0048] In step six, horizontal partition management and vertical partition management of the data partition management module are established to adapt to operations of storing and accessing the same record or storing and accessing the same attribute of different records, thereby shortening the operation response time.

[0049] The method for establishing an efficient index based on a double-layer index model in step seven is to establish a global index for the actual storage range of local data on the basis of indexing local data, so as to improve the time performance of data query and insertion operations. The storage query access service in step seven is to receive storage query requests for power big data from the system and users through the Internet cloud platform, and to call related statements to execute storage query operations through SQL statements and complex storage query interfaces.

[0050] Example 2

[0051] according to Figure 2 ,3 As shown in , 5, and 6, this embodiment provides a storage and analysis method for massive power network big data, including the following steps:

[0052] Step 1: Use a distributed storage model to temporarily store the massive power network big data collected from the client, and encrypt the big data after preliminary analysis and classification into structured data, semi-structured data, and unstructured data;

[0053] Step 2: After preliminary analysis of the collected data, some abnormal indicator data are extracted and exported separately for comparison, and the abnormal indicator data are output and displayed;

[0054] Step 3: Establish a buffer layer based on the distributed file system HDFS, store the encrypted structured data in the Hive data warehouse based on HDFS, and store the encrypted loose semi-structured data and unstructured data in the distributed database HBase based on HDFS;

[0055] Step 4: Build a panoramic database of the power grid based on Neo4j, establish a device mapping table according to the power network topology, and orderly integrate the massive structured data, semi-structured data, and unstructured data that are scattered and isolated in the distributed file system HDFS;

[0056] Step 5: Establish a data partition management module in the panoramic database, which is divided into horizontal partition management and vertical partition management for internal organization;

[0057] Step 6: Establish an efficient indexing method based on a double-layer indexing model, and provide storage query access services based on the Internet cloud platform.

[0058] The distributed storage model in step 1 uses multiple servers to share the storage load, and the storage performance of the distributed storage model is further improved through expansion in the later stage.

[0059] In the step 2, a warning is issued in a timely manner while displaying the abnormal indicator data, reminding relevant personnel to pay close attention, and to track the abnormal indicator data and make relevant processing in a timely manner when necessary.

[0060] The distributed file system HDFS in step 3 is a highly fault-tolerant distributed file system, which is deployed on cheap machines to provide high-throughput data access and is suitable for application on large-scale data sets.

[0061] The buffer layer in step three is a storage cache structure based on a parallel execution engine, which caches structured data, loose semi-structured data and unstructured data to reduce the storage load of the Hive data warehouse and the distributed database HBase. The parallel execution engine divides a storage or read operation into multiple independent storage or read operations and executes them, and summarizes the results after the execution is completed.

[0062] In step 4, Neo4j models the power network big data around the graph, using "node space" to express the big data in the power field. It traverses nodes and edges at the same speed as traditional relational databases.

[0063] In the step 5, the horizontal partition management and the vertical partition management of the data partition management module are established to adapt to the operations of storing and accessing the same record or storing and accessing the same attribute of different records, thereby shortening the operation response time.

[0064] The method for establishing an efficient index based on a double-layer index model in step six is ​​to establish a global index for the actual storage range of local data on the basis of indexing local data, so as to improve the time performance of data query and insertion operations; the storage query access service is to receive storage query requests for power big data from the system and users through the Internet cloud platform, and to call related statements to execute storage query operations through SQL statements and complex storage query interfaces.

[0065] The present invention establishes a first buffer layer and a second buffer layer to perform buffer storage on massive big data, effectively reducing the storage load caused by unified storage of big data, utilizing the characteristics of a distributed storage model, utilizing multiple servers to share the storage load, and improving storage performance;

[0066] The distributed file system HDFS and data partition management module of the present invention can effectively manage the distributed storage and multi-directional partitioning of massive big data, greatly shortening the response time of storage and subsequent queries;

[0067] The present invention establishes an efficient indexing method based on a double-layer indexing model, and re-establishes a global index on the basis of indexing local data. The double indexes take effect simultaneously, which greatly improves the efficiency of data storage and query and reduces the difficulty of data management.

[0068] The above shows and describes the basic principles, main features and advantages of the present invention. It should be understood by those skilled in the art that the present invention is not limited to the above embodiments. The above embodiments and descriptions are only for explaining the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which fall within the scope of the present invention. The scope of protection of the present invention is defined by the attached claims and their equivalents.

Claims

1. A storage and analysis method for massive power network big data, characterized in that: The following steps are involved: Step 1: Use a distributed storage model to temporarily store the massive power network big data collected from the client, and encrypt the big data after preliminary analysis and classification into structured data, semi-structured data, and unstructured data; Step 2: After preliminary analysis of the collected data, some abnormal indicator data are extracted and exported separately for comparison, and the abnormal indicator data are output and displayed; Step 3: Establish a first buffer layer based on the distributed file system HDFS, and store the encrypted structured data in the Hive data warehouse based on HDFS; The first buffer layer is a diversified storage medium cache structure based on DRAM and solid-state hard disk; Step 4: Establish a second buffer layer based on the distributed file system HDFS, and store the encrypted loose semi-structured data and unstructured data in the distributed database HBase based on HDFS; The second buffer layer is a storage cache structure based on a parallel execution engine, caching loose semi-structured data and unstructured data; Step 5: Build a panoramic database of the power grid based on Neo4j. Neo4j models the power grid big data around graphs and expresses the power field big data in node space. It traverses nodes and edges at the same speed as traditional relational databases. It establishes a device mapping table based on the power grid topology and orderly integrates the massive structured data, semi-structured data, and unstructured data that are scattered and isolated in the distributed file system HDFS. Step 6: Establish a data partition management module in the panoramic database, which is divided into horizontal partition management and vertical partition management for internal organization; establish the horizontal partition management and vertical partition management of the data partition management module to adapt to the operations of storing and accessing the same record or storing and accessing the same attribute of different records; Step 7: Establish an efficient indexing method based on a double-layer indexing model, and provide storage query access services based on the Internet cloud platform.

2. A storage and analysis method for massive power network big data according to claim 1, characterized in that: The distributed storage model in step 1 uses multiple servers to share the storage load, and the storage performance of the distributed storage model is further improved through expansion in the later stage.

3. The storage and analysis method for massive power network big data according to claim 1 is characterized in that: In the step 2, a warning is issued in a timely manner while displaying the abnormal indicator data, reminding relevant personnel to pay close attention, track the abnormal indicator data and make relevant processing in a timely manner.

4. The storage and analysis method for massive power network big data according to claim 1 is characterized in that: The distributed file system HDFS in step 3 and step 4 is a highly fault-tolerant distributed file system, which is deployed on cheap machines to provide high-throughput data access and is suitable for application on large-scale data sets.

5. The storage and analysis method for massive power network big data according to claim 1 is characterized by: The DRAM memory in step three stores both read data and write data, and the solid state drive only stores write data. The write data is stored in the DRAM memory and also in the solid state drive at the same time.

6. The storage and analysis method for massive power network big data according to claim 1 is characterized by: The parallel execution engine divides a storage or read operation into multiple independent storage or read operations and executes them, and summarizes the results after the execution is completed.

7. The storage and analysis method for massive power network big data according to claim 1 is characterized by: The method for establishing an efficient index based on a double-layer index model in step 7 is to establish a global index for the actual storage range of local data on the basis of indexing local data, thereby improving the time performance of data query and insertion operations. The storage query access service in step seven is to receive the storage query request of the system and the user for the power big data through the Internet cloud platform, and call the relevant statements to execute the storage query operation through SQL statements and complex storage query interfaces.

Images