Power internet of things data retrieval method, system and device based on spark platform

Abstract

The application discloses a power Internet of Things data retrieval method, system and device based on a Spark platform, mainly relates to the power Internet of Things data retrieval technical field, and is used to solve the problem that the existing power Internet of Things operation data retrieval scheme has a long retrieval time and consumes a large amount of memory, and there is a burden of continuously retrieving massive data in a centralized manner. The application comprises the following steps: S1, obtaining a data retrieval statement based on received query data; S2, creating a Dataframe containing retrieval data in HIVE, and then creating an RDD through a SparkContext instance; S3, judging whether the aggregation dimension of the RDD is less than the total column number of the Dataframe; when the aggregation dimension is less than the total column number, storing the RDD in HBase; when the aggregation dimension is greater than or equal to the total column number, storing the pruned RDD in HBase through a Spark operator and a BUC algorithm; and S4, returning the RDD data in HBase to a user. The application successfully retrieves data in a short time and with less memory consumption through the above method, and gets rid of the burden of continuously retrieving massive data in a centralized manner.

Description

Technical Field

[0001] This application relates to the field of power Internet of Things (IoT) data retrieval technology, and in particular to a power IoT data retrieval method, system and device based on the Spark platform. Background Technology

[0002] The Internet of Things (IoT) in the power grid is the application of IoT in smart grids. It will effectively integrate communication infrastructure resources and power system infrastructure resources, improve the informatization level of the power system, improve the utilization efficiency of existing power system infrastructure, and provide important technical support for the generation, transmission, transformation, distribution, and consumption of electricity in the power grid.

[0003] With the development of database technology and the increasing scale of databases, people hope to extract useful information from existing databases to serve decision-making.

[0004] However, the current data retrieval solutions in the operation of the power Internet of Things have long retrieval times and consume a lot of memory, resulting in the burden of continuously retrieving massive amounts of data. Summary of the Invention

[0005] To address the aforementioned shortcomings of existing technologies, this invention provides a power Internet of Things (IoT) data retrieval method, system, and device based on the Spark platform, in order to solve the aforementioned technical problems.

[0006] Firstly, this application provides a data retrieval method for the power Internet of Things based on the Spark platform. The method includes: S1: obtaining a data retrieval statement based on the received query data; S2: obtaining the retrieval data corresponding to the data retrieval statement in HBase, creating a DataFrame containing the retrieval data in HIVE, and then creating an RDD through a SparkContext instance; S3: determining whether the aggregation dimension of the RDD is less than the total number of columns in the DataFrame; if it is less than the total number of columns, storing the RDD in HBase; if it is greater than or equal to the total number of columns, storing the pruned RDD in HBase through Spark operators and the BUC algorithm; S4: returning the RDD data in HBase to the user.

[0007] Furthermore, the data retrieval statement contains several sub-retrieval statements; the retrieval data corresponding to the data retrieval statement in HBase is obtained, and a DataFrame containing the retrieval data is created in HIVE. Specifically, this includes: when the sub-retrieval statement specifies a data table and the corresponding table header name, obtaining the first number of table headers and the first column data corresponding to the table header name based on the sub-retrieval statement; when the sub-retrieval statement specifies a data table and all table header names, obtaining the second number of table headers and the second column data corresponding to the data table based on the number of columns in the data table; determining the total number of columns and table header data corresponding to the user's query data based on the first and second table header numbers; determining the retrieval data based on the first and second column data; and creating a DataFrame in HIVE based on the total number of columns, the table header data, and the retrieval data.

[0008] Furthermore, creating an RDD through a SparkContext instance specifically includes: when the retrieved data is detected as a collection, calling the parallelize() method in the SparkContext instance to create an RDD based on the retrieved data; and when the retrieved data is detected as a local file or an HDFS file, calling the textFile() method in the SparkContext instance to create an RDD based on the retrieved data.

[0009] Furthermore, when the number of columns is less than the total number of columns, the RDD is stored in HBase; when the number of columns is greater than or equal to the total number of columns, the pruned RDD is stored in HBase using Spark operators and the BUC algorithm. Specifically, this includes: determining whether the clustering dimension of the RDD is 1; if it is 1, using the groupby operator in Spark to count the occurrences of each dimension in the RDD; storing dimension data with occurrences greater than a preset threshold as RDD data in HBase; pruning and discarding dimension data with occurrences less than or equal to the preset threshold using the BUC algorithm; if it is not 1, merging the dimension data in the RDD, and using the groupby and count operators in Spark to group the merged results and count their occurrences; storing the merged results with occurrences greater than the preset threshold as RDD data in HBase.

[0010] Furthermore, the method also includes: pruning and discarding merged results whose occurrence frequency is less than or equal to a preset threshold using the BUC algorithm to update the RDD; repeating step S3 with the updated RDD until there is an RDD with fewer than the total number of columns, or dimension data with an occurrence frequency greater than the preset threshold, or merged results with an occurrence frequency greater than the preset threshold.

[0011] Secondly, this application provides a power Internet of Things (IoT) data retrieval system based on the Spark platform. The system includes: a creation module, used to obtain a data retrieval statement based on the received query data; it is also used to obtain the retrieval data corresponding to the data retrieval statement in HBase, create a DataFrame containing the retrieval data in HIVE, and then create an RDD through a SparkContext instance; a storage module, used to determine whether the aggregation dimension of the RDD is less than the total number of columns of the DataFrame; when it is less than the total number of columns, the RDD is stored in HBase; when it is greater than or equal to the total number of columns, the pruned RDD is stored in HBase through Spark operators and the BUC algorithm; and a return module, used to return the RDD data in HBase to the user.

[0012] Furthermore, the creation module also includes an RDD creation unit; the RDD creation unit is used to call the parallelize() method in the SparkContext instance to create an RDD based on the retrieved data when the retrieved data is detected as a collection; and to call the textFile() method in the SparkContext instance to create an RDD based on the retrieved data when the retrieved data is detected as a local file or an HDFS file.

[0013] Thirdly, this application provides a power Internet of Things (IoT) data retrieval device based on the Spark platform. The device includes: a processor; and a memory storing executable code thereon. When the executable code is executed, the processor performs a power IoT data retrieval method based on the Spark platform as described above.

[0014] Those skilled in the art will understand that the present invention has at least the following beneficial effects:

[0015] The solution of this invention employs a hierarchical strategy during the RDD aggregation process, which reduces the number of intermediate results requiring iteration in memory. This significantly improves the algorithm performance under the Spark distributed parallel framework and makes it more suitable for distributed operation. The solution uses BUC for rapid data computation and leverages Spark's fast data processing capabilities to effectively alleviate the computational pressure on massive amounts of power data, greatly enhancing data analysis and storage capabilities.

[0016] This solution provides a high-performance data retrieval solution for the operation of the power Internet of Things. It can retrieve distributed data in parallel, providing convenience for professionals. It can successfully retrieve data in a short time and with less memory consumption, freeing them from the burden of continuously retrieving massive amounts of data in a centralized manner. Attached Figure Description

[0017] The following description refers to some embodiments of this disclosure, in which:

[0018] Figure 1 This is a flowchart of a power Internet of Things (IoT) data retrieval method based on the Spark platform provided in an embodiment of this application.

[0019] Figure 2 This is a schematic diagram of the internal structure of a power Internet of Things data retrieval system based on the Spark platform provided in an embodiment of this application.

[0020] Figure 3 This is a schematic diagram of the internal structure of a power Internet of Things (IoT) data retrieval device based on the Spark platform, provided in an embodiment of this application. Detailed Implementation

[0021] Those skilled in the art should understand that the embodiments described below are merely preferred embodiments of this disclosure and do not imply that this disclosure can only be implemented through these preferred embodiments. These preferred embodiments are merely used to explain the technical principles of this disclosure and are not intended to limit the scope of protection of this disclosure. Based on the preferred embodiments provided by this disclosure, all other embodiments obtained by those skilled in the art without creative effort should still fall within the scope of protection of this disclosure.

[0022] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0023] This invention relates to a method, system, and device for data retrieval in the power Internet of Things (IoT) based on the Spark platform. Distributed file systems such as HDFS and S3 are used for data storage. When retrieving data, the scheme reads data files from the HDFS file system and creates a Resilient Distributed Dataset (RDD) of vertex cube All, which is then stored in memory. Next, the data format queried by the user is converted into a DataFrame, and then stored in a temporary table registered in HIVE. The RDD is continuously operated on using Spark's groupby and count operators until the aggregation dimension is less than the number of columns in the DataFrame. During this process, the RDD after each operation is evaluated to determine whether to store it in HBase. Finally, the user retrieves the data directly from HBase.

[0024] The technical solutions proposed in the embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0025] This application also provides a method for retrieving power Internet of Things (IoT) data based on the Spark platform, such as... Figure 1 As shown in the embodiments of this application, the method mainly includes the following steps:

[0026] S1: Based on the received query data, obtain the data retrieval statement.

[0027] S2: Obtain the corresponding retrieval data in HBase for the data retrieval statement, create a DataFrame containing the retrieval data in HIVE, and then create an RDD through a SparkContext instance.

[0028] It should be noted that a data retrieval statement is used to express the data requirements for retrieval. Furthermore, a data retrieval statement can contain several sub-retrieval statements.

[0029] Specifically, this involves retrieving the corresponding data in HBase for the data retrieval query, and creating a DataFrame containing the retrieved data in HIVE. This can be done as follows:

[0030] S21: When the sub-retrieval statement specifies the data table and the corresponding table header name, the first number of table headers and the first column data corresponding to the table header name are obtained according to the sub-retrieval statement; when the sub-retrieval statement specifies the data table and all table header names, the second number of table headers and the second column data corresponding to the data table are obtained according to the number of columns in the data table.

[0031] S22: Based on the number of the first and second table headers, determine the total number of columns and the number of header data corresponding to the user's query data; based on the first and second column data, determine the data to be retrieved;

[0032] S23: Create a Dataframe in HIVE based on the total number of columns, header data, and retrieval data.

[0033] Specifically, creating an RDD using a SparkContext instance can be done as follows:

[0034] When the retrieved data is detected as a collection, the `parallelize()` method in the `SparkContext` instance is called to create an RDD based on the retrieved data; when the retrieved data is detected as a local file or an HDFS file, the `textFile()` method in the `SparkContext` instance is called to create an RDD based on the retrieved data.

[0035] S3: Determine if the aggregation dimension of the RDD is less than the total number of columns in the DataFrame; if it is less than the total number of columns, store the RDD in HBase; if it is greater than or equal to the total number of columns, store the pruned RDD in HBase using Spark operators and the BUC algorithm.

[0036] Specifically, when the number of columns is less than the total number of columns, the RDD is stored in HBase; when the number of columns is greater than or equal to the total number of columns, the pruned RDD is stored in HBase using Spark operators and the BUC algorithm. More specifically:

[0037] S310: Determine if the aggregation dimension of the RDD is 1;

[0038] S311: If it is 1, use the groupby operator in Spark to count the number of times each dimension of data appears in the RDD; store the dimension data with the number of occurrences greater than the preset threshold as RDD data in HBase; prune and discard the dimension data with the number of occurrences less than or equal to the preset threshold using the BUC algorithm.

[0039] S312: If not 1, merge the dimensional data in the RDD, and use the groupby and count operators in Spark to group the merged results and count the number of occurrences; merged results with a number of occurrences greater than a preset threshold are stored as RDD data in HBase.

[0040] In addition, this application can iteratively compute the RDD, specifically by: pruning and discarding merged results whose occurrence frequency is less than or equal to a preset threshold using the BUC algorithm to update the RDD; repeating step S3 with the updated RDD until there is an RDD with fewer than the total number of columns, or dimension data with an occurrence frequency greater than the preset threshold, or merged results with an occurrence frequency greater than the preset threshold.

[0041] S4: Return the RDD data in HBase to the user.

[0042] besides, Figure 2 This application provides an embodiment of a power Internet of Things (IoT) data retrieval system based on the Spark platform. For example... Figure 2 As shown in the embodiments of this application, the system mainly includes:

[0043] The module is used to obtain the data retrieval statement based on the received query data; it is also used to obtain the retrieval data corresponding to the data retrieval statement in HBase, create a DataFrame containing the retrieval data in HIVE, and then create an RDD through a SparkContext instance;

[0044] The creation module also includes the RDD creation unit;

[0045] The RDD creation unit is used to call the `parallelize()` method in the `SparkContext` instance to create an RDD based on the retrieved data when the retrieved data is detected as a collection; and to call the `textFile()` method in the `SparkContext` instance to create an RDD based on the retrieved data when the retrieved data is detected as a local file or an HDFS file.

[0046] The storage module is used to determine whether the aggregation dimension of the RDD is less than the total number of columns of the DataFrame; when it is less than the total number of columns, the RDD is stored in HBase; when it is greater than or equal to the total number of columns, the pruned RDD is stored in HBase using Spark operators and the BUC algorithm.

[0047] The return module is used to return RDD data from HBase to the user.

[0048] In addition, embodiments of this application also provide a power Internet of Things (IoT) data retrieval device based on the Spark platform, such as... Figure 3 As shown, it stores executable instructions, which, when executed, implement a power IoT data retrieval method based on the Spark platform as described above. Specifically, the server sends execution instructions to the memory via the bus. When the memory receives the execution instructions, it sends an execution signal to the processor via the bus to activate the processor.

[0049] It should be noted that the processor is used to obtain the data retrieval statement based on the received query data; obtain the retrieval data corresponding to the data retrieval statement in HBase; create a DataFrame containing the retrieval data in HIVE; and then create an RDD through a SparkContext instance; determine whether the aggregation dimension of the RDD is less than the total number of columns in the DataFrame; if it is less than the total number of columns, store the RDD in HBase; if it is greater than or equal to the total number of columns, store the pruned RDD in HBase using Spark operators and the BUC algorithm; and return the RDD data in HBase to the user.

[0050] The technical solutions of this disclosure have been described in conjunction with the preceding embodiments. However, it will be readily understood by those skilled in the art that the scope of protection of this disclosure is not limited to these specific embodiments. Without departing from the technical principles of this disclosure, those skilled in the art can disassemble and combine the technical solutions in the above embodiments, and can also make equivalent changes or substitutions to the relevant technical features. Any changes, equivalent substitutions, improvements, etc., made within the technical concept and / or technical principles of this disclosure will fall within the scope of protection of this disclosure.

Claims

1. A method for retrieving power Internet of Things (IoT) data based on the Spark platform, characterized in that, The method includes: S1: Based on the received query data, obtain the data retrieval statement; the data retrieval statement contains several sub-retrieval statements; Obtain the retrieval data corresponding to the data retrieval statement in HBase, and create a DataFrame containing the retrieval data in HIVE, specifically including: When the data table and the corresponding table header name are specified in the sub-retrieval statement, the first number of table headers and the first column data corresponding to the table header name are obtained according to the sub-retrieval statement. When a data table and all table header names are specified in the sub-retrieval statement, the number of second table headers and the corresponding second column data of the data table are obtained based on the number of columns in the data table. Based on the number of the first table header and the number of the second table header, determine the total number of columns and the table header data corresponding to the user's query data; Based on the data in the first column and the data in the second column, determine the data to be retrieved; Based on the total number of columns, header data, and retrieval data, create a DataFrame in HIVE; S2: Obtain the retrieval data corresponding to the data retrieval statement in HBase, create a DataFrame containing the retrieval data in HIVE, and then create an RDD through a SparkContext instance; specifically including: When the retrieved data is detected to be a collection, the parallelize() method in the SparkContext instance is called to create an RDD based on the retrieved data; When the retrieved data is detected to be a local file or an HDFS file, the textFile() method in the SparkContext instance is called to create an RDD based on the retrieved data; S3: Determine whether the aggregation dimension of the RDD is less than the total number of columns in the DataFrame; if it is less than the total number of columns, store the RDD in HBase; if it is greater than or equal to the total number of columns, store the pruned RDD in HBase using Spark operators and the BUC algorithm; specifically, this includes: determining whether the aggregation dimension of the RDD is 1. If the value is 1, the groupby operator in Spark is used to count the number of times each dimension of data appears in the RDD; dimension data with a frequency greater than a preset threshold is stored in HBase as RDD data; dimension data with a frequency less than or equal to the preset threshold is pruned and discarded using the BUC algorithm. If it is not 1, the dimensional data in the RDD will be merged, and the groupby and count operators in Spark will be used to group the merged results and count the number of occurrences. The merged results with a number of occurrences greater than a preset threshold will be stored as RDD data in HBase. S4: Return the RDD data in HBase to the user.

2. The power IoT data retrieval method based on the Spark platform according to claim 1, characterized in that, The method further includes: The BUC algorithm is used to prune and discard merge results that occur less than or equal to a preset threshold in order to update the RDD; Repeat step S3 with the updated RDD until there is an RDD with fewer than the total number of columns, or dimension data with more than a preset threshold occurrences, or a merged result with more than a preset threshold occurrences.

3. A power Internet of Things (IoT) data retrieval system based on the Spark platform, characterized in that, The system includes: A module is created to obtain a data retrieval statement based on received query data. The data retrieval statement contains several sub-retrieval statements. The module retrieves the corresponding retrieval data in HBase based on the data retrieval statement and creates a DataFrame containing the retrieval data in HIVE. Specifically, this includes: when a sub-retrieval statement specifies a data table and the corresponding table header name, obtaining the first number of table headers and the first column data corresponding to the table header name based on the sub-retrieval statement; when a sub-retrieval statement specifies a data table and all table header names, obtaining the second number of table headers and the second column data corresponding to the data table based on the number of columns in the data table; determining the total number of columns and table header data corresponding to the user's query data based on the first and second table header numbers; and determining the total number of columns and table header data based on the first and second column data. The system retrieves column data to determine the data to be retrieved. Based on the total number of columns, header data, and retrieved data, a DataFrame is created in HIVE. It also retrieves the corresponding data in HBase for the data retrieval statement, creates a DataFrame containing the retrieved data in HIVE, and then creates an RDD through a SparkContext instance. Specifically, when the retrieved data is detected as a collection, the `parallelize()` method in the SparkContext instance is called to create an RDD based on the retrieved data; when the retrieved data is detected as a local file or HDFS file, the `textFile()` method in the SparkContext instance is called to create an RDD based on the retrieved data. The storage module is used to determine whether the clustering dimension of the RDD is less than the total number of columns of the DataFrame. When it is less than the total number of columns, the RDD is stored in HBase. When it is greater than or equal to the total number of columns, the pruned RDD is stored in HBase using Spark operators and the BUC algorithm. Specifically, this includes: determining whether the clustering dimension of the RDD is 1; if it is 1, using the groupby operator in Spark to count the occurrences of each dimension in the RDD; storing dimension data with occurrences greater than a preset threshold as RDD data in HBase; pruning and discarding dimension data with occurrences less than or equal to the preset threshold using the BUC algorithm; if it is not 1, merging the dimension data in the RDD, and using the groupby and count operators in Spark to group the merged results and count the occurrences; storing the merged results with occurrences greater than the preset threshold as RDD data in HBase. The return module is used to return RDD data from HBase to the user.

4. The power Internet of Things data retrieval system based on the Spark platform according to claim 3, characterized in that, The creation module also includes an RDD creation unit; The RDD creation unit is used to call the parallelize() method in the SparkContext instance to create an RDD based on the retrieved data when the retrieved data is detected to be a collection; and to call the textFile() method in the SparkContext instance to create an RDD based on the retrieved data when the retrieved data is detected to be a local file or an HDFS file.

5. A power Internet of Things (IoT) data retrieval device based on the Spark platform, characterized in that, The device includes: processor; And a memory having executable code stored thereon, which, when executed, causes the processor to perform a power Internet of Things data retrieval method based on the Spark platform as described in any one of claims 1-2.

Images