A heterogeneous energy data storage method based on lake-warehouse integration

Abstract

The present invention discloses a method for storing heterogeneous energy data based on a lake-warehouse integration, including collecting heterogeneous energy data and preprocessing the data; performing fuzzy clustering on the processed data using an FCM algorithm to extract characteristic keywords; and loading the source data in the HDFS component into a lake-warehouse table created based on a Hive component based on a Hive component. This method unifies heterogeneous energy data through FCM clustering and ETL tools, optimizes data loading and query efficiency, and reduces processing time. The use of a lake-warehouse integration architecture enhances data governance and cross-domain sharing, improves system scalability and flexibility, reduces data processing time, improves the efficiency of data query and execution, and effectively reduces the problem of difficult unified management of data islands.

Description

Technical Field

[0001] The present invention relates to the technical field of heterogeneous energy data storage, and in particular to a lake-warehouse integrated heterogeneous energy data storage method. Background Art

[0002] 1. Thesis "Research on Lake-Warehouse Integrated Architecture from the Perspective of Data Governance": By comparing the differences between data warehouses, data lakes, and lake-warehouse integration, analyzing the advantages of lake-warehouse integration, dividing business areas and mapping them to the data perspective to build a distributed lake-warehouse integrated architecture, integrating existing research and related technologies to build a lake-warehouse integrated functional module, and explaining the dynamic stream-batch integrated data flow process. The distributed lake-warehouse integrated architecture includes construction concepts such as data domain decoupling, cross-domain data sharing, and joint data governance; the lake-warehouse integrated functional module mainly includes data sources, lake-warehouse integrated core functional areas, and users; the stream-batch integrated data flow process includes batch data processes and real-time data processes. This study can integrate the lake-warehouse integration into the effective data governance process and build a relatively complete lake-warehouse integrated architecture system, thereby providing a reference for related research or enterprises.

[0003] 2. The paper "Analysis and Design of Distributed Lake-Warehouse Integrated Architecture for Railway Data": A brief analysis of the existing data lake architecture was conducted to determine the use of the concept of lake-warehouse integration, and railway data was classified by subject to meet business processing needs; secondly, a distributed lake-warehouse integrated architecture for railway data was designed, and the architecture and functions of the railway bureau-level sub-lake-warehouse integration and the national railway group's total lake-warehouse integration, as well as the data flow process between the two, were explained; finally, the characteristics and existing problems of the designed architecture were analyzed, providing a reference for further building an effective railway operation data lake.

[0004] 3. Patent "A Lake-Warehouse Integrated Architecture": Discloses a lake-warehouse integrated architecture in the field of data warehouses, including a data collection layer, a data warehouse and a business application layer; wherein: the data collection layer is used to collect business data and log data, and transmit the collected data to the data warehouse; the data warehouse is implemented based on the data lake, and is used to process and analyze the data transmitted by the data collection layer, and transmit the data obtained from the processing and analysis to the business application layer; the business application layer is used to provide the data transmitted by the data warehouse to external business systems, so that the business systems can realize the application of corresponding data. Summary of the invention

[0005] In view of the above existing problems, the present invention is proposed.

[0006] Therefore, the present invention provides a heterogeneous energy data storage method based on lake-warehouse integration, which can solve the problem that heterogeneous data is difficult to effectively integrate and uniformly manage, and does not propose an effective processing method. This method pre-processes and transforms heterogeneous energy data by using the FCM clustering algorithm (Fuzzy C-means algorithm, referred to as FCM algorithm) and ETL tools (Extract-Transform-Load, used to describe the process of extracting (extract), transforming (transform), and loading (load) data from the source end to the destination end), thereby achieving a unified format and classified storage of data, and effectively solving the problem of isolated islands of heterogeneous energy data.

[0007] In order to solve the above technical problems, the present invention provides the following technical solutions, a heterogeneous energy data storage method based on lake-warehouse integration, comprising:

[0008] Collecting heterogeneous energy data and preprocessing the data;

[0009] The processed data is subjected to fuzzy clustering using the FCM algorithm to extract characteristic keywords;

[0010] Based on the Hive component, the source data in the HDFS component is loaded into the lake warehouse table created based on the Hive component.

[0011] As a preferred solution of the heterogeneous energy data storage method based on lake-warehouse integration described in the present invention, the heterogeneous energy data includes power industry data, other energy data, and external support data.

[0012] As a preferred solution of the heterogeneous energy data storage method based on lake-warehouse integration described in the present invention, the preprocessing of the data includes specifying the number of clustering categories c (2≤c≤A), A is the number of processed data, ζ∈(0,∞) is used to specify the iteration threshold, initialize the clustering center β(0), and set the initial iteration index k=1.

[0013] As a preferred solution of the heterogeneous energy data storage method based on lake-warehouse integration described in the present invention, the preprocessing of the data also includes calculating the fuzzy partition matrix of the kth iteration.

[0014] As a preferred solution of the heterogeneous energy data storage method based on lake-warehouse integration described in the present invention, wherein: the fuzzy clustering of the processed data using the FCM algorithm to extract characteristic keywords includes sorting the data and obtaining a permutation function so that the function satisfies: The D function represents the Euclidean distance, that is that is With xt The Euclidean distance between

[0015] calculate uniform weight, where It refers to the cluster center weight after the permutation operation;

[0016] right Sort and take As the minimum parameter of the cluster center point in this fuzzy clustering, it is used for the calculation of subsequent steps;

[0017] If ||β (k+1) -β (k) ||>ζ, then return to the previous step and increment k by 1 until ||β (k+1) -β (k) ||≤ζ or the iteration index k reaches 50, and the final extracted feature keywords are obtained.

[0018] As a preferred solution of the heterogeneous energy data storage method based on lake-warehouse integration described in the present invention, wherein: the fuzzy clustering of the processed data using the FCM algorithm to extract feature keywords also includes optimizing the calculation formula of the fuzzy partition matrix as follows:

[0019]

[0020] Among them, D function represents the Euclidean distance, c is the number of cluster categories, A is the number of data after dimensionality reduction, m is the weighted index, m∈(1,∞); the cluster center weight after permutation operation for:

[0021]

[0022] Among them, A is the number of processed data, u ij is the fuzzy partition matrix membership, m is the weighted index, x j represents the jth data, and c is the number of clustering categories.

[0023] As a preferred solution of the heterogeneous energy data storage method based on lake-warehouse integration described in the present invention, wherein: the source data is loaded into the lake-warehouse table created based on the Hive component, the data storage layer of the lake-warehouse table includes the original data layer, the data warehouse layer and the data service layer, the source data is cleaned, and the data obtained after the processing is stored in the original data layer, and at the same time, basic operations and optimization operations are performed on the data stored in the original data layer, and the data obtained after the operation is stored in the data warehouse layer.

[0024] As a preferred solution of the heterogeneous energy data storage method based on lake-warehouse integration described in the present invention, wherein: the source data is loaded into the lake-warehouse table created based on the Hive component, including: Hive is a data warehouse tool built on Hadoop, providing Hive SQL language to analyze and process data stored in the HDFS file system, Iceberg provides Hive external table format storage to integrate Iceberg data into the Hive table through Hive.

[0025] A computer device includes a memory and a processor, wherein the memory stores a computer program, and is characterized in that when the processor executes the computer program, it implements the steps of a heterogeneous energy data storage method based on lake-warehouse integration.

[0026] A computer-readable storage medium having a computer program stored thereon, characterized in that when the computer program is executed by a processor, the steps of a heterogeneous energy data storage method based on lake-warehouse integration are implemented.

[0027] Beneficial effects of the present invention: This method unifies heterogeneous energy data through FCM clustering and ETL tools, optimizes data loading and query efficiency, and reduces processing time. The lake-warehouse integrated architecture enhances data governance and cross-domain sharing, improves system scalability and flexibility, reduces data processing time, improves data query and execution efficiency, and effectively reduces the problem of data islands being difficult to manage in a unified manner. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0029] Figure 1 A schematic diagram of a heterogeneous data lake-warehouse integrated architecture of a heterogeneous energy data storage method based on a lake-warehouse integration is provided for an embodiment of the present invention.

[0030] Figure 2 A schematic diagram of Hive table information of a heterogeneous energy data storage method based on lake-warehouse integration provided for an embodiment of the present invention.

[0031] Figure 3 A schematic diagram of Hive table information of a heterogeneous energy data storage method based on lake-warehouse integration provided for an embodiment of the present invention.

[0032] Figure 4A flowchart of a heterogeneous energy data storage framework based on a lake-warehouse integration is provided for a heterogeneous energy data storage method based on a lake-warehouse integration according to an embodiment of the present invention. DETAILED DESCRIPTION

[0033] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the drawings of the specification. Obviously, the described embodiments are part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary persons in the art without creative work should fall within the scope of protection of the present invention.

[0034] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0035] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The term "in one embodiment" that appears in different places in this specification does not necessarily refer to the same embodiment, nor does it refer to a separate or selective embodiment that is mutually exclusive with other embodiments.

[0036] The present invention is described in detail with reference to schematic diagrams. When describing the embodiments of the present invention, for the sake of convenience, the cross-sectional diagrams showing the device structure will not be partially enlarged according to the general scale, and the schematic diagrams are only examples, which should not limit the scope of protection of the present invention. In addition, in actual production, the three-dimensional dimensions of length, width and depth should be included.

[0037] At the same time, in the description of the present invention, it should be noted that the directions or positional relationships indicated by the terms "upper, lower, inner and outer" are based on the directions or positional relationships shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific direction, be constructed and operated in a specific direction, and therefore cannot be understood as limiting the present invention. In addition, the terms "first, second or third" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance.

[0038] In the present invention, unless otherwise clearly specified and limited, the terms "install, connect, connect" should be understood in a broad sense, for example: it can be a fixed connection, a detachable connection or an integral connection; it can also be a mechanical connection, an electrical connection or a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0039] Example 1, reference Figure 1-Figure 4 , which is the first embodiment of the present invention, and provides a heterogeneous energy data storage method based on lake-warehouse integration, including:

[0040] The present invention includes a connection component, an HDFS component and a Hive component; the connection component is used to realize the data link between the source data storage system and the lake warehouse integrated data processing system; the HDFS component is used to store the source data read by the connection component; the Hive component is used to load the source data in the HDFS component into the lake warehouse table created based on the Hive component;

[0041] The data storage layer of the lake warehouse table includes the original data layer, the data warehouse layer and the data service layer; the original data layer refers to the layer used to store the source data loaded from the HDFS component; the data warehouse layer refers to the layer used to store the data obtained after basic operations and optimization operations are performed on the data stored in the original data layer; the data service layer is used to store the data obtained after business processing and conversion of the data stored in the data warehouse layer. The data warehouse layer includes a first sublayer and a second sublayer. Among them, the first sublayer is used to store the data obtained after basic operations and optimization operations are performed on the data stored in the original data layer; the second sublayer is used to store the data obtained after the sub-table operation is performed based on the data stored in the first sublayer.

[0042] The data storage layer of the heterogeneous energy data lake warehouse integrated architecture constructed by the present invention includes an original data layer (ODS layer), a data warehouse layer (DW layer) and a data service layer (ADS layer); wherein the original data layer is used to store source data loaded from the hdfs component; the data warehouse layer is used to store data obtained after basic operations and optimization operations are performed on the data stored in the original data layer; and the data service layer is used to store data obtained after business processing and conversion of the data stored in the data warehouse layer.

[0043] Among them, the ODS layer is the layer closest to the data stored in the source data storage system. In order to consider the need to trace data issues later, this layer can be connected to the source data intact. Subsequent data denoising, deduplication, outlier processing and other processes can be placed in the subsequent DW layer. It can also be based on the need for standardization and consistency of subsequent data processing. The source data can be cleaned first and then stored in the ODS layer. Data cleaning processing refers to the process of cleaning, converting and normalizing data. In the ODS layer, because data from different data sources may have different data formats, data types and incomplete data values, the data cleaning process includes checking the consistency, completeness and accuracy of the data, and cleaning and repairing it. In a specific implementation, the DW layer is a core layer in the created lake warehouse table. Therefore, the optimization of the DW layer plays an important role in improving the performance of the lake warehouse integrated data system. In this embodiment, the DW layer can be subdivided into a first sublayer and a second sublayer during the specific optimization design; wherein, the first sublayer is used to store the data obtained after basic operations and optimization operations on the data stored in the original data layer;

[0044] The operation includes the following steps: using the ETL data processing algorithm to clean the data stored in the original data layer, such as removing null values ​​and dirty data, and desensitizing the data table after the data cleaning. The optimization operation includes: distinguishing the data into large tables and small tables according to the preset data volume; when the data is a small table, reading all small tables into the hash table in the memory, and serializing the hash table in the memory into a hash table file after reading; uploading the hash table file to the cache, and the cache sends the file to the local disk; when the method function is called, reading the hash table from the cache to the memory, scanning the large table, finding the corresponding small table according to the key value of the large table, adding a random value before the key value and performing local aggregation with the small table, and then removing the random value and performing global aggregation, and the final result is used as the data obtained after the optimization operation. Among them, the second sublayer is used to store the data obtained after the table division operation based on the data stored in the first sublayer. The table division operation includes performing table division operation according to time and / or performing table division operation according to business subject.

[0045] The specific steps are as follows: collect heterogeneous energy data, including power industry data, other energy data, and external support data. Preprocess the data and use the FCM algorithm to perform fuzzy clustering on the processed data to extract feature keywords, including:

[0046] S1: Specify the number of cluster categories c (2≤c≤A), A is the number of processed data, ζ∈(0,∞) is used to specify the iteration threshold, initialize the cluster center β(0), and set the initial iteration index k=1;

[0047] S2: Calculate the fuzzy partition matrix of the kth iteration;

[0048] S3: Sort the data and obtain the permutation function so that the function satisfies: The D function represents the Euclidean distance; that is With x t The Euclidean distance between

[0049] S4: Calculation uniform weight, where It refers to the cluster center weight after the permutation operation;

[0050] S5: Yes Sort and take As the minimum parameter of the cluster center point in this fuzzy clustering, it is used for the calculation of subsequent steps;

[0051] S6: If ||β (k+1) -β (k) ||>ζ, then return to step S2 and increment k by 1 until ||β (k+1) -β (k) ||≤ζ or the iteration index k reaches 50, and the final extracted feature keywords are obtained.

[0052] Then the calculation formula of the fuzzy partitioning matrix described in step S2 is:

[0053]

[0054] Among them, D function represents the Euclidean distance, c is the number of cluster categories, A is the number of data after dimensionality reduction, m is the weighted index, m∈(1,∞); the cluster center weight after permutation operation for:

[0055]

[0056] Among them, A is the number of processed data, u ij is the fuzzy partition matrix membership, m is the weighted index, x j represents the jth data, and c is the number of clustering categories.

[0057] Through the provided lake-warehouse integrated heterogeneous energy data storage system, offline / real-time data storage and calculation, as well as SQL statement-based data query services can be provided. By integrating the data of the data lake and the data warehouse on a service platform, the data of the data lake and the data warehouse can be determined according to the needs of business development, which energy data should be placed in the data warehouse and which in the data lake, thereby forming lake-warehouse integrated data. Commonly, the data lake can be based on Iceberg, and the underlying data of Iceberg can be stored in HDFS. The read.format method in Spark can be used to convert the image into binary data and store it on the HDFS file system, and the length, width, path, binary data, mode and other information of the image can be obtained to form an Iceberg table for storage. Add the prefix ODS to the Iceberg table name to realize the logical stratification of the data lake, store the data in the ODS original layer, and use the method Option("write.metadata.table-name","ODS_picture_01") to name the Iceberg table. The storage format of the Iceberg table on the HDFS file system is divided into metadata and data information.

[0058] Based on the Hive component, the source data in the HDFS component is loaded into the lake warehouse table created based on the Hive component. Specifically, the data storage layer of the lake warehouse table includes the original data layer, the data warehouse layer and the data service layer; this step is to perform data cleaning on the source data, and store the processed data in the original data layer. Perform basic operations and optimization operations on the data stored in the original data layer, and store the data obtained after the operation in the data warehouse layer. Hive is a data warehouse tool built on Hadoop, which provides Hive SQL language to analyze and process massive data stored in the HDFS file system. Iceberg provides Hive external table format storage, and Hive can also be used to integrate Iceberg data into Hive tables. This time, we choose to integrate Iceberg table data into Hive tables through Hive tools. First, create the Hive table ODS_picture_01. Similarly, add the ODS prefix to achieve logical stratification, store the data in the ODS original layer, and create the fields of the Hive table based on the fields of the Iceberg table that has been stored. The data field stores the binary data converted from the picture, so the field type should be Binary. Then, use the Row Format Serde, Stored as Inputrmat, Outputformat, and Location tools provided by Hive SQL to create the communication between the Hive table and iceberg. After the table is created, the Hive table integrates the Iceberg table data by inserting the partition data in the Iceberg table into the partition in the Hive table. The specific SQL format is ALTER TABLE TABLE_NAMEADD PARTITION(PARTITION_FIELD)LOCATION'PATH'. By converting unstructured data into binary data and storing it in the database, it is more convenient to integrate with structured data through SQL.

[0059] The data is divided into large tables and small tables according to the preset data volume; when the data is a small table, all small tables are read into the hash table in the memory, and the hash table in the memory is serialized into a hash table file after reading; the hash table file is uploaded to the cache, and the cache sends the file to the local disk. When the method function is called, the hash table is read from the cache to the memory, the large table is scanned, the corresponding small table is found according to the key value of the large table, a random value is added before the key value, and then the small table is locally aggregated, and then the random value is removed and then the global aggregation is performed, and the final result is used as the data obtained after the optimization operation. The data stored in the data warehouse layer is processed and converted for business purposes, and the obtained data is stored in the data service layer. Among them, the table partitioning operation includes table partitioning according to time, and / or table partitioning according to business subject, that is, creating a partition table with time or business subject as the partition field, which can cut the data into smaller fragments, and only need to access specific partitions of the data when querying, thereby improving the query efficiency of the data. At the same time, creating a partition table by time or subject can also make data management more convenient, and improve concurrency and availability. In addition, you can also convert the binary data in the Hive database into images by overriding the Start method in JavaFX, obtain the connection through the getConnection method, and then use the getBinaryStream method to obtain the binary data in the database and put it into the Image class to construct the image, which is convenient for image display.

[0060] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

[0061] Embodiment 2, the second embodiment of the present invention, is different from the previous embodiment in that:

[0062] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program codes.

[0063] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0064] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0065] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0066] Although the preferred embodiments of the present application have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.

[0067] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A heterogeneous energy data storage method based on lake-warehouse integration, characterized in that: include, Collecting heterogeneous energy data and preprocessing the data; The processed data is subjected to fuzzy clustering using the FCM algorithm to extract characteristic keywords; Based on the Hive component, the source data in the HDFS component is loaded into the lake warehouse table created based on the Hive component.

2. A method for storing heterogeneous energy data based on lake-warehouse integration as claimed in claim 1, characterized in that: The heterogeneous energy data includes power industry data, other energy data, and external support data.

3. A method for storing heterogeneous energy data based on lake-warehouse integration as claimed in claim 2, characterized in that: The preprocessing of the data comprises: Specify the number of cluster categories c (2≤c≤A), A is the number of processed data, ζ∈(0,∞) is used to specify the iteration threshold, initialize the cluster center β(0), and set the initial iteration index k=1.

4. A method for storing heterogeneous energy data based on lake-warehouse integration as claimed in claim 3, characterized in that: The preprocessing of the data further comprises: Compute the fuzzy partitioning matrix for the kth iteration.

5. A method for storing heterogeneous energy data based on lake-storage integration as claimed in claim 4, characterized in that: The method of performing fuzzy clustering on the processed data using the FCM algorithm to extract characteristic keywords includes sorting the data and obtaining a permutation function so that the function satisfies: The D function represents the Euclidean distance, that is that is With x t The Euclidean distance between calculate uniform weight, where It refers to the cluster center weight after the permutation operation; right Sort and take As the minimum parameter of the cluster center point in this fuzzy clustering, it is used for the calculation of subsequent steps; If ||β (k+1) -β (k) ||>ζ, then return to the previous step and increment k by 1 until ||β (k+1) -β (k) ||≤ζ or the iteration index k reaches 50, and the final extracted feature keywords are obtained.

6. A method for storing heterogeneous energy data based on lake-storage integration as claimed in claim 5, characterized in that: The method of using the FCM algorithm to perform fuzzy clustering on the processed data to extract characteristic keywords also includes optimizing the fuzzy partition matrix membership u ij The calculation formula is: Among them, D function represents the Euclidean distance, c is the number of cluster categories, A is the number of data after dimensionality reduction, m is the weighted index, m∈(1,∞0; the weight of the cluster center after the permutation operation for: Among them, A is the number of processed data, u ij is the fuzzy partition matrix membership, m is the weighted index, x j represents the jth data, and c is the number of clustering categories.

7. A method for storing heterogeneous energy data based on lake-storage integration as claimed in claim 6, characterized in that: The source data is loaded into the lake warehouse table created based on the Hive component, and the data storage layer of the lake warehouse table includes the original data layer, the data warehouse layer and the data service layer. The source data is cleaned and the processed data is stored in the original data layer. At the same time, basic operations and optimization operations are performed on the data stored in the original data layer, and the data obtained after the operation is stored in the data warehouse layer.

8. A method for storing heterogeneous energy data based on lake-storage integration as claimed in claim 7, characterized in that: The source data is loaded into the lake warehouse table created based on the Hive component, including: Hive is a data warehouse tool built on Hadoop, providing Hive SQL language to analyze and process data stored in the HDFS file system, and Iceberg provides Hive external table format storage to integrate Iceberg data into the Hive table through Hive.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 8 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Images