Lake entering method for multi-source heterogeneous energy data

Abstract

The method comprises the following steps: acquiring multi-source heterogeneous data from internal and external systems of an energy enterprise, and carrying out lake-entering data preprocessing; multi-source heterogeneous data enters a lake, Apache Flink serves as a unified data entering engine, the lake entering modes of all types of data are different, and data entering the lake is carried out through different lake entering controllers; and executing lake data archiving, deleting and migrating. According to the method, the Flink + Iceberg technical architecture is used for realizing real-time access of multi-source heterogeneous energy data, data access from different types of data sources is supported, diversification and comprehensiveness of the data sources are ensured, multiple data formats are adapted, and the requirements of real-time performance and flexibility of energy business processing are met.

Description

Technical Field

[0001] The present invention relates to the technical field of multi-source heterogeneous energy data entering a lake, and in particular to a method for entering multi-source heterogeneous energy data into a lake. Background Art

[0002] (1) Patent: The data entry method proposed in "A Hudi Data Ingestion Method and System for Multi-Source Heterogeneous Data" includes: publishing data entry tasks: uploading relevant information of the lake task and publishing the lake entry task; executing data entry tasks: using Apache Hudi as the data lake architecture, transferring the original data from the data source to the data lake for storage through the data entry processing engine, and collecting metadata information of the original data at the same time, and persistently storing the metadata information of the original data in the metadata database; building a data resource directory: building a data resource directory of the data lake based on the collected metadata and related information of the metadata in the data lake.

[0003] (2) Patent: "A method for importing multi-source heterogeneous data into a data lake" discloses a method for importing multi-source heterogeneous data into a data lake, comprising: obtaining the access interface address of external file-type data, importing the file-type data and saving it to the distributed file system of the local data lake server; or obtaining the access interface information of the external data source, connecting it to the local data lake server, importing the data of the external data source and saving it to the distributed file system in the form of a data file, or converting the non-relational data of the external data source into relational data and saving it to a relational database, or directly importing the relational data of the external data source and saving it to the relational database, or importing the non-relational data of the external data source and saving it to a document database.

[0004] (3) Patent: "Real-time data entry into the lake method, device, electronic device and storage medium" The method described includes: using a parsing tool to obtain CDC event files in the database log, and writing the CDC event files into a message queue to obtain CDC log data, using a log processing application to subscribe to the CDC log data to obtain real-time queue data; structuring the real-time queue data to obtain structured data, and using a blank decision tree to determine the SQL language rules corresponding to the structured data; using the SQL language rules to convert the structured data into SQL statements, and determining the interface type corresponding to the SQL statement through a neural network model; calculating the API interface corresponding to the maximum match between the interface type and the preset data lake API interface as the best transmission interface, and transmitting the SQL statement to the interactive application layer of the preset data lake through the API interface.

[0005] In summary: the existing method of entering multi-source heterogeneous data into the data lake is difficult to meet the needs of energy business processing in terms of real-time and flexibility. Summary of the invention

[0006] In view of the above existing problems, the present invention is proposed.

[0007] Therefore, the present invention provides a method for entering multi-source heterogeneous energy data into a data lake to solve the problem that the existing data entry method cannot meet the real-time and flexibility of energy business processing.

[0008] In order to solve the above technical problems, the present invention provides the following technical solution, a method for entering multi-source heterogeneous energy data into a lake, comprising:

[0009] Acquire multi-source heterogeneous data from internal and external systems of energy enterprises and pre-process the data before entering the lake;

[0010] Apache Flink is used as a unified data entry engine for multi-source heterogeneous data entering the lake. Different types of data are entered into the lake in different ways through different entry controllers.

[0011] Perform lake data archiving, deletion, and migration.

[0012] As a preferred solution for the method of entering the multi-source heterogeneous energy data into the lake described in the present invention, the multi-source heterogeneous data includes data on the entire chain of energy production, transmission, storage, consumption and energy market transactions.

[0013] As a preferred solution of the method for entering the multi-source heterogeneous energy data into the lake described in the present invention, the preprocessing of the data entering the lake includes using SeaTunnel to provide data cleaning, conversion and formatting tools to perform unified data management.

[0014] As a preferred solution of the method for entering the lake of multi-source heterogeneous energy data described in the present invention, the preprocessing of the data entering the lake also includes defining the preprocessing operator using the Spark SQL function, and writing the preprocessing operator based on the schema of the Kafka entry into the lake; the processing logic of each preprocessing operator is regarded as a SQL subquery, the operators are executed in the order of definition, and the subsequent operators write SQL functions based on the output fields of the previous operators.

[0015] As a preferred solution of the method for entering the lake of multi-source heterogeneous energy data described in the present invention, the preprocessing of the data entering the lake also includes standardizing the data before the data enters the lake, and standardizing the data entering the lake through custom rules.

[0016] As a preferred solution of the method for entering multi-source heterogeneous energy data into a lake according to the present invention, the multi-source heterogeneous data entering a lake includes: using Apache Flink as a unified data entering the lake processing engine, the gateway uniformly manages and distributes data entering the lake tasks, collects corresponding metadata according to data types, and executes corresponding data entering the lake tasks according to data classification;

[0017] The unified data entry processing engine is used to transfer the original data from the data source to the data lake for storage. Meanwhile, the metadata information of the original data is collected and stored in the metadata database.

[0018] Iceberg's data metadata management engine enables management of massive multi-source heterogeneous energy data while supporting dynamic partitioning.

[0019] As a preferred solution of the method for entering multi-source heterogeneous energy data into the lake described in the present invention, the multi-source heterogeneous data entering the lake also includes that Flink and Iceberg introduce transaction mechanism and fault tolerance mechanism, and automatically roll back and retry if an exception occurs during the synchronization process;

[0020] The way to store semi-structured data in the data lake includes data format conversion and mapping relationship establishment; data format conversion is to parse and convert semi-structured data into structured data, and store the converted data in the data lake, using HDFS as the underlying storage;

[0021] The mapping relationship is established by creating a mapping relationship between the original file of the semi-structured data and the corresponding data table, so as to realize the entry of the semi-structured data into the lake;

[0022] Among them, the unstructured data is entered into the lake according to the basic feature metadata, file analysis content and original file;

[0023] The basic feature metadata is entered into the lake by integrating the basic information of the document itself from the source end. The data content is still stored in the source system, and only the basic feature metadata of unstructured data is stored in the data lake.

[0024] The file parsing content is entered into the lake by performing text parsing and splitting the file content of the data source. The original file is still stored in the source system, and only the parsed content enhancement metadata is stored in the data lake.

[0025] The original file entry into the lake moves the original file from the data source end into the lake, directly stores the original file in the data lake and performs full life cycle management.

[0026] As a preferred solution for the method of entering the lake for multi-source heterogeneous energy data described in the present invention, the execution of lake data archiving, deletion, and migration includes: using metadata management and intelligent classification technology to automatically mark the data entering the lake, clarify the archived data, archiving time, and storage period after archiving, and dynamically adjust the storage level of the data according to the timeliness, importance, and frequency of use of the energy data. For data that is about to reach the archiving conditions, the archiving process is automatically triggered to migrate the data from hot storage to lower-cost cold storage or archival storage.

[0027] A computer device includes a memory and a processor, wherein the memory stores a computer program, and is characterized in that when the processor executes the computer program, it implements the steps of a method for entering multi-source heterogeneous energy data into a lake.

[0028] A computer-readable storage medium having a computer program stored thereon, characterized in that when the computer program is executed by a processor, the steps of a method for entering multi-source heterogeneous energy data into a lake are implemented.

[0029] Beneficial effects of the present invention: The present invention uses the Flink+Iceberg technical architecture to realize real-time entry of multi-source heterogeneous energy data into the lake, supports data access from different types of data sources, ensures the diversity and comprehensiveness of data sources, adapts to multiple data formats, and meets the real-time and flexibility requirements of energy business processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0031] Figure 1 A flowchart of a method for entering multi-source heterogeneous energy data into a lake is provided for an embodiment of the present invention. DETAILED DESCRIPTION

[0032] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the drawings of the specification. Obviously, the described embodiments are part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary persons in the art without creative work should fall within the scope of protection of the present invention.

[0033] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0034] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The term "in one embodiment" that appears in different places in this specification does not necessarily refer to the same embodiment, nor does it refer to a separate or selective embodiment that is mutually exclusive with other embodiments.

[0035] The present invention is described in detail with reference to schematic diagrams. When describing the embodiments of the present invention, for the sake of convenience, the cross-sectional diagrams showing the device structure will not be partially enlarged according to the general scale, and the schematic diagrams are only examples, which should not limit the scope of protection of the present invention. In addition, in actual production, the three-dimensional dimensions of length, width and depth should be included.

[0036] At the same time, in the description of the present invention, it should be noted that the directions or positional relationships indicated by the terms "upper, lower, inner and outer" are based on the directions or positional relationships shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific direction, be constructed and operated in a specific direction, and therefore cannot be understood as limiting the present invention. In addition, the terms "first, second or third" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance.

[0037] In the present invention, unless otherwise clearly specified and limited, the terms "install, connect, connect" should be understood in a broad sense, for example: it can be a fixed connection, a detachable connection or an integral connection; it can also be a mechanical connection, an electrical connection or a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0038] Example 1, reference Figure 1 , which is the first embodiment of the present invention, and provides a method for entering multi-source heterogeneous energy data into a lake, including:

[0039] S1: Acquire multi-source heterogeneous data from internal and external systems of energy enterprises.

[0040] Furthermore, the internal and external systems of energy enterprises involve metering systems, dispatching systems, meteorological systems, economic systems, etc., including data on all aspects and chains of energy production, transmission, storage, consumption and energy market transactions.

[0041] Furthermore, the data entering the lake is divided into three categories according to the organization form and characteristics of the data: structured data, semi-structured data and unstructured data.

[0042] S2: Preprocessing of data entering the lake, performing necessary data cleaning and conversion before the data enters the lake.

[0043] Furthermore, data preprocessing uses SeaTunnel to provide flexible data cleaning, conversion, and formatting tools to ensure the high quality of input data and promote unified data management.

[0044] S3: Multi-source heterogeneous data enters the lake, with Apache Flink as the unified data entry engine. Each type of data enters the lake in a different way, and is entered through different entry controllers.

[0045] Furthermore, database data such as MySQL and Oracle are structured data, and the structured inflow controller is responsible for data entry into the lake; JSON data and log files are semi-structured data, and the semi-structured inflow controller is responsible for data entry into the lake; image, audio, video and other data are unstructured data, and the unstructured inflow controller is responsible for data entry into the lake.

[0046] Optionally, structured data can be captured in real time through tools such as Flink CDC and OGG, and stored in Kafka. The binlog logs can be parsed in real time to generate add, delete, and modify commands to operate the data records in the data lake, so that data can be entered into the lake in real time, with a timeliness of seconds or minutes.

[0047] Optionally, semi-structured data is entered into the lake through data format conversion and mapping relationship establishment.

[0048] Optionally, unstructured data can be entered into the lake in different ways, such as basic feature metadata, file parsed content, and original files, as needed.

[0049] S4: Archiving, deleting, and migrating lake data. The archiving, deletion, and migration operations are used to ensure the standardization of data entering the lake, including the following:

[0050] Use advanced metadata management and intelligent classification technology to automatically tag data and clearly identify archived data, archiving time, and storage period after archiving.

[0051] Dynamically adjust the data storage level based on factors such as the timeliness, importance, and frequency of use of energy data. Automatically trigger the archiving process for data that is about to reach archiving conditions, and migrate the data from hot storage to lower-cost cold storage or archival storage. During the archiving process, use efficient data migration tools and compression algorithms to reduce data volume and storage costs.

[0052] Through high-performance indexing and query technology, archived data can be quickly retrieved and called in the data lake, providing support for enterprises to analyze historical trends and assist in decision-making.

[0053] In step S1, multi-source heterogeneous energy data from internal and external systems of energy enterprises are obtained, and data access from different types of data sources (relational databases, NoSQL databases, file systems, message queues, APIs, etc.) is supported to ensure the diversity and comprehensiveness of data sources, adapt to multiple data formats (CSV, JSON, Parquet, Avro, etc.), and meet the needs of entering the lake in different data formats.

[0055] In step S2, before the data enters the lake, SeaTunnel's data cleaning function is used to preprocess the data, including converting data types, deleting duplicate rows, filling missing values, removing outliers, etc. By defining appropriate data cleaning rules, the quality and accuracy of the data entering the lake are ensured.

[0056] Furthermore, data preprocessing uses Spark SQL functions to define preprocessing operators, and preprocessing operators are written based on the schema of Kafka entering the lake; the processing logic of each preprocessing operator is regarded as a SQL subquery, and the operators are executed in the order of definition. The subsequent operators write SQL functions based on the output fields of the previous operators.

[0057] Furthermore, before the data enters the lake, it is standardized and processed through custom rules, such as unifying the date format and converting numerical units, to ensure that similar data from different data sources have consistent formats and units.

[0058] In the S3 step, Apache Flink is used as the unified data entry processing engine. The gateway uniformly manages and distributes data entry tasks, collects metadata based on data types, and executes adapted data entry tasks based on data classification.

[0059] Furthermore, the original data is transferred from the data source to the data lake for storage through a unified data entry processing engine, and the metadata information of the original data is collected and persistently stored in the metadata database.

[0060] Furthermore, through Iceberg's data metadata management engine, efficient management of massive multi-source heterogeneous energy data can be achieved, dynamic partitioning can be supported and data fragmentation can be effectively reduced, thus improving the efficiency and manageability of data entry into the lake.

[0061] Optionally, structured data can be imported into the lake using Flink CDC. Flink CDC captures the changed data in the database by monitoring the binlog or wal log of the database. The data is output in the form of incremental streams to ensure the real-time nature of the data. Flink receives the CDC data stream and uses its powerful stream processing capabilities to perform real-time conversion and aggregation of data. Flink's processing process includes steps such as data reception, conversion, aggregation, and output. During the real-time reading and writing process, Flink writes the processed data to the Iceberg data lake. The write operation uses a distributed transaction to ensure data consistency and reliability.

[0062] Furthermore, Flink and Iceberg ensure the integrity and reliability of data synchronization by introducing transaction mechanisms and fault-tolerant mechanisms. Even if an exception occurs during the synchronization process, they can automatically roll back and retry to ensure that data is not lost.

[0063] Optionally, the method of entering semi-structured data into the lake includes data format conversion and mapping relationship establishment;

[0064] Furthermore, data format conversion is to parse and convert semi-structured data into structured data, and store the converted data in the data lake, using HDFS as the underlying storage, to facilitate users to query and analyze data;

[0065] Furthermore, the mapping relationship is established by creating a mapping relationship between the original file of the semi-structured data and the corresponding data table, so as to realize the entry of the semi-structured data into the lake.

[0066] Optionally, unstructured data can be entered into the lake in different ways, such as by using basic feature metadata, file parsing content, and original files.

[0067] Furthermore, basic feature metadata is entered into the lake through the basic information of the document itself integrated from the source end. The data content is still stored in the source system, and only the basic feature metadata of unstructured data is stored in the data lake.

[0068] Furthermore, the file parsing content is entered into the lake by performing text parsing and splitting on the file content of the data source. The original file is still stored in the source system, and only the parsed content enhancement metadata is stored in the data lake.

[0069] Furthermore, the original files are moved into the lake from the data source, and the original files are directly stored in the data lake and managed throughout their life cycle.

[0070] In step S4, metadata management and intelligent classification technology are used to automatically tag data entering the lake, clarify the archived data, archiving time and storage period after archiving, and dynamically adjust the storage level of the data based on factors such as the timeliness, importance and frequency of use of energy data. For data that is about to reach archiving conditions, the archiving process is automatically triggered to migrate the data from hot storage to lower-cost cold storage or archive storage.

[0071] Furthermore, during the archiving process, efficient data migration tools and compression algorithms are used to reduce data volume and storage costs. Through high-performance indexing and query technology, archived data can be quickly retrieved and called in the data lake, providing support for users to conduct data analysis and assist in decision-making.

[0072] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

[0073] Embodiment 2, the second embodiment of the present invention, is different from the previous embodiment in that:

[0074] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program codes.

[0075] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0076] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0077] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0078] Although the preferred embodiments of the present application have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.

[0079] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A method for entering multi-source heterogeneous energy data into a lake, characterized by: include, Acquire multi-source heterogeneous data from internal and external systems of energy enterprises and pre-process the data before entering the lake; Apache Flink is used as a unified data entry engine for multi-source heterogeneous data entering the lake. Different types of data are entered into the lake in different ways through different entry controllers. Perform lake data archiving, deletion, and migration.

2. The method for entering multi-source heterogeneous energy data into a lake as claimed in claim 1, characterized in that: The multi-source heterogeneous data include data on all aspects and chains of energy production, transmission, storage, consumption and energy market transactions.

3. The method for entering a multi-source heterogeneous energy data lake as claimed in claim 2, characterized in that: The preprocessing of the data entering the lake includes using SeaTunnel to provide data cleaning, conversion and formatting tools to perform unified data management.

4. The method for entering a multi-source heterogeneous energy data lake as claimed in claim 3, characterized in that: The data preprocessing for entering the lake also includes defining preprocessing operators using Spark SQL functions and writing preprocessing operators based on the schema of Kafka entering the lake; The processing logic of each preprocessing operator is regarded as a SQL subquery. Operators are executed in the order of definition, and subsequent operators write SQL functions based on the output fields of the previous operators.

5. The method for entering a multi-source heterogeneous energy data lake as claimed in claim 4, characterized in that: The pre-processing of the data entering the lake also includes standardizing the data before the data enters the lake, and standardizing the data entering the lake through custom rules.

6. A method for entering multi-source heterogeneous energy data into a lake as claimed in claim 5, characterized in that: The multi-source heterogeneous data entry into the lake includes using Apache Flink as a unified data entry processing engine, the gateway uniformly managing and distributing data entry tasks, collecting metadata according to data types, and executing corresponding data entry tasks according to data classification; The unified data entry processing engine is used to transfer the original data from the data source to the data lake for storage. Meanwhile, the metadata information of the original data is collected and stored in the metadata database. Iceberg's data metadata management engine enables management of massive multi-source heterogeneous energy data while supporting dynamic partitioning.

7. The method for entering a multi-source heterogeneous energy data lake according to claim 6, characterized in that: The multi-source heterogeneous data entering the lake also includes that Flink and Iceberg introduce transaction mechanisms and fault tolerance mechanisms. If an exception occurs during the synchronization process, it will automatically roll back and retry; The way to store semi-structured data in the data lake includes data format conversion and mapping relationship establishment; data format conversion is to parse and convert semi-structured data into structured data, and store the converted data in the data lake, using HDFS as the underlying storage; The mapping relationship is established by creating a mapping relationship between the original file of the semi-structured data and the corresponding data table, so as to realize the entry of the semi-structured data into the lake; Among them, the unstructured data is entered into the lake according to the basic feature metadata, file analysis content and original file; The basic feature metadata is entered into the lake by integrating the basic information of the document itself from the source end. The data content is still stored in the source system, and only the basic feature metadata of unstructured data is stored in the data lake. The file parsing content is entered into the lake by performing text parsing and splitting the file content of the data source. The original file is still stored in the source system, and only the parsed content enhancement metadata is stored in the data lake. The original file entry into the lake moves the original file from the data source end into the lake, directly stores the original file in the data lake and performs full life cycle management.

8. The method for entering a multi-source heterogeneous energy data lake as claimed in claim 7, characterized in that: The execution of lake data archiving, deletion, and migration includes: Metadata management and intelligent classification technology are used to automatically tag data entering the lake, clearly define the archived data, archiving time and storage period after archiving, and dynamically adjust the data storage level based on the timeliness, importance and frequency of use of energy data. For data that is about to reach archiving conditions, the archiving process is automatically triggered to migrate data from hot storage to lower-cost cold storage or archive storage.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 8 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Images