A multi-source heterogeneous energy data import method

Abstract

The application discloses a multi-source heterogeneous energy data lake entry method, which comprises the following steps: obtaining multi-source heterogeneous data from internal and external systems of an energy enterprise, and performing lake data preprocessing; multi-source heterogeneous data lake entry, using Apache Flink as a unified data lake entry engine, different lake entry modes for each type of data, and different lake entry controllers for data lake entry; and performing lake data archiving, deleting and migrating. The application realizes real-time lake entry of multi-source heterogeneous energy data by using a technical architecture of Flink+Iceberg, supports data access from different types of data sources, ensures the diversification and comprehensiveness of data sources, adapts to various data formats, and meets the real-time and flexibility requirements of energy business processing.

Description

Technical Field

[0001] This invention relates to the field of lake ingestion technology for multi-source heterogeneous energy data, and more particularly to a method for ingesting multi-source heterogeneous energy data into a lake. Background Technology

[0002] (1) Patent: The data ingestion method and system proposed in the patent "A Hudi data ingestion method and system for multi-source heterogeneous data" includes: publishing data ingestion task: uploading relevant information of the data ingestion task and publishing the data ingestion task; executing data ingestion task: using Apache Hudi as the data lake architecture, the raw data is transmitted from the data source to the data lake for storage through the data ingestion processing engine, while collecting the metadata information of the raw data and persistently storing the metadata information of the raw data in the metadata database; constructing data resource catalog: constructing the data resource catalog of the data lake based on the collected metadata and the relevant information of the metadata in the data lake.

[0003] (2) Patent: "A Method for Importing Multi-Source Heterogeneous Data into a Data Lake" discloses a method for importing multi-source heterogeneous data into a data lake, including: obtaining the access interface address of external file-type data, importing the file-type data and saving it to the distributed file system of the local data lake server; or obtaining the access interface information of external data sources, connecting it to the local data lake server, importing the data from the external data sources and saving it to the distributed file system in the form of data files, or converting the non-relational data from the external data sources into relational data and saving it to a relational database, or directly importing the relational data from the external data sources and saving it to a relational database, or importing the non-relational data from the external data sources and saving it to a document-type database.

[0004] (3) Patent: The method described in the "Real-time Data Ingestion Method, Apparatus, Electronic Device and Storage Medium" includes: using a parsing tool to obtain CDC event files in the database logs and writing the CDC event files into a message queue to obtain CDC log data; using a log processing application to subscribe to the CDC log data to obtain real-time queue data; performing structured processing on the real-time queue data to obtain structured data; using a blank decision tree to determine the SQL language rules corresponding to the structured data; using the SQL language rules to convert the structured data into SQL statements; using a neural network model to determine the interface type corresponding to the SQL statements; calculating the API interface with the maximum matching degree between the interface type and the preset data lake API interface as the best transmission interface; and transmitting the SQL statements to the interactive application layer of the preset data lake through the API interface.

[0005] In summary, existing methods for ingesting multi-source heterogeneous data into the lake are insufficient to meet the real-time and flexibility requirements of energy business processing. Summary of the Invention

[0006] In view of the aforementioned existing problems, the present invention is proposed.

[0007] Therefore, this invention provides a method for importing multi-source heterogeneous energy data into a lake to solve the problem that existing data import methods cannot meet the real-time and flexibility requirements of energy business processing.

[0008] To address the aforementioned technical problems, this invention provides the following technical solution: a method for importing multi-source heterogeneous energy data into a lake, comprising:

[0009] Acquire heterogeneous data from multiple sources and external systems of energy companies, and perform preprocessing of the data entering the lake;

[0010] Multi-source heterogeneous data is fed into the lake, with Apache Flink serving as the unified data feeding engine. Each type of data is fed into the lake in a different way, through different data feeding controllers.

[0011] Perform data archiving, deletion, and migration for the lake.

[0012] As a preferred embodiment of the method for importing multi-source heterogeneous energy data into a lake according to the present invention, the multi-source heterogeneous data includes data from the entire chain of energy production, transmission, storage, consumption, and energy market transactions.

[0013] As a preferred embodiment of the method for importing multi-source heterogeneous energy data into a lake according to the present invention, the preprocessing of the imported data includes using SeaTunnel to provide data cleaning, transformation and formatting tools for unified data management.

[0014] As a preferred embodiment of the method for importing multi-source heterogeneous energy data into a lake as described in this invention, the preprocessing of the imported data further includes defining preprocessing operators using Spark SQL functions and writing preprocessing operators based on the Kafka import schema; the processing logic of each preprocessing operator is regarded as a subquery of SQL, the operators are executed in the order of definition, and the subsequent operators are written SQL functions based on the output fields of the preceding operators.

[0015] As a preferred embodiment of the method for importing multi-source heterogeneous energy data into a lake according to the present invention, the preprocessing of the data into the lake further includes standardizing the data before it is imported into the lake by using custom rules.

[0016] As a preferred embodiment of the method for importing multi-source heterogeneous energy data into a lake as described in this invention, the multi-source heterogeneous data import into a lake includes: using Apache Flink as a unified data import processing engine, having a gateway to uniformly manage and distribute data import tasks, collecting corresponding metadata according to data type, and executing corresponding data import tasks according to data classification.

[0017] The unified data lake processing engine transmits raw data from the data source to the data lake for storage, while collecting metadata information of the raw data and storing the metadata information of the raw data in the metadata database.

[0018] Iceberg's data metadata management engine enables the management of massive amounts of multi-source heterogeneous energy data, while also supporting dynamic partitioning.

[0019] As a preferred embodiment of the method for importing multi-source heterogeneous energy data into a lake as described in this invention, the multi-source heterogeneous data import into the lake further includes Flink and Iceberg introducing transaction and fault tolerance mechanisms, so that if an anomaly occurs during the synchronization process, automatic rollback and retry will be performed.

[0020] The semi-structured data entry method includes data format conversion and mapping relationship establishment; data format conversion is to parse and convert semi-structured data into structured data, and store the converted data in the data lake, with HDFS as the underlying storage;

[0021] The mapping relationship is established by creating a mapping relationship between the original files of semi-structured data and the corresponding data tables, thereby enabling the semi-structured data to be imported into the lake.

[0022] Among them, unstructured data is entered into the lake according to three methods: basic feature metadata, file parsing content, and raw file.

[0023] The basic feature metadata is fed into the data lake by integrating the basic information of the documents themselves from the source end. The data content is still stored in the source system, and the data lake only stores the basic feature metadata of unstructured data.

[0024] The file parsing content is put into the data lake by parsing and splitting the file content of the data source into the lake. The original file is still stored in the source system, and the data lake only stores the parsed content to enhance metadata.

[0025] The process of importing original files into the data lake involves moving the original files from the data source into the data lake, storing the original files directly in the data lake, and managing them throughout their entire lifecycle.

[0026] As a preferred embodiment of the method for importing multi-source heterogeneous energy data into a lake as described in this invention, the execution of lake data archiving, deletion, and migration includes: using metadata management and intelligent classification technology to automatically mark the data imported into the lake, specifying the data to be archived, the archiving time, and the storage period after archiving; dynamically adjusting the storage level of the data based on the timeliness, importance, and frequency of use of the energy data; and automatically triggering the archiving process for data that is about to meet the archiving conditions, migrating the data from hot storage to lower-cost cold storage or archive storage.

[0027] A computer device includes a memory and a processor, the memory storing a computer program, characterized in that the processor executes the computer program to implement the steps of a method for importing multi-source heterogeneous energy data into a lake.

[0028] A computer-readable storage medium having a computer program stored thereon, characterized in that, when the computer program is executed by a processor, it implements the steps of a method for importing multi-source heterogeneous energy data into a lake.

[0029] The beneficial effects of this invention are as follows: This invention enables real-time import of multi-source heterogeneous energy data into the lake by using the Flink+Iceberg technical architecture, supports data access from different types of data sources, ensures the diversity and comprehensiveness of data sources, adapts to multiple data formats, and meets the real-time and flexibility requirements of energy business processing. Attached Figure Description

[0030] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0031] Figure 1 This is a schematic diagram of a method for importing multi-source heterogeneous energy data into a lake, provided as an embodiment of the present invention. Detailed Implementation

[0032] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the protection scope of the present invention.

[0033] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0034] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in different places in this specification does not necessarily refer to the same embodiment, nor is it a single or selective embodiment that is mutually exclusive with other embodiments.

[0035] This invention is described in detail with reference to the schematic diagrams. When detailing the embodiments of this invention, for ease of explanation, the cross-sectional views illustrating the device structure may be partially enlarged, not adhering to the usual scale. Furthermore, the schematic diagrams are merely examples and should not be construed as limiting the scope of protection of this invention. In actual fabrication, the three-dimensional spatial dimensions of length, width, and depth should be included.

[0036] Furthermore, in the description of this invention, it should be noted that the terms "upper," "lower," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. These terms are used solely for the convenience of describing the invention and for simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the invention. In addition, the terms "first," "second," or "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0037] Unless otherwise explicitly specified and limited, the terms "installation," "connection," and "joining" in this invention should be interpreted broadly. For example, they can refer to fixed connections, detachable connections, or integral connections; similarly, they can refer to mechanical connections, electrical connections, or direct connections, or indirect connections through an intermediate medium, or internal connections between two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0038] Example 1, referring to Figure 1 This is the first embodiment of the present invention, which provides a method for importing multi-source heterogeneous energy data into a lake, including:

[0039] S1: Acquire heterogeneous data from multiple sources within and outside the energy company's systems.

[0040] Furthermore, the internal and external systems of energy companies involve metering systems, dispatching systems, meteorological systems, economic systems, etc., including data from the entire chain of energy production, transmission, storage, consumption, and energy market transactions.

[0041] Furthermore, the data entering the lake is divided into three categories according to its organization and characteristics: structured data, semi-structured data, and unstructured data.

[0042] S2: Data preprocessing before entering the lake involves necessary data cleaning and transformation.

[0043] Furthermore, data preprocessing utilizes SeaTunnel's flexible data cleaning, transformation, and formatting tools to ensure high-quality input data and promote unified data management.

[0044] S3: Multi-source heterogeneous data ingestion, with Apache Flink as the unified data ingestion engine. Each type of data has a different ingestion method and is ingested through different ingestion controllers.

[0045] Furthermore, data from databases such as MySQL and Oracle are structured data, and the structured data entry controller is responsible for data entry into the lake; JSON data and log files are semi-structured data, and the semi-structured data entry controller is responsible for data entry into the lake; image, audio, and video data are unstructured data, and the unstructured data entry controller is responsible for data entry into the lake.

[0046] Optionally, structured data can be captured in real time from the database binlog using tools such as Flink CDC and OGG, stored in Kafka, and manipulated by generating CRUD commands to operate the data records in the data lake through real-time parsing of the binlog, achieving real-time data entry into the lake with a timeliness of seconds or minutes.

[0047] Optionally, semi-structured data can be incorporated into the lake through methods such as data format conversion and mapping relationship establishment.

[0048] Optionally, unstructured data can be imported into the lake in different ways, such as importing basic feature metadata, importing parsed file content, or importing the original file, as needed.

[0049] S4: Lake data archiving, deletion, and migration. Archiving, deletion, and migration operations ensure the standardization of data entering the lake, including the following:

[0050] By employing advanced metadata management and intelligent classification technologies, data is automatically tagged, clearly defining the archived data, the archiving time, and the storage period after archiving.

[0051] The storage tier of energy data is dynamically adjusted based on factors such as timeliness, importance, and frequency of use. For data nearing archiving status, the archiving process is automatically triggered, migrating the data from hot storage to lower-cost cold storage or archive storage. During the archiving process, efficient data migration tools and compression algorithms are employed to reduce data volume and lower storage costs.

[0052] High-performance indexing and querying technologies enable rapid retrieval and access to archived data within the data lake, supporting enterprises in analyzing historical trends and aiding decision-making. Detailed implementation method:

[0054] In step S1, multi-source heterogeneous energy data from internal and external systems of energy enterprises is acquired. It supports access to data from different types of data sources (relational databases, NoSQL databases, file systems, message queues, APIs, etc.), ensuring the diversity and comprehensiveness of data sources and adapting to various data formats (CSV, JSON, Parquet, Avro, etc.) to meet the needs of different data formats for entering the lake.

[0055] In step S2, before the data is fed into the lake, SeaTunnel's data cleaning function is used to preprocess the data, including converting data types, deleting duplicate rows, filling missing values, and removing outliers. By defining appropriate data cleaning rules, the quality and accuracy of the data fed into the lake are ensured.

[0056] Furthermore, data preprocessing uses Spark SQL functions to define preprocessing operators, which are written based on the schema of the Kafka lake. The processing logic of each preprocessing operator is regarded as a subquery of SQL. The operators are executed in the order they are defined, and the SQL functions of the later operators are written based on the output fields of the earlier operators.

[0057] Furthermore, before the data is fed into the lake, it is standardized by using custom rules, such as standardizing date formats and converting numerical units, to ensure that similar data from different data sources have consistent formats and units.

[0058] In step S3, Apache Flink serves as the unified data inbound processing engine. The gateway manages and distributes data inbound tasks, collects appropriate metadata based on data type, and executes appropriate data inbound tasks based on data classification.

[0059] Furthermore, the raw data is transferred from the data source to the data lake for storage through a unified data lake processing engine. At the same time, the metadata information of the raw data is collected and persistently stored in the metadata database.

[0060] Furthermore, Iceberg's data metadata management engine enables efficient management of massive amounts of multi-source heterogeneous energy data, supports dynamic partitioning, effectively reduces data fragmentation, and improves the efficiency and manageability of data entering the lake.

[0061] Optionally, structured data is fed into the Iceberg data lake using Flink CDC. Flink CDC captures changes in the database by monitoring the database's binlog or WAL log. This data is output as an incremental stream, ensuring real-time performance. Flink receives the CDC data stream and leverages its powerful stream processing capabilities to perform real-time transformation and aggregation. Flink's processing includes data reception, transformation, aggregation, and output. During real-time read / write operations, Flink writes the processed data to the Iceberg data lake. Write operations employ distributed transactions to ensure data consistency and reliability.

[0062] Furthermore, Flink and Iceberg introduce transaction and fault tolerance mechanisms to ensure the integrity and reliability of data synchronization. Even if anomalies occur during the synchronization process, they can automatically roll back and retry to ensure that data is not lost.

[0063] Optionally, the methods for importing semi-structured data into the lake include data format conversion and the establishment of mapping relationships;

[0064] Furthermore, data format conversion involves parsing and converting semi-structured data into structured data, storing the converted data in a data lake, using HDFS as the underlying storage, to facilitate user querying and analysis of data;

[0065] Furthermore, the mapping relationship is established by creating a mapping relationship between the original files of the semi-structured data and the corresponding data tables, thereby enabling the semi-structured data to be imported into the lake.

[0066] Optionally, unstructured data can be imported into the lake in different ways, such as by importing basic feature metadata, importing parsed file content, or importing the original file.

[0067] Furthermore, basic feature metadata is fed into the data lake by integrating basic information from the documents themselves from the source system. The data content is still stored in the source system, and the data lake only stores the basic feature metadata of unstructured data.

[0068] Furthermore, the data lake process involves parsing and splitting the file content from the data source before storing it in the lake. The original file remains stored in the source system, while the data lake only stores the parsed content to enhance metadata.

[0069] Furthermore, the original files are moved into the data lake from the data source, where they are stored directly and managed throughout their entire lifecycle.

[0070] In step S4, metadata management and intelligent classification technologies are used to automatically label the data entering the lake, clarify the data to be archived, the archiving time, and the storage period after archiving. Based on factors such as the timeliness, importance, and frequency of use of energy data, the storage level of the data is dynamically adjusted. For data that is about to meet the archiving conditions, the archiving process is automatically triggered to migrate the data from hot storage to lower-cost cold storage or archive storage.

[0071] Furthermore, during the archiving process, efficient data migration tools and compression algorithms are employed to reduce data volume and lower storage costs. High-performance indexing and querying technologies enable archived data to be quickly retrieved and accessed within the data lake, providing support for users to conduct data analysis and assist in decision-making.

[0072] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

[0073] Example 2, the second embodiment of the present invention, differs from the previous embodiment in that:

[0074] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0075] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0076] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0077] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0078] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0079] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A method for importing multi-source heterogeneous energy data into a lake, characterized in that: include, Acquire heterogeneous data from multiple sources and external systems of energy companies, and perform preprocessing of the data entering the lake; Multi-source heterogeneous data is fed into the lake, with Apache Flink serving as the unified data feeding engine. Each type of data is fed into the lake in a different way, through different data feeding controllers. Perform data archiving, deletion, and migration for the lake; The multi-source heterogeneous data ingestion includes using Apache Flink as a unified data ingestion processing engine, with a gateway uniformly managing and distributing data ingestion tasks, collecting corresponding metadata according to data type, and executing corresponding data ingestion tasks according to data classification. The unified data lake processing engine transmits raw data from the data source to the data lake for storage, while collecting metadata information of the raw data and storing the metadata information of the raw data in the metadata database. Iceberg's data metadata management engine enables the management of massive amounts of multi-source heterogeneous energy data, while also supporting dynamic partitioning; The multi-source heterogeneous data ingestion also includes Flink and Iceberg introducing transaction and fault tolerance mechanisms, which automatically roll back and retry if an anomaly occurs during the synchronization process; The semi-structured data entry method includes data format conversion and mapping relationship establishment; data format conversion is to parse and convert semi-structured data into structured data, and store the converted data in the data lake, with HDFS as the underlying storage; The mapping relationship is established by creating a mapping relationship between the original files of semi-structured data and the corresponding data tables, thereby enabling the semi-structured data to be imported into the lake. Among them, unstructured data is entered into the lake according to three methods: basic feature metadata, file parsing content, and raw file. The basic feature metadata is fed into the data lake by integrating the basic information of the documents themselves from the source end. The data content is still stored in the source system, and the data lake only stores the basic feature metadata of unstructured data. The file parsing content is put into the data lake by parsing and splitting the file content of the data source into the lake. The original file is still stored in the source system, and the data lake only stores the parsed content to enhance metadata. The process of importing original files into the data lake involves moving the original files from the data source into the data lake, storing the original files directly in the data lake, and managing them throughout their entire lifecycle.

2. The method for importing multi-source heterogeneous energy data into a lake as described in claim 1, characterized in that: The multi-source heterogeneous data includes data from the entire chain of energy production, transmission, storage, consumption, and energy market transactions.

3. The method for importing multi-source heterogeneous energy data into a lake as described in claim 2, characterized in that: The preprocessing of the data entering the lake includes using SeaTunnel to provide data cleaning, transformation, and formatting tools for unified data management.

4. The method for importing multi-source heterogeneous energy data into a lake as described in claim 3, characterized in that: The preprocessing of the data entering the lake also includes defining preprocessing operators using Spark SQL functions and writing preprocessing operators based on the schema of the Kafka data entering the lake; The processing logic of each preprocessing operator is regarded as a subquery of SQL. The operators are executed in the order they are defined, and the SQL functions of the subsequent operators are written based on the output fields of the previous operators.

5. The method for importing multi-source heterogeneous energy data into a lake as described in claim 4, characterized in that: The preprocessing of the data entering the lake also includes standardizing the data before it enters the lake by using custom rules.

6. The method for importing multi-source heterogeneous energy data into a lake as described in claim 5, characterized in that: The execution of lake data archiving, deletion, and migration includes, Using metadata management and intelligent classification technologies, data entering the lake is automatically labeled, and the data to be archived, the archiving time, and the storage period after archiving are clearly defined. Based on the timeliness, importance, and frequency of use of energy data, the storage level of the data is dynamically adjusted. For data that is about to meet the archiving conditions, the archiving process is automatically triggered, and the data is migrated from hot storage to lower-cost cold storage or archive storage.

7. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images