Batch stream integrated data processing method for energy big data

Abstract

The invention discloses a batch flow integrated data processing method for energy big data. The method comprises the following steps: performing energy big data acquisition and energy big data storage; the method comprises the following steps: uniformly accessing metadata of Kafka and Iceberg by using Flink Catalog, respectively creating a database space for the Kafka and Iceberg, and carrying out energy big data processing; and executing the energy big data analysis application. The invention aims to provide a batch-stream integrated data processing method for energy big data, and solves the problems that in the prior art, stream processing and batch processing of the energy big data cannot be effectively combined, the data processing efficiency is low, the expansibility and compatibility are insufficient, and the technical advancement is insufficient.

Description

Technical Field

[0001] The present invention relates to the technical field of batch-stream integrated data processing of energy big data, and in particular to a batch-stream integrated data processing method for energy big data. Background Art

[0002] (1) Paper: "Design and Research of a Stream-Batch Integrated Data Processing Platform Based on Stateful Real-Time Streams" discusses the design of the core architecture of stream-batch integrated data, proposes a stream-batch integrated data processing method based on stateful real-time streams, and implements the processing and calculation of stream-batch integrated data in a platform-based manner.

[0003] (2) Patent: The method proposed in "A stream-batch integrated data processing method, device, equipment and storage medium" includes capturing the real-time change data of the source database in real time through distributed stream processing, and forwarding the real-time change data to the target database; importing the offline data of the source database into the data operation layer of the data warehouse through incremental update or full update; connecting the target database and the data operation layer by constructing an external table of the target database; merging the real-time change data in the target database and the offline data of the data operation layer to obtain the total data.

[0004] (3) Patent: "A method for constructing a data lake service platform based on stream-batch integration" proposes a method for constructing a data lake service platform based on stream-batch integration, including: using Flink CDC to uniformly connect data sources to the underlying layer of data operations; building a data lake service platform and a data bus to access the operation task, and uniformly connecting data sources to the data lake; the data lake service platform stores the data in a distributed file and metadata uniformly, and uses a unified data lake table format for the accessed data; using Flink as a stream-batch integrated computing engine, building a gateway for a unified data lake service platform based on Flink SQL GateWay, and uniformly using Flink SQL as a development language for stream-batch data processing, so as to realize stream-batch processing of the accessed data according to business timeliness; the operation monitoring center determines whether the indicators of the metrics data have reached the preset alarm threshold, and sends an alarm notification to the relevant administrator when it reaches the threshold.

[0005] (4) Patent: "A data processing method, system and readable storage medium for energy big data" The method includes: obtaining energy big data information, preprocessing the energy big data to obtain energy type sub-data, classifying the preset database according to energy type to obtain an energy type sub-database, and sending the energy type sub-database to the corresponding energy type sub-database for storage. By classifying, arranging and categorizing the energy big data, the energy big data is stored in a regular manner.

[0006] In summary: As a batch-stream integrated data architecture that combines the advantages of stream processing and batch processing and can provide more flexible and efficient data processing capabilities, it has not been widely used in the field of energy big data processing; data is stored in data lakes or data warehouses, and lake-warehouse integrated storage is not achieved, which affects data processing efficiency; data processing tools such as Flink, Iceberg and StarRocks have not been widely integrated in the batch-stream integrated architecture for use. Summary of the invention

[0007] In view of the above existing problems, the present invention is proposed.

[0008] Therefore, the present invention provides a batch-stream integrated data processing method for energy big data, which solves the problems in the prior art that stream processing and batch processing of energy big data cannot be effectively combined, data processing efficiency is low, scalability and compatibility are insufficient, and technology is not advanced enough.

[0009] In order to solve the above technical problems, the present invention provides the following technical solutions, a batch-stream integrated data processing method for energy big data, comprising:

[0010] Carry out energy big data collection and energy big data storage;

[0011] Use Flink Catalog to uniformly access the metadata of Kafka and Iceberg, create a database space for Kafka and Iceberg respectively, and process energy big data;

[0012] Execute energy big data analysis applications.

[0013] As an optimal solution of the batch-stream integrated data processing method for energy big data described in the present invention, the energy big data comes from the internal and external business systems of specific energy enterprises, including metering data, scheduling data, Internet of Things data, meteorological data, and economic data; the energy big data is divided into structured data, time series data, external data, and unstructured data according to type.

[0014] As a preferred solution of the batch-stream integrated data processing method for energy big data described in the present invention, the energy big data collection includes: offline and real-time data use a unified data collection method Flink CDC, push the real-time captured data to the message queue, batch processing and stream processing read data from the message queue, and load it into the data storage layer.

[0015] As a preferred solution of the batch-stream integrated data processing method for energy big data described in the present invention, the energy big data storage includes: the structured data is uniformly collected through Flink CDC, the data is written to the Iceberg data lake, and at the same time, according to the needs of the scenario, it is synchronously pushed to Kafka for temporary storage;

[0016] Time series data and external data are collected and written into Kafka. The stream processing tool Flink is used to collect data into the Iceberg data lake, and Flume is used to write Kafka hotspot time series data into the time series library simultaneously.

[0017] Unstructured data is written to object storage through the service interface, and part or all of the data is written to the data lake as needed.

[0018] As a preferred solution of the batch-stream integrated data processing method for energy big data described in the present invention, the energy big data storage includes uniformly storing structured, time-series, external, and unstructured data into an Iceberg cluster, while utilizing the underlying homology capabilities of Hadoop and Flink for unified data processing to form a data warehouse in the Iceberg cluster.

[0019] As a preferred solution of the batch-stream integrated data processing method for energy big data described in the present invention, the energy big data processing includes: using Flink Catalog to uniformly access the metadata of Kafka and Iceberg, creating a database space for Kafka and Iceberg respectively, and using the database name to distinguish real-time data and offline data when writing tasks;

[0020] Use Flink Catalog to build data lineage relationships for data sets, helping developers analyze upstream and downstream data sets and all corresponding read and write jobs.

[0021] Flink SQL uses SQL as the interface language for processing data, converts SQL statements into data flow graphs, and then executes them through the Flink engine, simplifying the data processing process.

[0022] As a preferred solution of the batch-stream integrated data processing method for energy big data described in the present invention, the energy big data analysis application includes writing multi-source energy big data into a real-time data warehouse based on StarRocks through real-time access, and using a data warehouse hierarchical modeling method to aggregate, process and summarize the data.

[0023] As a preferred solution of the batch-stream integrated data processing method for energy big data described in the present invention, the energy big data analysis application includes querying the data stored in the Iceberg data lake through the External Catalog provided by StarRocks without data migration;

[0024] By using materialized views for data modeling within StarRocks, data links are simplified, timeliness is improved, and operation and maintenance work is simplified, supporting data analysis applications, including real-time data analysis, data interactive query, data exploration and mining.

[0025] A computer device includes a memory and a processor, wherein the memory stores a computer program, and is characterized in that when the processor executes the computer program, steps of a batch-stream integrated data processing method for energy big data are implemented.

[0026] A computer-readable storage medium having a computer program stored thereon, characterized in that when the computer program is executed by a processor, the steps of a batch-stream integrated data processing method for energy big data are implemented.

[0027] Beneficial effects of the invention: The purpose of the invention is to provide a batch-stream integrated data processing method for energy big data, so as to solve the problems in the prior art that stream processing and batch processing of energy big data cannot be effectively combined, data processing efficiency is low, scalability and compatibility are insufficient, and technology is not advanced enough.

[0028] (1) Efficient data processing capability: The batch-stream integrated framework proposed in the present invention can receive and process data streams in real time, meeting application scenarios with high real-time requirements. It can also efficiently process historical data and batch data, reducing data processing time and significantly improving data processing efficiency.

[0029] (2) High scalability and compatibility: The batch-stream integrated data processing method proposed in the present invention can support the concurrent execution of large-scale data tasks. With the development of business and the growth of data volume, the batch-stream integrated framework can be easily expanded and upgraded to meet the ever-changing data processing needs.

[0030] (3) Use of relatively advanced technologies and tools: The batch-stream integrated data processing method proposed in the present invention uses technical components and tools such as Flink, Iceberg and StarRocks to ensure the advanced advantages of the method. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0032] Figure 1 A flowchart of a batch-stream integrated data processing method for energy big data is provided for one embodiment of the present invention. DETAILED DESCRIPTION

[0033] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the drawings of the specification. Obviously, the described embodiments are part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary persons in the art without creative work should fall within the scope of protection of the present invention.

[0034] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0035] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The term "in one embodiment" that appears in different places in this specification does not necessarily refer to the same embodiment, nor does it refer to a separate or selective embodiment that is mutually exclusive with other embodiments.

[0036] The present invention is described in detail with reference to schematic diagrams. When describing the embodiments of the present invention, for the sake of convenience, the cross-sectional diagrams showing the device structure will not be partially enlarged according to the general scale, and the schematic diagrams are only examples, which should not limit the scope of protection of the present invention. In addition, in actual production, the three-dimensional dimensions of length, width and depth should be included.

[0037] At the same time, in the description of the present invention, it should be noted that the directions or positional relationships indicated by the terms "upper, lower, inner and outer" are based on the directions or positional relationships shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific direction, be constructed and operated in a specific direction, and therefore cannot be understood as limiting the present invention. In addition, the terms "first, second or third" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance.

[0038] In the present invention, unless otherwise clearly specified and limited, the terms "install, connect, connect" should be understood in a broad sense, for example: it can be a fixed connection, a detachable connection or an integral connection; it can also be a mechanical connection, an electrical connection or a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0039] Example 1, reference Figure 1 , which is the first embodiment of the present invention, and provides a batch-stream integrated data processing method for energy big data, comprising:

[0040] S1: Collect energy big data and store energy big data.

[0041] S2: Use Flink Catalog to uniformly access the metadata of Kafka and Iceberg, create a database space for Kafka and Iceberg respectively, and process energy big data.

[0042] S3: Execute energy big data analysis applications.

[0043] The energy big data comes from the internal and external business systems of specific energy enterprises, including metering data, dispatching data, Internet of Things data, meteorological data, and economic data;

[0044] The energy big data is divided into structured data, time series data, external data, and unstructured data according to type.

[0045] The energy big data collection includes using the unified data collection method Flink CDC for offline and real-time data, pushing the real-time captured data to the message queue, and batch processing and stream processing reading data from the message queue and loading it into the data storage layer.

[0046] The energy big data storage includes: structured data is collected uniformly through Flink CDC, the data is written to the Iceberg data lake, and according to the needs of the scenario, it is synchronously pushed to Kafka for temporary storage;

[0047] Time series data and external data are collected and written into Kafka. The stream processing tool Flink is used to collect data into the Iceberg data lake, and Flume is used to write Kafka hotspot time series data into the time series library simultaneously.

[0048] Unstructured data is written to object storage through the service interface, and part or all of the data is written to the data lake as needed.

[0049] The energy big data storage includes uniformly storing structured, time-series, external, and unstructured data into the Iceberg cluster, while utilizing the underlying homology capabilities of Hadoop and Flink for unified data processing to form a data warehouse in the Iceberg cluster.

[0050] The energy big data processing includes: using Flink Catalog to uniformly access the metadata of Kafka and Iceberg, creating a database space for Kafka and Iceberg respectively, and using the database name to distinguish real-time data and offline data when writing tasks;

[0051] Use Flink Catalog to build data lineage relationships for data sets, helping developers analyze upstream and downstream data sets and all corresponding read and write jobs.

[0052] Flink SQL uses SQL as the interface language for processing data, converts SQL statements into data flow graphs, and then executes them through the Flink engine, simplifying the data processing process.

[0053] The energy big data analysis application includes writing multi-source energy big data into a real-time data warehouse based on StarRocks through real-time access, and using a data warehouse hierarchical modeling method to aggregate, process and summarize the data.

[0054] The energy big data analysis application includes querying data stored in the Iceberg data lake through the External Catalog provided by StarRocks without the need for data migration;

[0055] By using materialized views for data modeling within StarRocks, data links are simplified, timeliness is improved, and operation and maintenance work is simplified, supporting data analysis applications, including real-time data analysis, data interactive query, data exploration and mining.

[0056] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

[0057] Embodiment 2, the second embodiment of the present invention, is different from the previous embodiment in that:

[0058] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program codes.

[0059] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0060] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0061] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0062] Although the preferred embodiments of the present application have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.

[0063] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A batch-stream integrated data processing method for energy big data, characterized by: include, Carry out energy big data collection and energy big data storage; Use Flink Catalog to uniformly access the metadata of Kafka and Iceberg, create a database space for Kafka and Iceberg respectively, and process energy big data; Execute energy big data analysis applications.

2. The batch-stream integrated data processing method for energy big data according to claim 1, characterized in that: The energy big data comes from the internal and external business systems of specific energy enterprises, including metering data, dispatching data, Internet of Things data, meteorological data, and economic data; The energy big data is divided into structured data, time series data, external data, and unstructured data according to type.

3. A batch-stream integrated data processing method for energy big data as claimed in claim 2, characterized in that: The energy big data collection includes using the unified data collection method Flink CDC for offline and real-time data, pushing the real-time captured data to the message queue, and batch processing and stream processing reading data from the message queue and loading it into the data storage layer.

4. The batch-stream integrated data processing method for energy big data according to claim 3 is characterized by: The energy big data storage includes: structured data is collected uniformly through Flink CDC, the data is written to the Iceberg data lake, and according to the needs of the scenario, it is synchronously pushed to Kafka for temporary storage; Time series data and external data are collected and written into Kafka. The stream processing tool Flink is used to collect data into the Iceberg data lake, and Flume is used to write Kafka hotspot time series data into the time series library simultaneously. Unstructured data is written to object storage through the service interface, and part or all of the data is written to the data lake as needed.

5. The batch-stream integrated data processing method for energy big data according to claim 4, characterized in that: The energy big data storage includes uniformly storing structured, time-series, external, and unstructured data into the Iceberg cluster, while utilizing the underlying homology capabilities of Hadoop and Flink for unified data processing to form a data warehouse in the Iceberg cluster.

6. A batch-stream integrated data processing method for energy big data as claimed in claim 5, characterized in that: The energy big data processing includes: using Flink Catalog to uniformly access the metadata of Kafka and Iceberg, creating a database space for Kafka and Iceberg respectively, and using the database name to distinguish real-time data and offline data when writing tasks; Use Flink Catalog to build data lineage relationships for data sets, helping developers analyze upstream and downstream data sets and all corresponding read and write jobs. Flink SQL uses SQL as the interface language for processing data, converts SQL statements into data flow graphs, and then executes them through the Flink engine, simplifying the data processing process.

7. A batch-stream integrated data processing method for energy big data according to claim 6, characterized in that: The energy big data analysis application includes writing multi-source energy big data into a real-time data warehouse based on StarRocks through real-time access, and using a data warehouse hierarchical modeling method to aggregate, process and summarize the data.

8. The batch-stream integrated data processing method for energy big data according to claim 7, characterized in that: The energy big data analysis application includes querying data stored in the Iceberg data lake through the ExternalCatalog provided by StarRocks without the need for data migration; By using materialized views for data modeling within StarRocks, data links are simplified, timeliness is improved, and operation and maintenance work is simplified, supporting data analysis applications, including real-time data analysis, data interactive query, data exploration and mining.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 8 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Images