A multi-source heterogeneous lake warehouse integrated data processing method, device and medium

Through the Flink cluster configuration and integrated flow batch acquisition technology of data warehouses and data lakes, integrated lake and warehouse management is realized, solving the problem of difficulty in integrating data warehouses and data lakes, improving data real-time and utilization, and adapting to more Multiple data acquisition scenarios.

CN116166757BActive Publication Date: 2024-11-15INSPUR GENERSOFT CO LTD
View PDF 7 Cites 0 Cited by

Patent Information

Application Number
CN202211553242.4
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-12-06
Publication Date
2024-11-15
Estimated Expiration
2042-12-06

AI Technical Summary

Technical Problem

The existing lake and warehouse integration technology is difficult to integrate data warehouses and data lakes, and each link is relatively independent, making it difficult to realize a true lake and warehouse integrated system.

Method used

By configuring the data warehouse and data lake in Flink cluster, combining the integrated flow batch collection technology, unified data management and real-time calculation are realized, data collection, lake entry and real-time processing are opened up, and traditional development is expanded based on the Hudi data lake storage base Storage capability of data warehouse.

Benefits of technology

It has realized the integration of Hujian, lowered the threshold for data collection and management, improved data utilization, enhanced real-time data and massive processing capabilities, and became the focus of enterprise digital transformation.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN116166757B_ABST
    Figure CN116166757B_ABST
Patent Text Reader

Abstract

The present invention discloses a multi-source heterogeneous lake-warehouse integrated data processing method, equipment and medium, which belongs to the field of data integration management technology, and is used to solve the technical problem that the existing lake-warehouse integration technology cannot well integrate the data warehouse and the data lake, and each link is relatively independent, making it difficult to truly realize the lake-warehouse integrated system. The method includes: configuring the data warehouse and the data lake with a Flink cluster to obtain a lake-warehouse integrated system; according to the lake-warehouse integrated system, classifying and configuring the collection of business data to obtain batch collection data and real-time collection data; according to the Flink cluster configured by the lake-warehouse integrated system, the lake-warehouse integrated system is referenced to the data source to obtain a reference relationship; according to the reference relationship, the batch collection data and the real-time collection data are calculated to obtain the data calculation results; and the data calculation results are stored in the lake-warehouse integrated system.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of data integration management, and in particular to a multi-source heterogeneous lake warehouse integrated data processing method, equipment and medium. Background Art

[0002] The lake-warehouse integration is a new open architecture that connects the data warehouse and the data lake and combines the advantages of the two architectures. Its underlying layer supports the coexistence of multiple data types and enables mutual sharing of data. The lake-warehouse integration architecture enhances the ability to process real-time business and manage unstructured data. Its advantages are mainly reflected in four aspects: first, it has perfect data management capabilities, second, data can be traced in a timely manner, third, it supports a variety of computing engines, and fourth, it has higher data real-time performance.

[0003] From the perspective of data collection and data processing, traditional data warehouses generally use a T+1 data collection model. Later, with the development of Change Data Capture (CDC) technology, data changes in business systems can now enter data warehouses in near real time.

[0004] The existing lake-warehouse integration technology is still basically an integration of data warehouse and data lake, that is, the product provides independent functions of data warehouse and data lake, but has not achieved lake-warehouse integration. The lake and warehouse management are still independent systems, and the data cannot be truly integrated and unified. In data collection and real-time computing, most manufacturers use task scheduling in the form of Flink sql, which requires professional practitioners to customize and write tasks. It is difficult for general users to get started and cannot well achieve integrated management of data collection into the lake and real-time data processing. Summary of the invention

[0005] The embodiments of the present application provide a multi-source heterogeneous lake-warehouse integrated data processing method, device and medium, which are used to solve the following technical problems: the existing lake-warehouse integration technology cannot well integrate the data warehouse and the data lake, and each link is relatively independent, making it difficult to truly realize the lake-warehouse integrated system.

[0006] The present application embodiment adopts the following technical solutions:

[0007] On the one hand, an embodiment of the present application provides a multi-source heterogeneous lake-warehouse integrated data processing method, the method comprising: configuring a Flink cluster for a data warehouse and a data lake to obtain a lake-warehouse integrated system; according to the lake-warehouse integrated system, classifying and configuring the collection of business data to obtain batch collected data and real-time collected data; according to the Flink cluster configured by the lake-warehouse integrated system, referencing the data source of the lake-warehouse integrated system to obtain a reference relationship; according to the reference relationship, performing data calculation on the batch collected data and the real-time collected data to obtain data calculation results; and storing the data calculation results in the lake-warehouse integrated system.

[0008] The embodiment of the present application integrates the integrated flow and batch collection technology, lake-warehouse integrated system management, and connects the links of data collection, entering the lake, and real-time calculation, thus truly achieving lake-warehouse integration. And you only need to define the collection tasks according to your own business scenarios, and then perform lake-warehouse integrated management, and then integrate it into the traditional data warehouse management system based on the Hudi data lake storage base, which expands the storage capacity of traditional data warehouses and solves the problems of scattered and messy data in the data lake. After the data enters the lake, the generated Flink Sql and Flink Jar task definitions can be used to perform real-time processing and calculation on the data. With the help of massive, real-time, and multi-mode data processing capabilities, the lake-warehouse integrated system not only realizes the continuous release of the value of the full amount of data, but also becomes the focus of attention in the process of enterprise digital transformation, bringing new meaning and value to users.

[0009] In a feasible implementation, a Flink cluster is configured for a data warehouse and a data lake to obtain a lake-warehouse integrated system, specifically including: adding a Flink cluster to the data warehouse and the data lake through a Flink data processing engine to obtain first configuration information; adding a data service to the carte service in the ETL server to obtain second configuration information; wherein, the addition of the data service includes: a data streaming processing and batch processing core engine; the core engine includes: the Flink data processing engine and the IDI data processing engine; connecting the preset data source end to the data warehouse and the data lake at the source end; wherein, the data source end includes: a CDC data source and a Hudi data source; according to the first configuration information, the second configuration information and the data source end, cluster configuration is performed on the data warehouse and the data lake to obtain the configured lake-warehouse integrated system.

[0010] In a feasible implementation manner, the preset data source end is connected to the data warehouse and the data lake for source data connection, specifically including: determining the CDC data source and the Hudi data source according to the connection information of the destination end in the data warehouse and the data lake; wherein the destination end is used to collect data or calculate the source end of data in real time; the CDC data source supports mysql database, postgresql database, oracle database and sqlserver database; wherein the mysql database and postgresql database are the default support of the core engine, and the oracle database and sqlserver database are the extended support based on the preset extended core program; through several databases supported by the CDC data source, the CDC data source is connected to the data warehouse and the data lake for source data connection; according to the hive data warehouse tool with preset core parameters, the Hudi data source is connected to the data warehouse and the data lake for source data connection.

[0011] The embodiment of the present application integrates data lake and data warehouse technologies to expand the scope of data management and improve data utilization.

[0012] In a feasible implementation manner, according to the lake-warehouse integrated system, the business data is classified, configured and collected to obtain batch collected data, specifically including: running the configured ETL server according to the basic information in the business data; wherein the basic information at least includes: basic data information, data abbreviation and data code; identifying the data source and data table in the business data according to the data connection at the source end; identifying the lake warehouse storage location of the data warehouse and the data lake according to the connection information of the destination end in the lake-warehouse integrated system; creating a batch data table and a batch lake warehouse theme model according to the data source and data table through the lake warehouse storage location; wherein the batch data table is used to define the number of concurrent batch data tasks and the scheduling cycle; generating a batch collection task according to the batch data table; and after generating the batch collection task, the method also includes: starting the batch collection task according to the ETL server; sending the batch collection task to the corresponding IDI core engine according to the configuration of the scheduling cycle to collect data and obtain the batch collection data; and storing the batch collection data in the batch lake warehouse theme model.

[0013] In a feasible implementation, according to the lake-warehouse integrated system, business data is classified, configured and collected to obtain real-time collected data, specifically including: creating a real-time data table and a real-time lake-warehouse theme model according to the data source and data table through the lake-warehouse storage location; wherein the real-time data table is used to define the number of concurrent real-time data tasks and the abnormal automatic restart configuration; generating a real-time collection task according to the real-time data table; and after generating the real-time collection task, the method also includes: starting the real-time collection task according to the ETL server; sending the real-time collection task to the Flink data processing engine to collect and monitor the real-time collection task to obtain the real-time collection data; and storing the real-time collection data in the real-time lake-warehouse theme model; wherein the real-time lake-warehouse theme model and the batch lake-warehouse theme model are storage location models created under the connection theme corresponding to the Hudi data source in the lake-warehouse integrated system.

[0014] The embodiment of the present application lowers the threshold for data collection through wizard-style collection task definition and management, and integrates batch collection data and real-time collection data to adapt to more data collection scenarios.

[0015] In a feasible implementation, according to the Flink cluster configured by the lake-warehouse integration system, the data source of the lake-warehouse integration system is referenced to obtain a reference relationship, specifically including: creating a Flink information management table according to the Flink cluster configured by the lake-warehouse integration system; wherein the Flink information management table is used for unified entry management of the batch collected data and the real-time collected data; according to the Flink information management table, determining the type of connector; wherein the types of connectors include: jdbc and cdc; according to the type of connector, determining the type of data source and the corresponding data table; wherein the types of data sources include: CDC data source and Hudi data source; according to the type of data source and the corresponding data table, generating a reference relationship of the Flink information management table; wherein the reference relationship is used for referencing sql type jobs in the Flink cluster.

[0016] The embodiment of the present application simplifies the definition of real-time development tasks by providing a unified Flink information management table.

[0017] In a feasible implementation manner, according to the reference relationship, data calculation is performed on the batch collected data and the real-time collected data to obtain a data calculation result, specifically including: according to the basic information of the data in the data source end, running the configured Flink cluster; wherein the basic information at least includes: basic data information, data abbreviation and data code; the data source end includes: CDC data source and Hudi data source; the Flink information management table in the reference relationship is identified to obtain a definition data table that conforms to the sql type job in the Flink cluster; according to the definition data table, the preset Flink sql statement is performed data logic calculation to obtain a data calculation task; wherein the data logic includes: defining the number of task parallelism, TaskManager configuration, checkpoint configuration and abnormal automatic restart configuration; starting the data calculation task; sending the data calculation task to the Flink data processing engine to perform real-time data calculation on the data in the data source end to obtain the data calculation result.

[0018] The embodiments of the present application provide Flink Sql and Flink Jar task definitions, which facilitate scalable real-time data processing.

[0019] In a feasible implementation manner, after running the configured Flink cluster according to the basic information of the data in the data source, the method further includes: combining the predefined jar package program with the Flink jar statement in the Flink information management table for data logic calculation to obtain a data calculation task based on the jar package program.

[0020] In the second aspect, an embodiment of the present application also provides a multi-source heterogeneous lake-warehouse integrated data processing device, the device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, so that the at least one processor can execute a multi-source heterogeneous lake-warehouse integrated data processing method described in any of the above embodiments.

[0021] In the third aspect, an embodiment of the present application also provides a non-volatile computer storage medium, characterized in that the storage medium is a non-volatile computer-readable storage medium, and the non-volatile computer-readable storage medium stores at least one program, each of which includes instructions, and when the instructions are executed by the terminal, the terminal executes a multi-source heterogeneous lake warehouse integrated data processing method described in any of the above embodiments.

[0022] The embodiments of the present application provide a multi-source heterogeneous lake-warehouse integrated data processing method, equipment and medium, which integrates the integrated stream-batch collection technology, lake-warehouse integrated system management, and connects the data collection, lake entry, real-time calculation and other links, thus truly achieving lake-warehouse integration. And you only need to define the collection tasks according to your own business scenarios, and then perform lake-warehouse integrated management, and then integrate it into the traditional data warehouse management system based on the Hudi data lake storage base, which expands the storage capacity of traditional data warehouses and solves the problems of data lake data dispersion, disorder and insufficiency. After the data enters the lake, the generated Flink Sql and Flink Jar task definitions can be used to perform real-time processing and calculation of the data. With the help of massive, real-time, and multi-mode data processing capabilities, the lake-warehouse integrated system realizes the continuous release of the value of the full amount of data, and has also become the focus of attention in the process of enterprise digital transformation, bringing new meaning and value to users. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0024] Figure 1 A flow chart of a multi-source heterogeneous lake warehouse integrated data processing method provided in an embodiment of the present application;

[0025] Figure 2 An overall framework diagram of a lake-warehouse integrated system provided in an embodiment of the present application;

[0026] Figure 3 A data source configuration diagram provided in an embodiment of the present application;

[0027] Figure 4 A target end configuration diagram provided in an embodiment of the present application;

[0028] Figure 5 A collection task scheduling diagram provided in an embodiment of the present application;

[0029] Figure 6 A schematic diagram of a topic model configuration provided in an embodiment of the present application;

[0030] Figure 7 A schematic diagram of SQL parameter maintenance provided in an embodiment of the present application;

[0031] Figure 8 A schematic diagram of operating parameter maintenance provided in an embodiment of the present application;

[0032] Fig. 9 A schematic diagram of real-time monitoring of a Flink job task provided in an embodiment of the present application;

[0033] Fig.10 A structural schematic diagram of a multi-source heterogeneous lake-warehouse integrated data processing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0034] In order to enable those skilled in the art to better understand the technical solutions in this application, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are only part of the embodiments of this application, not all of them. Based on the embodiments of this specification, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this application.

[0035] The present application embodiment provides a multi-source heterogeneous lake warehouse integrated data processing method, such as Figure 1 As shown, the method specifically includes steps S101-S104:

[0036] S101. Configure Flink clusters for the data warehouse and data lake to obtain a data warehouse-lake integrated system.

[0037] Specifically, through the Flink data processing engine, the Flink cluster is added to the data warehouse and the data lake to obtain the first configuration information. The carte service in the ETL server is added as a data service to obtain the second configuration information. The addition of the data service includes: data stream processing and batch processing core engines. The core engines include: Flink data processing engine and IDI data processing engine.

[0038] Furthermore, the preset data source end is connected to the data warehouse and the data lake at the source end; wherein the data source end includes: CDC data source and Hudi data source.

[0039] According to the connection information of the destination in the data warehouse and the data lake, determine the CDC data source and the Hudi data source. Among them, the destination is the source end for collecting data or calculating data in real time. The CDC data source supports mysql database, postgresql database, oracle database and sqlserver database. Among them, mysql database and postgresql database are the default support of the core engine, and oracle database and sqlserver database are the extended support based on the preset extended core program. Through several databases supported by the CDC data source, the CDC data source is connected to the data warehouse and the data lake for the source end data. According to the hive data warehouse tool with preset core parameters, the Hudi data source is connected to the data warehouse and the data lake for the source end data.

[0040] Furthermore, according to the first configuration information, the second configuration information and the data source, the data warehouse and the data lake are clustered to obtain a configured lake-warehouse integrated system.

[0041] In one embodiment, Figure 2 An overall framework diagram of a lake-warehouse integrated system provided in an embodiment of the present application, Figure 2 As shown in the figure, through the CDC data source and Hudi data source on the data source side, the CDC data source supports MySQL database, PostgreSQL database, Oracle database and SQL Server database. The Hudi data source supports the hive data warehouse tool with preset core parameters. Among them, ODS is the operation data layer, DW is the data warehouse layer, and DM is the data mart layer.

[0042] Figure 3 A data source configuration diagram provided in an embodiment of the present application, such as Figure 3 as well as Figure 2 As shown in the figure, by adding Flink cluster configuration and ETL server settings, basic information is maintained and used to connect the flink cluster and carte service. These two services provide data streaming processing and batch processing core engines. The connection method is accessed through the web api provided by the engine. The configuration of the source end is as follows Figure 3 shown. Figure 4 A target end configuration diagram provided in an embodiment of the present application, such as Figure 4 As shown in the figure, the newly added CDC data source and Hudi data source maintain the connection information of the data source or data lake, which is used for the data source end or destination end of the collection task or real-time calculation. The configuration method is as follows Figure 4 shown.

[0043] S102. According to the lake-warehouse integrated system, the business data is classified and configured for collection to obtain batch collection data and real-time collection data.

[0044] Specifically, according to the basic information in the business data, run the configured ETL server. Among them, the basic information at least includes: basic data information, data abbreviation and data code. According to the data connection on the source end, identify the data source and data table in the business data. According to the connection information of the destination end in the lake-warehouse integrated system, identify the lake warehouse storage location of the data warehouse and the data lake. Furthermore, according to the data source and data table, create batch data tables and batch lake warehouse subject models through the lake warehouse storage location. Among them, the batch data table is used to define the number of concurrent batch data tasks and the scheduling cycle.

[0045] Further, a batch collection task is generated according to the batch data table. After the batch collection task is generated, the method further includes: starting the batch collection task according to the ETL server. According to the configuration of the scheduling cycle, the batch collection task is sent to the corresponding IDI core engine to collect data and obtain batch collection data. The batch collection data is stored in the batch lake warehouse theme model.

[0046] In one embodiment, Figure 5 A collection task scheduling diagram provided in an embodiment of the present application, such as Figure 5 As shown in the figure, by configuring the batch collection task, the data configuration is synchronized in the lake warehouse integrated system. Figure 6 A schematic diagram of a topic model configuration provided in an embodiment of the present application is shown in FIG. Figure 6 As shown, the configuration of the topic model includes abbreviation, code, database table, separator, storage format, storage location, etc.

[0047] After identifying the lake warehouse storage location of the data warehouse and the data lake, it is also possible to create a real-time data table and a real-time lake warehouse topic model based on the data source and data table and the lake warehouse storage location. The real-time data table is used to define the number of concurrent real-time data tasks and configure automatic restart in case of exceptions. Generate real-time collection tasks based on the real-time data table.

[0048] After generating the real-time collection task, the method further includes: starting the real-time collection task according to the ETL server. Sending the real-time collection task to the Flink data processing engine to collect and monitor the real-time collection task to obtain real-time collection data. And storing the real-time collection data in the real-time lake warehouse theme model. Among them, the real-time lake warehouse theme model and the batch lake warehouse theme model are storage location models created under the connection theme corresponding to the Hudi data source in the lake warehouse integrated system.

[0049] In one embodiment, after the batch collection task definition is generated, the system will automatically generate a batch data collection task. When the task is started, the task will be sent to the idi engine for data collection according to the configuration of the scheduling cycle. The collected data will be stored in the model of the lake warehouse theme. Fig. 9 A schematic diagram of real-time monitoring of a Flink job task provided in an embodiment of the present application is shown as follows: Fig. 9 The following figure shows the real-time monitoring method for Flink collection task jobs. After the real-time collection task definition is generated, the system will automatically generate a real-time data collection task. When the task is started, the system will send the task to the Flink engine to monitor and collect source data in real time. The collected data will be stored in the model of the lake warehouse theme.

[0050] S103. According to the Flink cluster configured in the lake-warehouse integrated system, the data source of the lake-warehouse integrated system is referenced to obtain a reference relationship.

[0051] Specifically, according to the Flink cluster configured in the lake warehouse integration system, a Flink information management table is created. The Flink information management table is used for unified entry management of batch collection data and real-time collection data.

[0052] Furthermore, the type of connector is determined according to the Flink information management table, wherein the types of connectors include: jdbc and cdc.

[0053] Furthermore, according to the type of the connector, the type of the data source and the corresponding data table are determined, wherein the types of the data source include: CDC data source and Hudi data source.

[0054] Furthermore, according to the type of data source and its corresponding data table, a reference relationship of the Flink information management table is generated, wherein the reference relationship is used for the reference of the SQL type job in the Flink cluster.

[0055] S104: Perform data calculation on the batch collected data and the real-time collected data according to the reference relationship to obtain data calculation results, and store the data calculation results in the lake warehouse integrated system.

[0056] Specifically, according to the basic information of the data in the data source, the configured Flink cluster is run. The basic information at least includes: basic data information, data abbreviation and data code. The data source includes: CDC data source and Hudi data source. The Flink information management table in the reference relationship is identified to obtain the definition data table that meets the sql type job in the Flink cluster.

[0057] Furthermore, according to the defined data table, the preset Flink SQL statement is used for data logic calculation to obtain the data calculation task. The data logic includes: defining the number of parallel tasks, TaskManager configuration, checkpoint configuration, and abnormal automatic restart configuration.

[0058] After running the configured Flink cluster according to the basic information of the data in the data source, you can also combine the predefined jar package program with the Flink jar statement in the Flink information management table for data logic calculation to obtain a data calculation task based on the jar package program.

[0059] Furthermore, a data computing task is started and sent to the Flink data processing engine to perform real-time data computing on the data in the data source to obtain data computing results.

[0060] In one embodiment, Figure 7 A schematic diagram of SQL parameter maintenance provided in an embodiment of the present application is provided. Figure 8 A schematic diagram of operating parameter maintenance provided in an embodiment of the present application is shown in FIG. Figure 7 as well as Figure 8 As shown in the figure, the real-time development task is defined by FlinkSql and Flink jar, and the defined data table includes: abbreviation, table name, SQL statement, etc., and the data logic includes: defining the number of task parallelism, TaskManager configuration, checkpoint configuration, and abnormal automatic restart configuration, etc.

[0061] For the definition data table of sql type jobs in the Flink cluster, the implementation process is as follows: define basic information, abbreviation, code, and running Flink cluster. Add the Flink information management table and select the definition data table of sql type jobs from the Flink information management table. Define the data logic calculation according to the preset Flink sql statement. Then define the number of task parallelism, TaskManager configuration, checkpoint, automatic restart of exceptions and other configurations. Through the above definition, the lake warehouse integration system will automatically form a data calculation task. When the task is started, the lake warehouse integration system will send the task to the Flink data processing engine to perform real-time data calculation on the data source end, and the data calculation results will be stored in the lake warehouse topic model of the corresponding lake warehouse integration system.

[0062] For the definition data table of jar type jobs in the Flink cluster, the implementation process is as follows: define basic information, abbreviation, code name, and running Flink cluster. Add a custom jar package program to combine data logic calculations with the Flinkjar statement in the Flink information management table. Then define the number of task parallelism, TaskManager configuration, checkpoint, automatic restart on exceptions and other configurations. Through the above definition, the lake warehouse integration system will automatically form a data calculation task. When the task is started, the lake warehouse integration system will send the task to the Flink engine to perform real-time data calculations on the data source end. The data calculation results will be stored in the lake warehouse topic model of the corresponding lake warehouse integration system.

[0063] In addition, the present application embodiment also provides a multi-source heterogeneous lake warehouse integrated data processing device, such as Fig.10 As shown, the multi-source heterogeneous lake-warehouse integrated data processing device 100 specifically includes:

[0064] At least one processor 101; and a memory 102 in communication with the at least one processor 101; wherein the memory 102 stores instructions executable by the at least one processor 101, so that the at least one processor 101 can execute:

[0065] Configure Flink clusters for the data warehouse and data lake to obtain a lake-warehouse integrated system;

[0066] According to the lake warehouse integrated system, the business data is classified and configured for collection to obtain batch collection data and real-time collection data;

[0067] According to the Flink cluster configured in the lake-warehouse integration system, reference the data source of the lake-warehouse integration system to obtain the reference relationship;

[0068] According to the reference relationship, data calculation is performed on batch collected data and real-time collected data to obtain data calculation results; and the data calculation results are stored in the lake warehouse integrated system.

[0069] This application integrates the integrated flow and batch collection technology, lake-warehouse integrated system management, and connects the data collection, lake entry, real-time calculation and other links, thus truly achieving lake-warehouse integration. And you only need to define the collection tasks according to your own business scenarios, and then perform lake-warehouse integrated management, and then integrate it into the traditional data warehouse management system based on the Hudi data lake storage base, which expands the storage capacity of traditional data warehouses and solves the problems of scattered and messy data in the data lake. After the data enters the lake, the generated Flink Sql and Flink Jar task definitions can be used to perform real-time processing and calculation on the data. With the help of massive, real-time, and multi-mode data processing capabilities, the lake-warehouse integrated system not only realizes the continuous release of the value of the full amount of data, but also becomes the focus of attention in the process of enterprise digital transformation, bringing new meaning and value to users.

[0070] Each embodiment in this application is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device, equipment, and non-volatile computer storage medium embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0071] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0072] For the convenience of description, the above device is described in various units according to their functions. Of course, when implementing this specification, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0073] Those skilled in the art will appreciate that the embodiments of this specification may be provided as methods, systems, or computer program products. Therefore, the embodiments of this specification may be in the form of complete hardware embodiments, complete software embodiments, or embodiments in combination with software and hardware. Moreover, the embodiments of this specification may be in the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0074] This specification is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of this specification. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0075] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0076] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0077] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0078] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0079] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0080] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0081] This specification may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. This specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media including storage devices.

[0082] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device, equipment, and non-volatile computer storage medium embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0083] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0084] The above description is only one or more embodiments of this specification and is not intended to limit this specification. For those skilled in the art, one or more embodiments of this specification may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of one or more embodiments of this specification shall be included in the scope of the claims of this specification.

Claims

1. A multi-source heterogeneous lake warehouse integrated data processing method, characterized in that: The method comprises: Perform Flink cluster configuration on the data warehouse and data lake to obtain a lake-warehouse integrated system, specifically including: adding the Flink cluster to the data warehouse and data lake through the Flink data processing engine to obtain the first configuration information, adding the carte service in the ETL server to the data service to obtain the second configuration information, wherein the addition of the data service includes: data streaming processing and batch processing core engines, the core engines include: the Flink data processing engine and the IDI data processing engine, connecting the preset data source end with the data warehouse and the data lake for source data, wherein the data source end includes: CDC data source and Hudi data source, according to the first configuration information, the second configuration information and the data source end, cluster configuration is performed on the data warehouse and the data lake to obtain the configured lake-warehouse integrated system; According to the lake-warehouse integrated system, the business data is classified and configured for collection to obtain batch collection data and real-time collection data; According to the Flink cluster configured in the lake-warehouse integrated system, reference the data source of the lake-warehouse integrated system to obtain a reference relationship; According to the reference relationship, data calculation is performed on the batch collected data and the real-time collected data to obtain data calculation results; and the data calculation results are stored in the lake warehouse integrated system, specifically including: according to the basic information of the data in the data source end, running the configured Flink cluster, wherein the basic information at least includes: basic data information, data abbreviation and data code, the data source end includes: CDC data source and Hudi data source, the Flink information management table in the reference relationship is identified, and a definition data table that conforms to the sql type job in the Flink cluster is obtained, according to the definition data table, the preset Flink sql statement is performed data logic calculation to obtain a data calculation task, wherein the data logic includes: defining the number of task parallelism, TaskManager configuration, checkpoint configuration and abnormal automatic restart configuration, starting the data calculation task, and sending the data calculation task to the Flink data processing engine to perform real-time data calculation on the data in the data source end to obtain the data calculation result.

2. According to claim 1, a multi-source heterogeneous lake warehouse integrated data processing method is characterized in that: Connecting the preset data source end with the data warehouse and the data lake to the source end data specifically includes: Determine the CDC data source and the Hudi data source according to the connection information of the destination end in the data warehouse and the data lake; wherein the destination end is used to collect data or calculate the source end of data in real time; The CDC data source supports mysql database, postgresql database, oracle database and sqlserver database; wherein the mysql database and postgresql database are the default support of the core engine, and the oracle database and sqlserver database are the extended support based on the preset extended core program; Through several databases supported by the CDC data source, the CDC data source is connected to the data warehouse and the data lake for source data; According to the hive data warehouse tool with preset core parameters, the Hudi data source is connected to the data warehouse and the data lake at the source end.

3. According to claim 1, a multi-source heterogeneous lake warehouse integrated data processing method is characterized in that: According to the lake-warehouse integrated system, the business data is classified and configured for collection to obtain batch collection data, specifically including: According to the basic information in the business data, the configured ETL server is run; wherein the basic information at least includes: basic data information, data abbreviation and data code; According to the data connection of the source end, identifying the data source and data table in the business data; According to the connection information of the destination end in the lake-warehouse integrated system, the lake-warehouse storage location of the data warehouse and the data lake is identified; According to the data source and the data table, a batch data table and a batch lake warehouse subject model are created through the lake warehouse storage location; wherein the batch data table is used to define the number of concurrent batch data tasks and the scheduling period; According to the batch data table, a batch collection task is generated; and after the batch collection task is generated, the method further includes: According to the ETL server, starting the batch collection task; According to the configuration of the scheduling cycle, the batch collection task is sent to the corresponding IDI core engine to collect data and obtain the batch collection data; and the batch collection data is stored in the batch lake warehouse theme model.

4. A multi-source heterogeneous lake-warehouse integrated data processing method according to claim 3, characterized in that: According to the lake-warehouse integrated system, the business data is classified and configured for collection to obtain real-time collection data, including: According to the data source and the data table, a real-time data table and a real-time lake warehouse subject model are created through the lake warehouse storage location; wherein the real-time data table is used to define the number of concurrent real-time data tasks and configure automatic restart in case of abnormalities; According to the real-time data table, a real-time collection task is generated; and after the real-time collection task is generated, the method further includes: According to the ETL server, starting the real-time collection task; Sending the real-time collection task to the Flink data processing engine to collect and monitor the real-time collection task to obtain the real-time collection data; and storing the real-time collection data in the real-time lake warehouse theme model; Among them, the real-time lake warehouse theme model and the batch lake warehouse theme model are storage location models created under the connection theme corresponding to the Hudi data source in the lake warehouse integrated system.

5. According to claim 1, a multi-source heterogeneous lake warehouse integrated data processing method is characterized in that: According to the Flink cluster configured in the lake-warehouse integrated system, the data source of the lake-warehouse integrated system is referenced to obtain a reference relationship, which specifically includes: According to the Flink cluster configured in the lake warehouse integrated system, a Flink information management table is created; wherein the Flink information management table is used for unified entry management of the batch collection data and the real-time collection data; According to the Flink information management table, determine the type of connector; wherein the type of connector includes: jdbc and cdc; According to the type of the connector, determine the type of the data source and the data table corresponding to it; wherein the type of the data source includes: CDC data source and Hudi data source; According to the type of the data source and its corresponding data table, a reference relationship of the Flink information management table is generated; wherein the reference relationship is used for referencing SQL type jobs in the Flink cluster.

6. The multi-source heterogeneous lake-warehouse integrated data processing method according to claim 1 is characterized in that: After running the configured Flink cluster according to the basic information of the data in the data source, the method further includes: The predefined jar package program is combined with the Flink jar statement in the Flink information management table for data logic calculation to obtain a data calculation task based on the jar package program.

7. A multi-source heterogeneous lake-warehouse integrated data processing device, characterized in that: The device comprises: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, so that the at least one processor can execute a multi-source heterogeneous lake-warehouse integrated data processing method according to any one of claims 1-6.

8. A non-volatile computer storage medium, characterized in that: The storage medium is a non-volatile computer-readable storage medium, which stores at least one program. Each of the programs includes instructions. When the instructions are executed by a terminal, the terminal executes a multi-source heterogeneous lake-warehouse integrated data processing method according to any one of claims 1-6.

Citation Information

Patent Citations

  • Data analysis system and data analysis method and device

    CN111241078A

  • Data processing method and device and data lake architecture

    CN112597218A

  • Factor data acquisition method and device, computer equipment and storage medium

    CN112883091A

  • Data processing method based on data medium table and related equipment thereof

    CN113836235A

  • Data processing method and device based on data lake, and readable storage medium

    CN115357654A