Lake and warehouse integrated data processing method, device, equipment, medium and program product

Abstract

The invention relates to a lake and warehouse integrated data processing method, device and equipment, a medium and a program product. The method comprises the following steps: collecting heterogeneous data from a data source in batches and / or in real time; cleaning is carried out according to the types of the heterogeneous data, and the cleaned data are stored in a constructed real-time lake bin; constructing an ETL (Extract-Transform-Load) link for the data in the real-time lake and warehouse, and processing the data through the ETL link to obtain processed data; wherein the ETL link is an extraction, conversion and loading link; and querying the processed data from the real-time lake cabin through a calculation engine, and providing corresponding data service. Therefore, the timeliness of data entering the lake can be improved as much as possible under the condition that the throughput rate is guaranteed, the stability of the whole processing process is high, the intelligence of ETL link processing is well improved, and the lake-warehouse integrated data management efficiency is remarkably improved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a lake-warehouse integrated data processing method, device, computer equipment, computer-readable storage medium and computer program product. Background Art

[0002] "Lake and warehouse integration" is a new architectural model that fully combines the advantages of data warehouse and data lake. By storing data on the low-cost storage architecture of the data lake, it can have the flexibility of data lake data format and inherit the governance capabilities of data warehouse data.

[0003] In traditional technology, in order to improve the governance capabilities of data lakes, a data warehouse layered design is usually adopted to uniformly collect heterogeneous data at the ODS layer (source layer); and integrate real-time data with offline data wide tables at the DWD layer (detail layer) to build a streaming ETL (Extract-Transform-Load) data processing link; finally, public indicators are formed at the DWS layer (aggregation layer) to expose data interfaces to provide services.

[0004] However, the above method consumes a lot of system resources, and each ETL process needs to be manually designed. When processing large-scale data, it is difficult to guarantee the data throughput rate. Summary of the invention

[0005] Based on this, it is necessary to provide a lake-warehouse integrated data processing method, device, computer equipment, computer-readable storage medium and computer program product that can realize large-scale data uploading to the cloud and into the lake, ensure data throughput, and do not require human intervention in response to the above technical problems.

[0006] In a first aspect, the present application provides a lake-warehouse integrated data processing method, comprising:

[0007] Ingest heterogeneous data from data sources in batch and / or real-time;

[0008] Clean the heterogeneous data according to their types, and store the cleaned data in the constructed real-time lake warehouse;

[0009] An ETL link is constructed for the data in the real-time lake warehouse, and the data is processed through the ETL link to obtain processed data; wherein the ETL link is an extraction, transformation and loading link;

[0010] The processed data is queried from the real-time lake warehouse through a computing engine, and corresponding data services are provided.

[0011] In one embodiment, the collecting heterogeneous data from the data source in batches and / or in real time includes:

[0012] If the data source is real-time data, the computing engine is used to collect heterogeneous data in real time;

[0013] If the data source is offline data, the computing engine collects heterogeneous data in batches;

[0014] The data sources include: relational database, non-relational database, file system, and message queue.

[0015] In one embodiment, the cleaning is performed according to the types of the heterogeneous data, including:

[0016] If the heterogeneous data is minute-level real-time data, or the heterogeneous data is daily-level quasi-real-time data obtained through crontab scheduled tasks, the similarity between two data points is determined by circular sampling, and data deduplication is performed based on the comparison result of the similarity with the preset threshold, and data with similarity greater than the preset threshold is removed to obtain cleaned data; wherein, crontab is a periodic execution process;

[0017] If the heterogeneous data is offline data, a preset detection method is used to find abnormal values ​​or outliers, and the data with abnormal values ​​or outliers is eliminated to obtain cleaned data; the preset detection method includes: any one of statistics, machine learning, and deep learning.

[0018] In one embodiment, the real-time lake warehouse includes: a flow layer, a detail layer, and a summary layer; wherein:

[0019] The flow layer is used to store the cleaned data into the real-time data lake and / or the offline data lake;

[0020] The detail layer is used to build an ETL link and complete data processing through the ETL link;

[0021] The aggregation layer is used to aggregate the data processed by the ETL link and expose the data interface, which is used to provide data services.

[0022] In one embodiment, the ETL link includes multiple models, and the model adopts an incremental learning method to perform real-time update of model parameters; wherein the multiple models are used to perform at least one of the following operations:

[0023] Extracting data from the real-time data lake and / or offline data lake of the real-time lake warehouse;

[0024] Performing logical conversion processing on the extracted data to obtain converted data, wherein the logical conversion processing includes at least one of: dimension table widening, window query, conditional filtering, multi-stream splicing, and index optimization;

[0025] The conversion data is processed and incremented to obtain processed data.

[0026] In one embodiment, querying the processed data from the real-time lake warehouse through the computing engine and providing corresponding data services include:

[0027] querying the processed data from the real-time lake warehouse through a computing engine, and analyzing the processed data to obtain an analysis result;

[0028] The analysis results are displayed on a user-visualized front-end interface through a data interface, and corresponding data services are provided on the user-visualized front-end interface, wherein the data services include at least one of data analysis, data mining, content recommendation, and data visualization.

[0029] In a second aspect, the present application also provides a lake-warehouse integrated data processing device, including:

[0030] A data collection module, used to collect heterogeneous data from data sources in batches and / or in real time;

[0031] A data cleaning and storage module, used to clean the heterogeneous data according to their types and store the cleaned data into the constructed real-time lake warehouse;

[0032] A data processing module is used to construct an ETL link for the data in the real-time lake warehouse, and process the data through the ETL link to obtain processed data; wherein the ETL link is an extraction, transformation and loading link;

[0033] The data service module is used to query the processed data from the real-time lake warehouse through the computing engine and provide corresponding data services.

[0034] In a third aspect, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0035] Ingest heterogeneous data from data sources in batch and / or real-time;

[0036] Clean the heterogeneous data according to their types, and store the cleaned data in the constructed real-time lake warehouse;

[0037] An ETL link is constructed for the data in the real-time lake warehouse, and the data is processed through the ETL link to obtain processed data; wherein the ETL link is an extraction, transformation and loading link;

[0038] The processed data is queried from the real-time lake warehouse through a computing engine, and corresponding data services are provided.

[0039] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the following steps are implemented:

[0040] Ingest heterogeneous data from data sources in batch and / or real-time;

[0041] Clean the heterogeneous data according to their types, and store the cleaned data in the constructed real-time lake warehouse;

[0042] An ETL link is constructed for the data in the real-time lake warehouse, and the data is processed through the ETL link to obtain processed data; wherein the ETL link is an extraction, transformation and loading link;

[0043] The processed data is queried from the real-time lake warehouse through a computing engine, and corresponding data services are provided.

[0044] In a fifth aspect, the present application further provides a computer program product, including a computer program, which implements the following steps when executed by a processor:

[0045] Ingest heterogeneous data from data sources in batch and / or real-time;

[0046] Clean the heterogeneous data according to their types, and store the cleaned data in the constructed real-time lake warehouse;

[0047] An ETL link is constructed for the data in the real-time lake warehouse, and the data is processed through the ETL link to obtain processed data; wherein the ETL link is an extraction, transformation and loading link;

[0048] The processed data is queried from the real-time lake warehouse through a computing engine, and corresponding data services are provided.

[0049] The above-mentioned lake-warehouse integrated data processing method, device, computer equipment, computer-readable storage medium and computer program product collect heterogeneous data in batches and / or in real time from the data source; thus, large-scale heterogeneous data can be processed to improve processing efficiency. The heterogeneous data is cleaned according to the type of the data, and the cleaned data is stored in the constructed real-time lake warehouse; thus, redundant and abnormal data can be intelligently removed, the efficiency of subsequent data processing can be improved, and the throughput rate can be guaranteed. An ETL link is constructed for the data in the real-time lake warehouse, and the data is processed through the ETL link to obtain the processed data; wherein the ETL link is an extraction, transformation and loading link; thus, the governance efficiency of the data lake can be improved. The processed data is queried from the real-time lake warehouse through the computing engine, and corresponding data services are provided. Therefore, while ensuring the throughput rate, the timeliness of data entering the lake can be improved as much as possible. The entire processing process has high stability, which greatly improves the intelligence of the ETL link processing, so that the data governance efficiency of the lake warehouse integration is significantly improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0051] Figure 1 This is an application environment diagram of a lake-warehouse integrated data processing method in one embodiment;

[0052] Figure 2 1. It is a flow chart of a lake-warehouse integrated data processing method in one embodiment;

[0053] Figure 3 It is a flowchart of a lake-warehouse integrated data processing method in another embodiment;

[0054] Figure 4 1 is a structural block diagram of a lake-warehouse integrated data processing device in one embodiment;

[0055] Figure 5 It is a structural block diagram of a lake-warehouse integrated data processing system in one embodiment;

[0056] Figure 6 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0057] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0058] The lake-warehouse integrated data processing method provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown. Among them, the unified entrance of the upper layer of data application is used to provide interface services, and the computing engine (such as Spark, Flink, Trino, AI+ML) is equipped with a unified resource scheduling Yarn / K85 platform to query and analyze data from the constructed real-time lake warehouse, and can feedback the results of data processing to the real-time lake warehouse, which can be used to optimize the processing flow of the real-time lake warehouse (such as the model contained in the ETL link). The process of collecting data from the data source and entering the lake warehouse is managed as a unified metadata Catalog. Among them, the data source may include: Kafka, HDFS, Hive, MySQL, etc. For different types of data in different data sources, data collection can be carried out in batches or in real time, and the collected data is imported into the constructed real-time lake warehouse. The real-time lake warehouse may include: source layer (Hudi), detail layer (Hudi), and summary layer (Hudi), and the data flow between each layer adopts the form of stream batch integration. In addition, in order to increase the speed of data processing, a data lake acceleration module (such as Alluxio, JuiceFS) can also be added to the real-time lake warehouse. For example, the intelligence of lake-warehouse integrated data processing can also be improved in terms of data storage, basic services, etc. Data storage methods can include: HDFS2, HDFS3, object storage, cloud storage; basic services can include: security verification (Kerberos), tenant management (LDAP), and alarm monitoring (Prometheus).

[0059] The embodiment shown in the above application environment includes the complete process of the lake-warehouse integrated data processing method, namely, firstly, heterogeneous data is collected in batches and / or in real time from the data source; the heterogeneous data are cleaned according to the type of the heterogeneous data, and the cleaned data are stored in the constructed real-time lake warehouse; an ETL link is constructed for the data in the real-time lake warehouse, and the data is processed through the ETL link to obtain the processed data; wherein the ETL link is an extraction, transformation and loading link; the processed data is queried from the real-time lake warehouse through the computing engine, and corresponding data services are provided.

[0060] This embodiment combines machine learning technology to design an intelligent lake-warehouse full-link solution for multi-dimensional heterogeneous data in a batch-stream integrated manner, which can include steps such as data collection into the lake, data cleaning and integration, intelligent ETL construction and optimization, data landing and application, etc. This can achieve large-scale data migration to the cloud and lake, and improve the efficiency of data lake governance.

[0061] In an exemplary embodiment, Figure 2 As shown, a lake-warehouse integrated data processing method is provided, including the following steps 201 to 204.

[0062] Step 201 : Collect heterogeneous data from data sources in batches and / or in real time.

[0063] In this embodiment, the collection method supports the industry-wide common data collection methods such as JDBC collection, HTTP collection, FTP collection, MQ collection, and Web Service collection. The collected data is heterogeneous data and can be divided into the following three categories:

[0064] The first category: real-time log data at the minute level, which contains a large number of duplicate or redundant logs.

[0065] The second category: daily quasi-real-time data obtained through crontab scheduled tasks. This type of data contains a large number of duplicate or redundant logs.

[0066] The third category: offline data of MySQL, Oracle, SQL Server, etc. obtained by establishing a source database connection and synchronizing it through Flink CDC technology.

[0067] Exemplarily, if the data of the data source is real-time data, the heterogeneous data is collected in real time through the computing engine.

[0068] Exemplarily, if the data of the data source is offline data, the heterogeneous data is collected in batches through the computing engine.

[0069] In this embodiment, the data sources include: a relational database, a non-relational database, a file system, and a message queue.

[0070] Step 202: Clean the heterogeneous data according to their types, and store the cleaned data into the constructed real-time lake warehouse.

[0071] Exemplarily, if the heterogeneous data is real-time data at the minute level, or the heterogeneous data is quasi-real-time data at the day level obtained through a crontab scheduled task, the similarity between each pair of data points is determined through a cyclic sampling method, and data deduplication processing is performed based on the comparison result of the similarity with a preset threshold, and data with a similarity greater than the preset threshold is eliminated to obtain cleaned data; wherein, the crontab is a periodic execution process.

[0072] Exemplarily, if the heterogeneous data is offline data, a preset detection method is used to find abnormal values ​​or outliers, and the data containing the abnormal values ​​or outliers is eliminated to obtain cleaned data; the preset detection method includes: any one of statistics, machine learning, and deep learning.

[0073] In this embodiment, the three types of data shown in step 201 may be combined to perform data cleaning respectively.

[0074] For the first and second types of data, the main strategy is data deduplication, that is, through circular sampling, the similarity between two data points can be calculated, and the threshold can be set to determine whether to perform data deduplication operations.

[0075] The similarity calculation formula is as follows:

[0076]

[0077] similarity indicates similarity, and Num of common attributes between teo data points indicates the number of common attributes between two data points. Total number of attributes indicates the total number of attributes.

[0078] In this embodiment, after the above deduplication processing, pruned log set data can be obtained, reducing system storage pressure.

[0079] For the third type of data, the main strategy is to clean up the outliers or outliers in the data. For example, outliers or outliers in the data can be discovered and cleaned up based on statistics, machine learning or deep learning to improve data quality. The calculation formula is as follows:

[0080]

[0081] Here, x represents the data point,

[0082] μ represents the mean value of the data,

[0083] σ represents the standard deviation of the data,

[0084] z represents the degree of deviation of the data point from the mean value and can be used to determine whether the data is an outlier.

[0085] In this embodiment, after the above-mentioned abnormal value or outlier data is eliminated, unified and standardized DB (database) type data can be obtained.

[0086] In this embodiment, the real-time lake warehouse includes: a flow layer, a detail layer, and a summary layer; wherein: the flow layer is used to store the cleaned data into the real-time data lake and / or the offline data lake; the detail layer is used to build an ETL link and complete data processing through the ETL link; the summary layer is used to summarize the data processed by the ETL link and expose the data interface; the data interface is used to provide data services.

[0087] In this embodiment, data collection and cleaning can be automated. Compared with traditional data collection and cleaning, which requires a lot of manpower and time and is prone to errors, in this embodiment, machine learning technology is combined with some common clustering and deduplication algorithms to realize data collection, cleaning and classification automation, thereby greatly improving the efficiency and accuracy of data processing.

[0088] Step 203: construct an ETL link for the data in the real-time lake warehouse, and process the data through the ETL link to obtain processed data.

[0089] In this embodiment, the ETL link includes multiple models, and the model adopts an incremental learning method to perform real-time update of model parameters; wherein the multiple models are used to perform at least one of the following operations:

[0090] Extracting data from the real-time data lake and / or offline data lake of the real-time lake warehouse;

[0091] Performing logical conversion processing on the extracted data to obtain converted data, wherein the logical conversion processing includes at least one of: dimension table widening, window query, conditional filtering, multi-stream splicing, and index optimization;

[0092] The conversion data is processed and incremented to obtain processed data.

[0093] In this embodiment, by introducing multiple incremental learning models, intelligent automation of ETL construction is achieved, thereby reducing manual intervention and errors and improving data processing efficiency.

[0094] For example, the incremental learning example of the linear regression model is used to update the model parameters in real time. Assume that the linear regression model formula is:

[0095] y=w0+w1x1+w2x2+...+wnxn

[0096] Among them, the loss function of the model is the mean squared error:

[0097] Loss=2m1∑i=1m(prediction(i)-y(i))2

[0098] Where m is the number of samples, pre di ct i on (i) It is i The predicted value of the sample y(i) is the true value of the sample. In incremental learning, each time a new sample is received, the model parameters can be updated in real time by following the steps below:

[0099] For new samples xnew Calculate the predicted value: pre d ictionnew =w0+w1x new1 +w2x new2 +...+w n x newn

[0100] According to the true value of the new sample ynew , update the model parameters: Among them, α is the learning rate, which is used to control the step size of parameter update.

[0101] By repeating the above steps, the model can dynamically update parameters when receiving new data to adapt to changes in data and maintain accuracy.

[0102] Step 204: query the processed data from the real-time lake warehouse through the computing engine and provide corresponding data services.

[0103] Exemplarily, the processed data is queried from the real-time lake warehouse through a computing engine, and the processed data is analyzed to obtain analysis results; the analysis results are displayed on a user-visualized front-end interface through a data interface, and corresponding data services are provided on the user-visualized front-end interface, and the data services include: at least one of data analysis, data mining, content recommendation, and data visualization.

[0104] In this embodiment, the lake warehouse DWS layer data is deeply mined and analyzed through machine learning technology, so as to discover the value and potential behind the data. In addition, the data processing results can be converted into upper-level services by using visualization technology, providing more accurate and personalized services for enterprises and users.

[0105] In the above-mentioned lake-warehouse integrated data processing method, heterogeneous data is collected in batches and / or in real time from the data source; thus, large-scale heterogeneous data can be processed and processing efficiency can be improved. The heterogeneous data is cleaned according to the type of the data, and the cleaned data is stored in the constructed real-time lake warehouse; thus, redundant and abnormal data can be intelligently removed, the efficiency of subsequent data processing can be improved, and the throughput rate can be guaranteed. An ETL link is constructed for the data in the real-time lake warehouse, and the data is processed through the ETL link to obtain the processed data; wherein the ETL link is an extraction, transformation and loading link; thus, the governance efficiency of the data lake can be improved. The processed data is queried from the real-time lake warehouse through the computing engine, and corresponding data services are provided. Therefore, while ensuring the throughput rate, the timeliness of data entering the lake can be improved as much as possible. The entire processing process has high stability, which greatly improves the intelligence of the ETL link processing, so that the data governance efficiency of the lake warehouse integration is significantly improved.

[0106] In another exemplary embodiment, Figure 3 As shown, a lake-warehouse integrated data processing method is provided, including: pre-data collection and cleaning, global data aggregation, real-time data lake and connection data lake construction, intelligent real-time ETL process construction and optimization, data processing incremental calculation, and data application services. Figure 3 The method shown is Figure 2 A further refinement of the method shown.

[0107] Exemplarily, massive log data (such as host logs, component logs, job logs, etc.) are collected through application programming interfaces (such as JDBC), hypertext transfer protocol (HTTP), file transfer protocol (FTP), message queues (MQ), etc. These massive log data are generally real-time data. After cleaning these massive data, they are processed by real-time incremental synchronization through the computing engine Flink, and then stored in the real-time data lake using queue caching and incremental entry into the lake. In the real-time data lake, the stored real-time data can be classified through Kafka, Hudi / iceberg, ODS layer (source layer), and ACID management, and then batch and stream integration is realized through Flink.

[0108] For example, the data obtained through the RDB database (i.e., relational database) is generally offline data, so it can be stored in the offline data lake after offline batch synchronization processing using tools such as Canal, Flink, and cdc. In the offline data lake, the stored offline data can be classified through Hudi / iceberg, ODS layer, and ACID management, and then batch and stream integration can be achieved through Flink. Offline processing can also be performed directly through Spark / Hive.

[0109] For example, the data retrieved by Flink enters the ODS layer-DWD layer (detail layer) for intelligent real-time ETL process construction and optimization, and enters the DWD layer after logical conversion such as dimension table widening, multi-stream splicing, index optimization, etc. Further, data processing incremental calculation is performed in the DWD layer-DWS layer (aggregation layer) to obtain processed data. These processed data are stored in the DWS layer, and the interfaces exposed by the DWS layer provide data application services (for example: data governance, data modeling, data services, BI reports, content recommendations, etc.).

[0110] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0111] In this embodiment, intelligent technology is used to improve processing efficiency in multiple links such as pre-data cleaning, ETL link construction, and final collection and landing in the data lake; while ensuring the throughput rate, the timeliness of data entry into the lake can be improved as much as possible. A more complete ETL intelligent construction optimization and intelligent display function are achieved, further improving the data governance efficiency of the lake-warehouse integrated system.

[0112] Based on the same inventive concept, the embodiment of the present application also provides a lake-warehouse integrated data processing device for implementing the above-mentioned lake-warehouse integrated data processing method. The implementation solution provided by the device to solve the problem is similar to the implementation solution recorded in the above-mentioned method, so the specific limitations in one or more lake-warehouse integrated data processing device embodiments provided below can refer to the above limitations on the lake-warehouse integrated data processing method, which will not be repeated here.

[0113] In an exemplary embodiment, Figure 4 As shown, a lake-warehouse integrated data processing device is provided, including: a data acquisition module 401, a data cleaning and storage module 402, a data processing module 403 and a data service module 404, wherein:

[0114] The data collection module 401 is used to collect heterogeneous data from data sources in batches and / or in real time;

[0115] A data cleaning and storage module 402 is used to clean the heterogeneous data according to their types and store the cleaned data into the constructed real-time lake warehouse;

[0116] The data processing module 403 is used to construct an ETL link for the data in the real-time lake warehouse, and process the data through the ETL link to obtain processed data; wherein the ETL link is an extraction, transformation and loading link;

[0117] The data service module 404 is used to query the processed data from the real-time lake warehouse through the computing engine and provide corresponding data services.

[0118] Exemplarily, the data acquisition module 401 is specifically used to collect heterogeneous data in real time through a computing engine when the data of the data source is real-time data; when the data of the data source is offline data, it is used to collect heterogeneous data in batches through a computing engine; wherein the data source includes: a relational database, a non-relational database, a file system, and a message queue.

[0119] Exemplarily, the data cleaning storage module 402 is specifically used to determine the similarity between any two data points through a cyclic sampling method when the heterogeneous data is real-time data at the minute level, or when the heterogeneous data is quasi-real-time data at the day level obtained through a crontab scheduled task, and perform data deduplication processing based on a comparison result of the similarity with a preset threshold, thereby eliminating data with a similarity greater than the preset threshold to obtain cleaned data; wherein the crontab is a periodic execution process; when the heterogeneous data is offline data, a preset detection method is used to discover abnormal values ​​or outliers, and data with abnormal values ​​or outliers is eliminated to obtain cleaned data; the preset detection method includes: any one of statistics, machine learning, and deep learning.

[0120] Exemplarily, the real-time lake warehouse includes: a flow layer, a detail layer, and a summary layer; wherein:

[0121] The flow layer is used to store the cleaned data into the real-time data lake and / or the offline data lake;

[0122] The detail layer is used to build an ETL link and complete data processing through the ETL link;

[0123] The aggregation layer is used to aggregate the data processed by the ETL link and expose the data interface; the data interface is used to provide data services.

[0124] Exemplarily, the ETL link includes multiple models, and the models use incremental learning to perform real-time updates of model parameters; wherein the multiple models are used to perform at least one of the following operations:

[0125] Extracting data from the real-time data lake and / or offline data lake of the real-time lake warehouse;

[0126] Performing logical conversion processing on the extracted data to obtain converted data, wherein the logical conversion processing includes at least one of: dimension table widening, window query, conditional filtering, multi-stream splicing, and index optimization;

[0127] The conversion data is processed and incremented to obtain processed data.

[0128] Exemplarily, the data service module 404 is specifically used to: query the processed data from the real-time lake warehouse through a computing engine, and analyze the processed data to obtain analysis results; display the analysis results on a user-visualized front-end interface through a data interface, and provide corresponding data services on the user-visualized front-end interface, and the data services include: at least one of data analysis, data mining, content recommendation, and data visualization.

[0129] Each module in the above-mentioned lake warehouse integrated data processing device can be implemented in whole or in part by software, hardware and their combination. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0130] In an exemplary embodiment, Figure 5As shown, a lake-warehouse integrated data processing system is provided, including: a data acquisition layer 501, a data cleaning storage layer 502, a data processing layer 503 and a data service layer 504, wherein: the data acquisition layer 501 is used to collect heterogeneous data in batches and / or in real time from a data source; the data cleaning storage layer 502 is used to clean the heterogeneous data according to the type of the heterogeneous data, and store the cleaned data into the constructed real-time lake warehouse; the data processing layer 503 is used to build an ETL link for the data in the real-time lake warehouse, and process the data through the ETL link to obtain the processed data; wherein the ETL link is an extraction, transformation and loading link; the data service layer 504 is used to query the processed data from the real-time lake warehouse through a computing engine, and provide corresponding data services.

[0131] Exemplarily, the data collection layer 501 is specifically used to collect heterogeneous data in real time through a computing engine when the data of the data source is real-time data; when the data of the data source is offline data, the data collection layer 501 is used to collect heterogeneous data in batches through a computing engine; wherein the data source includes: a relational database, a non-relational database, a file system, and a message queue.

[0132] In this embodiment, the data collection layer is used to provide a series of methods to collect other application data. For example, the collection method supports industry-wide common data collection methods such as JDBC collection, HTTP collection, FTP collection, MQ collection, and Web Service collection. The collected data is heterogeneous data and can be divided into the following three categories:

[0133] The first category: real-time log data at the minute level, which contains a large number of duplicate or redundant logs.

[0134] The second category: daily quasi-real-time data obtained through crontab scheduled tasks. This type of data contains a large number of duplicate or redundant logs.

[0135] The third category: offline data of MySQL, Oracle, SQL Server, etc. obtained by establishing a source database connection and synchronizing it through Flink CDC technology.

[0136] For the first and second types of data, the main strategy is data deduplication, that is, through circular sampling, the similarity between two data points can be calculated, and the threshold can be set to determine whether to perform data deduplication operations.

[0137] The similarity calculation formula is as follows:

[0138]

[0139] similarity indicates similarity, and Num of common attributes between teo data points indicates the number of common attributes between two data points. Tota l number of attributes indicates the total number of attributes.

[0140] In this embodiment, after the above deduplication processing, pruned log set data can be obtained, reducing system storage pressure.

[0141] For the third type of data, the main strategy is to clean up the outliers or outliers in the data. For example, outliers or outliers in the data can be discovered and cleaned up based on statistics, machine learning or deep learning to improve data quality. The calculation formula is as follows:

[0142]

[0143] Here, x represents the data point,

[0144] μ represents the mean value of the data,

[0145] σ represents the standard deviation of the data,

[0146] z represents the degree of deviation of the data point from the mean value and can be used to determine whether the data is an outlier.

[0147] In this embodiment, after the above-mentioned abnormal value or outlier data is eliminated, unified and standardized DB (database) type data can be obtained.

[0148] Exemplarily, the data cleaning storage layer 502 is specifically used to determine the similarity between any two data points through a cyclic sampling method when the heterogeneous data is real-time data at the minute level, or when the heterogeneous data is quasi-real-time data at the day level obtained through a crontab scheduled task, and perform data deduplication processing based on a comparison result of the similarity with a preset threshold, thereby eliminating data with a similarity greater than the preset threshold to obtain cleaned data; wherein the crontab is a periodic execution process; when the heterogeneous data is offline data, a preset detection method is used to discover abnormal values ​​or outliers, and data with abnormal values ​​or outliers is eliminated to obtain cleaned data; the preset detection method includes: any one of statistics, machine learning, and deep learning.

[0149] Exemplarily, after completing data cleaning and integration, all data is stored in the lake as raw data in Iceberg, and an additional prefix ods (this is common knowledge in the industry) is added to the directory. At this time, Iceberg's partitioning method uses the default method or intelligent recommendation. After the data enters the lake, it enters the data processing layer on demand. The data processing layer 503 performs intelligent ETL link construction and optimization, that is, extracts all required data from the data source, performs the required data processing and conversion logic (such as dimension table widening, window query, conditional filtering, etc.), and obtains the execution result. At the same time, the directory adds an additional prefix dwd (this is common knowledge in the industry).

[0150] In this embodiment, a real-time data analysis and decision support system can be introduced into the intelligent ETL construction process of the data processing layer, and the dynamic learning and optimization mechanism can be triggered by real-time monitoring of changes in data characteristics and rules, and the ETL algorithm and process can be adaptively adjusted according to the learning conclusions and reward and punishment steps, and finally the ETL script or super-long SQL statement can be automatically generated. This method can improve the efficiency and accuracy of ETL construction, thereby realizing intelligent data processing.

[0151] For example, the ETL construction of the Hucang system will include multiple intelligent models. For example, an incremental learning method can be used, which allows the model to be continuously updated iteratively when new data is received without the need to retrain the entire model. This method is very suitable for processing streaming data and can maintain the accuracy and real-time performance of the model when the data source continuously generates new data.

[0152] For example, the incremental learning example of the linear regression model is used to update the model parameters in real time. Assume that the linear regression model formula is:

[0153] y=w0+w1x1+w2x2+...+wnxn

[0154] Among them, the loss function of the model is the mean squared error:

[0155] Loss=2m1∑i=1m(prediction(i)-y(i))2

[0156] Where m is the number of samples, pre di ct i on (i) It is i The predicted value of the sample y(i) is the true value of the sample. In incremental learning, each time a new sample is received, the model parameters can be updated in real time by following the steps below:

[0157] For new samples xnew Calculate the predicted value: pre d ictionnew =w0+w1x new1 +w2x new2 +...+w n x newn

[0158] According to the true value of the new sample ynew , update the model parameters: Among them, α is the learning rate, which is used to control the step size of parameter update.

[0159] By repeating the above steps, the model can dynamically update parameters when receiving new data to adapt to changes in data and maintain accuracy.

[0160] Exemplarily, the real-time lake warehouse includes: a flow layer, a detail layer, and a summary layer; wherein:

[0161] The flow layer is used to store the cleaned data into the real-time data lake and / or the offline data lake;

[0162] The detail layer is used to build an ETL link and complete data processing through the ETL link;

[0163] The aggregation layer is used to aggregate the data processed by the ETL link and expose the data interface; the data interface is used to provide data services.

[0164] Exemplarily, the ETL link includes multiple models, and the models use incremental learning to perform real-time updates of model parameters; wherein the multiple models are used to perform at least one of the following operations:

[0165] Extracting data from the real-time data lake and / or offline data lake of the real-time lake warehouse;

[0166] Performing logical conversion processing on the extracted data to obtain converted data, wherein the logical conversion processing includes at least one of: dimension table widening, window query, conditional filtering, multi-stream splicing, and index optimization;

[0167] The conversion data is processed and incremented to obtain processed data.

[0168] Exemplarily, the data service layer 504 is specifically used to: query the processed data from the real-time lake warehouse through a computing engine, and analyze the processed data to obtain analysis results; display the analysis results on a user-visualized front-end interface through a data interface, and provide corresponding data services on the user-visualized front-end interface, and the data services include: at least one of data analysis, data mining, content recommendation, and data visualization.

[0169] In this embodiment, the data service layer stores the DWS layer data, and based on this, provides data display, business services, data monitoring, real-time BI and other capabilities. The DWS layer data is processed by the DWD layer data. The data service layer obtains the series of interfaces exposed by the data storage and processing layer, and then provides the user with a visual front-end interface to facilitate lake warehouse management and data empowerment. At this time, the lake warehouse integrated system can display and apply data in a variety of ways, such as data analysis, data mining, content recommendation, data visualization, etc. Among them, artificial intelligence technology can provide more accurate data analysis and prediction, help enterprises better utilize data resources, and provide support and reference for the company's refined decision-making.

[0170] In this embodiment, the data collection layer 501, the data cleaning storage layer 502, the data processing layer 503 and the data service layer 504 can be independent of each other, or can be combined and integrated together to implement the following. Figure 2 , Figure 3 Steps in the illustrated method embodiment.

[0171] In an exemplary embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as shown in FIG. Figure 6As shown. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface, the display unit and the input device are connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless method can be implemented through WIFI, a mobile cellular network, near field communication (Near Field Communication, NFC) or other technologies. When the computer program is executed by the processor, a lake warehouse integrated data processing method is implemented. The display unit of the computer device is used to form a visually visible picture, which can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device shell, or an external keyboard, touchpad or mouse.

[0172] Those skilled in the art will understand that Figure 6 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0173] In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.

[0174] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0175] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0176] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0177] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0178] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0179] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A lake-warehouse integrated data processing method, characterized in that: The method comprises: Ingest heterogeneous data from data sources in batch and / or real-time; Clean the heterogeneous data according to their types, and store the cleaned data in the constructed real-time lake warehouse; An ETL link is constructed for the data in the real-time lake warehouse, and the data is processed through the ETL link to obtain processed data; wherein the ETL link is an extraction, transformation and loading link; The processed data is queried from the real-time lake warehouse through a computing engine, and corresponding data services are provided.

2. The method according to claim 1, characterized in that: The batch and / or real-time collection of heterogeneous data from data sources includes: If the data source is real-time data, the computing engine is used to collect heterogeneous data in real time; If the data source is offline data, the computing engine collects heterogeneous data in batches; The data sources include: relational database, non-relational database, file system, and message queue.

3. The method according to claim 1, characterized in that The cleaning is performed respectively according to the types of the heterogeneous data, including: If the heterogeneous data is minute-level real-time data, or the heterogeneous data is daily-level quasi-real-time data obtained through crontab scheduled tasks, the similarity between two data points is determined by circular sampling, and data deduplication is performed based on the comparison result of the similarity with the preset threshold, and data with similarity greater than the preset threshold is removed to obtain cleaned data; wherein, crontab is a periodic execution process; If the heterogeneous data is offline data, a preset detection method is used to find abnormal values ​​or outliers, and the data with abnormal values ​​or outliers is eliminated to obtain cleaned data; the preset detection method includes: any one of statistics, machine learning, and deep learning.

4. The method according to claim 1, characterized in that: The real-time lake warehouse includes: a flow layer, a detail layer, and a summary layer; wherein: The flow layer is used to store the cleaned data into the real-time data lake and / or the offline data lake; The detail layer is used to build an ETL link and complete data processing through the ETL link; The aggregation layer is used to aggregate the data processed by the ETL link and expose the data interface; the data interface is used to provide data services.

5. The method according to claim 1, characterized in that The ETL link includes multiple models, and the model adopts an incremental learning method to perform real-time update of model parameters; wherein the multiple models are used to perform at least one of the following operations: Extracting data from the real-time data lake and / or offline data lake of the real-time lake warehouse; Performing logical conversion processing on the extracted data to obtain converted data, wherein the logical conversion processing includes at least one of: dimension table widening, window query, conditional filtering, multi-stream splicing, and index optimization; The conversion data is processed and incremented to obtain processed data.

6. The method according to any one of claims 1 to 5, characterized in that: The processing data is queried from the real-time lake warehouse by the computing engine, and corresponding data services are provided, including: querying the processed data from the real-time lake warehouse through a computing engine, and analyzing the processed data to obtain an analysis result; The analysis results are displayed on a user-visualized front-end interface through a data interface, and corresponding data services are provided on the user-visualized front-end interface, wherein the data services include at least one of data analysis, data mining, content recommendation, and data visualization.

7. A lake-warehouse integrated data processing device, characterized in that: The device comprises: A data collection module, used to collect heterogeneous data from data sources in batches and / or in real time; A data cleaning and storage module, used to clean the heterogeneous data according to their types and store the cleaned data into the constructed real-time lake warehouse; A data processing module is used to construct an ETL link for the data in the real-time lake warehouse, and process the data through the ETL link to obtain processed data; wherein the ETL link is an extraction, transformation and loading link; The data service module is used to query the processed data from the real-time lake warehouse through the computing engine and provide corresponding data services.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images