Hudi-based data acquisition and processing system and method, electronic equipment and medium
Patent Information
- Application Number
- CN202411014985.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-26
- Publication Date
- 2024-11-08
AI Technical Summary
但是,随着数据量的持续增加和实时业务需求越来越多,同时,业务数据伴随着经常的数据修改和新增等,当前框架难以满足系统对海量数据的实时计算处理
[0032] Embodiments of this application provide a data collection and processing system, method, electronic equipment and media based on Hudi. The system includes a data collection module, a lake-warehouse integrated processing module and a data application module; through the data collection module, the system collects data from multiple business databases. Collect offline data and real-time data at the same time, and store offline data Data and real-time data are incrementally synchronized to the Hucang integrated processing module; through the Hucang integrated processing module, offline data and real-time data are stored and processed uniformly to obtain application indicator data; through the data application module, direct processing of data according to user instructions is achieved. Related query of application indicator data in the integrated processing module of Hucang. The system's data collection module integrates offline collection and real-time collection links to collect real-time data on changing data of each business database, which can improve data freshness; the lake warehouse integrated data processing module can uniformly store and process data , simplifying the data processing link; the data application module can realize related queries of heterogeneous data sources in the lake. Therefore, the above-mentioned data collection and processing system based on Hudi optimizes the links of the data processing system and improves the timeliness of data processing and resource utilization efficiency.
Smart Images

Figure CN118916432A_ABST
Abstract
Description
Technical Field
[0001] The present application belongs to the field of big data collection and processing technology, and in particular, relates to a HUDI-based data collection and processing system, method, electronic device and medium. Background Art
[0003] The current government big data processing system is usually built with Lambda architecture, mainly based on offline collection and offline data warehouse of Hadoop+HIVE, combined with technologies such as spark to realize stream batch processing. However, with the continuous increase in data volume and more and more real-time business needs, and the frequent data modification and addition of business data, the current framework is difficult to meet the system's real-time computing and processing of massive data. For example, data processing and query efficiency is low, and data synchronization cannot be effectively achieved in real time for newly added / modified data. Moreover, the current data processing system is usually divided into two data links: offline data warehouse and real-time data warehouse, and these two links are usually developed and maintained by two development teams. Such a system architecture is difficult to ensure data consistency, and the architecture is complex, the links are long, and the resources consumed are high. Summary of the invention
[0004] In order to overcome the problems existing in the related art, the embodiments of the present application provide a data acquisition and processing system, method, electronic device and medium based on HUDI to optimize the link of the data processing system and improve the timeliness of data processing and the utilization efficiency of resources.
[0005] This application is implemented through the following technical solutions:
[0006] In the first aspect, the embodiment of the present application provides a data acquisition and processing system based on hudi, including a data acquisition module, a lake warehouse integrated processing module and a data application module;
[0007] The data collection module is used to collect business data from multiple business databases and synchronize the business data increments to the lake-warehouse integrated processing module; wherein the business data includes offline data and real-time data;
[0008] The lake-warehouse integrated processing module is used to uniformly store and process the offline data and the real-time data to obtain application indicator data;
[0009] The data application module is used to perform associated query on the application indicator data in the lake-warehouse integrated processing module according to user instructions.
[0010] In combination with the first aspect, in some embodiments, the lake-warehouse integrated processing module is specifically used for:
[0011] The offline data and the real-time data are preprocessed through FlinkSQL, and dimension tables and fact tables are constructed; the dimension tables and the fact tables are aggregated and associated to construct a data wide table set; based on different preset indicators, preset dimensions and the data wide table set, an indicator data table is constructed to obtain application indicator data of different data marts.
[0012] In combination with the first aspect, in some embodiments, the lake-warehouse integrated processing module includes a hudi table; and the data acquisition module is specifically used for:
[0013] The binlog logs of the multiple business databases are captured in real time through flinkcdc, the change data of each business database is collected in real time, and the data is uniformly stored in the hudi table.
[0014] In combination with the first aspect, in some embodiments, the lake-warehouse integrated processing module includes a hudi table; the data acquisition module is further used for:
[0015] For the first business scenario, the merge on read strategy is adopted to write the business data into the hudi table; wherein the first business scenario is a business scenario in which the number of write requests to the database is greater than the number of read requests;
[0016] For the second business scenario, a copy on write strategy is adopted to write the business data into the hudi table; wherein, the second business scenario is a business scenario in which the amount of write requests to the database is less than the amount of read requests.
[0017] In combination with the first aspect, in some embodiments, the lake-warehouse integrated processing module is also used to store the metadata information of the hudi table based on a preset periodic Savepoint; when accessing the Savepoint, it is mapped to the real data file corresponding to the hudi table based on the metadata information.
[0018] In combination with the first aspect, in some embodiments, the lake-warehouse integrated processing module is also used to filter the error data through the filter preset in the Savepoint when accessing the snapshot view; wherein the error data is data whose processing time and event time are not the same date.
[0019] In combination with the first aspect, in some embodiments, the data application module includes a data science application unit, a visualization large screen unit, a report unit and a data analysis unit; the user instruction includes a first instruction, a second instruction, a third instruction and a fourth instruction; the lake warehouse integrated processing module includes multiple heterogeneous data sources, and the multiple heterogeneous data sources include HDFS, Hudi, Hbase, ElasticSearch, and Doris;
[0020] The data science application unit is used to query the first application indicator data of at least one heterogeneous data source through a unified SQL engine according to the first instruction input by the user, and write the first application indicator data into a heterogeneous database through a unified SQL interface; wherein the heterogeneous database is a database other than the multiple business databases;
[0021] The visualization large screen unit is used to query the second application indicator data in the lake warehouse integrated processing module through a unified SQL engine according to the second instruction input by the user, and display the second application indicator data;
[0022] The report unit is used to query the third application indicator data in the lake-warehouse integrated processing module through a unified SQL engine according to the third instruction input by the user, and generate a target report based on the third application indicator data;
[0023] The data analysis unit is used to query the fourth application indicator data of at least one heterogeneous data source through the unified SQL engine according to the fourth instruction input by the user, and write the fourth application indicator data into the heterogeneous database through the unified SQL interface for data analysis to obtain data analysis results.
[0024] In a second aspect, an embodiment of the present application provides a data collection and processing method based on hudi, including:
[0025] Collect business data from multiple business databases and synchronize the business data increments to hudi; wherein the business data includes offline data and real-time data;
[0026] The offline data and the real-time data are uniformly stored and processed in Hudi to obtain application indicator data;
[0027] According to the user's instructions, the application indicator data in hudi is queried in association, and the application indicator data queried by the user is visualized.
[0028] In a third aspect, an embodiment of the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the HUDI-based data acquisition and processing method as described in the second aspect is implemented.
[0029] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the hudi-based data acquisition and processing method as described in the second aspect is implemented.
[0030] In a fifth aspect, an embodiment of the present application provides a computer program product. When the computer program product runs on an electronic device, the electronic device executes the hudi-based data acquisition and processing method described in the second aspect above.
[0031] Compared with the related art, the embodiments of the present application have the following beneficial effects:
[0032] The embodiment of the present application provides a data acquisition and processing system, method, electronic device and medium based on hudi. The system includes a data acquisition module, a lake-warehouse integrated processing module and a data application module; through the data acquisition module, offline data and real-time data are collected from multiple business databases at the same time, and the offline data and real-time data are incrementally synchronized to the lake-warehouse integrated processing module; through the lake-warehouse integrated processing module, offline data and real-time data are uniformly stored and processed to obtain application index data; through the data application module, the associated query of the application index data in the lake-warehouse integrated processing module is realized directly according to the user's instructions. The data acquisition module of the system integrates the links of offline collection and real-time collection, and performs real-time data collection on the change data of each business database, which can improve the freshness of the data; the lake-warehouse integrated data processing module can uniformly store and process the data, simplifying the data processing link; the data application module can realize the associated query of heterogeneous data sources in the lake. Therefore, the above-mentioned data acquisition and processing system based on hudi optimizes the link of the data processing system, improves the timeliness of data processing and the efficiency of resource utilization.
[0033] It can be understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here.
[0034] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present specification. BRIEF DESCRIPTION OF THE DRAWINGS
[0035] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.
[0036] Figure 1 It is a data architecture diagram of an existing government data processing system provided by an embodiment of the present application;
[0037] Figure 2 It is a structural diagram of a hudi-based data acquisition and processing system provided in an embodiment of the present application;
[0038] Figure 3 It is a schematic diagram of batch processing of a collection link in an existing government data processing system provided in an embodiment of the present application;
[0039] Figure 4 This is a schematic diagram of a data acquisition link of a data acquisition module provided in an embodiment of the present application;
[0040] Figure 5 This is a schematic diagram of an optimized data link of a lake-warehouse integrated processing module provided in one embodiment of the present application;
[0041] Figure 6 It is a schematic diagram of using a unified SQL engine to query heterogeneous data sources in a data lake for application according to an embodiment of the present application;
[0042] Figure 7 It is a flowchart of a data collection and processing method based on hudi provided in an embodiment of the present application;
[0043] Figure 8 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0044] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.
[0045] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.
[0046] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.
[0047] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.
[0048] Figure 1 A schematic diagram of the data architecture of an existing government data processing system provided in an embodiment of the present application. Figure 1 The original government data processing system was built with Lambda architecture. The offline data warehouse and real-time data warehouse used two data links, requiring two development teams to complete development and operation and maintenance. This data architecture has the following problems:
[0049] 1. Data consistency is difficult to guarantee, the architecture is complex, the links are long, and downstream storage needs to implement data deduplication by itself to remove duplicate data and ensure the final consistency of the data.
[0050] 2. Complex architecture and long links. Since such an architecture requires two computing engines and a message queue Kafka to write data into the data warehouse, such an architecture involves many components, long links, high resource consumption, and complex maintenance.
[0051] 3. Newly added, modified, and deleted data cannot be synchronized in real time and effectively; real-time data is prone to omissions and duplications, and offline data needs to be re-run for correction and backup.
[0052] Based on the above problems, the embodiments of the present application provide a data acquisition and processing system, method, electronic device and medium based on hudi. The system includes a data acquisition module, a lake-warehouse integrated processing module and a data application module; through the data acquisition module, offline data and real-time data are collected from multiple business databases at the same time, and the offline data and real-time data increments are synchronized to the lake-warehouse integrated processing module; through the lake-warehouse integrated processing module, offline data and real-time data are uniformly stored and processed to obtain application index data; through the data application module, the associated query of the application index data in the lake-warehouse integrated processing module is realized directly according to the user's instructions. The data acquisition module of the system integrates the links of offline collection and real-time collection, and performs real-time data collection on the change data of each business database, which can improve the freshness of the data; the lake-warehouse integrated data processing module can uniformly store and process the data, simplifying the data processing link; the data application module can realize the associated query of heterogeneous data sources in the lake. Therefore, the hudi-based data acquisition and processing system provided by the embodiments of the present application optimizes the link of the data processing system, improves the timeliness of data processing and the efficiency of resource utilization.
[0053] See also Figure 2 The hudi-based data acquisition and processing system provided in an embodiment of the present application includes a data acquisition module 210, a lake-warehouse integrated processing module 220 and a data application module 230.
[0054] Among them, the data collection module 210 is used to collect business data from multiple business databases and synchronize the business data increments to the lake-warehouse integrated processing module 220.
[0055] The lake-warehouse integrated processing module 220 is used to uniformly store and process offline data and real-time data to obtain application indicator data.
[0056] The data application module 230 is used to perform associated query on the application indicator data in the lake-warehouse integrated processing module 220 according to user instructions.
[0057] In the embodiment of the present application, the above-mentioned business data includes offline data and real-time data. The above-mentioned multiple business databases can be Figure 2 The data source 240 shown includes Mysql, Oracle, and PG (PostgreSQL) databases.
[0058] Figure 3 This is a schematic diagram of batch processing of the collection link in the original government data processing system. Figure 3In the original system, the entire business data of the previous day was synchronized to Hive in batches every morning. Since this process cannot identify deletion and update operations, it is necessary to associate and merge the data added, modified, and deleted on the previous day with the Hive history table, and then overwrite the original Hive table to complete it. Figure 3 The light-colored blocks in the temporary table and historical storage table represent modified or deleted data, and the dark-colored blocks represent added data. Therefore, the collection link of the original system has the following problems:
[0059] Data is synchronized daily, which is a daily synchronization with poor timeliness. The results can only be seen the next day. The deletion and update operations cannot be recognized and need to be associated and merged with historical data, which has high development costs. Each time data is entered into the warehouse, the full amount of historical data and the newly added data need to be associated. The amount of data and resource consumption are high at night and idle during the day. Real-time tasks also require the construction of an additional set of flink+kafka real-time links to complete (see Figure 1 ).
[0060] Figure 4 It is a collection link diagram of the data collection module 210 provided in an embodiment of the present application. The lake-warehouse integrated processing module 220 includes a hudi table; the data collection module 210 is specifically used to: capture the binlog logs of multiple business databases in real time through flinkcdc, collect the change data of each business database in real time, and store them uniformly in the hudi table.
[0061] The above embodiment simplifies the data collection link, integrates offline and real-time collection, and uniformly adopts FlinkCDC stream processing. The data is scattered and distributed throughout the day for incremental entry into the lake, so that the data freshness can reach the minute level.
[0062] In some embodiments, the data collection module 210 is further configured to:
[0063] For the first business scenario, the merge on read strategy is adopted to write the business data into the hudi table; wherein, the first business scenario is a business scenario in which the number of write requests to the database is greater than the number of read requests.
[0064] For the second business scenario, the copy on write strategy is adopted to write the business data into the hudi table; wherein, the second business scenario is a business scenario in which the number of write requests to the database is less than the number of read requests.
[0065] In the above embodiment, the incremental data is directly written into the hudi table. According to different business scenarios, the merge on read strategy is adopted for data with more writes and less reads to speed up the write rate, and the data set is integrated and processed when querying. For data with more reads and less writes, the copy on write strategy is set to merge the data when writing, appropriately reduce the write performance, and improve the data processing efficiency and query efficiency.
[0066] In some embodiments, the data acquisition module 210 can also be used to merge business data according to a preset number of commits when synchronizing business data to the lake warehouse integrated processing module 220, and the merging form is a file merge. In this way, when merging files, there is no need to merge the entire table, which reduces resource consumption, and when merging data according to the number of commits, there is no need to centralize the data for associated merging. For example, if the preset number of commits is 5 times, then the above merging process is to merge the business data according to every 5 commits.
[0067] The above commit represents the process of writing a batch of records into a data set. Each commit corresponds to a point on the timeline, and a commit identifies a set of operations.
[0068] The data acquisition module 210 in the embodiment of the present application optimizes the data acquisition link, unifies the collection of full and incremental heterogeneous data, provides unified data integration and data sharing functions, and integrates data from different sources into a unified storage layer, so that it can be seamlessly integrated with other systems and applications, eliminate data silos, and improve data accessibility and availability.
[0069] In the original government data warehouse scenario, after importing business data into the data warehouse, a large wide table is constructed by joining multiple tables, and the calculated results are output to MySQL for storage after aggregation. The report business system will directly connect to MySQL for data display. In addition, an offline link will be built to perform T+1 batch runs to correct and back up online data. This will cause problems such as duplicate construction of real-time and offline links, high computing and storage costs, and high development and operation costs.
[0070] Specifically, the offline link uses T+1 offline batch processing, and processes the data of the previous day through batch processing in the early morning, and loads the data into the system. The data results can only be seen the next day. Therefore, the timeliness of offline cannot meet some business needs, so a real-time link needs to be developed to handle some needs with higher timeliness requirements. In this way, real-time data processing and offline processing are two separate links, and the data in Kafka cannot be queried and observed ad hoc, and needs to be migrated to HDFS for query. Data anomalies cannot be rerun on the real-time link, but need to be corrected through the offline link. Offline and real-time require different system occupancy and resource overhead, and real-time data needs to copy a copy of the data to HDFS, which will cause certain data inconsistency problems. Offline tasks are basically completed between 0:00 and 5:00 in the morning, which will lead to peak resource usage at night, tasks will be queued and waiting, and batch running will fail. Correspondingly, resources are idle during the day, resulting in insufficient resource utilization. Tasks are serially scheduled, forward dependent, high scheduling complexity, and relatively long links.
[0071] Figure 5 It is a schematic diagram of the optimized data link of the lake-warehouse integrated processing module 220 provided in one embodiment of the present application. The lake-warehouse integrated processing module 220 is specifically used for: preprocessing offline data and real-time data through flinksql, and constructing dimension tables and fact tables; aggregating and associating dimension tables and fact tables to construct data wide table sets; constructing indicator data tables based on different preset indicators, preset dimensions and data wide table sets to obtain application indicator data of different data marts.
[0072] See also Figure 5 , the business data collected by the data acquisition module 210 first arrives at the data source layer ODS of the hudi data lake. The data in this layer can be named using the ods_xsss_table name_sys_db type. Next is the data warehouse layer DW, which is divided into the DWD data detail layer and the DWS data service layer (not shown). The data of the data source layer ODS is preprocessed by flinksql, and then the dimension table and fact table of the DWD data detail layer are constructed. The table naming methods of the fact table and the dimension table can be dwd_fact_xsss_table name and dwd_dim_xsss_table name respectively; in the data detail layer DWD, the dimension table data and the fact table data are aggregated and associated, and a data wide table set is constructed and stored in the DWS data service layer. Afterwards, the indicator result data table of the analysis data layer ADS is constructed according to different indicators and dimensions, and the indicator data table is named using ads_xsss_table name.
[0073] Apache hudi is an open source data lake framework. Hudi supports stream-table duality, that is, it can be used as a stream to process data changes, and it can also be used as a table to query data. The embodiment of the present application uses the flink+hudi method to integrate all offline and real-time businesses into the hudi table for storage and processing. At the same time, the flink checkpoint frequency is set to 5 minutes to ensure that the data freshness can reach the minute level, which can meet the needs of business projects. Offline and real-time use hudi for unified full-volume persistent storage and processing, and data consistency is guaranteed. There is no need to copy data and correct data through offline batch processing. Multi-tasks can also be run in streaming and processed in parallel in flink.
[0074] The above-mentioned lake-warehouse integrated processing module 220 optimizes the links within the warehouse, adopts a set of architectures to meet both real-time and offline requirements, and unifies streams and batches through the flink stream processing engine, achieving the effect of reducing costs and unifying data caliber. At the same time, data can be directly connected to the business system without leaving the warehouse, and the Hudi table can be directly queried for data display.
[0075] In some embodiments, the lake-warehouse integrated processing module 220 is also used to store the metadata information of the hudi table based on a preset periodic Savepoint; when accessing the Savepoint, it is mapped to the real data file corresponding to the hudi table based on the metadata information.
[0076] The above-mentioned storage of metadata information of the hudi table based on the preset periodic Savepoint means that all files under the data path that are less than the snapshot time are converted and saved in the files under the metadata directory. It saves the data before this time point, and the data before the snapshot can be queried by specifying the time of the snapshot point to realize offline data analysis query on a daily and monthly granularity.
[0077] Hudi can be used to maintain metadata information. The real data files are stored in the hdfs file system in txt / orc format. The location field in the metadata information maintains the real storage address of the data. Therefore, when accessing the Savepoint, it can be mapped to the real data file corresponding to the hudi table based on the metadata information.
[0078] Snapshot view refers to storing metadata and accessing actual data files through mapping. In the above embodiment, the problem of data redundancy is solved through the hudi Snapshot View snapshot view. By setting a periodic Savepoint, the metadata information of the current timeline of hudi is stored. When the Savepoint is accessed again, the stored metadata information is mapped to the real file of hudi, thereby avoiding redundant data storage. It is equivalent to storing a snapshot every day, so that when offline ETL is required, the snapshot data is accessed; when responding to real-time demands, the latest data of hudi can be directly accessed.
[0079] There are also data delay and drift problems when data enters the lake. The drift problem is caused by the periodicity of flink's incremental writing to hudi, which cannot effectively align the data time and commit. As a result, data at around 24:00 yesterday and data at around 0:00 today will be written in the same commit. In addition, the data delay problem is caused by the time error between event time and processing time when writing data. Event time is the real time of business data, and processing time is the time when data processing is completed and written. For example, when the real time reaches 0 o'clock, it may still be processing data at 23 o'clock the previous day. At this time, if a snapshot view is taken, the snapshot view data read by the user is the data before 23 o'clock. Because commit is the processing time, not the event time of the business.
[0080] Based on the above problems, the lake-warehouse integrated processing module 220 in the embodiment of the present application is also used to filter the error data through the filter preset in the Savepoint when accessing the snapshot view; wherein the error data is the data whose processing time and event time are not the same date. The filter can be implemented by the filter predicate after the Sql where.
[0081] The lake-warehouse integrated processing module 220 is also used to identify the data processing progress through the preset identifier in the snapshot view.
[0082] In the above embodiment, a filter is embedded in the savepoint, and when accessing the snapshot view, the error data is filtered out by the filter to solve the data drift problem. An identifier is added to the snapshot view to identify the processing progress to solve the data delay problem.
[0083] In some embodiments, the data application module 230 may include a data science application unit, a visualization large screen unit, a report unit and a data analysis unit; user instructions include a first instruction, a second instruction, a third instruction and a fourth instruction; the lake-warehouse integrated processing module includes multiple heterogeneous data sources, and the multiple heterogeneous data sources include HDFS, Hudi, Hbase, ElasticSearch, and Doris.
[0084] Among them, the data science application unit can be used to query the first application indicator data of at least one heterogeneous data source through a unified SQL engine according to a first instruction input by a user, and write the first application indicator data into a heterogeneous database through a unified SQL interface; wherein the heterogeneous database is a database other than multiple business databases.
[0085] Figure 6 A schematic diagram of using a unified SQL engine to query heterogeneous data sources in a data lake for applications.
[0086] The visualization large screen unit can be used to query the second application indicator data in the lake-warehouse integrated processing module through a unified SQL engine according to the second instruction input by the user, and display the second application indicator data.
[0087] The reporting unit can be used to query the third application indicator data in the lake-warehouse integrated processing module through a unified SQL engine according to the third instruction input by the user, and generate a target report based on the third application indicator data.
[0088] The data analysis unit can be used to query the fourth application indicator data of at least one heterogeneous data source through the unified SQL engine according to the fourth instruction input by the user, and write the fourth application indicator data into the heterogeneous database through the unified SQL interface for data analysis to obtain the data analysis results.
[0089] In the above embodiment, a unified sql engine is built by integrating presto. In the data application module 230, the sql unified engine is used to perform associated queries on the application indicator data in different libraries such as Hudi, mysql, and Elasticsearch, and a report unit, a visualization large screen unit, etc. are built in the application module in a restful manner. The indicator data can also be queried through the unified engine and written into a KV database or a business mysql database for further data analysis and data science applications.
[0090] A specific application scenario of the embodiment of the present application is given below:
[0091] The hudi-based data acquisition and processing system provided in the embodiment of the present application is used to construct a risk hidden danger prevention and control system for a housing fund center. The housing provident fund risk hidden danger prevention and control system constructed in this embodiment mainly extracts the data of the existing housing provident fund management system by timed extraction and incremental extraction into the housing provident fund risk hidden danger prevention and control system to form a data lake, and then processes, aggregates and multi-dimensionally analyzes all data to check and screen out suspicious indicator data. At the same time, a comprehensive multi-dimensional analysis of important indicators such as business operations and risk analysis is carried out, and it is displayed in the form of reports and large-screen visualizations to assist business managers to understand the current status of the center's business in real time, monitor and warn of funding risks, and promptly discover operational anomalies, so as to make scientific decisions. The entire data processing process is completed independently in the data lake system and will not have any impact on the existing housing provident fund management business system.
[0092] The hudi-based data acquisition and processing system provided in the embodiment of the present application integrates the links of offline acquisition and real-time acquisition through the data acquisition module 210, performs real-time data acquisition on the change data of each business database, and can improve the freshness of the data; the lake warehouse integrated processing module 220 uniformly stores and processes the data, simplifying the data processing link; and the data application module 230 realizes the associated query of heterogeneous data sources in the lake. Therefore, the hudi-based data acquisition and processing system provided in the embodiment of the present application optimizes the link of the data processing system, improves the timeliness of data processing and the efficiency of resource utilization.
[0093] Figure 7 This is a flow chart of a data collection and processing method based on hudi provided in an embodiment of the present application, referring to Figure 7 , the data collection and processing method based on hudi is described in detail as follows:
[0094] S101, collect business data from multiple business databases, and synchronize the business data increments to hudi; wherein the business data includes offline data and real-time data.
[0095] In some embodiments, step S101 may include: capturing binlog logs of multiple business databases in real time through flinkcdc, collecting change data of each business database in real time, and storing them uniformly in a hudi table.
[0096] In some embodiments, step S101 may include: for a first business scenario, adopting a merge on read strategy to write business data into a hudi table; wherein the first business scenario is a business scenario in which the amount of write requests to the database is greater than the amount of read requests; for a second business scenario, adopting a copy on write strategy to write business data into a hudi table; wherein the second business scenario is a business scenario in which the amount of write requests to the database is less than the amount of read requests.
[0097] S102, offline data and real-time data are uniformly stored and processed in Hudi to obtain application indicator data.
[0098] In some embodiments, step S102 may include: storing metadata information of the hudi table based on a preset periodic Savepoint; and mapping to a real data file corresponding to the hudi table based on the metadata information when accessing the Savepoint.
[0099] In some embodiments, step S102 may include: when accessing the snapshot view, filtering the error data through the filter preset in the Savepoint; wherein the error data is data whose processing time and event time are not the same date.
[0100] S103, performing an associated query on the application indicator data in hudi according to the user instruction, and visually displaying the application indicator data queried by the user.
[0101] In some embodiments, step S103 may include:
[0102] According to the first instruction input by the user, the first application indicator data of at least one heterogeneous data source is queried through the unified SQL engine, and the first application indicator data is written into a heterogeneous database through a unified SQL interface; wherein the heterogeneous database is a database other than the multiple business databases.
[0103] According to the second instruction input by the user, the second application indicator data in the lake-warehouse integrated processing module is queried through the unified SQL engine, and the second application indicator data is displayed.
[0104] The reporting unit is used to query the third application indicator data in the lake-warehouse integrated processing module through a unified SQL engine according to the third instruction input by the user, generate a target report based on the third application indicator data, and perform a visual display.
[0105] The data analysis unit is used to query the fourth application indicator data of at least one heterogeneous data source through the unified SQL engine according to the fourth instruction input by the user, and write the fourth application indicator data into the heterogeneous database through the unified SQL interface for data analysis to obtain data analysis results.
[0106] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.
[0107] The present application also provides an electronic device, see Figure 8 The electronic device 800 may include: at least one processor 810, a memory 820, and a computer program stored in the memory 820 and executable on the at least one processor 810. When the processor 810 executes the computer program, the steps in any of the above-mentioned method embodiments are implemented, for example: Figure 7 Steps S101 to S103 in the illustrated embodiment.
[0108] Exemplarily, the computer program may be divided into one or more modules / units, one or more modules / units are stored in the memory 820, and executed by the processor 810 to complete the present application. The one or more modules / units may be a series of computer program segments capable of completing specific functions, and the program segments are used to describe the execution process of the computer program in the electronic device 800.
[0109] Those skilled in the art will understand that Figure 8 These are merely examples of electronic devices and do not constitute a limitation of the electronic device. The electronic device may include more or fewer components than those shown in the figure, or a combination of certain components, or different components, such as input and output devices, network access devices, buses, etc.
[0110] The processor 810 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc.
[0111] The memory 820 may be an internal storage unit of the electronic device, or an external storage device of the electronic device, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. The memory 820 is used to store the computer program and other programs and data required by the electronic device. The memory 820 may also be used to temporarily store data that has been output or is to be output.
[0112] The bus may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus in the drawings of the present application is not limited to only one bus or one type of bus.
[0113] The HUDI-based data acquisition and processing method provided in the embodiments of the present application can be applied to computers, wearable devices, vehicle-mounted devices, tablet computers, laptops, netbooks, personal digital assistants (PDAs), augmented reality (AR) / virtual reality (VR) devices, mobile phones and other electronic devices. The embodiments of the present application do not impose any restrictions on the specific types of electronic devices.
[0114] An embodiment of the present application further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps in each embodiment of the above-mentioned HUDI-based data acquisition and processing method can be implemented.
[0115] An embodiment of the present application provides a computer program product. When the computer program product runs on a mobile terminal, the mobile terminal can implement the steps in each embodiment of the above-mentioned HUD-based data acquisition and processing method when executing the computer program product.
[0116] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may at least include: any entity or device that can carry the computer program code to the camera / electronic device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. For example, a USB flash drive, a mobile hard disk, a disk or an optical disk.
[0117] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.
[0118] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.
[0119] In the embodiments provided in the present application, it should be understood that the disclosed devices / network equipment and methods can be implemented in other ways. For example, the device / network equipment embodiments described above are merely schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.
[0120] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0121] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.
Claims
1. A data acquisition and processing system based on hudi, characterized in that: It includes data acquisition module, lake-warehouse integrated processing module and data application module; The data collection module is used to collect business data from multiple business databases and synchronize the business data increments to the lake-warehouse integrated processing module; wherein the business data includes offline data and real-time data; The lake-warehouse integrated processing module is used to uniformly store and process the offline data and the real-time data to obtain application indicator data; The data application module is used to perform associated query on the application indicator data in the lake-warehouse integrated processing module according to user instructions.
2. The data acquisition and processing system based on hudi as claimed in claim 1, characterized in that: The lake-warehouse integrated processing module is specifically used for: The offline data and the real-time data are preprocessed through FlinkSQL, and dimension tables and fact tables are constructed; the dimension tables and the fact tables are aggregated and associated to construct a data wide table set; based on different preset indicators, preset dimensions and the data wide table set, an indicator data table is constructed to obtain application indicator data of different data marts.
3. The hudi-based data acquisition and processing system according to claim 1, characterized in that: The lake-warehouse integrated processing module includes a hudi table; the data acquisition module is specifically used for: The binlog logs of the multiple business databases are captured in real time through flinkcdc, the change data of each business database is collected in real time, and the data is uniformly stored in the hudi table.
4. The hudi-based data acquisition and processing system according to claim 1, characterized in that: The lake-warehouse integrated processing module includes a hudi table; the data acquisition module is also used for: For the first business scenario, the merge on read strategy is adopted to write the business data into the hudi table; wherein the first business scenario is a business scenario in which the number of write requests to the database is greater than the number of read requests; For the second business scenario, a copy on write strategy is adopted to write the business data into the hudi table; wherein, the second business scenario is a business scenario in which the amount of write requests to the database is less than the amount of read requests.
5. The hudi-based data acquisition and processing system according to claim 1, characterized in that: The lake-warehouse integrated processing module is also used to store the metadata information of the hudi table based on a preset periodic Savepoint; when accessing the Savepoint, it is mapped to the real data file corresponding to the hudi table based on the metadata information.
6. The hudi-based data acquisition and processing system according to claim 1, characterized in that: The lake-warehouse integrated processing module is also used to filter the error data through the filter preset in the Savepoint when accessing the snapshot view; wherein the error data is the data whose processing time and event time are not the same date.
7. The hudi-based data acquisition and processing system according to claim 1, characterized in that: The data application module includes a data science application unit, a visualization large screen unit, a report unit and a data analysis unit; the user instructions include a first instruction, a second instruction, a third instruction and a fourth instruction; the lake warehouse integrated processing module includes multiple heterogeneous data sources, and the multiple heterogeneous data sources include HDFS, Hudi, Hbase, ElasticSearch, and Doris; The data science application unit is used to query the first application indicator data of at least one heterogeneous data source through a unified SQL engine according to the first instruction input by the user, and write the first application indicator data into a heterogeneous database through a unified SQL interface; wherein the heterogeneous database is a database other than the multiple business databases; The visualization large screen unit is used to query the second application indicator data in the lake warehouse integrated processing module through a unified SQL engine according to the second instruction input by the user, and display the second application indicator data; The report unit is used to query the third application indicator data in the lake-warehouse integrated processing module through a unified SQL engine according to the third instruction input by the user, and generate a target report based on the third application indicator data; The data analysis unit is used to query the fourth application indicator data of at least one heterogeneous data source through the unified SQL engine according to the fourth instruction input by the user, and write the fourth application indicator data into the heterogeneous database through the unified SQL interface for data analysis to obtain data analysis results.
8. A data collection and processing method based on hudi, characterized in that: include: Collect business data from multiple business databases and synchronize the business data increments to hudi; wherein the business data includes offline data and real-time data; The offline data and the real-time data are uniformly stored and processed in Hudi to obtain application indicator data; According to the user's instructions, the application indicator data in hudi is queried in association, and the application indicator data queried by the user is visualized.
9. An electronic device comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor, characterized in that: When the processor executes the computer program, the method according to claim 8 is implemented.
10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to claim 8 is implemented.
Citation Information
Patent Citations
Multi-source heterogeneous lake and reservoir integrated data processing method, equipment and medium
CN116166757A
Lake-bin integrated structure
CN116610759A
Lake and warehouse integrated data processing method and system and storage medium
CN117251514A
Data association method, device and equipment and storage medium thereof
CN117332012A
Cited By
Data synchronization method, data synchronization system, computer equipment and medium
CN120162383A
Data processing method and device, equipment, medium and program product
CN120353847A
Data processing method, device, equipment, medium and program product
CN120353847B
Data processing method and device, equipment and storage medium
CN120849412A
