Method, device and equipment for constructing lightweight number bin based on MPP architecture

Abstract

The invention relates to the technical field of data processing, and discloses a method, device and equipment for constructing a lightweight data warehouse based on an MPP architecture, and the method comprises the steps: collecting data in a plurality of mainstream databases through a DataX collection assembly, and writing the collected data into a Dores data lake to obtain offline data; the method comprises the following steps: acquiring data of a plurality of data sources through a RoutingLoad component and a Kafka component, and writing the data into a Dores data lake to obtain real-time data; establishing a data hierarchical structure; performing calculation and analysis processing to generate a data calculation result; constructing a data asset catalog based on a metadata system of a Dores data lake to perform classified management of libraries, tables and fields; the data service interface is constructed based on the data asset directory, the BI report tool is connected with the data service interface to generate the data analysis report, and the data calculation result and the data analysis report are output to the outside, so that the construction and maintenance cost of an enterprise is reduced, and the requirements in a big data environment can be better met.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a method, device and equipment for building a lightweight data warehouse based on an MPP architecture. Background Art

[0002] With the development of technologies such as the Internet and the Internet of Things, the amount of data in enterprises is growing exponentially. Traditional data warehouses are often unable to cope with large-scale data, which is mainly reflected in the following aspects: First, traditional data warehouses often require high hardware and software costs, which is unaffordable for small and medium-sized enterprises; second, with the continuous changes in business needs, enterprises need a data warehouse solution that can respond and adjust quickly. Traditional data warehouses are usually based on batch processing and are difficult to meet the needs of real-time data processing and analysis. Traditional data warehouses have obvious deficiencies in flexibility, scalability, real-time performance, etc., and are costly and cannot cope well with the needs of the big data environment. Summary of the invention

[0003] The present application provides a method, device and equipment for building a lightweight data warehouse based on the MPP architecture, thereby reducing the construction and maintenance costs of the enterprise and being able to better meet the needs in a big data environment.

[0004] In a first aspect, the present application provides a method for constructing a lightweight data warehouse based on an MPP architecture. The method for constructing a lightweight data warehouse based on an MPP architecture includes: Collect data from multiple mainstream databases through the DataX collection component, and write the collected data into the Doris data lake to obtain offline data; Collect data from multiple data sources through RoutingLoad components and Kafka components, and write them into the Doris data lake to obtain real-time data; Establishing a data hierarchical structure of ODS layer, DWD layer, DWS layer, ADS layer and DM layer in the Doris data lake for the offline data and the real-time data; The Doris JDBC engine is used to calculate and analyze the data in the data hierarchical structure to generate data calculation results; Building a data asset catalog based on the metadata system of the Doris data lake, and classifying and managing the data calculation results by library, table, and field; A data service interface is constructed based on the data asset catalog, a data analysis report is generated by connecting to the data service interface through a BI reporting tool, and the data calculation results and the data analysis report are output externally.

[0005] A second aspect of the present application provides a device for constructing a lightweight data warehouse based on an MPP architecture, wherein the device for constructing a lightweight data warehouse based on an MPP architecture includes: The first collection module is used to collect data from multiple mainstream databases through the DataX collection component, and write the collected data into the Doris data lake to obtain offline data; The second collection module is used to collect data from multiple data sources through the RoutingLoad component and the Kafka component, and write them into the Doris data lake to obtain real-time data; Establishing a module for establishing a data hierarchical structure of an ODS layer, a DWD layer, a DWS layer, an ADS layer, and a DM layer in the Doris data lake for the offline data and the real-time data; A calculation module is used to calculate and analyze the data in the data hierarchical structure through the Doris JDBC engine to generate data calculation results; A management module is used to build a data asset catalog based on the metadata system of the Doris data lake and to classify and manage the data calculation results by library, table, and field; The output module is used to build a data service interface based on the data asset directory, connect the data service interface through a BI reporting tool to generate a data analysis report, and output the data calculation results and the data analysis report to the outside.

[0006] The third aspect of the present application provides an electronic device, comprising: a memory and at least one processor, wherein the memory stores instructions; the at least one processor calls the instructions in the memory so that the electronic device executes the above-mentioned method of building a lightweight data warehouse based on the MPP architecture.

[0007] Compared with the prior art, the present application has the following beneficial effects: through the DataX component and the RoutingLoad component, efficient collection of multi-source heterogeneous data is achieved, ensuring the real-time and integrity of the data; distributed parallel computing based on the MPP architecture significantly improves data processing efficiency, and the multi-level parallel scheduling scheme and three-dimensional data segmentation strategy optimize the utilization of computing resources; the five-layer data hierarchical architecture makes the data processing process clearer, and the unified metadata management system and standardized data service interface enhance the data asset management capabilities; the use of lightweight architecture design reduces the cost of system construction and maintenance, supports elastic expansion and facilitates rapid adjustment; by establishing a complete data quality inspection mechanism, full-process monitoring and alarm, and a multi-level data verification system, the accuracy of the data and the stability of the system operation are ensured. The present invention solves the problems of traditional data warehouses in terms of performance, cost, and scalability. The present invention: 1. Reduces the construction and maintenance costs of enterprises; 2. Has high flexibility and scalability, and can be quickly adjusted and expanded according to needs; 3. Supports real-time data stream processing, and can better meet the needs of enterprises for real-time data analysis; 4. Supports real-time data stream processing, and can better meet the needs of enterprises for real-time data analysis; 5. Through a distributed architecture and an efficient data processing engine, it can better cope with the needs in a big data environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0009] The structures, proportions, sizes, etc. illustrated in the drawings of this specification are only used to match the contents disclosed in the specification so as to facilitate understanding and reading by persons familiar with this technology. They are not used to limit the conditions under which the present invention can be implemented, and therefore have no substantive technical significance. Any structural modification, change in proportion or adjustment of size, without affecting the effects and purposes that can be achieved by the present invention, should still fall within the scope of the technical contents disclosed by the present invention.

[0010] Figure 1 It is a flowchart of a method for building a lightweight data warehouse based on an MPP architecture provided by an embodiment of the present invention; Figure 2 It is a schematic block diagram of the structure of a device for building a lightweight data warehouse based on an MPP architecture provided by an embodiment of the present invention; Figure 3 It is a schematic block diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0011] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0012] The flowcharts shown in the accompanying drawings are only examples and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may also be decomposed, combined or partially merged, so the actual execution order may change according to actual conditions.

[0013] It should also be understood that the terms used in this application specification are only for the purpose of describing specific embodiments and are not intended to limit the application. As used in this application specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0014] It should be further understood that the term "and / or" used in the specification and appended claims of this application refers to any combination and all possible combinations of one or more of the associated listed items, and includes these combinations. Figure 1 In the embodiment of the present application, an embodiment of the method for building a lightweight data warehouse based on the MPP architecture includes: Step 100: Collect data from multiple mainstream databases through the DataX collection component, and write the collected data into the Doris data lake to obtain offline data; It is understandable that the execution subject of the present application can be a device for building a lightweight data warehouse based on the MPP architecture, or a terminal or a server, which is not limited here. The embodiment of the present application is described by taking the server as the execution subject as an example.

[0015] Step 200: Collect data from multiple data sources through RoutingLoad components and Kafka components, and write them into the Doris data lake to obtain real-time data; Specifically, create a data receiving channel in the Kafka component and perform basic configuration on Kafka's Topic, including Topic name, number of Topic partitions, number of Topic copies, message storage time, and message size upper limit. Through these configurations, determine the logical partitioning of data and the message storage mechanism, thereby ensuring the scalability and reliability of the data receiving channel and providing guarantee for the efficient flow of real-time data. Configure the parameters of the data source according to the basic configuration of Topic. Configure the connection information of the data source to ensure that the data source can be smoothly connected to the Kafka component, and set the data change capture rules to accurately capture and synchronize the changes in the data source. The setting of the message sending mode determines whether the data is sent to Kafka in batch or streaming mode, and the message confirmation mechanism is used to ensure the accurate arrival of the message to avoid data loss due to network fluctuations or system failures. Through these steps, the data source acquisition configuration is obtained, which guides how to collect data from multiple data sources and process it through the Kafka component. Based on the data source acquisition configuration, configure the data loading task of the RoutingLoad component. For the data loading task, set the concurrency of the task to ensure that the system can efficiently handle multiple concurrent data requests when the data flow is large. The data batch size is also an important configuration item, which determines the amount of data loaded each time. Through reasonable configuration, a balance is achieved between data throughput and system performance. At the same time, the setting of data format conversion rules enables data from different data sources to be uniformly processed and converted into the format required by Doris Data Lake, ensuring data standardization and compatibility. For the processing of erroneous data, clarify the processing strategy of erroneous data, such as discarding, warning or fallback. These strategies help the system to automatically respond when abnormal data occurs. Through these configurations, data loading parameters are obtained to control various aspects of the data loading process. According to the data loading parameters, a data synchronization channel from Kafka to Doris Data Lake is created. To ensure efficient data consumption, set the consumer group identifier so that consumers can be effectively grouped and managed. Configure the consumer partition allocation strategy to ensure that each consumer instance can reasonably allocate Kafka partitions and improve parallel processing capabilities. At the same time, the setting of the site submission method determines the confirmation mechanism of data consumption, whether it is automatic submission or manual submission, which directly affects the stability of the system and the integrity of the data. The configuration of the consumption timeout is to prevent consumers from being blocked for a long time when processing data. Through these configurations, the synchronization channel configuration is obtained to ensure that data can be effectively synchronized from Kafka to the Doris data lake. The synchronization channel configuration is deployed to the scheduling platform, and the monitoring indicators of the task running status, resource usage, and data processing volume of the RoutingLoad component are configured to obtain the operation monitoring parameters.Running monitoring parameters help you understand the running status of the data loading task in real time, discover problems in time and take corresponding measures to ensure the continuous stability of data processing.

[0016] Step 300: Establish a data hierarchical structure of ODS layer, DWD layer, DWS layer, ADS layer and DM layer in Doris data lake for offline data and real-time data; It should be noted that offline data and real-time data from multiple data sources are merged and written into the ODS layer in the Doris data lake. In this process, the original data structure of the source system is retained, including data fields and formats from data sources such as MySQL, SQLServer, Oracle, and DAMO, so that the data in the ODS layer is consistent with the source system. The data structure of the DWD layer is constructed based on the data in the ODS layer. The DWD layer aims to establish a subject-oriented detailed table structure. Through the understanding of the business process and combined with data processing rules, business events are mapped into detailed data detailed tables. At this stage, the blood relationship of the fields is described to ensure that each data field can trace its source and enhance the transparency and interpretability of the data. Through this step, the detailed data table of the DWD layer is obtained. The detailed data table of the DWD layer is cleaned and the data standardization rules are applied to ensure the consistency of the data format; the desensitization rules are applied to sensitive fields to ensure data security, especially in scenarios involving personal privacy data. The data quality rules are uniformly configured, including the verification of the integrity, accuracy, and consistency of the data, to obtain the required and standardized detailed data. Import the standardized detailed data into the DWS layer for light aggregation. The design goal of the DWS layer is to lightly aggregate business indicators to support subsequent analysis and application scenarios. In this process, business indicators are defined, such as sales, transaction volume, etc., and the statistical dimensions and aggregation rules of these indicators are clearly set. For example, data is aggregated by dimensions such as time, region, and product category. Through this process, thematic aggregated data is obtained. On the basis of the DWS layer, the ADS layer is built based on these aggregated data to design specific application indicators. The data of the ADS layer is oriented to specific application scenarios and business calibers. Therefore, at this stage, it is necessary to design specific application scenarios and clarify the analysis requirements and calculation calibers of the business. At the same time, combined with different analysis dimensions, the business indicators are further refined to provide more direct and targeted indicator data for specific applications. Through these designs, application-oriented indicator data is obtained to directly support the applications of different business departments, such as supporting application scenarios such as financial analysis and sales forecasting. Based on the application-oriented indicator data, the data mart of the DM layer is constructed. The data mart at the DM layer is designed to provide independent data views for different business topics. Therefore, it is necessary to divide the data domains for different business topics to clarify the data contained in each topic.

[0017] Step 400: Calculate and analyze the data in the data hierarchical structure through the Doris JDBC engine to generate data calculation results; Specifically, the database connection of the Doris JDBC engine is configured to establish effective communication with the Doris data lake. The database address, authentication information, transaction isolation level, database name and other parameters are input into the connection pool to obtain the JDBC connection configuration information. The use of the connection pool improves the reuse rate of database connections, reduces the overhead caused by frequent establishment and disconnection of connections, and ensures the efficiency and stability of database connections. The setting of the transaction isolation level determines the degree of mutual influence between different transactions, thereby ensuring the consistency and isolation of data. The MPP computing parameters are set according to the JDBC connection configuration information. The core of the MPP architecture lies in its powerful parallel computing capability, so it is necessary to configure the MPP parallelism, number of data shards, query memory limit, number of computing threads and other parameters in detail. By reasonably setting the MPP parallelism and number of data shards, the computing resources of the system are fully utilized to improve the execution efficiency of queries, while the setting of the query memory limit and the number of computing threads helps to avoid excessive consumption of memory resources, thereby ensuring the stable operation of the system under high concurrency. Through the settings, the execution configuration of the MPP is obtained. Based on the MPP execution configuration, the standard SQL syntax is parsed to achieve further processing of the data. Register SQL statements such as data cleaning, data conversion, data aggregation, and data association into the SQL parser to obtain the SQL syntax tree. The SQL syntax tree is a structured representation of SQL statements. It decomposes complex SQL statements into multiple levels of expressions, allowing the system to understand and process these statements more efficiently. Through SQL parsing, ensure that the data processing logic can be correctly understood and executed by the system. Input the SQL syntax tree into the semantic analyzer to perform detailed verification of the semantics of the SQL statement. In this process, the semantic analyzer performs semantic verification on field references, table associations, aggregate functions, and calculation expressions to ensure that these references and operations are logically correct. For example, whether the field exists in the specified table, whether the table association conforms to the business logic, and whether the use of aggregate functions meets the requirements. Through these verifications, the query execution plan is obtained to ensure that the task will not fail due to semantic errors during the execution phase. Based on the query execution plan, the system's functions are expanded to support specific business needs. Register custom functions, expand the definitions of business calculation functions, statistical analysis functions, data processing functions, etc., and obtain a complete function registry. The registration of custom functions enhances the system's data processing capabilities in specific scenarios, such as introducing more complex logic in specific business calculations, or using dedicated functions in statistical analysis to improve calculation accuracy and efficiency. Build distributed computing tasks based on the function registry. In distributed computing, configure the data sharding strategy to ensure that data can be reasonably distributed among multiple computing nodes to achieve load balancing. At the same time, configure the allocation of computing nodes to maximize the use of computing resources on each node and improve computing efficiency.Set the aggregation method for intermediate results to ensure that the results of distributed computing can be correctly merged to obtain complete computing results.

[0018] Step 500: Build a data asset catalog based on the metadata system of the Doris data lake, and classify and manage the data calculation results by library, table, and field; Specifically, the metadata system of the Doris data lake is initialized and configured, and the library table structure information, field attribute information, and update cycle information of the ODS layer, DWD layer, DWS layer, ADS layer, and DM layer in the data lake are input into the metadata system to obtain the basic metadata information. The basic metadata information includes the library table design and field definition of each level of the data warehouse, and records the detailed attributes and update frequency of each data entity. Based on the basic metadata information, a hierarchical asset catalog is constructed for the data. In order to facilitate the management and positioning of different data assets, the data assets at each level are classified and configured by business domain division, data subject division, and data update method to obtain the asset classification architecture. For example, the business domain division classifies the data according to the business department to which it belongs, while the data subject division is subdivided according to different analysis topics, and the data update method helps to understand the timeliness and change frequency of the data. Asset tag management is performed on the data calculation results based on the asset classification architecture. Appropriate tags are added to each data asset, including data source tags, business attribute tags, calculation rule tags, and quality level tags. The data source tag identifies the source system of the data, the business attribute tag describes the business meaning of the data, the calculation rule tag records the calculation process of data generation, and the quality level tag is used to rate the quality of the data. These tags are input into the asset identification library to form a complete data asset tag system and obtain a unified identification of data assets. According to the data asset tags, a unified retrieval rule is constructed. Index construction and retrieval configuration are performed on the contents such as the library table field naming rules, data storage format, and metadata version information. By standardizing the index of the library table field naming rules, it is ensured that users can quickly locate the target table or field when searching for data; the index of the data storage format helps users understand the physical storage of the data and then choose the appropriate operation method; the retrieval configuration of the metadata version information enables users to query the historical version of the data at any time, which is convenient for data backtracking and change management. Through this step, a set of efficient asset retrieval solutions is obtained. The asset retrieval solution is applied to the data display module. Functional configuration is performed on the data table structure display, field attribute display, data sample display, and data statistical information display to obtain the data preview strategy. Based on the data preview strategy, a storage management system is constructed. Data storage management involves statistical analysis of data storage location, storage space occupancy, data compression ratio, and storage cost.

[0019] Step 600: Build a data service interface based on the data asset catalog, connect to the data service interface through a BI reporting tool to generate a data analysis report, and output data calculation results and data analysis reports to the outside.

[0020] Specifically, the table structure information and field information in the data asset catalog are input into the Doris query engine, and a standardized data access interface is formed by encapsulating the standard SQL query interface, custom function interface, and data service interface. Based on the data access interface specification, a unified data service view is constructed. The complex field relationships, data models, and statistical calibers in the data calculation results are standardized to obtain a standard data service model to eliminate the complexity in the data structure. A data access permission system is established based on the standard data service model to ensure the security and standardized use of data. The construction of the data access permission system involves the management of access credentials, access frequency restrictions, and data security levels for application systems. By setting access credentials, it is ensured that only authorized application systems can access the data service interface, thereby preventing unauthorized access threats to data security. At the same time, restrictions on access frequency can prevent the system from being overwhelmed by high-frequency requests and ensure the stability of the system. The control settings of the data security level can ensure that access to sensitive data is limited to specific users and systems, thereby reducing the risk of data leakage. Through these measures, a complete access control rule is obtained. The access control rule is connected to the BI reporting tool. The report type, display component, data indicator, and business scope are uniformly configured to build a standardized report development template. The configuration of report type includes common chart types, such as line charts, bar charts, pie charts, etc., while the display component involves the visualization component settings of data in the BI tool. The configuration of data indicators and business scope enables the data in each report to be consistent, ensuring that the interpretation and use of data conform to the business logic of the enterprise. Through configuration, a development template is obtained that allows developers to quickly build data analysis reports. A dynamic refresh mechanism is built based on the report development template to ensure the timeliness of data. Differentiate the configuration of data loading strategy, timed refresh strategy, and incremental update strategy. The setting of data loading strategy determines how data is obtained during the initial loading, while the timed refresh strategy is used to ensure that data is updated within a fixed time interval, which is suitable for scenarios with low real-time requirements. The incremental update strategy only updates the newly added or changed data in the case of frequent data changes, thereby reducing system overhead and improving the efficiency of data updates.

[0021] In a specific embodiment, the process of executing step 100 may specifically include the following steps: Integrate DataX components in the scheduling platform, configure the connection parameters of MySQL data source, SQL Server data source, Oracle data source, and DAMO data source, and obtain the source database configuration information; Set the DataX Reader plug-in according to the source database configuration information, configure the database connection address, port number, user name, access password, and database name, and obtain the Reader reading parameters; Set the data reading task based on the Reader reading parameters, configure the data table name, field configuration, data sharding strategy, and batch reading size to obtain the data source reading configuration; Configure the connection information of Doris Data Lake, enter the database address, port number, access account, and access password into the DataX Writer plug-in, and obtain the Writer write parameters; Set the data writing strategy based on the Writer writing parameters, configure the target data table, field mapping, writing mode, batch size, and error handling strategy to obtain the data writing configuration; Submit the data source reading configuration and data writing configuration to the scheduling platform, set the task execution cycle, concurrency, timeout, and retry strategy, and obtain the offline collection task; Monitor and configure offline collection tasks, set alarm thresholds and processing strategies for task execution status monitoring, data volume monitoring, and performance monitoring, and obtain task monitoring rules; Execute offline collection tasks according to task monitoring rules, and synchronously write data from multiple mainstream databases into the Doris data lake through the DataX component to generate offline data.

[0022] Specifically, the DataX component is integrated into the scheduling platform. The scheduling platform manages the execution of data flows, the triggering of scheduled tasks, and the retry after task failure. As a data integration tool, DataX is integrated into the scheduling platform to achieve data synchronization between different databases. After integrating the DataX component, multiple source databases are connected and configured, including MySQL, SQLServer, Oracle, and DAMO databases. The connection information of these databases, such as database address, port number, user name, access password, etc., are key parameters, and the configuration information of the source database is obtained through these configurations. According to the source database configuration information, configure the DataX Reader plug-in. The Reader plug-in is the module in DataX responsible for reading data from the data source. Set the database connection address, port number, user name, access password, and database name to establish a connection with the source database. After the Reader reading parameter configuration is completed, set specific data reading tasks based on these parameters. Configure the data table name and fields, that is, determine which tables need to read data from and the specific fields involved. For example, if the data table name is (in represents the table number), the field name is (in represents the field number), the reading task description is: ; in, Represents a data reading task, is the total number of tables, The number of fields in each table. The data sharding strategy is configured in this process to improve reading efficiency. The data sharding strategy is expressed as: ; in, Indicates the sharding strategy. The primary key field used for sharding. By sharding the data according to the primary key field, the data processing efficiency is effectively improved during the parallel reading process. At the same time, the batch reading size is configured. , determines the amount of data to be read in each batch, by setting , strike a balance between read performance and system resource consumption. Finally, the complete data source read configuration is obtained through configuration. After completing the read configuration, configure the connection information of Doris Data Lake, including the database address , port number , access account and access password , and input these parameters into the DataX Writer plug-in to obtain the Writer write parameters. For the configuration of the Writer plug-in, ensure that the data can be correctly written to the target system. Its configuration is expressed as: ; Based on the Writer write parameters, set the data write strategy. The write strategy involves the selection of the target data table, field mapping, write mode, batch size, and error handling strategy. For example, the target data table is set to , the field mapping relationship is expressed as: ; in, Represents a collection of mappings from source fields to target fields. The write mode selects append or overwrite, and the batch size Determine the amount of data to be written each time, by setting , ensuring that data can be written efficiently without putting too much pressure on the system. In addition, the error handling strategy It is used to determine the processing method when writing fails, such as setting it to retry or skip. Through the above configuration, a complete data writing configuration is obtained. Submit the data source reading configuration and data writing configuration to the scheduling platform to form a complete offline collection task. In the scheduling platform, the execution cycle of the task , concurrency , Timeout And retry strategy The execution cycle determines when the task is triggered, such as scheduled execution every day; the concurrency determines the number of tasks that can be executed simultaneously; the timeout setting is used to prevent the task from being stuck for a long time, and the retry strategy is used to automatically retry after the task fails. In order to ensure that the execution of the collection task can be monitored in time, the offline collection task is monitored and configured. The monitoring configuration includes task execution status monitoring, data volume monitoring, performance monitoring, etc., and alarm thresholds are set. and treatment strategies To form task monitoring rules. For example, set an alarm to be triggered when the task execution time exceeds a certain threshold, or notify when the amount of data read is lower than expected. The set of monitoring rules is expressed as: ; Through task monitoring rules, you can understand the running status of tasks in real time, take timely measures to deal with abnormal situations, and ensure the stability of the data synchronization process. According to the configured task monitoring rules, execute offline collection tasks, and synchronize data from multiple mainstream databases to the Doris data lake through the DataX component to generate offline data. During the entire data synchronization process, DataX reads, shards, transmits, and writes source data according to the previously set reading and writing parameters, and follows the task execution cycle and concurrency settings in the scheduling platform to ensure that tasks are completed as planned.

[0023] In a specific embodiment, the process of executing step 200 may specifically include the following steps: Create a data receiving channel in the Kafka component, set the Topic name, number of Topic partitions, number of Topic replicas, message storage time, and upper limit of message size, and obtain the basic Topic configuration; Configure the parameters of the data source end according to the basic configuration of Topic, input the data source connection information, data change capture rules, message sending mode, and message confirmation mechanism into the Kafka component, and obtain the data source collection configuration; Based on the data source acquisition configuration, configure the data loading task of the RoutingLoad component, set the task concurrency, data batch size, data format conversion rules, and error data processing strategy to obtain data loading parameters; Create a data synchronization channel from Kafka to Doris data lake according to the data loading parameters, set the consumer group identifier, consumer partition allocation strategy, site submission method, and consumer timeout to obtain the synchronization channel configuration; Deploy the synchronization channel configuration to the scheduling platform, configure monitoring indicators for the task running status, resource usage, and data processing volume of the RoutingLoad component, and obtain the running monitoring parameters; Build data quality inspection rules based on operation monitoring parameters, set inspection standards and processing procedures for data integrity, data accuracy, and data timeliness, and obtain quality control configuration; Establish task alarm rules according to quality control configuration, set alarm trigger conditions, alarm level classification, alarm information template, alarm notification channel, and obtain alarm processing solutions; Input the alarm processing plan into the scheduling platform, start the real-time collection task of the RoutingLoad component, write and monitor the quality of the collected data in real time, and generate real-time data in the Doris data lake.

[0024] Specifically, create a data receiving channel in the Kafka component. Perform basic configuration on Kafka's Topic, including setting the Topic name, number of partitions, number of replicas, message storage time, and upper limit of message size. For example, set the number of partitions to 10 so that efficient data parallel processing can be achieved through multiple partitions. At the same time, the number of replicas is set to 3 to ensure data reliability. Even if some Kafka nodes fail, the data can still be effectively restored. The message storage time is set to 7 days to ensure that even if temporary problems occur, the data can be reprocessed within a certain period of time. The upper limit of the message size is used to ensure that each message does not exceed the acceptable range to avoid affecting the stability of the system. After configuring Kafka's Topic, set the corresponding parameters on the data source according to the basic configuration of the Topic. These parameter configurations include the definition of the connection information of the data source, data change capture rules, message sending mode, and message confirmation mechanism. For example, when configuring to capture order data from a MySQL data source, set the connection information of the data source such as the connection address, user name, and password, and define the change capture rules to ensure that all new, modified, or deleted order information can be captured. The message sending mode selects the synchronous mode to ensure that each message is confirmed to be sent only after it is successfully written to Kafka, ensuring the integrity of data transmission. The message confirmation mechanism selects "all copies confirmed" to ensure that each copy of the Kafka cluster receives the message, thereby improving the reliability of the data. Configure the data loading task of the RoutingLoad component. The role of the RoutingLoad component is to efficiently load the data in Kafka into the target system, that is, the Doris data lake. In this process, set the task concurrency, data batch size, data format conversion rules, and error handling strategies. For example, if a higher concurrency is set, such as 10 concurrent tasks, parallel processing is used to speed up data loading when the data flow is large. The setting of the data batch size helps to balance the efficiency of data transmission and the use of resources. It is usually set to thousands of records so that the amount of data transmitted at one time is large enough but does not cause too much pressure on the system. At the same time, configure the data format conversion to convert the data format at the source end into a format that can be understood by the Doris data lake to ensure the availability of data in the target system. The error handling strategy is also very important. For example, when an error is found in a data record, choose to skip the record, but continue to process other data to ensure the continuity of the data loading task. Based on the above data loading parameters, create a data synchronization channel from Kafka to Doris data lake. At this stage, configure the Kafka consumer, including setting the consumer group ID, consumer partition allocation strategy, site submission method, and consumer timeout. The consumer group ID is used to classify multiple consumers into a group to ensure that each message will only be processed by one consumer in the group, so as to effectively realize parallel consumption.The allocation strategy of the consumption partition determines how consumers allocate partitions in Kafka, such as using the "polling strategy" to ensure load balancing between consumers. The site submission method determines whether to submit the offset to record the progress of consumption after the message is successfully processed. The manual submission method is selected so that the consumer can continue to consume from the last submitted position when the consumer is unexpectedly interrupted to ensure accurate data processing. The consumption timeout is used to prevent the consumer from automatically disconnecting when it is unresponsive for a long time to ensure the stable operation of the system. Complete the synchronization channel configuration and deploy it to the scheduling platform to ensure that the entire data synchronization process is effectively managed and monitored. In order to achieve efficient monitoring, the task running status, resource usage and data processing volume of the RoutingLoad component are configured with monitoring indicators. The running status monitoring understands in real time whether the data synchronization task is proceeding normally, and the resource usage monitoring ensures that the system resources are not overloaded, such as whether the CPU and memory usage are within the normal range. For the monitoring of data processing volume, it helps to determine whether the data flow meets expectations. If there is a sudden drop in data processing volume, it means that there is a problem with the data source or there is an abnormality in network transmission. After completing the monitoring configuration, build data quality inspection rules based on the monitoring parameters. The first is to check the integrity of the data to ensure that all the data read from Kafka is successfully transmitted to the Doris data lake without loss. The second is to verify the accuracy of the data. By comparing the source data with the target data, ensure that no data is damaged or mistransmitted during the transmission process. Check the timeliness of the data to ensure that the data can be consumed and stored within a reasonable time window. According to the data quality inspection rules, establish the alarm rules of the task to ensure that when the data synchronization task is abnormal, the alarm can be triggered in time and the relevant operation and maintenance personnel can be notified. The alarm rules include setting alarm trigger conditions, such as task failure or too long execution time; dividing the alarm level, and determining the priority of the alarm according to the severity of the problem, such as setting the problem of data loss as a high priority alarm; defining the template of the alarm information to ensure that the alarm information sent to the operation and maintenance personnel is detailed enough so that they can quickly locate the problem. The alarm notification channel selects SMS, email or IM tools to ensure that the relevant personnel can obtain the alarm information and respond in the first time. Enter the alarm processing plan into the scheduling platform and start the real-time collection task of the RoutingLoad component. After the collection task is started, the data in Kafka is written to the Doris data lake in real time, and the entire process is monitored in real time through the quality control and alarm rules configured previously. During the data synchronization process, the Doris data lake can achieve efficient data storage and query, which is suitable for the needs of real-time data analysis.

[0025] In a specific embodiment, the process of executing step 300 may specifically include the following steps: Merge offline data and real-time data into the ODS layer of Doris Data Lake, retain the original data structure of data sources such as MySQL, SQLServer, Oracle, and DAMO, and obtain ODS layer data that is consistent with the source system; Based on the ODS layer data, the DWD layer detailed table structure for the subject is constructed, and the business process, data processing rules, and field blood relationship are mapped to obtain the DWD layer detailed data table; Perform data cleaning according to the DWD layer detailed data table, uniformly configure data standardization rules, desensitization rules, and data quality rules to obtain standardized detailed data; Import the standardized detailed data into the DWS layer for light aggregation, define business indicators, statistical dimensions, and aggregation rules, and obtain thematic aggregated data; Build ADS layer application indicators based on thematic summary data, design specific application scenarios, business scope, and analysis dimensions, and obtain application-oriented indicator data; Build a DM layer data mart based on application-oriented indicator data, divide data domains, data granularity, and access permissions for different business themes, and obtain a thematic data mart; Based on the subject-based data mart, the scheduling dependency between data layers is established, and the data update cycle, task priority, and execution strategy are configured to obtain a data layer scheduling solution; Register the data hierarchical scheduling plan to the scheduling platform, start the hierarchical data processing task, perform ETL processing on the data at each level, and generate a data hierarchical structure.

[0026] Specifically, offline data and real-time data are merged and written into the ODS layer (Operational Data Store) of Doris Data Lake to build the initial data storage structure. Data from multiple data sources such as MySQL, SQL Server, Oracle, and DAMO are loaded simultaneously and their original data structures are maintained. Assume that the data of each data source is , , ,and , the data after merging into the ODS layer is represented as: ; in, Represents the merged ODS data, symbol Represents a union operation on different data sources. This operation ensures that the original data from different systems maintains its original structure, including field types, table structures, etc., and provides data storage consistent with the source system for easy traceability and verification. Based on the data in the ODS layer, a subject-oriented detail table structure of the DWD layer (Data WarehouseDetail) is constructed. The goal of the DWD layer is to perform detailed segmentation and processing of the business topics of the data in the ODS layer so that it can reflect specific business processes. Ensure that the evolution of each field in the data source and processing process can be traced. Based on the establishment of the DWD layer, data cleaning is performed to ensure the accuracy and consistency of the data. The cleaning process involves data standardization, desensitization, and data quality control. Let the standardization processing function be , unify the format of the data. Desensitization function Sensitive information is processed, such as the customer's ID number and phone number, and only part of it is visible to ensure data privacy. Data quality control function Used to check the integrity and consistency of data, such as checking whether the order data has corresponding customer information, whether the order amount conforms to the business logic, etc. Normalized data after cleaning It is expressed as: ; Through this process, the data in the DWD layer is ensured to be strictly standardized, desensitized and quality controlled to form high-quality detailed data. The standardized detailed data is imported into the DWS layer (Data Warehouse Service) for light aggregation. The DWS layer performs preliminary aggregation calculations on specific business indicators to make the data easier to analyze and use. At this stage, aggregate calculations are performed on specific business indicators, such as sales, order quantity, etc. Through aggregation processing, the data in the DWS layer can quickly reflect the overall operation of the business. Based on the DWS layer, the ADS layer (Application Data Store) application indicator layer is constructed. The data in the ADS layer is designed according to specific application scenarios to provide support for specific business decisions and analysis. Suppose there is a function , used to extract key business indicators required by applications from the DWS layer, such as "Customer Lifetime Value" (CLV) and "Monthly Active Users" (MAU). It is expressed as: ; By applying this function, we can obtain specific application indicator data, such as the number of active customers in a certain period of time, the average purchase value of customers, etc. After obtaining the application indicators at the ADS layer, we build the DM layer (DataMart) data mart based on the application-oriented indicator data. The DM layer data mart is tailored for different business themes and different business departments to quickly obtain the required data. At this stage, we divide the data domain according to different business themes, and set the division function as , and combined with access control strategies , then the DM layer data It is expressed as: ; in, The indicator data of the ADS layer is divided into business themes (such as sales, customers, inventory, etc.). By setting data access permissions, different business departments can only access data related to their business, ensuring data security and reasonable use. After completing the construction of each layer of data, establish scheduling dependencies between data layers to ensure that data at each layer can be updated on time. The establishment of scheduling dependencies requires detailed configuration of data update cycles, task priorities, and execution strategies. For example, if the update cycle is , the task priority is , the execution strategy is , then the scheduling scheme of data layering It is expressed as: ; By setting these parameters, we can ensure that the update of each layer of data is consistent with the processing logic of the upper layer data. For example, the data of the ODS layer is fully updated every day, while the DWD layer and DWS layer are incrementally updated every hour, and the DM layer can choose real-time or periodic updates based on actual business needs. Register to the scheduling platform and start the hierarchical data processing task. The scheduling platform is responsible for performing ETL (extraction, transformation, loading) processing on each data layer according to the set scheduling plan to ensure the flow and processing of data between various levels. For example, after extracting the original data from the ODS layer, it is cleaned and normalized, then the business subject is refined in the DWD layer, and then the aggregation calculation is performed in the DWS layer, and finally the ADS and DM layers provide business-oriented data for specific applications.

[0027] In a specific embodiment, the process of executing step 400 may specifically include the following steps: Configure the database connection for the Doris JDBC engine, input the database address, authentication information, transaction isolation level, and database name parameters into the connection pool, and obtain the JDBC connection configuration information; Set the MPP computing parameters according to the JDBC connection configuration information, set the MPP parallelism, number of data shards, query memory limit, and number of computing threads, and obtain the MPP execution configuration; Parse the standard SQL syntax based on the MPP execution configuration, register data cleaning statements, data conversion statements, data aggregation statements, and data association statements into the SQL parser, and obtain the SQL syntax tree; Input the SQL syntax tree into the semantic analyzer, perform semantic verification on field references, table associations, aggregate functions, and calculation expressions, and obtain the query execution plan; Register custom functions based on the query execution plan, extend the definition of business calculation functions, statistical analysis functions, and data processing functions, and obtain a function registry; Build distributed computing tasks based on the function registry, configure data sharding strategy, computing node allocation, and intermediate result aggregation method to obtain a distributed execution plan; Optimize queries based on distributed execution solutions, optimize execution plan rewriting, index selection, data pre-reading, and memory management to obtain query optimization strategies; Apply query optimization strategies to the data hierarchical structure, perform distributed parallel computing, perform distributed aggregation and merging processing on the calculation results, and generate data calculation results.

[0028] Specifically, configure the database connection for the Doris JDBC engine so that the system can access the Doris data lake. Enter the database address, authentication information, transaction isolation level, and database name parameters to obtain the JDBC connection configuration information. After obtaining the JDBC connection configuration information, set the MPP computing parameters to ensure that the computing tasks can be efficiently executed in parallel. The MPP computing parameters include parallelism, the number of data shards, the query memory limit, and the number of computing threads. Set the parallelism to , indicating that 16 query threads can be run in parallel; the number of data shards is , which means that the data is divided into 64 parts for parallel processing; the query memory limit is GB, each query uses up to 8 GB of memory; the number of computing threads is , that is, the entire computation process will be processed in parallel by 128 threads. Then the MPP execution configuration is expressed as: ; By setting these parameters, hardware resources can be used rationally to ensure that query tasks can be completed with optimal performance. High parallelism means that multiple queries can be executed at the same time, thereby shortening the overall response time; data sharding and the number of computing threads directly affect the parallelism and throughput of the calculation. After configuring the MPP execution parameters, the standard SQL syntax is parsed based on these configurations. The input SQL is converted into a tree structure that is easy to calculate and optimize, called a SQL syntax tree. Assume that there are the following types of SQL operations: data cleaning statements: used to remove invalid or redundant data; data conversion statements: used to convert data from one format to another, such as unifying date formats; data aggregation statements: used to calculate aggregation operations such as sums and averages; data association statements: used to connect multiple tables, such as associating order tables and customer tables. These SQL statements are recorded as The SQL parser parses these statements and forms an SQL syntax tree: ; Among them Represents the SQL syntax tree. The SQL parser SQLParser organizes different types of SQL statements into a tree structure to facilitate subsequent semantic analysis and query optimization. The SQL syntax tree is input into the semantic analyzer for semantic verification. The main task of semantic analysis is to check field references, table associations, aggregate functions, calculation expressions, etc. to ensure that the logic of SQL is correct. For example, check whether the referenced field exists in the table, whether the connection conditions between tables are reasonable, whether the use of aggregate functions meets the grammatical requirements, etc. Let the semantic analyzer be a function , then the query execution plan obtained for: ; Through semantic analysis, ensure that there are no logical errors when executing queries, such as missing required field references or incorrect table associations. After obtaining the query execution plan, extend the definition of some business requirements. At this time, register custom functions so that specific business calculation requirements can be met during execution. Custom functions include business calculation functions (such as customer lifetime value calculation), statistical analysis functions (such as standard deviation calculation), and data processing functions (such as string formatting). Suppose these custom functions are , , , then the registry of the custom function for: ; These functions are called during the query process, providing customized computing capabilities that standard SQL does not have, making the computing results more in line with specific business needs. After obtaining the query execution plan and function registry, build a distributed computing task. The construction of a distributed computing task involves how to shard data, how to allocate computing nodes, and how to aggregate intermediate results. For example, assuming the data sharding strategy is HASH ,in represents the shard key, such as "order number"; the computing nodes are allocated as RoundRobin , indicating that the allocation is done in a round-robin manner The data is sharded to each computing node; the intermediate results are summarized as follows TreeCombine ,in is the result of distributed computing. The distributed execution scheme is expressed as: ; By setting the data sharding strategy , ensuring that data can be reasonably distributed to different computing nodes; computing node allocation strategy It determines which node will calculate each shard; how to summarize the intermediate results It is used to merge the calculation results of each node to obtain the final data calculation results. Based on the distributed execution plan, query optimization is performed to improve query performance. Query optimization involves rewriting the execution plan, index selection, data pre-reading, and memory management. Suppose the optimization function is , including the following aspects: The execution plan is rewritten as , reorganize the original execution plan to improve efficiency; the index selection is , select appropriate indexes for SQL queries to speed up data retrieval; data pre-reading is , load the required data into the memory in advance; memory management is , manages memory allocation during query execution. Query optimization strategy It is expressed as: ; By applying these optimization strategies, the query execution efficiency is significantly improved, such as reducing unnecessary data scanning, improving cache hit rate, and reducing 1 / O operations. The query optimization strategy is applied to the entire data hierarchy and distributed parallel computing is performed. During the execution process, the data is sharded according to the distributed execution plan and the shards are assigned to different computing nodes. Each node independently completes its assigned data computing task, and finally merges the results through tree aggregation. Let the final calculation function be , including distributed aggregation ,in is the calculation result of each node, and the final data calculation result for: ; in, The calculation results of each node are summarized to obtain the final data calculation results. The results are further used for business analysis, such as calculating the purchasing habits of each customer and predicting future sales trends.

[0029] In a specific embodiment, the execution step applies the query optimization strategy to the data hierarchical structure, executes distributed parallel computing, and performs distributed aggregation and merging processing on the computing results. The process of generating the data computing results may specifically include the following steps: The query optimization strategy is configured for three-level task splitting according to the computing node allocation strategy of the first priority, the data sharding rule of the second priority, and the computing resource restriction of the third priority, to obtain a hierarchical distributed subtask list; Perform DAG task arrangement on the hierarchical distributed subtask list, build an execution plan through the first-level parent-child task dependency, the second-level parallelism configuration, and the third-level data partition mapping strategy to obtain a multi-level parallel scheduling solution; Based on the master-slave node allocation strategy in the multi-level parallel scheduling scheme, the data hierarchical structure is divided into three-dimensional data according to the first-level time partition, the second-level value partition and the third-level hash partition to obtain distributed intermediate results. The distributed intermediate results are input into the data consistency check module in the first phase, the data accuracy verification module in the second phase, and the outlier identification module in the third phase for triple data quality verification to obtain a multi-dimensional verification solution. According to the multi-dimensional verification scheme, a distributed computing tree is constructed, and the first-layer summary function, the second-layer sorting function, and the third-layer deduplication function are configured in a tree-like manner to obtain a hierarchical summary strategy. Execute distributed MapReduce calculations based on a hierarchical aggregation strategy, and process the first-stage Map results, the second-stage Shuffle results, and the third-stage Reduce results in a streaming manner according to preset rules to obtain multi-stage merged data; A three-level verification mechanism of first-level integrity verification, second-level calculation accuracy verification, and third-level business logic verification is applied to multi-stage merged data to obtain a progressive verification scheme; According to the progressive verification scheme, the first round of data sampling verification, the second round of full data verification and the third round of business rule verification are performed on the multi-stage merged data. After passing all verifications, it is written into the Doris data lake to generate data calculation results.

[0030] Specifically, the query optimization strategy is configured to split the tasks according to three priorities to build a hierarchical distributed subtask list. The first priority is determined to be the allocation strategy of the computing node. Suppose each computing node is ,in , is the total number of computing nodes. The computing node allocation strategy defines which tasks are performed by which nodes. The second priority is the data sharding rule. Suppose the data set is , the data is divided into Shards , then the sharding rule It is expressed as: ; The third priority is computing resource constraints. The available computing resources are , such as the maximum limit of memory and CPU. The hierarchical distributed subtask list represents the set of shard tasks assigned to each node, denoted as , defined as: ; Based on the hierarchical distributed subtask list, DAG (directed acyclic graph) tasks are arranged, and the execution plan is built by defining the dependencies between parent and child tasks, configuring parallelism and data partition mapping strategies. The first level of DAG task arrangement defines the dependencies of tasks, such as tasks Required tasks It can only start after completion, which is represented by The second level is the configuration of parallelism. Set the parallelism to , determines the number of tasks that can be executed simultaneously, The third level is the data partition mapping strategy. By partition mapping function Distribute to different nodes to obtain partitioned data sets , then the multi-level parallel scheduling scheme is expressed as: ; Through the multi-level scheduling scheme, the system can coordinate the dependencies and resource allocation between multiple tasks to ensure that each task can be executed in parallel on the appropriate node. Based on the multi-level parallel scheduling scheme, the data segmentation calculation is performed. The data is segmented in three dimensions according to the first-level time partition, the second-level value partition, and the third-level hash partition to balance the data load in different dimensions. Let the time partition function be , the data Divide into multiple shards according to time; the numerical partition function is , divide the data by numerical range; the hash partition function is , based on the hash key Evenly distribute the data. Result of three-dimensional data segmentation It is expressed as: ; The three-dimensional data segmentation strategy ensures the balanced distribution of data in multiple dimensions such as time, value, and hash, reduces the possibility of data skew, and improves the overall computing performance. The quality of the distributed intermediate results is verified. Data quality verification includes data consistency check in the first stage, data accuracy verification in the second stage, and outlier identification in the third stage. Suppose the data consistency check function is , used to check the consistency between multiple data shards, for example, whether the data in the same time period is consistent in each node. The data accuracy verification function is , which is used to ensure the numerical accuracy of the calculation results, such as whether the summary value is correct. The outlier identification function is , used to detect outliers in the data, such as abnormal large transactions. Final data quality verification results It is expressed as: ; Through triple data quality verification, the system can ensure the integrity, accuracy and consistency of data in different dimensions, ensuring the reliability of subsequent calculation results. After passing the quality verification, a distributed computing tree is built based on the verification scheme to perform further data processing. The first layer of the distributed computing tree includes summary functions , used to aggregate and summarize data; the second layer includes sorting functions , sort the data by a specified field; the third layer includes the deduplication function , used to remove duplicate records. Distributed computing tree It is expressed as: ; Through the hierarchical configuration of the distributed computing tree, data can be efficiently summarized, sorted, and deduplicated, further improving the quality and availability of data. After obtaining the results of the distributed computing tree, distributed MapReduce calculations are performed based on the hierarchical summary strategy. MapReduce calculations are divided into three stages: the Map stage is used to map the input data and divide the data into key-value pairs; the Shuffle stage groups these key-value pairs; and the Reduce stage summarizes the grouped data. Let the Map function be , the Shuffle function is , the Reduce function is , then multi-stage merging data It is expressed as: ; In this step, the data is initially segmented through the Map function, the data with the same key is grouped in the Shuffle phase, and finally the data is aggregated through the Reduce phase to obtain the final merged data. In order to ensure the quality of the merged data, a three-level verification mechanism is applied to it. The first level is data integrity verification, which checks whether the data of each shard is complete; the second level is calculation accuracy verification, which ensures that the results of the aggregate calculation are consistent with expectations; the third level is business logic verification, which ensures that the data complies with business rules. Suppose the integrity verification function is , the accuracy verification function is , the business logic verification function is , the progressive verification scheme is expressed as: ; Through the progressive three-level verification, the system can ensure the data quality of the final result to the greatest extent. According to the progressive verification scheme, multiple rounds of verification are performed on the data merged in multiple stages. The first is data sampling verification, which extracts some data samples for quality inspection; the second is full data verification, which verifies the consistency and accuracy of all data; and the last is the verification of business rules to ensure that each record meets the business logic requirements. After passing all verifications, the data is finally written into the Doris data lake to generate the final data calculation results. .

[0031] In a specific embodiment, the process of executing step 500 may specifically include the following steps: Initialize the configuration of the metadata system of Doris Data Lake, input the library table structure information, field attribute information, and update cycle information of the ODS layer, DWD layer, DWS layer, ADS layer, and DM layer into the metadata system, and obtain the basic metadata information; Build a hierarchical asset catalog based on metadata basic information, and perform multi-dimensional classification configuration on the business domain division, data subject division, and data update method of data assets at each level to obtain an asset classification architecture; Perform asset tag management on data calculation results based on the asset classification architecture, input data source tags, business attribute tags, calculation rule tags, and quality level tags into the asset identification library to obtain data asset tags; Build unified search rules based on data asset tags, perform index construction and search configuration on library table field naming rules, data storage format, and metadata version information, and obtain asset search solutions; Apply the asset retrieval solution to the data display module, configure the functions of data table structure display, field attribute display, data sample display, and data statistical information display to obtain the data preview strategy; Build a storage management system based on data preview strategy, conduct statistical analysis on data storage location, storage space occupancy, data compression ratio, and storage cost, and obtain storage information configuration; Establish a hierarchical authority management mechanism based on the storage information configuration, classify and set rules for data access rights, data modification rights, data management rights, and data export rights, and obtain an authority management plan; Apply the permission management solution to the access control of data calculation results, perform permission verification and access control on data access applications of users of different levels, and generate a data asset catalog.

[0032] Specifically, the metadata system is initialized and configured, and all basic information such as the library table structure information, field attribute information, and data update cycle of the ODS layer, DWD layer, DWS layer, ADS layer, and DM layer is entered into the metadata system. This information together constitutes the basis of metadata. For example, the ODS layer where the original data is stored, the DWD layer after cleaning, and the information of the data layer summarized in different dimensions are all integrated into the system to track and manage the entire life cycle of the data. Based on this basic information, the metadata is hierarchically divided to build a hierarchical asset catalog. The division of the asset catalog is multi-dimensional, for example, by business field, financial data, sales data, and customer data are divided into different business domains; by data subject, for example, in sales data, there are subjects such as orders, customers, and commodities. Multi-level division enables managers to more intuitively understand how data is organized and help better control the flow and access of data. After building a hierarchical asset catalog, add asset tags to each data set. Each data set will have multiple tags to describe its attributes, such as whether its data source is generated by an internal system or from an external partner; what category does its business attribute belong to, such as finance, sales, or customers; whether its calculation rules are obtained through simple aggregation or generated through complex model calculation; what is the quality of its data, whether it is high-quality and accurate data or data that needs further cleaning. Through these tags, data assets are managed and tracked to ensure consistency and reliability during data storage, calculation, and use. Based on asset tags, a unified search rule is established. For example, a specific data set can be found by library table name, field name, or version information. After applying the search scheme to the data display module, users can view the data in a variety of ways through the system. The data display module needs to support viewing the structure of the data table, field attributes, specific data samples, and data statistics. Build a storage management system to effectively manage the storage location, storage space occupancy, compression ratio, and storage cost of data. By counting and analyzing the distribution of data on different storage nodes, reasonable management of hot data and cold data is made. For example, frequently accessed data is stored in storage media with faster access speeds, while historical and less frequently accessed data is stored in storage media with lower costs. At the same time, adjustments are made based on the data compression ratio to reduce the overall storage cost. A hierarchical authority management mechanism is established to ensure that different types of data access are performed within a safe range. The authority management mechanism needs to classify data access rights, data modification rights, data management rights, and data export rights, and set specific rules for each authority. For example, ordinary business personnel can only view data but cannot modify it; data administrators have the authority to modify and manage data. Through the refined management of permissions, the safe use of data throughout the enterprise is ensured to avoid unnecessary risks and data leakage.Apply the permission management scheme to the access control of data calculation results. Perform permission verification on the user's access request to ensure that only users with corresponding permissions can access the relevant data. For example, if an analyst needs to view a data set, the system first checks whether his permission level allows it. If the permission is sufficient, the user can access the data; if the permission is insufficient, access will be denied. Finally, a data asset catalog is generated.

[0033] In a specific embodiment, the process of executing step 600 may specifically include the following steps: Input the table structure information and field information in the data asset catalog into the Doris query engine, encapsulate the standard SQL query interface, custom function interface, and data service interface to obtain the data access interface specification; Build a unified data service view based on the data access interface specification, normalize the complex field relationships, data models, and statistical calibers of data calculation results, and obtain a standard data service model; Establish a data access permission system based on the standard data service model, manage and set application system access credentials, access frequency restrictions, and data security levels to obtain access control rules; Connect access control rules with BI reporting tools, configure report types, display components, data indicators, and business calibers in a unified manner, and obtain report development templates; Build a dynamic refresh mechanism based on the report development template, and configure the data loading strategy, timed refresh strategy, and incremental update strategy differently to obtain the data update strategy; Set data service quality parameters according to the data update strategy, set thresholds for service response time, concurrent access volume, and data throughput, and obtain service quality standards; Apply service quality standards to the operation monitoring system, perform full-link monitoring of interface call status, data synchronization status, and service operation status, obtain operation monitoring parameters, and output data analysis reports and data calculation results based on the operation monitoring parameters.

[0034] Specifically, the table structure information and field information in the data asset catalog are input into the Doris query engine, and various data interfaces are encapsulated. The data asset catalog contains basic information such as the structure and field attributes of various tables. This information is loaded into the Doris query engine, and then the standard SQL query interface, custom function interface, and data service interface are encapsulated to form a unified data access interface specification. Ensure that all access to data complies with the unified interface standard to simplify query operations and improve the consistency and security of data use. Suppose the table structure information is , the field information is , the encapsulated data access interface specification is expressed as: ; in, It is an encapsulation function that encapsulates table structure information and field information through a combination of various query interfaces to obtain standardized access interface specifications. . Build a unified data service view according to the data access interface specification. The construction of the data service view is mainly to normalize the complex field relationships, data models, statistical calibers, etc. involved in the data calculation results to form a standard data service model, and convert various complex underlying data relationships into concise and easy-to-use view forms for business systems to query and use. For example, for the data of an order system, there is a complex many-to-one relationship between the order table and the customer table. Through the construction of the service view, these relationships are mapped to higher-level business indicators, such as the average customer order amount, customer order volume, etc. Assume that the data calculation result is , the normalization function is , Standard Data Service Model It is expressed as: ; in, Standardize the complex relationships, models, and business definitions in data calculation results to ensure that different business users get consistent and understandable results when using data. Establish a data access permission system based on the standard data service model to ensure that data access meets the enterprise's security standards. The establishment of a data access permission system requires the management of application system access credentials, while limiting access frequency and data security levels. Through this permission system, ensure that only authorized users can access the corresponding data, avoiding illegal access and abuse of data. Suppose the access credential is , the access frequency is limited to , the data security level is , access control rules It is expressed as: ; in, It is an access control function that combines access credentials, frequency limits, and security levels to formulate specific access control rules. These rules ensure the security and compliance of data when it is accessed. Connect the access control rules with the BI (business intelligence) reporting tool. The goal of the connection is to achieve a visual display of the data and ensure that the displayed content complies with the access control rules. Unify the configuration of report types, display components, data indicators, and business calibers in the BI reporting tool to form a report development template, so that users from different business departments can use the same template to generate reports to ensure the consistency and standardization of data display. Establish a dynamic refresh mechanism based on the report development template. The dynamic refresh mechanism is used to ensure the timeliness of data display, especially for frequently updated data, the system requires an effective data loading strategy. The scheduled refresh strategy is used to refresh the data at a predetermined time point, while the incremental update strategy is used to process data that only needs to update the changed parts. Assume that the data loading strategy is , the timing refresh strategy is , the incremental update strategy is , data update strategy It is expressed as: ; Through the refresh mechanism, the data in the report can be kept up to date in the shortest time possible to ensure that the data seen by business personnel is accurate and timely. For example, at the end of the financial month, the data is automatically updated according to the set refresh cycle to ensure that the financial personnel see the latest data of the month when generating reports. Based on the data update strategy, set data service quality parameters to ensure that the data service meets certain standards in terms of performance and reliability. Service quality parameters include settings for service response time, concurrent access volume, and data throughput, such as ensuring that data queries return results within 3 seconds, supporting simultaneous multi-user access, and a certain bandwidth throughput. By setting these parameters, ensure that data access and services are not affected during peak hours to meet business needs. After applying the service quality standards to the operation monitoring system, the entire data access link is monitored throughout the link. The monitoring scope includes the call status of the interface, the status of data synchronization, and the overall operation status of the service. For example, by monitoring the call status of the interface, it is found whether some interfaces frequently fail and repair them in a timely manner; by monitoring the data synchronization status, it is ensured that there is no loss or delay in the transmission of data from the data lake to the analysis platform. Operation monitoring parameters include the number of calls to each interface, average response time, data synchronization success rate, etc. Ultimately, these monitoring information are used to output data analysis reports and data calculation results to help managers understand and evaluate the health status of the system and data usage.

[0035] The above describes the method for constructing a lightweight data warehouse based on the MPP architecture in the embodiment of the present application. The following describes the device for constructing a lightweight data warehouse based on the MPP architecture in the embodiment of the present application. Figure 2 In the embodiment of the present application, an embodiment of a device for constructing a lightweight data warehouse based on the MPP architecture includes: The first collection module 11 is used to collect data from multiple mainstream databases through the DataX collection component, and write the collected data into the Doris data lake to obtain offline data; The second collection module 12 is used to collect data from multiple data sources through the RoutingLoad component and the Kafka component, and write them into the Doris data lake to obtain real-time data; Establishing module 13, used to establish the data hierarchical structure of ODS layer, DWD layer, DWS layer, ADS layer and DM layer in Doris data lake for offline data and real-time data; The calculation module 14 is used to calculate and analyze the data in the data hierarchical structure through the Doris JDBC engine to generate data calculation results; Management module 15, used to build a data asset catalog based on the metadata system of Doris data lake, and classify and manage the data calculation results by library, table, and field; The output module 16 is used to build a data service interface based on the data asset directory, connect the data service interface through the BI report tool to generate a data analysis report, and output data calculation results and data analysis reports to the outside.

[0036] See also Figure 3 , Figure 3 This is a schematic block diagram of the structure of an electronic device 300 provided in an embodiment of the present application. The electronic device 300 includes a processor 301 and a memory 302. The processor 301 and the memory 302 are connected via a device bus 303, wherein the memory 302 may include a non-volatile storage medium and an internal memory.

[0037] The non-volatile storage medium can store a computer program. The computer program includes program instructions, and when the program instructions are executed by the processor 301, the processor 301 can execute any of the above methods for building a lightweight data warehouse based on the MPP architecture.

[0038] The processor 301 is used to provide computing and control capabilities to support the operation of the entire electronic device 300 .

[0039] The internal memory provides an environment for the operation of the computer program in the non-volatile storage medium. When the computer program is executed by the processor 301, the processor 301 can execute any of the above-mentioned methods for building a lightweight data warehouse based on the MPP architecture.

[0040] Those skilled in the art will understand that Figure 3 The structure shown in the figure is only a block diagram of a partial structure related to the present application scheme, and does not constitute a limitation on the electronic device 300 involved in the present application scheme. The specific electronic device 300 may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0041] It should be understood that the processor 301 may be a central processing unit (CPU), and the processor 301 may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0042] It should be noted that technical personnel in the relevant field can clearly understand that, for the convenience and simplicity of description, the specific working process of the electronic device 300 described above can refer to the corresponding process of the aforementioned method for constructing a lightweight data warehouse based on the MPP architecture, and will not be repeated here.

[0043] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0044] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions to enable an electronic device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc., various media that can store program codes.

[0045] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for building a lightweight data warehouse based on MPP architecture, characterized in that: The method comprises: Collect data from multiple mainstream databases through the DataX collection component, and write the collected data into the Doris data lake to obtain offline data; Collect data from multiple data sources through RoutingLoad components and Kafka components, and write them into the Doris data lake to obtain real-time data; Establishing a data hierarchical structure of ODS layer, DWD layer, DWS layer, ADS layer and DM layer in the Doris data lake for the offline data and the real-time data; The Doris JDBC engine is used to calculate and analyze the data in the data hierarchical structure to generate data calculation results; Building a data asset catalog based on the metadata system of the Doris data lake, and classifying and managing the data calculation results by library, table, and field; A data service interface is constructed based on the data asset catalog, a data analysis report is generated by connecting to the data service interface through a BI reporting tool, and the data calculation results and the data analysis report are output externally.

2. The method for constructing a lightweight data warehouse based on the MPP architecture according to claim 1, characterized in that: The data from multiple mainstream databases are collected through the DataX collection component, and the collected data is written into the Doris data lake to obtain offline data, including: Integrate DataX components in the scheduling platform, configure the connection parameters of MySQL data source, SQL Server data source, Oracle data source, and DAMO data source, and obtain the source database configuration information; Set the DataX Reader plug-in according to the source database configuration information, configure the database connection address, port number, user name, access password, and database name, and obtain the Reader reading parameters; Based on the Reader reading parameters, a data reading task is set, and the data table name, field configuration, data sharding strategy, and batch reading size are configured to obtain a data source reading configuration; Configure the connection information of the Doris data lake, input the database address, port number, access account, and access password into the DataX Writer plug-in, and obtain the Writer write parameters; Set a data writing strategy based on the Writer writing parameters, configure the target data table, field mapping, writing mode, batch size, and error handling strategy, and obtain a data writing configuration; Submit the data source reading configuration and the data writing configuration to the scheduling platform, set the task execution cycle, concurrency, timeout time, and retry strategy, and obtain an offline collection task; Perform monitoring configuration on the offline acquisition task, set alarm thresholds and processing strategies for task execution status monitoring, data volume monitoring, and performance monitoring, and obtain task monitoring rules; The offline collection task is executed according to the task monitoring rules, and the data in the multiple mainstream databases are synchronously written into the Doris data lake through the DataX component to generate offline data.

3. The method for constructing a lightweight data warehouse based on the MPP architecture according to claim 2, characterized in that: The data from multiple data sources are collected through the RoutingLoad component and the Kafka component, and written into the Doris data lake to obtain real-time data, including: Create a data receiving channel in the Kafka component, set the Topic name, number of Topic partitions, number of Topic copies, message storage time, and upper limit of message size, and obtain the basic Topic configuration; According to the Topic basic configuration, the data source end is configured with parameters, and the data source connection information, data change capture rules, message sending mode, and message confirmation mechanism are input into the Kafka component to obtain the data source collection configuration; Based on the data source acquisition configuration, configure the data loading task of the RoutingLoad component, set the task concurrency, data batch size, data format conversion rules, and error data processing strategy to obtain data loading parameters; Create a data synchronization channel from Kafka to the Doris data lake according to the data loading parameters, set the consumer group identifier, consumer partition allocation strategy, site submission method, and consumer timeout, and obtain the synchronization channel configuration; Deploy the synchronization channel configuration to the scheduling platform, configure monitoring indicators for the task running status, resource usage, and data processing volume of the RoutingLoad component, and obtain operation monitoring parameters; Building data quality inspection rules based on the operation monitoring parameters, setting inspection standards and processing procedures for data integrity, data accuracy, and data timeliness, and obtaining quality control configuration; Establish task alarm rules according to the quality control configuration, set alarm trigger conditions, alarm level division, alarm information template, alarm notification channel, and obtain alarm processing solutions; The alarm processing solution is input into the scheduling platform, the real-time collection task of the RoutingLoad component is started, the collected data is written in real time and the quality is monitored, and real-time data is generated in the Doris data lake.

4. The method for constructing a lightweight data warehouse based on the MPP architecture according to claim 3 is characterized in that: The data hierarchical structure of establishing the ODS layer, the DWD layer, the DWS layer, the ADS layer and the DM layer in the Doris data lake for the offline data and the real-time data includes: The offline data and the real-time data are merged and written into the ODS layer of the Doris data lake, and the original data structure of data sources such as MySQL, SQLServer, Oracle and DAMO is retained to obtain ODS layer data consistent with the source system; Based on the ODS layer data, a subject-oriented detailed table structure of the DWD layer is constructed, and the business process, data processing rules, and field blood relationship are mapped to obtain a DWD layer detailed data table; Perform data cleaning processing according to the DWD layer detailed data table, uniformly configure data standardization rules, desensitization rules, and data quality rules to obtain standardized detailed data; Import the standardized detailed data into the DWS layer for light aggregation, define business indicators, statistical dimensions, and aggregation rules, and obtain thematic aggregated data; Based on the thematic summary data, ADS layer application indicators are constructed, and specific application scenarios, business scopes, and analysis dimensions are designed to obtain application-oriented indicator data; Constructing a DM layer data mart based on the application-oriented indicator data, dividing data domains, data granularity, and access permissions for different business themes, and obtaining a thematic data mart; Based on the thematic data mart, scheduling dependencies between data layers are established, data update cycles, task priorities, and execution strategies are configured to obtain a data layer scheduling solution; The data hierarchical scheduling scheme is registered in the scheduling platform, the hierarchical data processing task is started, and the ETL processing is performed on the data of each level to generate a data hierarchical structure.

5. The method for constructing a lightweight data warehouse based on the MPP architecture according to claim 4 is characterized in that: The Doris JDBC engine is used to calculate and analyze the data in the data hierarchical structure to generate data calculation results, including: Perform database connection configuration on the Doris JDBC engine, input the database address, authentication information, transaction isolation level, and database name parameters into the connection pool, and obtain JDBC connection configuration information; According to the JDBC connection configuration information, MPP computing parameters are set, MPP parallelism, number of data shards, query memory limit, and number of computing threads are set to obtain MPP execution configuration; Parse the standard SQL grammar based on the MPP execution configuration, register the data cleaning statements, data conversion statements, data aggregation statements, and data association statements into the SQL parser, and obtain the SQL grammar tree; Input the SQL syntax tree into a semantic analyzer, perform semantic verification on field references, table associations, aggregate functions, and calculation expressions, and obtain a query execution plan; Register a custom function based on the query execution plan, extend and define the business calculation function, statistical analysis function, and data processing function, and obtain a function registration table; Construct a distributed computing task according to the function registry, configure the data sharding strategy, computing node allocation, and intermediate result aggregation method, and obtain a distributed execution plan; Based on the distributed execution scheme, query optimization is performed, and execution plan rewriting, index selection, data pre-reading, and memory management are optimized and configured to obtain a query optimization strategy; The query optimization strategy is applied to the data hierarchical structure, distributed parallel computing is performed, and distributed aggregation and merging processing is performed on the computing results to generate data computing results.

6. The method for constructing a lightweight data warehouse based on the MPP architecture according to claim 5, characterized in that: The step of applying the query optimization strategy to the data hierarchical structure, executing distributed parallel computing, and performing distributed aggregation and merging processing on the computing results to generate data computing results includes: The query optimization strategy is configured to split the three-level tasks according to the computing node allocation strategy of the first priority, the data sharding rule of the second priority, and the computing resource restriction of the third priority, to obtain a hierarchical distributed subtask list; Performing DAG task arrangement on the hierarchical distributed subtask list, building an execution plan through the first-level parent-child task dependency, the second-level parallelism configuration and the third-level data partition mapping strategy, and obtaining a multi-level parallel scheduling solution; Based on the master-slave node allocation strategy in the multi-level parallel scheduling scheme, the data hierarchical structure is divided and calculated in three dimensions according to the first-level time partition, the second-level value partition and the third-level hash partition to obtain a distributed intermediate result; The distributed intermediate results are input into the data consistency check module of the first stage, the data accuracy verification module of the second stage, and the outlier identification module of the third stage to perform triple data quality verification to obtain a multi-dimensional verification solution; Construct a distributed computing tree according to the multi-dimensional verification scheme, perform tree-shaped combination configuration on the first-layer summary function, the second-layer sorting function, and the third-layer deduplication function, and obtain a hierarchical summary strategy; Execute distributed MapReduce calculation based on the hierarchical aggregation strategy, perform streaming processing on the first-stage Map result, the second-stage Shuffle result and the third-stage Reduce result according to preset rules, and obtain multi-stage merged data; Applying a three-level verification mechanism of first-level integrity verification, second-level calculation accuracy verification, and third-level business logic verification to the multi-stage merged data to obtain a progressive verification scheme; According to the progressive verification scheme, the multi-stage merged data is subjected to the first round of data sampling verification, the second round of full data verification and the third round of business rule verification. After passing all verifications, the data is written into the Doris data lake to generate data calculation results.

7. The method for constructing a lightweight data warehouse based on the MPP architecture according to claim 6, characterized in that: The metadata system based on the Doris data lake builds a data asset catalog and classifies and manages the data calculation results by library, table, and field, including: Initialize and configure the metadata system of the Doris data lake, input the library table structure information, field attribute information, and update cycle information of the ODS layer, DWD layer, DWS layer, ADS layer, and DM layer into the metadata system, and obtain metadata basic information; Build a hierarchical asset catalog based on the metadata basic information, perform multi-dimensional classification configuration on the business domain division, data subject division, and data update method of each level of data assets, and obtain an asset classification architecture; Perform asset tag management on the data calculation results based on the asset classification framework, input the data source tag, business attribute tag, calculation rule tag, and quality level tag into the asset identification library to obtain a data asset tag; Build a unified search rule based on the data asset tag, perform index construction and search configuration on the library table field naming rules, data storage format, and metadata version information, and obtain an asset search solution; Apply the asset retrieval scheme to the data display module, configure the functions of data table structure display, field attribute display, data sample display, and data statistical information display, and obtain a data preview strategy; Building a storage management system based on the data preview strategy, performing statistical analysis on data storage location, storage space occupancy, data compression ratio, and storage cost, and obtaining storage information configuration; Establish a hierarchical authority management mechanism according to the storage information configuration, classify data access rights, data modification rights, data management rights, and data export rights, and set rules to obtain an authority management plan; The authority management scheme is applied to the access control of the data calculation results, and authority verification and access control are performed on data access applications of users of different levels to generate a data asset catalog.

8. The method for constructing a lightweight data warehouse based on the MPP architecture according to claim 7, characterized in that: The data service interface is constructed based on the data asset catalog, a data analysis report is generated by connecting the data service interface through a BI report tool, and the data calculation results and the data analysis report are output externally, including: Input the table structure information and field information in the data asset catalog into the Doris query engine, encapsulate the standard SQL query interface, custom function interface, and data service interface to obtain the data access interface specification; Construct a unified data service view according to the data access interface specification, normalize the complex field relationship, data model, and statistical caliber of the data calculation results, and obtain a standard data service model; Establish a data access permission system based on the standard data service model, manage and set application system access credentials, access frequency restrictions, and data security levels, and obtain access control rules; Connect the access control rules with the BI reporting tool, uniformly configure the report type, display components, data indicators, and business caliber, and obtain a report development template; Building a dynamic refresh mechanism based on the report development template, differentially configuring the data loading strategy, the timed refresh strategy, and the incremental update strategy to obtain the data update strategy; According to the data update strategy, data service quality parameters are set, and thresholds are set for service response time, concurrent access volume, and data throughput to obtain service quality standards; The service quality standard is applied to the operation monitoring system to perform full-link monitoring of the interface call status, data synchronization status, and service operation status to obtain operation monitoring parameters, and the data analysis report and the data calculation results are output based on the operation monitoring parameters.

9. A device for constructing a lightweight data warehouse based on an MPP architecture, characterized in that: The method for constructing a lightweight data warehouse based on the MPP architecture according to any one of claims 1 to 8 is used, and the device for constructing a lightweight data warehouse based on the MPP architecture comprises: The first collection module is used to collect data from multiple mainstream databases through the DataX collection component, and write the collected data into the Doris data lake to obtain offline data; The second collection module is used to collect data from multiple data sources through the RoutingLoad component and the Kafka component, and write them into the Doris data lake to obtain real-time data; Establishing a module for establishing a data hierarchical structure of an ODS layer, a DWD layer, a DWS layer, an ADS layer, and a DM layer in the Doris data lake for the offline data and the real-time data; A calculation module is used to calculate and analyze the data in the data hierarchical structure through the Doris JDBC engine to generate data calculation results; A management module is used to build a data asset catalog based on the metadata system of the Doris data lake and to classify and manage the data calculation results by library, table, and field; The output module is used to build a data service interface based on the data asset catalog, connect the data service interface through a BI reporting tool to generate a data analysis report, and output the data calculation results and the data analysis report to the outside.

10. An electronic device, characterized in that: The electronic device comprises: a memory and at least one processor, wherein instructions are stored in the memory; The at least one processor calls the instructions in the memory so that the electronic device executes the method for building a lightweight data warehouse based on the MPP architecture as described in any one of claims 1-8.

Images