Establishment method based on Apache Dores lake and warehouse data analysis platform

Abstract

The invention belongs to the technical field of data management, and particularly relates to a method for constructing a lake and warehouse data analysis platform based on Apache Dores, which comprises the steps of configuring a data source access module, deploying an Apache Dores analysis engine, configuring a data management module, configuring an SQL query and management module, deploying a BI and visual integration module and configuring an operation and maintenance monitoring module. According to the method, S1, dynamic parallelism degree adjustment based on the data arrival rate, S2, barrel division strategy optimization based on the gradient, S4, resource allocation based on the cost model and S7, fault self-healing based on anomaly detection form a complete intelligent regulation and control system, compared with a traditional static configuration scheme, the platform resource utilization rate is improved, and automatic adjustment can be achieved according to service fluctuation.

Description

Technical Field

[0001] This invention belongs to the field of data management technology, specifically relating to a method for building a data analysis platform based on Apache Doris Lakeware. Background Technology

[0002] As enterprises deepen their digital transformation, existing data analytics platforms face a severe trade-off between performance, real-time capabilities, and cost. Traditional data warehouse technology stacks are complex, requiring the integration of more than ten components such as Kafka, Spark, and Presto, leading to high operational and maintenance difficulties and frequent data silos. While real-time OLAP engines such as ClickHouse and Druid offer excellent performance, they lack native lake warehouse integration capabilities, resulting in significant issues with redundant data storage and cross-source query latency.

[0003] Apache Doris, with its MPP architecture, vectorized execution engine, and MySQL protocol compatibility, offers a new path to simplify the technology stack. However, its deep integration with data lake systems such as Hudi, Iceberg, and Delta Lake still lacks standardized engineering methods, particularly in core areas like metadata consistency assurance, multi-tenant resource isolation, and intelligent fault self-healing. Existing technologies do not provide quantifiable and replicable platform construction methods; module configuration relies on manual experience and lacks automated optimization mechanisms based on dynamic parameters such as data arrival rate, skewness, and query cost, making it difficult to achieve end-to-end intelligent management from data access and lifecycle management to monitoring and alerting. Therefore, a systematic construction method is urgently needed to transform Doris' core capabilities into an enterprise-ready platform with adaptive partitioning, elastic resource scheduling, and fault self-healing, solving the challenges of data governance and service quality assurance in lake warehouse scenarios. Summary of the Invention

[0004] To address the shortcomings of existing technologies where the configuration of each module relies on manual experience when building a platform based on Apache Doris, and lacks an automated optimization mechanism based on dynamic parameters such as data arrival rate, skewness, and query cost, making it difficult to guarantee the reliability of data analysis results, this invention provides a method for building a data analysis platform based on Apache Doris Lakehouse and a storage medium.

[0005] In a first aspect, the present invention provides a method for building a data analysis platform based on the Apache Doris lakehouse, comprising: S1. Configure the data source access module to obtain the raw multi-source data of the data lake system and dynamically adjust the parallelism parameters of the stream processing; deploy the Apache Doris analysis engine to build an MPP distributed architecture of FE / BE nodes, and import the raw data into Doris internal storage or keep it in the external data lake. S2. Configure the data management module to perform lifecycle management on the data imported in step S1; S3. Use the data lake federated query module to establish a metadata mapping and query acceleration channel between Doris and the data lake system, and force a refresh of metadata through version number verification. S4. Configure the SQL query and management module, provide SQL query interface and MySQL protocol compatibility layer, build multi-tenant resource queue and perform query resource allocation and isolation based on cost prediction factor; S5 deploys a BI and visualization integration module, using the Doris analytics engine as a data source to connect with business intelligence tools, providing graphical data exploration and reporting capabilities.

[0006] Further improvements to this technical solution include the following method for obtaining the raw multi-source data of the data lake system in step S1: S111. Configure the data source access module, establish a unified configuration library containing data source type identifiers, connection addresses, and authentication credentials, and determine the access weight of each data source through data source priority scoring; the calculation formula for data source priority scoring is: ; in, Rate the priority of the i-th data source; These are preset data source weight coefficients based on business importance. This is the connection stability coefficient; S112. Collect raw data streams in real time from business databases, message queues, log systems, data lakes, and external APIs, and prioritize the data sources configured in step S111. As a basis for data sampling frequency, the data arrival rate of each data source within a unit time window is statistically analyzed, and the fluctuation coefficient of data flow is calculated to identify abnormal peaks. S113. Based on the data arrival rate and fluctuation coefficient obtained in step S112, dynamically calculate the parallelism parameter of the stream processing task, send the parallelism adjustment value to the execution engine, and continuously monitor the task CPU utilization and queue backlog duration to trigger iterative optimization.

[0007] Further improvements to this technical solution include the following specific steps in step S1 for constructing the MPP distributed architecture of the FE / BE nodes: S121. Based on the total number of data sources n obtained in step S11 and the average daily data volume of each data source... Total data size of the computing platform is estimated. The number of FE nodes is determined based on high availability constraints. With the number of BE nodes Among them, FE nodes meet the odd-number election requirement, and BE nodes meet the replica redundancy multiple relationship; S122, Based on the number of BE nodes determined in step S121 Estimated total data size of the platform Calculate the storage path capacity of a single BE node. With memory quota The calculation results are then written to the BE node configuration file. S123. Based on the configuration file generated in step S122, complete the distributed deployment of FE / BE nodes and establish a heartbeat detection and metadata synchronization channel between nodes; import the raw data accessed in step S1 into Doris internal storage or register it as an external lake table according to the pre-stored routing strategy, wherein the routing decision is based on the data access frequency. With query latency requirements They jointly decide and generate a data distribution topology map. Further improvements to this technical solution include step S2, which includes: S21. Based on the number of BE nodes deployed in step S12 With single-node storage capacity Calculate the global data partitioning strategy parameters; and monitor the number of rows in each partition of each data table. , when the number of rows When the partition deviates from the threshold range, partition merging or splitting operations are automatically triggered, partition operation logs are generated, and the metadata directory is updated. S22. Based on the partition operation log generated in step S21, obtain the actual data distribution of each partition, calculate the bucket skew for the table using hash bucketing, and when the skew exceeds the threshold, dynamically adjust the number of buckets based on the number of CPU cores and parallel efficiency coefficient of the BE node, and synchronize the new bucket key configuration to the Doris analysis engine. S23. Based on the initial timestamp of the data recorded when importing data in step S11 and the bucket distribution adjusted in step S22, construct a historical query popularity evaluation model, and evaluate the historical query popularity over a preset time period. The query logs are analyzed to calculate the access popularity value of each data partition. Based on the popularity threshold, the data is divided into three levels: hot, warm, and cold, and the storage media and caching strategies are configured differently. S24. Based on the storage level determined in step S23, configure a low compression ratio high-speed access format for hot data and a high compression ratio archive format for cold data; dynamically select the ZSTD compression level in the data loading task, and calculate the data integrity index based on the quality rule engine. When the index is lower than the threshold, trigger an alarm and block downstream writing.

[0008] Further improvements to this technical solution include step S3, which includes: S31. Identify the type of external data lake system and extract its metadata service interface address. Establish a connection channel between the Doris FE node and the data lake metadata service. Calculate the initial cache refresh cycle using the metadata synchronization frequency formula and configure the refresh cycle in the Doris FE metadata cache management component. S32. Register external table mapping relationships in the Doris FE metadata cache layer, subscribe to the incremental change event stream of the data lake, monitor the change in the commit frequency of the data lake commit log, and dynamically adjust the cache expiration time based on the changed commit frequency. S33. Before each federated query is executed, obtain the metadata version number of the Doris cache and the current version number of the data lake, and calculate the version difference; when the version difference exceeds the threshold, trigger a forced refresh and roll back the query plan to the latest version, while recording the refresh time and the number of version jumps. S34. Attach a data freshness tag to the federated query results, return the freshness value along with the query response, and write it to the monitoring log. The freshness tag includes the metadata delay timestamp and the latest commit timestamp of the data lake.

[0009] Further improvements to this technical solution include step S4, which includes: S41. Configure the MySQL protocol compatibility layer and SQL query interface, establish a role-permission-resource-based access control model, and collect data on each tenant's access over a preset period of time. Extract feature data from historical query logs, including query frequency, average scan data volume, and SQL complexity score, and construct tenant query profiles. S42. Based on the tenant query profile constructed in step S41, create a multi-tenant resource queue in the Doris FE layer, and allocate the total cluster resources according to the query load weight of each tenant. With business priority Dynamic allocation generates resource quotas for each tenant. The quota value is written to the resource manager's configuration library for query scheduling; tenant resource quota The calculation formula is: ; in, Set the resource queue quota for tenant i; The business priority weight for tenant i; Total number of tenants; This represents the total resources of the cluster. S43. Receive the SQL query request submitted by the user. After the query is parsed, call the cost estimation tool to calculate the estimated query cost based on statistical information metadata. Compare the estimated cost with the tenant resource quota generated in step S42. Perform a comparison, and based on the comparison results, decide whether to execute immediately, wait in the queue, or trigger a query degradation, and record the decision results in the audit log; S44. When executing query scheduling in step S43, priority labels are assigned to different query types. When cluster resources are insufficient, high-priority queries can trigger a preemption mechanism, suspending low-priority queries and releasing their resources according to the preemption threshold.

[0010] Further improvements to this technical solution include step S5, which includes: S51. Configure the Doris data source connection pool in the BI and visualization integration module. Based on the historical query concurrency and average response time of each tenant recorded in step S4, calculate the connection parameters including the initial size and maximum expansion capacity of the connection pool. Encrypt and store the connection parameters in the configuration center, and generate dedicated driver plugins for Tableau, Power BI, and FineBI tools. S52. Based on the SQL query history recorded in step S4 and the hot data partitioning information generated in step S2, an intelligent query recommendation engine is built in the BI module. Cluster analysis is performed on the query patterns of each user group over a preset duration, high-frequency query templates are extracted and the results are pre-calculated and cached. When users explore by dragging dimensions, the optimal query path and pre-aggregated view are automatically recommended. S53. Based on the query template recommended in step S52 and the data lake freshness tag obtained in step S3, dynamically select the visualization rendering strategy, use real-time rendering mode for hot data and asynchronous loading mode for warm and cold data, and predict the report loading time based on data volume and network bandwidth; when the prediction time exceeds the threshold, automatically switch to sampling display or incremental loading mode. S54. Embed user behavior tracking components in the BI report display interface to collect user click, drill-down, and filter operation sequences. Calculate the usability score of functional modules based on operation frequency and dwell time. Feed back functions with scores below the threshold to the development team for optimization. And condense high-frequency operation paths into standardized report templates for new users to reuse.

[0011] Further improvements to this technical solution include: providing a unified API through the application service interface module for upper-layer business systems to call and analyze the results. The specific steps are as follows: S61. Based on the SQL query interface configured in step S4 and the partitions and buckets defined in step S2, build a unified API specification, define RESTful endpoint paths, request parameter structures and response data formats, manage interface evolution through version number calculation formula, and register the API definition file to the service discovery center for callers to query and dynamically discover. S62. Deploy the API gateway component in the application service interface module to integrate the total cluster resources in step S1 and the tenant quotas in step S4. The computation rate limiting strategy uses the total cluster resources, which is the sum of the resources of all BE nodes, including the total storage path capacity and the total memory quota. Based on the rate limit, JWT authentication and fine-grained permission verification are performed on the call requests of the upper-layer business system. The number of concurrent API calls is dynamically allocated according to the remaining quota of each tenant, and requests exceeding the limit are added to the waiting queue. At the same time, API call logs are generated. S63. Push the API call logs to the operation and maintenance monitoring module configured in step S8 in real time; S64. Inject the data freshness tag obtained in step S3 and the data integrity index calculated in step S2 into the API response body to provide the caller with a data credibility score; when the credibility is lower than the business threshold, add a warning label to the response header to guide the business system to perform degradation processing or trigger the data compensation process.

[0012] Further improvements to this technical solution include: configuring an operation and maintenance monitoring module to monitor the platform's operating status, performance indicators, and resource usage in real time; identifying performance anomalies through an anomaly detection model and triggering a fault self-healing mechanism to generate alarm information; the specific steps are as follows: S71. Collect operational metrics from the probes deployed in each module from steps S1 to S6, including CPU utilization, memory usage, disk IOPS, network throughput of FE / BE nodes, as well as query QPS, failure rate, and P99 latency recorded in step S4; aggregate the collected data by time window and input it into the time series database, and calculate the health baseline for each metric. S72. Based on the health baseline calculated in step S71, the data freshness label obtained in step S3, and the data integrity index calculated in step S2, an anomaly detection model is constructed. The anomaly detection model identifies performance degradation or data quality deterioration through residual analysis. When the residual exceeds the dynamic threshold, an anomaly event is triggered and the root cause label is recorded. S73. When an abnormal event is triggered in step S72 and the abnormal residual is greater than the alarm threshold, query the node architecture information deployed in step S1 and the tenant quota information in step S4, and automatically perform fault self-healing actions according to the fault type and scope of impact, including BE node replica migration, query task degradation, cache cleanup or switching federated queries to internal tables, and record the success rate and recovery time of the self-healing operation. S74. Based on the anomaly type identified in step S72 and the self-healing operation result in step S73, generate structured alarm information and push the alarm information to the corresponding responsible persons through multiple channels. At the same time, archive the alarm information to the knowledge base for optimizing threshold parameters.

[0013] In a second aspect, the present invention provides a computer storage medium storing instructions which, when executed on a computer, cause the computer to perform the methods described in the above aspects.

[0014] The beneficial effects of this invention are as follows: Through the systematic technical integration of steps S1-S5, a performance breakthrough was achieved across the entire data analysis chain. Based on the Apache Doris MPP architecture and vectorized execution engine, combined with intelligent data partitioning management in S2, data lake federated query optimization in S3, and cost prediction resource scheduling in S4, the platform can achieve sub-second query response times when processing massive amounts of data. In particular, the adaptive partitioning strategy based on data volume thresholds in S2 and the dynamic resource allocation based on query complexity in S4 effectively avoid performance bottlenecks caused by data skew or resource contention in traditional solutions, improve query performance, and fully meet the needs of high-performance analysis scenarios such as real-time reporting and ad-hoc queries.

[0015] This invention simplifies the complex technology stack that traditionally required integrating more than 10 components into a unified platform by leveraging the unified architecture of the Doris core engine in S1 and the native lake warehouse integration capabilities of S3. S7's intelligent monitoring and fault self-healing mechanisms automate operations and maintenance, significantly reducing manpower investment. Actual deployment shows that this reduces platform operation and maintenance costs, and S5's BI integration module enables business personnel to directly explore data, reducing reliance on professional data analysts.

[0016] A complete data governance framework was built through S2's full lifecycle data management, S3's metadata consistency assurance, and S7's quality monitoring system. S2's data popularity-based hierarchical management and dynamic selection of compression algorithms reduce storage costs while ensuring performance for accessing frequently accessed data; S3's version verification mechanism ensures strong consistency for data lake queries; and S6's data credibility scoring mechanism provides reliable data quality assessments for business systems. This comprehensive quality assurance system effectively solves problems such as data silos and uncontrollable quality in traditional solutions.

[0017] The key advantage of this invention lies in its ability to achieve adaptive optimization of the platform. S1 is based on dynamic parallelism adjustment of data arrival rate, S2 is based on bucketing strategy optimization of skewness, S4 is based on cost model for resource allocation, and S7 is based on fault self-healing based on anomaly detection. These components form a complete intelligent control system, which improves the platform's resource utilization compared to traditional static configuration schemes and can automatically adjust according to business fluctuations.

[0018] By leveraging the multi-replica architecture of S1, the multi-tenant isolation of S4, the API gateway protection of S6, and the fault self-healing mechanism of S7, an enterprise-grade high-availability assurance system has been built. The health scoring model and dynamic calculation of circuit breaker thresholds in S7 enable the system to identify potential risks in advance and intervene automatically, improving platform availability. This reliability assurance ensures the continuous and stable operation of critical business processes, providing solid support for enterprise digital transformation. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a schematic flowchart illustrating a method according to an embodiment of the present invention.

[0021] Figure 2 This is a schematic diagram of the overall architecture of a data analysis platform built on Doris in one embodiment of the present invention.

[0022] Figure 3 This is a schematic diagram illustrating the workflow of the Apache Doris core analysis engine in one embodiment of the present invention.

[0023] Figure 4 This is a schematic diagram of the data lake federated query and data analysis process in one embodiment of the present invention. Detailed Implementation

[0024] To make the objectives, features, and advantages of this invention more apparent and understandable, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings of the specific embodiments. Obviously, the embodiments described below are only some embodiments of this invention, and not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0025] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein in the description of the invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention.

[0026] The key terms used in this invention will be explained below.

[0027] Apache Doris: An open-source analytical database based on an MPP architecture, characterized by high performance, real-time capabilities, and ease of use. In this invention, it serves as the core analytical engine, responsible for the storage, computation, and analysis of massive amounts of data.

[0028] ClickHouse and Druid are both open-source real-time OLAP (Online Analytical Processing) engines. Mentioned as comparative solutions in the background section, their commonality is that while they offer excellent query performance, they lack native lakeware integration capabilities, resulting in redundant data storage and cross-source query latency issues.

[0029] Kafka, Spark, and Presto: core components of the big data ecosystem; Kafka: a distributed message queue for high-throughput real-time data streaming; Spark: a unified analytics engine for large-scale data processing; Presto: a distributed SQL query engine for interactive queries.

[0030] API: Application Programming Interface; a set of well-defined communication protocols and methods between software components. In this invention, it specifically refers to the unified RESTful API provided externally through the application service interface module (step S7), which is called by upper-layer business systems to obtain analysis results.

[0031] SQL: Structured Query Language; a standard programming language used for managing and querying relational databases. In this invention, the SQL query and management module (step S5) provides a standard SQL interface and a MySQL protocol compatibility layer, enabling users to perform data querying and analysis using familiar SQL syntax.

[0032] Enterprise-ready platform: This refers to a software platform that possesses high availability, high reliability, security, scalability, and comprehensive operational support, capable of meeting the stringent production environment requirements of large enterprises. This invention aims to transform Apache Doris into an "enterprise-ready" lakehouse data analytics platform through systematic methods (such as multi-tenant resource isolation and fault self-healing mechanisms).

[0033] FE (Frontend) refers to the front-end node of Apache Doris, which is responsible for metadata management and query scheduling; BE (Backend) refers to the back-end node, which is responsible for data storage and computation execution. The two work together through an MPP architecture.

[0034] Hudi, Iceberg, and Delta Lake are all open-source data lake storage format frameworks used to manage massive datasets. This invention uses them as external data source systems for integration.

[0035] MPP: Massively Parallel Processing architecture, refers to a technical architecture that distributes query tasks to multiple BE nodes for parallel computation to improve performance.

[0036] Figure 1 This is a schematic flowchart illustrating a method for building a data analysis platform based on the Apache Doris lakehouse, as provided by this invention. Figure 1 The method shown can be used to build something like this. Figure 2 The data analysis platform shown is illustrated. The order of steps in this flowchart can be changed, and some steps can be omitted, depending on different needs.

[0037] like Figure 1 As shown, the method includes: S1. Configure the data source access module to obtain the raw multi-source data of the data lake system and dynamically adjust the parallelism parameters of the stream processing; deploy the Apache Doris analysis engine to build an MPP distributed architecture of FE / BE nodes, and import the raw data into Doris internal storage or keep it in the external data lake. S2. Configure the data management module to perform lifecycle management on the data imported in step S1; S3. Use the data lake federated query module to establish a metadata mapping and query acceleration channel between Doris and the data lake system, and force a refresh of metadata through version number verification. S4. Configure the SQL query and management module, provide SQL query interface and MySQL protocol compatibility layer, build multi-tenant resource queue and perform query resource allocation and isolation based on cost prediction factor; S5 deploys a BI and visualization integration module, using the Doris analytics engine as a data source to connect with business intelligence tools, providing graphical data exploration and reporting capabilities.

[0038] To facilitate understanding of the present invention, the following describes the principle of the method for building a lakehouse data analysis platform based on Apache Doris, combined with the process of building the lakehouse data analysis platform based on Apache Doris in the embodiments, and also incorporates... Figure 3and Figure 4 The present invention further describes the construction method of the Apache Doris Lakehouse Data Analysis Platform provided by the present invention.

[0039] First, the methods for obtaining the raw multi-source data of the data lake system in step S1 include: S111. Configure the data source access module, establish a unified configuration library containing data source type identifiers, connection addresses, and authentication credentials, and set up standardized access templates for each type of data source, including retry policies and timeout thresholds. Determine the access weight of each data source through data source priority scoring; the calculation formula for data source priority scoring is: ; in, The priority score for the i-th data source is given (the value ranges from 0 to 100). The data source weight coefficients are preset according to business importance (values ​​range from 0.8 to 1.0 for business databases and from 0.3 to 0.5 for log systems). This is the connection stability coefficient; S112. Collect raw data streams in real time from business databases, message queues, log systems, data lakes, and external APIs. Use the priority score configured in step S11 as the basis for data sampling frequency, calculate the data arrival rate of each data source within a unit time window, and calculate the data flow fluctuation coefficient to identify abnormal peaks. ; ; in, The data arrival rate of the i-th data source (unit: data items / second); In the time window (Recommended value: 10-60 seconds) Number of data entries collected within this period; This refers to the flow fluctuation coefficient. and These are the maximum and minimum arrival rates over the past 10 time windows, respectively. Average arrival rate; S113. Based on the data arrival rate and fluctuation coefficient obtained in step S112, dynamically calculate the parallelism parameter of the stream processing task, send the parallelism adjustment value to the execution engine, and continuously monitor the task CPU utilization and queue backlog duration to trigger iterative optimization; the formula for calculating the parallelism parameter is: ; in, The dynamic parallelism (positive integer) of the Flink stream processing task; Total number of data sources; Average processing time (seconds) for a single data entry; The number of data entries processed per batch. This is a redundancy factor (with a value of 1.2 to 1.5, used to cope with sudden traffic surges).

[0040] Secondly, the specific steps in step S1 to construct the MPP distributed architecture of FE / BE nodes include: S121. Based on the total number of data sources n obtained in step S11 and the average daily data volume of each data source... Total data size of the computing platform is estimated. The number of FE nodes is determined based on high availability constraints. With the number of BE nodes Among them, FE nodes meet the odd-number election requirement, and BE nodes meet the replica redundancy multiple relationship; the total data scale of the platform is estimated. The calculation formula is: ; ; ; in, To estimate the total data volume (GB); Reserve a coefficient for data growth (value 1.3~1.5); The number of front-end nodes (must be an odd number and not less than 3); This refers to the number of backend nodes. The number of data replicas (values ​​can be 1, 2, or 3); S122, Based on the number of BE nodes determined in step S121 Estimated total data size of the platform Calculate the storage path capacity of a single BE node. With memory quota The storage path capacity needs to consider columnar storage compression rate and system reserved space overhead. Memory quotas need to meet query execution and caching requirements, and the calculation results should be written to the BE node configuration file. The formula for the storage capacity of a single BE node is: ; in, Storage path capacity for a single BE node (unit: GB); The compression ratio for Doris columnar storage (value range: 0.1~0.3). Reserve a certain percentage of space for the system (range: 0.15~0.2) to cover metadata and temporary file overhead; The formula for memory quota for a single BE node is: ; in, Memory quota for a single BE node (in GB); The number of concurrent queries that a single node can handle (usually 150~300). Average memory consumption per query (unit: GB); Memory safety utilization rate (value range: 0.7~0.8); Reserve memory (in GB) for PageCache and Query Cache; S123. Based on the configuration file generated in step S122, complete the distributed deployment of FE / BE nodes and establish a heartbeat detection and metadata synchronization channel between nodes; import the raw data accessed in step S1 into Doris internal storage or register it as an external lake table according to the pre-stored routing strategy, wherein the routing decision is based on the data access frequency. With query latency requirements The decision is made jointly, and a data distribution topology map is generated for use in subsequent step S3; Data routing strategy formula: ; in, Let be the average daily number of queries for the i-th data table over the past 7 days; The query latency SLA (in seconds) required by the i-th business party; The threshold for hot data (usually 100 times / day); The threshold for cold data (typically 10 times / day); Internal indicates importing to Doris internal storage; External indicates retaining data in an external data lake; Hybrid indicates dual-copy storage.

[0041] Among them, dual-replica storage: the hybrid storage strategy defined in this invention refers to the same data being stored in Doris in a columnar compressed format to ensure high-performance queries, and also retained in an external data lake in an open format (such as Parquet) to support federated analysis, maintaining consistency through metadata synchronization.

[0042] Additionally, step S2 includes: S21. Based on the number of BE nodes deployed in step S12 With single-node storage capacity Calculate the global data partitioning strategy parameters; and monitor the number of rows in each partition of each data table. When the number of rows deviates from the threshold range, partition merging or splitting operations are automatically triggered, generating partition operation logs and updating the metadata directory for subsequent query routing; the maximum number of partitions is calculated using the following formula: ; in, This represents the maximum number of partitions. Number of days to retain data; Ideal partition granularity (unit: day); Reserved number of hot partitions (value range: 3~5) to ensure the availability of frequently accessed partitions; Ideal partitioning granularity The adaptive adjustment rule is: ; in, This represents the number of rows in a single partition. This represents the current partition granularity (unit: days). This rule dynamically optimizes the partition granularity based on the data growth rate to ensure query efficiency.

[0043] S22. Based on the partition operation log generated in step S21, obtain the actual data distribution of each partition, and calculate the bucket skew for tables using hash bucketing. When the tilt When the threshold is exceeded, the number of CPU cores based on BE nodes The number of buckets is dynamically adjusted along with the parallel efficiency coefficient, and the new bucket key configuration is synchronized to the Doris analysis engine; the bucket skewness calculation formula is: ; in, Let i be the row number of the i-th bucket; This represents the current number of buckets; the skewness reflects the uniformity of the data distribution, with a larger value indicating a more uneven distribution. Adjusted formula for calculating the number of buckets: ; in, This is the adjusted number of buckets; This is the parallel efficiency coefficient (value range: 0.7~0.9). This is the skew suppression factor (value range: 1.0~1.3); this formula ensures that the number of buckets matches the cluster resources, optimizing parallel query performance; S23. Based on the initial timestamp of the data recorded when importing data in step S12 and the bucket distribution adjusted in step S22, construct a historical query popularity evaluation model, and evaluate the historical preset duration. The query logs for 30 days were analyzed to calculate the access popularity value of each data partition. Based on the popularity threshold, the data was divided into three levels: hot, warm, and cold, and storage media and caching strategies were configured differently accordingly. The formula for calculating partition access popularity is: ; in, The access frequency of the j-th partition (unit: times / GB / day); The time decay weights are 1.0 for the most recent 7 days, 0.5 for 7-14 days, and 0.2 for 14-30 days. Let be the number of queries to partition j on day t; The data volume of partition j (in GB); The data classification rules are as follows: ; in, The threshold for hot data (typically 100 times / GB / day); Set a threshold for cold data (typically 10 times / GB / day); hot data is stored on SSDs with compression disabled, warm data is stored on SAS disks, and cold data is archived to object storage. S24. Based on the storage level determined in step S23, configure a low compression ratio high-speed access format for hot data (e.g., a compression ratio of 1.5:1 to 3.0:1, corresponding to ZSTD level 1 to 2 compression levels), and configure a high compression ratio archive format for cold data (e.g., a compression ratio of 5.0:1 to 8.0:1, corresponding to ZSTD level 5 to 6 compression levels); dynamically select the ZSTD compression level in the data loading task, and calculate the data integrity index based on the quality rule engine. When the index is lower than the threshold, trigger an alarm and block downstream writing.

[0044] The formula for dynamically selecting the ZSTD compression level is: ; in, ZSTD compression level (value range: 1~6); Queries frequency (unit: times / hour); The loading frequency (unit: times / hour); this formula balances compression efficiency and performance based on access patterns.

[0045] The formula for data integrity scoring is: ; in, Score the data integrity (value range: 0~1); This represents the number of valid rows that passed the validation. This represents the total number of rows. The number of rows with null values ​​in the k-th non-null constraint field; when A quality alarm is triggered at that time.

[0046] Partitioning and Bucketing: Partitioning refers to horizontally dividing a data table into multiple sub-tables (Partition) based on time or value range, used for data lifecycle management; Bucketing refers to further subdividing data within a partition based on hash value (Bucket), used to balance data distribution and improve query performance.

[0047] ZSTD compression: Zstandard compression algorithm, a columnar storage compression method supported by Doris, provides 1 to 6 compression levels. The higher the level, the greater the compression ratio, but the higher the CPU consumption. This invention dynamically selects a balance between performance and storage efficiency based on the query / load frequency.

[0048] Hot, warm, and cold data: Storage tiers are divided based on data access frequency. Hot data (high-frequency access) resides on SSDs and is configured with low compression, warm data (medium-frequency access) is stored on SAS disks, and cold data (low-frequency access) is archived to object storage, achieving the optimal balance between cost and performance.

[0049] Next, step S3 includes: S31. Identify the type of external data lake system (e.g., Hudi, Iceberg, Delta Lake), extract its metadata service interface address, establish a connection channel between the Doris FE node and the data lake metadata service, calculate the initial cache refresh cycle through the metadata synchronization frequency formula, and configure the refresh cycle in the metadata cache management component of Doris FE. The formula for cache refresh latency is: ; in, Cache refresh delay time (in seconds); Metadata submission frequency for the data lake (unit: submissions / second); calculated by monitoring submission logs over the past hour. and To dynamically adjust parameters, The value range is 0.5 to 1.5 seconds. The value range is 1 to 3 seconds, and it can be manually configured according to the business's tolerance for metadata freshness. S32. Register external table mapping relationships in the Doris FE metadata cache layer, subscribe to the incremental change event stream of the data lake, monitor the commit frequency changes of the data lake commit log, and dynamically adjust the cache expiration time based on the changed commit frequency to avoid performance jitter or data inconsistency caused by cache expiration too early or too late. The formula for cache expiration time is: ; in, Cache expiration time (in seconds); Maximum cache time (usually 300 seconds); Minimum cache time (usually 10 seconds); The stability coefficient (range: 0.8~1.2, depending on the standard deviation of the submission frequency) is used. If the value is 1.0, then use 0.8; otherwise, use 0.8. S33. Before each federated query is executed, obtain the metadata version number of the Doris cache and the current version number of the data lake, and calculate the version difference. When the version difference exceeds the threshold, trigger a forced refresh and roll back the query plan to the latest version. At the same time, record the refresh time and the number of version jumps for anomaly analysis in step S7. The formula for calculating version difference is: ; in, Metadata version difference (positive integer); The current metadata version number of the data lake (obtained in real time from the data lake metadata service); The version number of metadata cached for Doris FE; The rules for the forced refresh flag are as follows: ; in, Version difference threshold (value range: 3~5); when If a serious cache lag is detected, a forced refresh is triggered. S34. Attach a data freshness label to the federated query results, return the freshness value along with the query response, and write it to the monitoring log for quality assessment in step S7. The freshness label includes the metadata delay timestamp and the latest data lake commit timestamp.

[0050] The formula for calculating the data freshness label is: ; in, Data freshness label (unit: seconds); To query the system timestamp of the execution time; The Doris FE caches the last update timestamp of the table's metadata; when If this happens, a warning indicator will be returned in the query results.

[0051] In addition, step S4 includes: S41. Configure the MySQL protocol compatibility layer and SQL query interface, establish a role-permission-resource-based access control model, and collect data on each tenant's access over a preset period of time. Historical query logs within 30 days were used to extract feature data including query frequency, average scan data volume, and SQL complexity score, and a tenant query profile was constructed. The formula for calculating tenant query load weight is: ; in, The query load weight for tenant i; For tenant i during the statistical window period (Usually, the total number of queries within 30 days is taken); Score the SQL complexity of tenant i (calculated by parsing the number of JOIN layers, subqueries, and aggregate functions in the SQL, with a value range of 0.5 to 3.0). This is the complexity impact coefficient (value range: 0.3~0.5). S42. Based on the tenant query profile constructed in step S41, create a multi-tenant resource queue in the Doris FE layer, and allocate the total cluster resources according to the query load weight of each tenant. With business priority Dynamic allocation generates resource quotas for each tenant. The quota value is written to the resource manager's configuration library for query scheduling; tenant resource quota The calculation formula is: ; in, Resource queue quota for tenant i (unit: GB memory or CPU cores); The business priority weight for tenant i (set by the business party according to the Service Level Agreement (SLA), with high priority values ​​ranging from 0.8 to 1.0, medium priority values ​​ranging from 0.5 to 0.7, and low priority values ​​ranging from 0.2 to 0.4). Total number of tenants; The total resources of the cluster (based on the total memory of the BE nodes in step S2) (Calculated from the total number of CPU cores). S43. Receive the SQL query request submitted by the user, and after the query is parsed, call the cost estimation tool to calculate the estimated query cost based on statistical information and metadata. Combine the estimated cost with the tenant resource quota generated in step S42. A comparison is performed, and based on the comparison results, a decision is made to execute immediately, wait in the queue, or trigger query degradation, and the decision results are recorded in the audit log; the formula for calculating the estimated query cost is: ; in, To query the estimated cost (unit: resource quota equivalent); To estimate the number of rows to scan (obtained from the Doris statistics table tablet_meta); Average row size (in bytes); The number of JOIN operations; Let n be the number of rows in the input table for the k-th JOIN; The amount of Shuffle data that needs to be transmitted over the network (unit: GB); This is the network overhead weighting coefficient (value range: 1.5~2.0). Cluster network bandwidth (unit: GB / s); when When a query degradation is triggered, the cost can be reduced to within the quota by decreasing the parallelism or adding filtering conditions. S44. When performing query scheduling in step S43, priority labels are assigned to different query types (OLAP / ETL / Ad-hoc). When cluster resources are insufficient, high-priority queries can trigger a preemption mechanism, based on the preemption threshold. Pause low-priority queries and release their resources to achieve query-level resource isolation and service quality assurance.

[0052] The query priority label rules are as follows: ; in, For query priority tags (values ​​range from 1 to 3, with higher values ​​indicating higher priority); The decision-making rules for seizing control are as follows: ; in, Prioritizes newly arriving queries; Priority for currently running queries; This is the preemption threshold (usually set to 1, indicating that only preemption by a higher priority level is allowed). In order to seize the key position in decision-making.

[0053] Then, step S5 includes: S51. Configure the Doris data source connection pool in the BI and visualization integration module. Based on the historical query concurrency and average response time of each tenant recorded in step S5, calculate the initial size and maximum expansion capacity of the connection pool, and encrypt and store the connection parameters (such as address, port, authentication information) in the configuration center. Generate dedicated driver plugins for Tableau, Power BI, and FineBI tools. The formula for calculating the initial size of the connection pool is: ; in, Initial number of connections for the connection pool (unit: connections); Number of BI tool instances (unit: instances); Average concurrent queries for tenant i (unit: queries / second, extracted from the audit log in step S4); This is the peak value coefficient (range: 0.2~0.5), used to handle peak query volumes; Formula for calculating the maximum capacity of the connection pool: ; in, Maximum number of connections in the pool (unit: connections); To ensure the connection pool can be flexibly expanded, the expansion factor (value range: 2~3) is set. S52. Based on the SQL query history recorded in step S4 and the hot data partitioning information generated in step S2, an intelligent query recommendation engine is built in the BI module. Cluster analysis is performed on the query patterns of each user group over a preset duration, high-frequency query templates are extracted and the results are pre-calculated and cached. When users explore by dragging and dropping dimensions, the optimal query path and pre-aggregated view are automatically recommended to improve analysis efficiency. Query template recommendation rating formula: ; in, The query template recommendation score (value range: 0~1) indicates that the higher the value, the higher the recommendation priority. This is the historical number of times the template has been executed (unit: times). Total number of queries by users (unit: times); The heat value of the affected partition (referencing the result of step S2); The threshold for hot data (typically 100 times / GB / day); For the weighting coefficients, satisfying (Default values: 0.5, 0.3, 0.2); Let be the cosine similarity between the current user behavior vector u and the historical user behavior vector v; S53. Based on the query template recommended in step S52 and the data lake freshness tag obtained in step S3, dynamically select the visualization rendering strategy, use real-time rendering mode for hot data and asynchronous loading mode for warm and cold data, and predict the report loading time based on data volume and network bandwidth; when the prediction time exceeds the threshold, automatically switch to sampling display or incremental loading mode to balance experience and performance. Formula for predicting report rendering time: ; in, Render the report with the predicted time (in seconds); The size of the query result data (unit: MB); Client network bandwidth (unit: MB / s, detected via BI tool SDK); This is the compression transmission factor (value range: 0.3~0.6). To query the execution time (in seconds, refer to the return value of step S4); The rules for selecting the rendering mode are as follows: ; in, Rendering mode; Data freshness (unit: seconds, referencing the results of step S3); Real-time mode returns results synchronously, Async mode loads in the background and displays a progress bar, and Sample mode returns sampled data to accelerate the response.

[0054] S54. Embed user behavior tracking components in the BI report display interface to collect user click, drill-down, and filter operation sequences. Calculate the usability score of functional modules based on operation frequency and dwell time. Feed back functions with scores below the threshold to the development team for optimization. And condense high-frequency operation paths into standardized report templates for new users to reuse, continuously improving the platform's usability.

[0055] Functional module usability rating formula: ; in, Rate the usability of the functional modules (value range: 0~1), with higher values ​​indicating better usability; Average daily number of clicks for the feature (unit: clicks); Average dwell time (in seconds); Average session duration (in seconds); The number of unique users using this feature (unit: users); Total number of users (unit: number); when When this happens, mark it as a feature to be optimized and trigger the improvement process.

[0056] BI tools: Business Intelligence tools, such as Tableau, Power BI, FineBI, etc., are used for data analysis and report generation through a graphical interface. In this invention, they are used as query clients for the Doris analytics engine.

[0057] In addition, this method also includes: providing a unified API to the outside world through the application service interface module, allowing upper-layer business systems to call the analysis results. The specific steps are as follows: S61. Based on the SQL query interface configured in step S4 and the partitions and buckets defined in step S2, build a unified API specification, define RESTful endpoint paths, request parameter structures and response data formats, manage interface evolution through version number calculation formula, ensure backward compatibility of different versions of business systems, and register the API definition file to the service discovery center for callers to query and dynamically discover. API major version number calculation formula: ; in, The API major version number (positive integer); Current timestamp (in seconds); The date of the API's first release (in seconds); The major version iteration cycle (unit: days, usually 365 days); this formula ensures that the version number increases linearly with the iteration cycle, avoiding frequent changes; Formula for maximum allowed size of a single request: ; in, Maximum allowed size for a single request (in bytes); The number of request parameters (unit: number of parameters); This is the average parameter length (unit: bytes, usually 128 bytes). The overhead of encapsulating JSON / XML (in bytes, typically 512 bytes). This constraint prevents excessively large requests from overloading resources; S62. Deploy the API gateway component in the application service interface module to integrate the total cluster resources based on step S12. With tenant quota in step S5 The computation rate limiting strategy uses the total cluster resources, which is the sum of the resources of all BE nodes, including the total storage path capacity and the total memory quota. JWT authentication and fine-grained permission verification are performed on the call requests of the upper-layer business system. The number of concurrent API calls is dynamically allocated according to the remaining quota of each tenant, and requests exceeding the limit are added to the waiting queue to ensure fairness. Tenant API rate limit calculation formula: ; in, API rate limit for tenant i (in times / second); Total cluster resources (unit: GB memory or CPU cores, refer to the result of step S12). Resource quota for tenant i (unit: GB memory or CPU cores, refer to the result of step S5); API resource consumption coefficient (value range: 0.1~0.3), representing the proportion of resources used in a single API call; Formula for the maximum number of concurrent users in a tenant API: ; in, The maximum number of concurrent API connections for tenant i (in units). The average query time (unit: seconds, referencing the record value from step S4). The concurrency efficiency coefficient (range: 0.7~0.9) takes into account the overhead of connection pooling and thread switching. S63. Push API call logs to the operation and maintenance monitoring module configured in step S8 in real time, calculate API health score through interface latency distribution model; trigger circuit breaker mechanism when error rate or response time exceeds circuit breaker threshold and generate graded alarm; at the same time, record circuit breaker events for subsequent fault self-healing module analysis to ensure system resilience; API health score formula: ; in, API health score (value range: 0~1), the higher the value, the better the health status; Interface error rate (unit: percentage, obtained from monitoring data in step S7); The SLA commitment delay for the interface (unit: seconds, typically 2 seconds); 99th percentile delay (in seconds); Current load (unit: QPS); To maximize the QPS (refer to step S62) (Total). The weighting coefficients satisfy... ,and , , ; Formula for calculating the circuit breaker threshold: ; in, Circuit breaker threshold (unit: seconds); Historical average latency (unit: seconds); The historical delay standard deviation (unit: seconds) is obtained from the statistical values ​​in step S7; when the delay exceeds The circuit breaker is triggered at any time; S64. Inject the data freshness tag obtained in step S3 and the data integrity index calculated in step S2 into the API response body to provide the caller with a data reliability score. When the reliability is lower than the business threshold, add a warning flag to the response header to guide the business system to perform degradation processing or trigger a data compensation process to improve data reliability.

[0058] Data credibility scoring formula: ; in, The data is rated on its reliability (value range: 0~1), with a higher value indicating that the data is more reliable. This is a data integrity indicator (referencing the result of step S3, with a value range of 0~1). The freshness decay coefficient (value range: 0.01~0.1). Data freshness (unit: seconds, referencing the results of step S3); Warning flag rules: ; in, This is the business credibility threshold (usually set to 0.85). In response to the warning flag. When the flag is 1, add the “Data-Warning: Low-Trust” flag to the response header.

[0059] Finally, the method also includes: configuring an operation and maintenance monitoring module to monitor the platform's operating status, performance indicators, and resource usage in real time; identifying performance anomalies through an anomaly detection model and triggering a fault self-healing mechanism to generate alarm information; the specific steps are as follows: S71. Collect operational metrics from the probes deployed in each module from steps S1 to S6, including CPU utilization, memory usage, disk IOPS, and network throughput of the FE / BE nodes, as well as query QPS, failure rate, and P99 latency recorded in step S4. Aggregate the collected data by time window and input it into the time series database, and calculate the health baseline for each metric for anomaly detection. Formula for calculating average CPU utilization: ; Where, is the average CPU utilization (range: 0~1); is the aggregation window (unit: seconds, usually 60 seconds); is the CPU usage (unit: number of cores); is the total CPU (unit: number of cores); this formula calculates the average utilization within the window period through integration, smoothing out instantaneous fluctuations.

[0060] Query the formula for calculating the delayed health baseline: ; in, To query the baseline for delayed health status (unit: milliseconds); The historical 30-day average latency (in milliseconds); The standard deviation of delay (in milliseconds); This is the confidence coefficient (value 1.96, corresponding to a 95% confidence level). This baseline is used to identify abnormal delays; S72. Based on the health baseline calculated in step S71 and the data freshness label obtained in step S3 Step S2 calculates the data integrity index An anomaly detection model is constructed to identify performance degradation or data quality deterioration through residual analysis. When the residual exceeds the dynamic threshold, an anomaly event is triggered and the root cause label is recorded. Formula for calculating abnormal residuals: ; in, Abnormal residuals (unit: milliseconds, positive values ​​indicate abnormality); Current query latency (in milliseconds); This is a dynamic drift threshold (unit: milliseconds) used to adapt to cyclical changes in business operations; Dynamic drift threshold formula: ; in, The amplitude is periodic (range: 0.1~0.2 times the baseline). Business cycle (unit: hours, usually 24 hours); The trend slope (estimated from historical data via linear regression); This represents the number of days since the deployment date. This formula adapts to both cyclical and trending changes in business operations. S73, When an abnormal event is triggered in step S72 and the residual hour, For abnormal residuals, query the node architecture information deployed in step S12 and the tenant quota information in step S4, and automatically execute fault self-healing actions according to the fault type and scope of impact, including BE node replica migration, query task degradation, cache cleanup or switching federated queries to internal tables, and record the success rate and recovery time of self-healing operations. Formula for calculating self-healing success rate: ; in, The self-healing success rate (value range: 0~1); Number of times for successful self-healing; This represents the total number of times self-healing has been triggered. This is a smoothing coefficient (set to 0.01 to avoid division by zero errors). This metric assesses the effectiveness of the self-healing mechanism; Data recovery time target formula: ; in, Data recovery time target (in seconds); The number of tablets on the faulty node (unit: tablets); Average tablet size (unit: GB); Cluster network bandwidth (unit: GB / s); Bandwidth utilization (value range: 0.6~0.8); The time taken for FE master node election (in seconds, referring to step S12 configuration); this formula estimates the time required for fault recovery. S74. Based on the anomaly type identified in step S72 and the self-healing operation result in step S73, generate structured alarm information. The alarm includes alarm level, a list of affected tenants, root cause tags (e.g., insufficient resources, data skew, metadata delay), and suggested actions. It is pushed to the corresponding responsible persons through multiple channels (e.g., DingTalk, email, WeChat Work), and the alarm data is archived in the knowledge base for optimizing threshold parameters.

[0061] Alarm level determination rules: ; in, Alarm level; Alarm trigger threshold (unit: milliseconds); For data integrity metrics (referencing the results of step S3); Alarm trigger threshold formula: ; in, The standard deviation of the residuals (in milliseconds); Monitoring period length (unit: seconds); The false alarm rate tolerance is set (usually 0.001); this formula sets a dynamic threshold based on extreme value theory.

[0062] The present invention also provides a computer storage medium, wherein the computer storage medium may store a program, which, when executed, may include some or all of the steps provided in the embodiments of the present invention. The storage medium may be a magnetic disk, an optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0063] Those skilled in the art will clearly understand that the techniques in the embodiments of the present invention can be implemented using software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solutions in the embodiments of the present invention, or the parts that contribute to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium such as a USB flash drive, mobile hard drive, read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk, or other media capable of storing program code. It includes several instructions to cause a computer terminal (which may be a personal computer, server, or a second terminal, network terminal, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention.

[0064] Although the present invention has been described in detail with reference to the accompanying drawings and preferred embodiments, the present invention is not limited thereto. Various equivalent modifications or substitutions can be made to the embodiments of the present invention by those skilled in the art without departing from the spirit and essence of the invention, and such modifications or substitutions should all be within the scope of the present invention. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should also be covered within the protection scope of the present invention.

Claims

1. A method for building a data analysis platform based on the Apache Doris lakehouse, characterized in that, include: S1. Configure the data source access module to obtain the raw multi-source data of the data lake system and dynamically adjust the parallelism parameters of the stream processing. Deploy the Apache Doris analytics engine to build an MPP distributed architecture of FE / BE nodes, and import the raw data into Doris internal storage or keep it in an external data lake; S2. Configure the data management module to perform lifecycle management on the data imported in step S1; S3. Use the data lake federated query module to establish a metadata mapping and query acceleration channel between Doris and the data lake system, and force a refresh of metadata through version number verification. S4. Configure the SQL query and management module, provide SQL query interface and MySQL protocol compatibility layer, build multi-tenant resource queue and perform query resource allocation and isolation based on cost prediction factor; S5 deploys a BI and visualization integration module, using the Doris analytics engine as a data source to connect with business intelligence tools, providing graphical data exploration and reporting capabilities.

2. The method for building a data analysis platform based on the Apache Doris lakehouse according to claim 1, characterized in that, The methods for obtaining raw multi-source data from the data lake system in step S1 include: S111. Configure the data source access module, establish a unified configuration library containing data source type identifiers, connection addresses, and authentication credentials, and determine the access weight of each data source through data source priority scoring; the calculation formula for data source priority scoring is: ； in, Rate the priority of the i-th data source; These are preset data source weight coefficients based on business importance. This is the connection stability coefficient; S112. Collect raw data streams in real time from business databases, message queues, log systems, data lakes, and external APIs, and prioritize the data sources configured in step S111. As a basis for data sampling frequency, the data arrival rate of each data source within a unit time window is statistically analyzed, and the fluctuation coefficient of data flow is calculated to identify abnormal peaks. S113. Based on the data arrival rate and fluctuation coefficient obtained in step S112, dynamically calculate the parallelism parameter of the stream processing task, send the parallelism adjustment value to the execution engine, and continuously monitor the task CPU utilization and queue backlog duration to trigger iterative optimization.

3. The method for building a data analysis platform based on the Apache Doris lakehouse as described in claim 1, characterized in that, The specific steps in step S1 for constructing the MPP distributed architecture of FE / BE nodes include: S121. Based on the total number of data sources n obtained in step S11 and the average daily data volume of each data source... Total data size of the computing platform is estimated. The number of FE nodes is determined based on high availability constraints. With the number of BE nodes Among them, FE nodes meet the odd-number election requirement, and BE nodes meet the replica redundancy multiple relationship; S122, Based on the number of BE nodes determined in step S121 Estimated total data size of the platform Calculate the storage path capacity of a single BE node. With memory quota The calculation results are then written to the BE node configuration file. S123. Based on the configuration file generated in step S122, complete the distributed deployment of FE / BE nodes and establish a heartbeat detection and metadata synchronization channel between nodes; import the raw data accessed in step S1 into Doris internal storage or register it as an external lake table according to the pre-stored routing strategy, wherein the routing decision is based on the data access frequency. With query latency requirements They jointly decide and generate a data distribution topology map.

4. The method for building a data analysis platform based on the Apache Doris lakehouse according to claim 3, characterized in that, Step S2 includes: S21. Based on the number of BE nodes deployed in step S12 With single-node storage capacity Calculate the global data partitioning strategy parameters; and monitor the number of rows in each partition of each data table. , when the number of rows When the partition deviates from the threshold range, partition merging or splitting operations are automatically triggered, partition operation logs are generated, and the metadata directory is updated. S22. Based on the partition operation log generated in step S21, obtain the actual data distribution of each partition, calculate the bucket skew for the table using hash bucketing, and when the skew exceeds the threshold, dynamically adjust the number of buckets based on the number of CPU cores and parallel efficiency coefficient of the BE node, and synchronize the new bucket key configuration to the Doris analysis engine. S23. Based on the initial timestamp of the data recorded when importing data in step S11 and the bucket distribution adjusted in step S22, construct a historical query popularity evaluation model, and evaluate the historical query popularity over a preset time period. The query logs are analyzed to calculate the access popularity value of each data partition. Based on the popularity threshold, the data is divided into three levels: hot, warm, and cold, and the storage media and caching strategies are configured differently. S24. Based on the storage level determined in step S23, configure a low compression ratio high-speed access format for hot data and a high compression ratio archive format for cold data; dynamically select the ZSTD compression level in the data loading task, and calculate the data integrity index based on the quality rule engine. When the index is lower than the threshold, trigger an alarm and block downstream writing.

5. The method for building a data analysis platform based on the Apache Doris lakehouse according to claim 1, characterized in that, Step S3 includes: S31. Identify the type of external data lake system and extract its metadata service interface address. Establish a connection channel between the Doris FE node and the data lake metadata service. Calculate the initial cache refresh cycle using the metadata synchronization frequency formula and configure the refresh cycle in the Doris FE metadata cache management component. S32. Register external table mapping relationships in the Doris FE metadata cache layer, subscribe to the incremental change event stream of the data lake, monitor the change in the commit frequency of the data lake commit log, and dynamically adjust the cache expiration time based on the changed commit frequency. S33. Before each federated query is executed, obtain the metadata version number of the Doris cache and the current version number of the data lake, and calculate the version difference; when the version difference exceeds the threshold, trigger a forced refresh and roll back the query plan to the latest version, while recording the refresh time and the number of version jumps. S34. Attach a data freshness tag to the federated query results, return the freshness value along with the query response, and write it to the monitoring log. The freshness tag includes the metadata delay timestamp and the latest commit timestamp of the data lake.

6. The method for building a data analysis platform based on the Apache Doris lakehouse according to claim 1, characterized in that, Step S4 includes: S41. Configure the MySQL protocol compatibility layer and SQL query interface, establish a role-permission-resource-based access control model, and collect data on each tenant's access over a preset period of time. Extract feature data from historical query logs, including query frequency, average scan data volume, and SQL complexity score, and construct tenant query profiles. S42. Based on the tenant query profile constructed in step S41, create a multi-tenant resource queue in the Doris FE layer, and allocate the total cluster resources according to the query load weight of each tenant. With business priority Dynamic allocation generates resource quotas for each tenant. The quota value is written to the resource manager's configuration library for query scheduling; tenant resource quota The calculation formula is: ； in, Set the resource queue quota for tenant i; The business priority weight for tenant i; Total number of tenants; This represents the total resources of the cluster. S43. Receive the SQL query request submitted by the user. After the query is parsed, call the cost estimation tool to calculate the estimated query cost based on statistical information metadata. Compare the estimated cost with the tenant resource quota generated in step S42. Perform a comparison, and based on the comparison results, decide whether to execute immediately, wait in the queue, or trigger a query degradation, and record the decision results in the audit log; S44. When executing query scheduling in step S43, priority labels are assigned to different query types. When cluster resources are insufficient, high-priority queries can trigger a preemption mechanism, suspending low-priority queries and releasing their resources according to the preemption threshold.

7. The method for building a data analysis platform based on the Apache Doris lakehouse according to claim 6, characterized in that, Step S5 includes: S51. Configure the Doris data source connection pool in the BI and visualization integration module. Based on the historical query concurrency and average response time of each tenant recorded in step S4, calculate the connection parameters including the initial size and maximum expansion capacity of the connection pool. Encrypt and store the connection parameters in the configuration center, and generate dedicated driver plugins for Tableau, Power BI, and FineBI tools. S52. Based on the SQL query history recorded in step S4 and the hot data partitioning information generated in step S2, an intelligent query recommendation engine is built in the BI module. Cluster analysis is performed on the query patterns of each user group over a preset duration, high-frequency query templates are extracted and the results are pre-calculated and cached. When users explore by dragging dimensions, the optimal query path and pre-aggregated view are automatically recommended. S53. Based on the query template recommended in step S52 and the data lake freshness tag obtained in step S3, dynamically select the visualization rendering strategy, use real-time rendering mode for hot data and asynchronous loading mode for warm and cold data, and predict the report loading time based on data volume and network bandwidth; when the prediction time exceeds the threshold, automatically switch to sampling display or incremental loading mode. S54. Embed user behavior tracking components in the BI report display interface to collect user click, drill-down, and filter operation sequences. Calculate the usability score of functional modules based on operation frequency and dwell time. Feed back functions with scores below the threshold to the development team for optimization. And condense high-frequency operation paths into standardized report templates for new users to reuse.

8. The method for building a data analysis platform based on the Apache Doris lakehouse according to claim 7, characterized in that, Also includes: The application service interface module provides a unified API for upper-layer business systems to access and analyze the results. The specific steps are as follows: S61. Based on the SQL query interface configured in step S4 and the partitions and buckets defined in step S2, build a unified API specification, define RESTful endpoint paths, request parameter structures and response data formats, manage interface evolution through version number calculation formula, and register the API definition file to the service discovery center for callers to query and dynamically discover. S62. Deploy the API gateway component in the application service interface module to integrate the total cluster resources in step S1 and the tenant quotas in step S4. The computation rate limiting strategy uses the total cluster resources, which is the sum of the resources of all BE nodes, including the total storage path capacity and the total memory quota. Based on the rate limit, JWT authentication and fine-grained permission verification are performed on the call requests of the upper-layer business system. The number of concurrent API calls is dynamically allocated according to the remaining quota of each tenant, and requests exceeding the limit are added to the waiting queue. At the same time, API call logs are generated. S63. Push the API call logs to the operation and maintenance monitoring module configured in step S8 in real time; S64. Inject the data freshness tag obtained in step S3 and the data integrity index calculated in step S2 into the API response body to provide the caller with a data credibility score; when the credibility is lower than the business threshold, add a warning label to the response header to guide the business system to perform degradation processing or trigger the data compensation process.

9. The method for building a data analysis platform based on the Apache Doris lakehouse according to claim 1, characterized in that, Also includes: Configure the operation and maintenance monitoring module to monitor the platform's operating status, performance indicators, and resource usage in real time. Identify performance anomalies through the anomaly detection model and trigger the fault self-healing mechanism to generate alarm information. The specific steps are as follows: S71. Collect operational metrics from the probes deployed in each module from steps S1 to S6, including CPU utilization, memory usage, disk IOPS, network throughput of FE / BE nodes, as well as query QPS, failure rate, and P99 latency recorded in step S4; aggregate the collected data by time window and input it into the time series database, and calculate the health baseline for each metric. S72. Based on the health baseline calculated in step S71, the data freshness label obtained in step S3, and the data integrity index calculated in step S2, an anomaly detection model is constructed. The anomaly detection model identifies performance degradation or data quality deterioration through residual analysis. When the residual exceeds the dynamic threshold, an anomaly event is triggered and the root cause label is recorded. S73. When an abnormal event is triggered in step S72 and the abnormal residual is greater than the alarm threshold, query the node architecture information deployed in step S1 and the tenant quota information in step S4, and automatically perform fault self-healing actions according to the fault type and scope of impact, including BE node replica migration, query task degradation, cache cleanup or switching federated queries to internal tables, and record the success rate and recovery time of the self-healing operation. S74. Based on the anomaly type identified in step S72 and the self-healing operation result in step S73, generate structured alarm information and push the alarm information to the corresponding responsible persons through multiple channels. At the same time, archive the alarm information to the knowledge base for optimizing threshold parameters.

10. A computer-readable storage medium storing a computer program, characterized in that, When the program is executed, it implements the method as described in any one of claims 1-9.

Images