A data processing method, apparatus, device, and medium

Abstract

The application discloses a data processing method, device, equipment and medium, the data processing method comprises the steps that the original data is acquired from multiple data sources, and the original data is preprocessed to obtain preprocessed data; the preprocessed data is processed in real time according to a business rule to obtain business index data; the business index data is stored in a preset stream data lake; the business index data in the stream data lake is synchronized to a data warehouse corresponding to the stream data lake, and the data warehouse is pre-constructed according to the index data characteristics of the stream data lake. The method integrates multiple data sources, constructs an efficient stream data lake and data warehouse, realizes high-performance query, provides comprehensive, real-time and accurate data support for enterprises, and helps enterprises to realize data-driven decision and business innovation.

Description

Technical Field

[0001] This invention relates to the field of database technology, and in particular to a data processing method, apparatus, device, and medium. Background Technology

[0002] In today's digital age, enterprises face the challenge of massive, diverse, and rapidly growing data. This data comes from a wide range of sources, including but not limited to relational databases, non-relational databases, API interfaces, message queues, and various file systems. Traditional data storage and processing methods struggle to meet enterprises' demands for real-time, accurate, and high-performance data analysis. Existing data storage architectures often suffer from data silos, making it difficult to effectively integrate and share data from different sources, thus hindering the full realization of data value. Data warehouse construction and query performance also face bottlenecks, failing to meet enterprises' requirements for real-time analysis and rapid decision-making. Summary of the Invention

[0003] This invention provides a data processing method, apparatus, device, and medium to address the problem that existing data processing methods cannot meet enterprises' needs for real-time data analysis and decision-making.

[0004] In a first aspect, this application provides a data processing method, comprising the steps of: acquiring raw data from multiple data sources and preprocessing the raw data to obtain preprocessed data; performing real-time streaming processing on the preprocessed data according to business rules to obtain business indicator data; storing the business indicator data in a preset streaming data lake; and synchronizing the business indicator data in the streaming data lake to a data warehouse corresponding to the streaming data lake, wherein the data warehouse is pre-constructed based on the indicator data characteristics of the streaming data lake.

[0005] Secondly, this application provides a data processing apparatus, comprising: a data acquisition module for acquiring raw data from multiple data sources and preprocessing the raw data to obtain preprocessed data; a stream processing module for performing real-time stream processing on the preprocessed data according to business rules to generate business indicator data; a data storage module for storing the business indicator data in a preset streaming data lake; and a data synchronization module for synchronizing the business indicator data in the streaming data lake to a data warehouse corresponding to the streaming data lake, wherein the data warehouse is pre-constructed based on the indicator data characteristics of the streaming data lake.

[0006] Thirdly, this application provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the above-described data processing method.

[0007] Fourthly, this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described data processing method.

[0008] The aforementioned data processing method, apparatus, computer equipment, and storage medium include the following steps: acquiring raw data from multiple data sources and preprocessing the raw data to obtain preprocessed data; performing real-time stream processing on the preprocessed data according to business rules to obtain business indicator data; storing the business indicator data in a pre-defined streaming data lake; and synchronizing the business indicator data in the streaming data lake to a corresponding data warehouse, where the data warehouse is pre-built based on the characteristics of the streaming data lake's indicator data. This method integrates multiple data sources to construct an efficient streaming data lake and data warehouse, and enables high-performance queries, providing enterprises with comprehensive, real-time, and accurate data support, thus helping them achieve data-driven decision-making and business innovation. Attached Figure Description

[0009] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0010] Figure 1 This is a flowchart of a data processing method according to an embodiment of the present invention; Figure 2 This is a specific flowchart of step S2 in the data processing method of one embodiment of the present invention; Figure 3 This is a specific flowchart of step S3 in the data processing method in another embodiment of the present invention; Figure 4 This is a specific flowchart of the data processing method for constructing a data warehouse in one embodiment of the present invention; Figure 5 This is a specific flowchart of step S4 in the data processing method of one embodiment of the present invention; Figure 6 This is a specific flowchart of a data query and analysis process in a data processing method according to an embodiment of the present invention; Figure 7 This is a schematic diagram of a data processing device according to an embodiment of the present invention; Figure 8 This is a schematic diagram of a computer device according to an embodiment of the present invention. Detailed Implementation

[0011] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0012] In one embodiment, such as Figure 1 As shown, a data processing method is provided, including the following steps: Step S1: Obtain raw data from multiple data sources and preprocess the raw data to obtain preprocessed data.

[0013] It should be noted that the data source is a dynamically scalable heterogeneous data source, including but not limited to relational databases (such as MySQL, Oracle, etc.), non-relational databases (such as MongoDB, Redis, etc.), API interfaces, message queues (such as Kafka, RabbitMQ, etc.), and various file systems (such as local files, HDFS, etc.). Raw data is unprocessed information directly collected from various data sources, with significant differences in format, structure, semantics, and update frequency. Preprocessed data, on the other hand, is intermediate data resulting from preprocessing operations such as cleaning, format standardization, field mapping, and semantic alignment of the raw data. It eliminates noise, redundancy, and conflicts, laying a structured and consistent foundation for subsequent data storage and analysis.

[0014] In this embodiment, a unified acquisition adapter is designed to adapt to multiple data sources. It is specifically matched with acquisition strategies for heterogeneous data sources such as databases, message queues, and file systems. It supports multiple acquisition modes such as incremental capture, full synchronization, and change data capture (CDC). Dynamic configuration and hot-swappable expansion are achieved through metadata-driven mechanisms. The adapter has built-in fault-tolerant retry, breakpoint resume, and flow control mechanisms to ensure high concurrency, low latency, and high reliability of data access capabilities, providing stable and high-quality input guarantees for the subsequent construction of a unified data foundation.

[0015] Specifically, the unified data acquisition adapter can be designed with a plug-in architecture, supporting hot-swappable expansion of data source adaptation plug-ins, and adding specialized adaptation plug-ins for IoT device data streams and third-party SaaS platform APIs. Each plug-in interfaces with the core scheduling engine through standardized interfaces, enabling automatic metadata registration, dynamic orchestration of acquisition tasks, and real-time monitoring of execution status. During the acquisition process, the unified data acquisition adapter also tags the raw data with timestamps, source identifiers, and checksums to ensure data traceability and verifiable consistency.

[0016] Furthermore, during the data acquisition process, the collected raw data undergoes preliminary real-time preprocessing, including data cleaning, format conversion, and data filtering, to obtain preprocessed data and ensure data quality and consistency. Data cleaning includes missing value imputation, outlier identification and removal, duplicate record deduplication, and illegal character removal; format conversion covers timestamp standardization, encoding unification (e.g., UTF-8), numerical unit normalization, and unstructured text parsing; data filtering dynamically extracts subsets of data within valid time periods, geographical ranges, or key indicator thresholds based on business rules. All operations are executed with low latency in an in-memory streaming engine, ensuring that preprocessing results are available in real-time, semantically accurate, and traceable.

[0017] In this embodiment, a unified data acquisition adapter is developed, and corresponding acquisition strategies and technologies are adopted based on the characteristics of different data sources to ensure accurate, complete, and real-time data acquisition. Combined with a real-time preprocessing workflow, data cleaning and format standardization are achieved, breaking down data silos at the source to improve data quality and consistency, providing a reliable foundation for subsequent data storage and analysis.

[0018] In some embodiments, an AI-assisted cleaning mechanism can be introduced into the preprocessing stage. By training a lightweight data quality detection model, abnormal data can be automatically identified and missing values ​​can be filled in, replacing the traditional rule-based preprocessing logic and improving the intelligence level of data cleaning.

[0019] In other embodiments, appropriate data acquisition adapters can be selected according to the type and characteristics of each data source, and data acquisition tasks can be started independently to collect raw data from each data source in real time. The data is then aggregated in parallel to a unified acquisition bus, and unified scheduling and intelligent routing are completed through the bus. The data collected by each adapter carries metadata tags and integrity verification information before entering the bus, ensuring that cross-source data can achieve semantic alignment and quality verification at the aggregation layer.

[0020] Step S2: Perform real-time stream processing on the preprocessed data according to business rules to obtain business indicator data.

[0021] It's important to note that business rules are the mapping logic between data and business metrics, encompassing metric calculation formulas, dimensional slicing conditions, timeliness constraints, and compliance verification thresholds. They drive data filtering and aggregation strategies in the preprocessing stage and serve as semantic anchors for subsequent analysis and modeling. Real-time stream processing is the crucial bridge connecting preprocessed data and business metric data. By real-time splitting, aggregation, and calculation of preprocessed data according to business rules, it outputs a stream of metric data conforming to business definitions within millisecond-level latency.

[0022] In this embodiment, Apache Flink is used as the real-time processing engine. Preprocessed data is input into the Flink real-time processing engine, where it undergoes real-time stream processing including data transformation, aggregation, and computation according to preset business rules to obtain corresponding business metric data. Flink features low latency, high throughput, and strong fault tolerance, enabling it to process data streams from various data sources in real time and perform data transformation, aggregation, and computation according to preset business rules to generate valuable business metric data. For example, real-time statistics of user behavior data can be generated to produce user activity metrics; real-time analysis of transaction data can detect abnormal transaction behavior, etc.

[0023] like Figure 2 As shown, specifically, step S2 includes the following sub-steps: Step S21: Transform the preprocessed data to obtain structured data.

[0024] Data transformation refers to mapping preprocessed data into a unified structure that conforms to business semantics. This includes operations such as field standardization, unit unification, timestamp alignment, and semantic tag injection to ensure that data from different sources are logically comparable, interconnected, and traceable. For example, the "order amount" field is uniformly mapped to RMB yuan and the business context tag "transaction scenario = online retail" is added to support subsequent multi-dimensional real-time analysis and dynamic attribution.

[0025] Step S22: Perform sliding window aggregation calculations on the structured data to generate time series indicators.

[0026] Among them, time series indicator data is generated based on sliding window aggregation calculation. The window size and step size are dynamically configured according to the business timeliness requirements. For example, a 10-second sliding window is used for second-level monitoring and a 60-second sliding window is used for minute-level analysis. The aggregation operation covers counting, summation, mean, deduplication statistics and abnormal mutation detection, and outputs standardized time series indicator data containing timestamps, indicator names, dimension combinations and values.

[0027] For example, a 10-second sliding window is used for the "new users per second" metric to count the peak registrations from various channels in real time; a 60-second sliding window is used for the "order payment success rate," combined with deduplication and abnormal change detection to identify instantaneous system fluctuations. All outputs carry a unified schema: timestamp, metric_name, dimensions (JSON format), value, and status_code, ensuring that downstream BI dashboards, alarm engines, and AI attribution models can directly consume them.

[0028] Step S23: Combine business rules to perform rule matching on time series indicators to obtain business indicator data that conforms to business logic.

[0029] The rule matching is based on a dynamic rule engine (such as Drools or a self-developed lightweight rule center). It compares time-series indicators with preset thresholds, trend patterns, and related dependency conditions in real time to obtain business indicator data that conforms to business logic. For example, when the "payment success rate" is below 99.5% three times in a 60-second window and is accompanied by a sudden increase in error codes 401, the rule engine immediately triggers a "payment link anomaly" alarm and outputs structured business indicators containing root cause dimensions (such as channel=APP, version=Android_8.2.1).

[0030] Furthermore, the business rules support hot updates and canary releases, ensuring that rule iterations do not affect the stability of the real-time link; all matching results are automatically tagged and injected with source metadata, including trigger rule ID, matching time, confidence level and original indicator snapshot, providing a complete chain of evidence for subsequent root cause analysis and strategy backtracking.

[0031] In this embodiment, Flink's low-latency characteristics are used to achieve real-time data stream processing, enabling millisecond-level matching between business rules and time-series metrics. This ensures the rapid acquisition of business metric data that conforms to business logic, supporting a second-level decision-making loop.

[0032] Step S3: Store business metrics data in a pre-defined streaming data lake.

[0033] It should be noted that a streaming data lake is a new architecture that integrates stream processing and lake storage. It can directly write real-time collected business metric data streams into a hierarchical lake warehouse, supporting ACID transactions and millisecond-level visibility.

[0034] In this embodiment, the streaming data lake is built using Paimon and HDFS. Paimon is a high-performance streaming data lake storage format with fast data write and query performance, supporting real-time data storage and analysis. HDFS, as a distributed file system, provides highly reliable and scalable storage capabilities, ensuring secure data storage and long-term preservation. By combining Paimon and HDFS, the needs of real-time data processing can be met while ensuring data reliability and scalability. The Paimon layer automatically partitions data by business domain, supports schema evolution and merge-on-read, ensuring that adding or deleting metric fields does not affect downstream consumption. The HDFS underlying layer enables Erasure Coding to reduce storage overhead by 30%, while implementing field-level permission control through Ranger. All written data is automatically marked with watermarks and event-time stamps to ensure accurate window calculations in out-of-order scenarios.

[0035] Furthermore, the streaming data lake comprises a raw data layer, a preprocessed data layer, and a business indicator data layer, enabling layered storage of raw data, preprocessed data, and business indicator data. These three layers strictly adhere to the principles of "immutable raw data, traceable preprocessing logic, and verifiable business indicators": the raw data layer stores raw data collected from various data sources, preserving the data's integrity and original characteristics; the preprocessed data layer stores preprocessed data generated after further cleaning, transformation, and integration of the raw data, resulting in more standardized and consistent data; and the business indicator data layer stores business indicator data that has been verified, aggregated, and annotated with business semantics. Each layer's data integrity is verified through a unique data index, and the entire indicator generation process is traceable, facilitating subsequent data analysis and querying.

[0036] Therefore, while performing step S3, it is also necessary to simultaneously write the raw data and preprocessed data to their respective data layers. Specifically, by creating a streaming data lake storage directory structure on HDFS, different types of data are stored in their respective directories according to the principle of hierarchical data storage, and the Paimon format is used to store the data, optimizing the data storage structure and indexes, and improving data query performance.

[0037] Furthermore, the data in the streaming data lake is regularly cleaned and archived, and expired and invalid data is deleted to free up storage space.

[0038] In another embodiment, based on the raw data layer, preprocessed data layer, and business indicator data layer, a three-tiered storage strategy (hot, warm, and cold) is implemented for the business indicator data in the business indicator data layer. Business indicator data is automatically categorized according to access frequency and timeliness: hot data is migrated to an SSD (Solid State Drive) cache cluster for storage, ensuring sub-second response times; warm data remains in Paimon+HDFS hybrid storage, balancing performance and cost; cold data is migrated to object storage and enabled with lifecycle management, retaining ten years of compliance traceability. The tiered storage strategy is dynamically adjusted in real-time by Flink jobs based on data popularity, ensuring precise matching of resource investment with business value.

[0039] like Figure 3 As shown, specifically, step S3 includes the following sub-steps: Step S31: Obtain the data popularity level of business indicator data.

[0040] It should be noted that data popularity level is a multi-dimensional feature describing data access frequency, update frequency, and business priority. Data popularity level includes first popularity level, second popularity level, and third popularity level. The first popularity level corresponds to hot data, requiring a response latency ≤100ms, a daily update frequency ≥100 times, and a business priority of P0; the second popularity level corresponds to warm data, with a response latency between 100ms and 2s, a daily update frequency of 10-99 times, and a business priority of P1; the third popularity level corresponds to cold data, with a response latency >2s, a daily update frequency <10 times, and a business priority of P2 or P3.

[0041] Specifically, a popularity scoring model is constructed by continuously collecting query logs, write logs, and scheduling metadata through the Flink real-time computing engine. This model integrates time decay factors and business weight coefficients to generate dynamic popularity scores for each indicator table, partition, and even field, and synchronously injects them into the data directory service as the core basis for subsequent tiered storage strategy execution. The popularity score is quantified in the range of 0 to 100, with scores ≥85 representing the first popularity level, 70-84 representing the second popularity level, and <70 representing the third popularity level.

[0042] By acquiring the data popularity level of business indicator data, we can accurately identify its storage capacity requirements and value timeliness boundaries, thereby driving the intelligent orchestration and elastic scheduling of subsequent storage resources.

[0043] Step S32: Based on the data popularity level, allocate the business indicator data to the storage partitions of the corresponding popularity level in the streaming data lake.

[0044] It should be noted that storage partitions include three types: hot storage partitions, warm storage partitions, and cold storage partitions. Hot storage partitions correspond to SSD cache clusters and are used to store business metric data of the highest priority, ensuring low latency and high concurrency access. Warm storage partitions, based on a Paimon+HDFS architecture, carry business metric data of the second highest priority, balancing throughput performance and storage costs. Cold storage partitions rely on object storage to store business metric data of the third highest priority, and are automatically archived, encrypted, and subject to compliance auditing through lifecycle policies. All storage partitions are interconnected through a unified metadata service, supporting transparent cross-partition queries and seamless migration during priority jumps.

[0045] Specifically, based on the data popularity level, business indicator data is allocated to the corresponding storage medium in the streaming data lake. That is, hot data corresponding to the first popularity level is migrated to the SSD cache cluster; warm data corresponding to the second popularity level is still stored in the Paimon+HDFS hybrid storage layer; and cold data corresponding to the third popularity level is migrated to object storage, and compliance retention policies and access audit tags are automatically bound. All migration actions are triggered through a unified metadata service to ensure cross-level data consistency and transaction atomicity.

[0046] Furthermore, the entire migration process is visualized, with the popularity score, ownership level, storage path, and policy effective time of each data point being synchronized to the operations and maintenance dashboard in real time. When the popularity score of a certain indicator table is lower than the corresponding scoring threshold for 30 consecutive minutes, a re-evaluation and dynamic migration are automatically triggered to avoid "hot and cold mismatch".

[0047] In other embodiments, Z-order indexes can be used to replace the original conventional data index design, optimizing data retrieval efficiency in multi-dimensional query scenarios. Z-order indexes map multi-dimensional data into a one-dimensional ordered sequence through a space-filling curve, significantly reducing file scans and I / O overhead during queries. By combining Bloom filters and Min / Max statistics at the Paimon layer, irrelevant partitions can be skipped, significantly reducing the response time for typical multi-dimensional analysis queries. This optimization, in conjunction with a popularity-based ranking strategy, ensures that high-value indicators are both "stored efficiently" and "queried quickly." Furthermore, a dynamic re-evaluation mechanism for popularity levels can be supported, automatically updating weight coefficients hourly based on the latest access logs and model call chains, and triggering storage-level migration tasks. All migration operations are atomically guaranteed through transaction snapshots, avoiding query interruptions or data inconsistencies.

[0048] Therefore, this application adopts a collaborative architecture of Flink+Paimon+HDFS, utilizes Flink's low latency to achieve real-time data streaming processing, leverages Paimon's streaming storage advantages and HDFS's high reliability and scalability to build a streaming data lake, and uses a tiered storage design to achieve gradient management from raw data to business indicator data, balancing real-time processing and long-term storage requirements.

[0049] Step S4: Synchronize the business metric data in the streaming data lake to the data warehouse corresponding to the streaming data lake.

[0050] It should be noted that the data warehouse is pre-built based on the metric data characteristics of the streaming data lake. The data warehouse consists of multiple data tables, which are linked together through predefined foreign key relationships.

[0051] In this embodiment, the data tables include fact tables and dimension tables. Based on the data in the data lake, a dimensional modeling approach is used to construct the data warehouse. Specifically, data relationships are established by constructing these two types of data tables—fact tables and dimension tables—to form a structured data model. The fact table carries core business metrics. The business meaning of each row of data in the fact table, i.e., the smallest unit of analysis (such as "a product item purchased by a user in an order"), is the granularity of analysis. The granularity of analysis must be clear and unambiguous, and determine the alignment of subsequent dimensions with the fact design, such as aggregateable metrics like order amount and user activity duration. The dimension table describes the context of the business scenario, such as attributes like time, region, and product category. By identifying the analytical perspective describing the business process, dimensions (such as time, user, product, and channel) are extracted through questions like "who, what, when, where, and how," thus constructing a dimension table with rich attributes (such as the user dimension including ID, region, and membership level). The two are linked through foreign keys, ensuring the traceability of historical data while supporting flexible management of slowly changing dimensions.

[0052] like Figure 4 As shown, specifically, the methods for building a data warehouse include: Step S01: Based on the business indicator data, determine the granularity and perspective of the business to be analyzed.

[0053] It's important to note that the business processes to be analyzed are core business processes such as user activity, order conversion, and product exposure. Clearly defining the subjects, actions, and business contexts involved is crucial, as these processes are closely related to business metrics. Therefore, based on these metrics, the granularity and perspective of the analysis can be precisely defined. The granularity of analysis is the smallest unit of business logic, such as "single click," "single session," or "single payment," which directly determines the row-level structure of the fact table and the method of metric aggregation. The perspective corresponds to the key dimension paths in dimensional modeling. For example, user activity analysis needs to be linked to the three dimensions of time, device, and region; order conversion needs to be associated with marketing activities, channel sources, and user segments.

[0054] By analyzing business metrics data, the granularity and perspective of the analysis can be determined, providing precise support for building fact tables and dimension tables.

[0055] Step S02: Construct a data warehouse based on the granularity and perspective of the analysis.

[0056] The system comprises two parts: a fact table built based on analytical granularity, using measurable business behaviors (such as click count, order amount, and session duration) as fact fields; and a dimension table built based on analytical perspectives, storing contextual attributes such as time, user, product, and channel in a structured manner. These two tables are linked by foreign keys to form a star schema, ensuring query efficiency and semantic clarity. Foreign key constraints not only guarantee data consistency but also support multi-dimensional drill-down and roll-up analysis. For example, the order fact table is linked to the user, time, and product dimension tables via user_id, time_id, and product_id respectively, enabling real-time dashboards to respond in seconds to complex queries such as "repurchase rate of high-value members in East China over the past seven days."

[0057] In other embodiments, the data warehouse can adopt a hybrid modeling approach of "wide table + dimension table" instead of pure dimension modeling. Wide tables are pre-built for high-frequency query scenarios to reduce the overhead of join queries. The synchronization mechanism introduces incremental synchronization optimization using CDC (Change Data Capture) technology, which realizes real-time capture and synchronization of data changes by parsing database logs, replacing the original periodic ETL incremental synchronization scheme and reducing data synchronization latency.

[0058] Furthermore, after determining that a data warehouse has been obtained, step S4 can be executed to synchronize the business indicator data to the corresponding data warehouse.

[0059] like Figure 5 As shown, specifically, step S4 includes the following sub-steps: Step S41: Based on the data storage structure of the data warehouse, extract business indicator data that conforms to the structure mapping relationship from the streaming data lake as data to be synchronized.

[0060] It should be noted that the data warehouse structure is a joint definition of fact tables and dimension tables, encompassing field naming conventions, data type constraints, primary and foreign key relationships, and partitioning strategies. The data to be synchronized consists of cleaned and standardized business metric data from the streaming data lake, ready to be synchronized to the data warehouse. Its field semantics, granularity level, and dimension paths must strictly match the corresponding data warehouse structure.

[0061] In this embodiment, based on the data warehouse's data storage structure, the data warehouse's ETL (Extract-Transform-Load) program periodically extracts business indicator data that conforms to the structural mapping relationship from the streaming data lake as data to be synchronized, and performs field mapping, type conversion, and primary / foreign key verification. The synchronization process adopts an idempotent write mechanism to ensure that the same business event is written only once under multiple triggers, preventing data duplication. At the same time, the ETL program embeds a real-time quality probe to provide real-time alerts for anomalies such as null value rate, distribution offset, and out-of-bounds values, and automatically triggers the data repair process.

[0062] Step S42: Perform dimensional association on the data to be synchronized to obtain the associated dimensional association data.

[0063] It should be noted that dimension-related data refers to hashing the business keys (such as user_id, product_id) in the data to be synchronized with the primary key of the dimension table to complete the dimension attributes (such as user region, product category, time period, etc.) and generate a wide table record with complete context. The association process adopts the optimization strategy of broadcast small table + bucketed large table, and realizes millisecond-level dimension pushdown in Flink SQL to ensure that the attribute integrity and timeliness are both met when the wide table is built.

[0064] In this embodiment, after the data to be synchronized is associated with dimensions, dimension-related data with a complete business context is generated, which is then written into the data warehouse. This dimension-related data has embedded timestamps, business partition identifiers, and lineage tags, supporting downstream ad-hoc queries and real-time analysis.

[0065] Step S43: Synchronize the dimension-related data to the corresponding data tables in the data warehouse.

[0066] In this embodiment, dimension-related data is ingested into the data warehouse in real time via a Kafka message queue, written to the corresponding fact table partition of the data warehouse according to the business partition key. During synchronization, the uniqueness of primary keys and the integrity of foreign key references are automatically verified, and the metadata lineage graph is updated. All write operations are persisted through transaction logs to ensure end-to-end exactly-Once semantics.

[0067] Furthermore, a data update mechanism is established to perform quality checks and verifications on the data in the data warehouse, ensuring its accuracy and completeness. New and changed data from the streaming data lake are regularly synchronized to the data warehouse to ensure consistency between the two. Simultaneously, the data in the data warehouse is regularly maintained and optimized, including data cleaning, index rebuilding, and performance tuning, to guarantee its performance and stability.

[0068] In this application, an enterprise-level data warehouse is constructed based on data from a streaming data lake using dimensional modeling. An automated ETL and incremental synchronization process is designed to ensure real-time consistency between the data warehouse and the data stored in the streaming data lake. Regular optimization and maintenance ensure the data warehouse's structure and stability, improving data reuse efficiency. Furthermore, the data warehouse provides enterprises with a more structured and standardized data view, facilitating multi-dimensional data analysis and decision support.

[0069] Furthermore, the data processing method also includes: using a data warehouse that has completed data synchronization to perform data query and analysis to obtain the target analysis results.

[0070] It's worth noting that StarRocks is a high-performance, real-time analytical database with fast query response times and powerful concurrency capabilities. By building external tables, you can fully leverage StarRocks' query optimization engine and parallel computing capabilities to perform high-performance queries on data in the data warehouse.

[0071] In this embodiment, by creating a table in StarRocks and configuring its connection parameters and data mapping rules, data from the data warehouse is mapped to StarRocks, ensuring accurate data mapping. When users perform multi-dimensional and complex data queries and analyses on StarRocks tables using SQL statements, StarRocks can leverage its high-performance query engine and parallel computing capabilities to quickly return query results. Simultaneously, StarRocks provides a visual query interface and reporting tools, allowing users to intuitively view and analyze query results, generate various reports and visualizations, and provide strong support for enterprise decision-making.

[0072] like Figure 6 As shown, using a data warehouse that has completed data synchronization, data query and analysis are performed to obtain the target analysis results. This includes the following sub-steps: Step S51: Obtain the user's query analysis request and extract the query keywords and target analysis dimensions that contain the user's query intent.

[0073] It's important to note that query analysis requests are based on user-input structured SQL queries. The system parses the SQL syntax tree to identify key fields, filtering conditions, aggregation logic, and relationships that contain the user's query intent. Query keywords refer to field names, table names, condition values, and function identifiers explicitly specified by the user in the SQL statement, such as "sales revenue," "Q1 2026," or "GROUP BY region." The system dynamically constructs an execution plan based on these. Target analysis dimensions are business perspectives specified by the user in the query, such as time, region, or product line, which automatically associates dimension tables with fact tables.

[0074] In this embodiment, based on the structured SQL query statement input by the user, query keywords and target analysis dimensions containing the user's query intent are obtained, providing accurate semantic support for subsequent query execution.

[0075] Step S52: Based on the query keywords, retrieve the business indicator data with the highest matching degree from the data warehouse as candidate data.

[0076] It should be noted that candidate data are business metrics that directly reflect the user's query intent and are awaiting further analysis.

[0077] In this embodiment, based on the query keywords, the semantic matching degree between them and the metadata in the data warehouse is calculated. Priority is given to matching field names, comments, business tags, and historical query popularity. The business indicator data with the highest matching degree is retrieved from the data warehouse as candidate data for further data analysis.

[0078] Step S53: Perform multidimensional analysis on the candidate data according to the target analysis dimensions to generate interactive target analysis results.

[0079] It should be noted that the target analysis results are structured analysis results generated after performing data aggregation, filtering, sorting, and association on the candidate data. The format is a wide table or summary view that meets the user's expectations.

[0080] In this embodiment, multidimensional analysis is performed on candidate data based on the target analysis dimensions. Time, region, product line and other dimension tables are automatically associated. Preset aggregation rules and business logic functions are applied to generate target analysis results that meet the user's intent and present them in real time in a dual-modal form of visual charts and structured tables.

[0081] Furthermore, it synchronously records the query context and analysis path, supports one-click tracing of data lineage, and backtracking of original fact table and dimension table versions, ensuring that the analysis process is auditable and the results are reproducible.

[0082] In this embodiment, data warehouse data is mapped through StarRocks external tables, making full use of its query optimization engine and parallel computing capabilities to support multi-dimensional SQL queries and visualization analysis, significantly improving query response speed and concurrent processing capabilities, and solving the performance bottleneck of traditional data warehouse queries.

[0083] In other embodiments, a data federation query architecture can also be adopted, using Trino as a query middleware to uniformly connect to streaming data lakes, data warehouses and original data sources, and realize joint queries across data sources; at the same time, a new query result caching mechanism is added, which caches high-frequency query results based on the LRU algorithm, further improving query response speed and avoiding strong dependence on specific query engines.

[0084] In other embodiments, Spark Streaming can be used instead of Flink for near real-time stream processing, Apache Iceberg can replace Paimon for building a streaming data lake, and ClickHouse can replace StarRocks for high-performance queries. This solution avoids the specific technical dependencies of the original components by combining mature components at the same functional level, while retaining the core process of "multi-source acquisition - lakehouse storage - efficient querying," meeting the needs of different technology stack selections. Edge nodes can also be added at the multi-source data acquisition end to first complete local lightweight preprocessing (deduplication, format conversion), and then synchronize the data to the cloud streaming data lake / data warehouse through a lightweight message protocol (such as MQTT). This variant solution is suitable for data source access scenarios in cross-regional and weak network environments. By distributing the processing pressure through edge nodes, the cloud transmission and processing load is reduced. The core innovation still revolves around "layered processing + lakehouse linkage."

[0085] In some embodiments, a modified data encryption solution is designed for highly compliant scenarios such as finance and government affairs: Sensitive data is anonymized during the data collection phase; transparent encryption is used for streaming data lake / data warehouse storage; and the query phase supports the display of anonymized results and hierarchical access control. An operation log auditing module is added to the synchronization phase to retain records of the entire data flow, meeting compliance traceability requirements. While maintaining the core technical framework, this solution adapts to specific industry needs by adding a compliance enhancement module, strengthening data sovereignty and privacy protection capabilities, achieving security control and accountability in corresponding scenarios, and making this method feasible for implementation in highly sensitive businesses.

[0086] In summary, the data processing method of this application includes the following steps: acquiring raw data from multiple data sources and preprocessing the raw data to obtain preprocessed data; performing real-time stream processing on the preprocessed data according to business rules to obtain business indicator data; storing the business indicator data in a pre-defined streaming data lake; and synchronizing the business indicator data in the streaming data lake to a data warehouse corresponding to the streaming data lake, wherein the data warehouse is pre-built based on the indicator data characteristics of the streaming data lake. This method integrates multiple data sources to construct an efficient streaming data lake and data warehouse, and achieves high-performance queries, providing enterprises with comprehensive, real-time, and accurate data support, and helping enterprises achieve data-driven decision-making and business innovation. This technology can be widely applied to various scenarios requiring metadata management of multi-source heterogeneous data, such as enterprise data platform construction and big data analysis platform construction, providing strong support for accurate data management and efficient retrieval.

[0087] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0088] In one embodiment, a data processing apparatus is provided, which corresponds one-to-one with the data processing methods described in the above embodiments. For example... Figure 7 As shown, the data processing device includes a data acquisition module 101, a stream processing module 102, a data storage module 103, and a data synchronization module 104. Detailed descriptions of each functional module are as follows: The data acquisition module 101 is used to acquire raw data from multiple data sources and preprocess the raw data to obtain preprocessed data.

[0089] The stream processing module 102 is used to perform real-time stream processing on preprocessed data according to business rules to generate business indicator data.

[0090] The data storage module 103 is used to store business indicator data to a preset streaming data lake.

[0091] The data synchronization module 104 is used to synchronize business indicator data in the streaming data lake to the data warehouse corresponding to the streaming data lake. The data warehouse is pre-built according to the characteristics of the indicator data of the streaming data lake.

[0092] Specific limitations regarding the data processing device can be found in the limitations regarding the data processing method described above, and will not be repeated here. Each module in the aforementioned data processing device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in the computer device in hardware form, or stored in the memory of the computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0093] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 8 As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used for communication with external terminals via a network connection. When the computer program is executed by the processor, it implements a data processing method.

[0094] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the data processing method described in the above embodiment, for example... Figure 1As shown in S1-S4, or Figures 2 to 6 As shown, to avoid repetition, it will not be described again here. Alternatively, when the processor executes a computer program, it implements the functions of each module / unit in this embodiment of the data processing device, for example... Figure 7 The functions of the data acquisition module 101, stream processing module 102, data storage module 103, and data synchronization module 104 shown are not described again here to avoid duplication.

[0095] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When executed by a processor, the computer program implements the data processing method described in the above embodiments, for example... Figure 1 As shown in S1-S4, or Figures 2 to 6 As shown, to avoid repetition, it will not be described again here. Alternatively, when the computer program is executed by the processor, it implements the functions of each module / unit in this embodiment of the data processing apparatus, for example... Figure 7 The functions of the data acquisition module 101, stream processing module 102, data storage module 103, and data synchronization module 104 shown are not described again here to avoid repetition. The computer-readable storage medium can be non-volatile or volatile.

[0096] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0097] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0098] It should be noted that any AI models, software tools, or components not belonging to this company appearing in the embodiments of this application are merely illustrative examples and do not represent actual use. All user personal information involved in the embodiments of this application has been authorized (with the knowledge and consent) by the relevant parties or has been fully authorized by all parties, and the executing entity may obtain it through various legal and compliant means. The collection, storage, use, processing, transmission, provision, and disclosure of the information, data, and signals involved all comply with relevant laws and regulations and do not violate public order and good morals.

[0099] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A data processing method, characterized in that, Including the following steps: Raw data is obtained from multiple data sources, and the raw data is preprocessed to obtain preprocessed data; The preprocessed data is processed in real time according to business rules to obtain business indicator data. The business metrics data are stored in a pre-defined streaming data lake; The business metric data in the streaming data lake is synchronized to the data warehouse corresponding to the streaming data lake, and the data warehouse is pre-built according to the metric data characteristics of the streaming data lake.

2. The data processing method according to claim 1, characterized in that, The data processing method further includes: Obtain user query analysis requests and extract query keywords and target analysis dimensions that contain user query intent; Based on the query keywords, the business indicator data with the highest matching degree is retrieved from the data warehouse as candidate data; Perform multidimensional analysis on the candidate data according to the target analysis dimensions to generate interactive target analysis results.

3. The data processing method according to claim 1, characterized in that, The step of performing real-time stream processing on the preprocessed data according to business rules to obtain business indicator data includes: The preprocessed data is then transformed to obtain structured data; The structured data is subjected to sliding window aggregation calculation to generate time series indicators; Based on the aforementioned business rules, the time series indicators are matched according to the rules to obtain the business indicator data that conforms to the business logic.

4. The data processing method according to claim 1, characterized in that, The step of storing the business metric data in a preset streaming data lake includes: Obtain the data popularity level of the aforementioned business indicator data; Based on the data popularity level, the business indicator data is allocated to the storage partition of the corresponding popularity level in the streaming data lake.

5. The data processing method according to claim 1, characterized in that, The data warehouse includes multiple data tables, which are linked through predefined foreign key relationships; synchronizing business indicator data from the streaming data lake to the corresponding data warehouse includes: Based on the data storage structure of the data warehouse, the business indicator data that conforms to the structure mapping relationship is extracted from the streaming data lake as data to be synchronized; Perform dimensional correlation on the data to be synchronized to obtain the correlated dimensional correlation data; The dimension-related data is synchronized to the corresponding data table in the data warehouse.

6. The data processing method according to claim 1, characterized in that, The method for constructing the data warehouse includes: Based on the aforementioned business indicator data, determine the granularity and perspective of the analysis of the business to be analyzed; The data warehouse is constructed based on the stated granularity and perspective of analysis.

7. The data processing method according to claim 1, characterized in that, The preprocessing includes data cleaning, format conversion, and data filtering; the preprocessing of the original data to obtain preprocessed data includes: The raw data is cleaned, format converted, and filtered to obtain the preprocessed data.

8. A data processing apparatus, characterized in that, include: The data acquisition module is used to acquire raw data from multiple data sources and preprocess the raw data to obtain preprocessed data. The stream processing module is used to perform real-time stream processing on the preprocessed data according to business rules to generate business indicator data; The data storage module is used to store the business indicator data to a preset streaming data lake; The data synchronization module is used to synchronize the business indicator data in the streaming data lake to the data warehouse corresponding to the streaming data lake. The data warehouse is pre-built according to the indicator data characteristics of the streaming data lake.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the data processing method as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the data processing method as described in any one of claims 1 to 7.

Images