Operational statistics consistency verification method and device based on dynamic data cube

Abstract

This invention discloses a method and apparatus for verifying operational statistics consistency based on a dynamic data cube. By constructing a dynamically scalable multidimensional data cube, it achieves flexible dimensional expansion and optimized query performance. Through a multi-layered intelligent verification mechanism, it ensures data consistency across all processing stages. Furthermore, through data lineage analysis, it enables comprehensive tracking of data sources and impacts. This invention effectively solves the technical problems of poor dimensional expansion flexibility, insufficient query performance, difficulty in guaranteeing data consistency, and unclear data lineage in existing technologies, providing enterprises with a more efficient and reliable operational statistics system.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and specifically to a method and apparatus for verifying operational statistics consistency based on dynamic data cubes. Background Technology

[0002] Operational statistics systems are core tools for enterprise data analysis, used to aggregate and analyze various types of business data to provide decision support for management. Traditional operational statistics systems typically employ a fixed-dimensional data analysis framework, constructing reports and analytical views through predefined combinations of dimensions.

[0003] Common operational statistics techniques include traditional OLAP (Online Transaction Processing) systems and data warehouse solutions. These techniques typically use star or snowflake schemas to build data structures, load business data through ETL processes, and then perform statistical analysis based on fixed dimensions. For example, some systems use predefined dimensions such as time, region, and product to build multidimensional data cubes, allowing users to analyze the data through operations such as slicing, dicing, and drill-down.

[0004] However, existing operational statistics systems primarily employ predefined dimensional models when dealing with multi-dimensional statistics and analysis. Developers define a fixed dimensional structure during the system design phase. When new statistical dimensions are needed, developers must modify the system code and redeploy, resulting in slow response times and difficulty adapting to rapidly changing business requirements. Furthermore, performance bottlenecks exist when handling large-scale time-span data queries, leading to long query response times. More seriously, during the generation of multi-dimensional reports, inconsistencies in key indicator data frequently occur due to diverse data sources and complex processing steps, impacting data reliability.

[0005] The main problems with existing technologies include: poor flexibility in dimensional expansion, requiring developers to manually modify code; insufficient performance for large-scale data queries, especially over long time spans; difficulty in ensuring data consistency across multi-dimensional reports; and unclear data lineage, making it difficult to trace data sources and calculation processes. These problems severely restrict the operational statistics system's ability to support business decision-making. Summary of the Invention

[0006] The purpose of this invention is to provide an operational statistics consistency verification method and apparatus based on dynamic data cubes, which solves the technical problems in the prior art such as poor flexibility in dimensional expansion, insufficient query performance, difficulty in guaranteeing data consistency, and unclear data lineage.

[0007] The operational statistics consistency verification method based on dynamic data cubes provided by this invention includes:

[0008] The system acquires raw data including customer information, product information, and transaction records. Based on the raw data, it constructs an OLAP cube structure that includes time, region, and product category dimensions. Based on the OLAP cube structure, it pre-calculates the aggregation results corresponding to the target query scenario. Based on the aggregation results, it obtains a multidimensional OLAP data structure.

[0009] The data integrity and format are verified through the original data verification layer based on the multidimensional OLAP data structure. Then, in the ETL intermediate layer, the data verified by the original data verification layer is verified for business logic. Based on the data verified by the ETL intermediate layer, preset dimension indicators are calculated to obtain indicator data. In the indicator calculation layer, the indicator data is verified using linear congruent check codes and inverted index technology. Finally, the data format and business meaning are verified in the presentation layer to obtain the verified reliable data.

[0010] Based on the verified reliable data, metadata information is collected and assigned a unique identifier. SQL statements are parsed, and data relationships are extracted based on the parsing results. A data lineage graph is constructed based on the metadata information and the data relationships using a graph algorithm. Data tracing and impact analysis results are obtained based on the data lineage graph.

[0011] Specifically, the process involves constructing an OLAP cube structure based on the original data, including dimensions of time, region, and product category; pre-calculating aggregation results corresponding to the target query scenario based on the OLAP cube structure; and obtaining a multidimensional OLAP data structure based on the aggregation results, including:

[0012] The raw data is cleaned, converted in format, and standardized to obtain standardized business data.

[0013] Based on the standardized business data, a multi-dimensional structure is formed according to the dimensions of time, region, and product category, resulting in an OLAP cube structure.

[0014] For the target query scenario in the OLAP cube structure, the aggregation result of each dimension combination corresponding to the OLAP cube structure is pre-calculated;

[0015] After caching the aggregation results, a multidimensional OLAP data structure is obtained.

[0016] After obtaining the multidimensional OLAP data structure, the method further includes:

[0017] Receive a user's data query request, parse the time dimension parameter in the data query request, and obtain the time query conditions;

[0018] For the time query conditions, a time bucketing function is used to perform bucketing, dividing the time into different granularities such as hour, day, week, and month to obtain the bucketed time dimensions;

[0019] Based on the time dimension after bucketing, multi-level aggregated views are pre-calculated and stored;

[0020] Based on the time range of the data query request, the most suitable aggregated view is selected, and data is obtained from the selected aggregated view and the minimum necessary calculations are performed to obtain the optimized query results.

[0021] The step of verifying the indicator data using linear congruential checksums and inverted indexes at the indicator calculation layer includes:

[0022] For the aforementioned indicator data, a corresponding check code is generated using a linear congruential algorithm to obtain the indicator check code;

[0023] Construct an inverted index of the indicator check code, establish a mapping relationship between the indicator value and its corresponding indicator check code, and obtain the check code index;

[0024] When the same indicator is calculated multiple times under different dimension combinations, the consistency of the calculation results is verified by the check code index to obtain the consistency verification result;

[0025] When inconsistency is determined based on the consistency verification results, a traceability process is initiated to obtain corrected consistency index data.

[0026] The step of constructing a data lineage graph based on the metadata information and the data relationships using a graph algorithm includes:

[0027] Based on the metadata and relational information, the graph database is initialized to obtain the graph storage structure;

[0028] Based on the graph storage structure, data elements, indicators, and report-type entities are used as nodes in the graph to obtain lineage nodes;

[0029] Based on the dependencies obtained from the parsing, directed edges representing data flow and computational dependencies are established between related bloodline nodes to obtain bloodline relationship edges;

[0030] The graph structure containing the bloodline nodes and bloodline relationship edges is optimized and its integrity is verified using graph algorithms to obtain a data bloodline relationship graph.

[0031] The process of obtaining the verified reliable data further includes:

[0032] Based on the aforementioned multidimensional OLAP data structure, a temporal logic model for data attributes is established, and business rules are transformed into LTLf formulas to obtain temporal attribute definitions.

[0033] Maintain a timestamp version chain for each data element to record its state at different processing stages, thus obtaining a temporal state chain.

[0034] The reactive synthesis algorithm of LTLfMT is executed to automatically generate a monitor program based on the temporal attribute definition, thus obtaining a temporal monitor.

[0035] The temporal state chain is used to perform cross-time point data comparison to verify whether the growth rate or year-on-year change meets business expectations and obtain temporal verification results.

[0036] When the temporal monitor detects a violation of temporal attributes, it records the violation information, analyzes the cause of the violation, and performs automatic repair according to a preset repair strategy to obtain the temporally repaired data.

[0037] The process of obtaining the multidimensional OLAP data structure further includes:

[0038] Based on the multidimensional OLAP data structure and CSLPerm rules, a score-based permission is assigned to each data resource, and the score-based permission is adjusted according to the type of concurrent operation to generate dynamic permission allocation;

[0039] Based on the dynamic permission allocation and separation logic strategy, a separation verification mechanism is implemented for the heap memory accessed by concurrent operations.

[0040] When performing a dynamic dimension expansion operation, the CSLPerm rule is used to control concurrent access and lock the affected data area;

[0041] Based on the residual heap inference algorithm, the heap state changes before and after data operations are automatically analyzed and inferred to track the impact range of each data operation and obtain the heap state change analysis results.

[0042] The process of obtaining the multidimensional OLAP data structure further includes:

[0043] Based on the aforementioned multidimensional OLAP data structure, historical query logs are collected and analyzed, and the state space of a Markov decision process is constructed based on the analysis results.

[0044] Design a multi-dimensional reward function, the optimization objectives of which include query response time, resource consumption and data accuracy, and dynamically adjust the weight of each objective of the reward function according to the business scenario to obtain a multi-objective reward function;

[0045] An optimal query strategy is generated based on the state space and the multi-objective reward function using a reinforcement learning algorithm, wherein long-term query behavior patterns are incorporated through ω-regularization.

[0046] For each query decision, a formalized certificate and decision basis record are generated. The certificate is used to characterize the optimality of the decision, and the decision basis record is used to record the reasoning basis of the decision.

[0047] The system executes a query strategy that carries certificates and decision-making basis records, monitors the actual results of the query execution, compares the actual results with the expected results, and automatically adjusts the model parameters and reward function of the state space based on the resulting deviations to obtain an optimized query strategy.

[0048] The process of obtaining the multidimensional OLAP data structure further includes:

[0049] A Markov decision process model is constructed based on the data lineage graph as a data quality MDP model. The quality indicators of the data quality MDP model include data integrity, accuracy, and consistency.

[0050] Based on the data quality MDP model, multiple data quality objectives are defined, including accuracy, completeness, timeliness, and consistency. A corresponding reward function is set for each data quality objective. The optimal data management strategy matching the multiple data quality objectives is determined by the multi-objective MDP solution algorithm, thus obtaining the multi-objective quality optimization strategy.

[0051] Based on the multi-objective quality optimization strategy, long-term behavior norms for data quality are defined using ω-regular expressions. By simulating the execution process of the multi-objective quality optimization strategy, behavior trajectories are generated, and it is verified whether the behavior trajectories satisfy the ω-regularity attribute to obtain long-term compliance verification results.

[0052] By integrating the data quality MDP model, the multi-objective quality optimization strategy, the lineage analysis results, and the long-term compliance verification results, an intelligent decision support engine is constructed. Based on historical data, the current state, and the long-term compliance verification results, the intelligent decision support engine adjusts the prediction model and outputs decision recommendations through the adjusted prediction model. For each decision recommendation, a formalized certificate and decision basis record are generated.

[0053] This application also provides an operational statistics consistency verification device based on a dynamic data cube, including:

[0054] The OLAP data structure generation module is used to acquire raw data including customer information, product information, and transaction records, and then construct an OLAP cube structure containing time, region, and product category dimensions based on the raw data. It also pre-calculates the aggregation results corresponding to the target query scenario based on the OLAP cube structure, and obtains a multidimensional OLAP data structure based on the aggregation results.

[0055] The verification data acquisition module is used to perform data integrity verification and data format verification based on the multidimensional OLAP data structure through the original data verification layer, and then perform business logic verification on the data verified by the original data verification layer in the ETL intermediate layer, and calculate the preset dimension indicators based on the data verified by the ETL intermediate layer to obtain indicator data. In the indicator calculation layer, the indicator data is verified using linear congruent check codes and inverted index technology, and the data format verification and business meaning verification are performed in the presentation layer to obtain the verified reliable data.

[0056] The analysis result acquisition module is used to collect metadata information and assign it a unique identifier based on the verified reliable data, parse SQL statements, extract data relationships based on the parsing results, construct a data lineage graph based on the metadata information and the data relationships using a graph algorithm, and obtain data tracing and impact analysis results based on the data lineage graph.

[0057] The operational statistics consistency verification method and apparatus based on dynamic data cubes provided by this invention achieves flexible dimensional expansion and optimized query performance by constructing a dynamically scalable multidimensional data cube; it ensures data consistency across all processing stages through a multi-layered intelligent verification mechanism; and it enables comprehensive tracking of data sources and impacts through data lineage analysis. This invention effectively solves the technical problems of poor dimensional expansion flexibility, insufficient query performance, difficulty in guaranteeing data consistency, and unclear data lineage in existing technologies, providing enterprises with a more efficient and reliable operational statistics system. Attached Figure Description

[0058] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0059] Figure 1 A flowchart of an operational statistics consistency verification method based on a dynamic data cube provided in this embodiment of the invention;

[0060] Figure 2This is an overall architecture diagram of an operational statistics consistency verification method based on dynamic data cubes, provided in an embodiment of the present invention. Detailed Implementation

[0061] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings.

[0062] In some of the processes described in the specification, claims, and accompanying drawings of this application, multiple operations appearing in a specific order are included. However, it should be clearly understood that these operations may not be executed in the order they appear herein, or may be executed in parallel. The operation numbers, such as S1, S2, etc., are merely used to distinguish different operations and do not themselves represent any execution order. Furthermore, these processes may include more or fewer operations, and these operations may be executed sequentially or in parallel. It should be noted that the descriptions such as "first," "second," etc., in this document are used to distinguish different messages, devices, modules, etc., and do not represent a sequential order, nor do they limit "first" and "second" to different types.

[0063] It will be understood by those skilled in the art that, unless specifically stated otherwise, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. It should be further understood that the term “comprising” as used in this application’s specification means the presence of the stated features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. It should be understood that when we say an element is “connected” or “coupled” to another element, it may be directly connected or coupled to the other element, or there may be intermediate elements. Furthermore, “connected” or “coupled” as used herein may include wireless connections or wireless coupling. The term “and / or” as used herein includes all or any units and all combinations of one or more associated listed items.

[0064] It will be understood by those skilled in the art that, unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by those skilled in the art to which this application pertains. It should also be understood that terms such as those defined in general dictionaries should be understood to have a meaning consistent with their meaning in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless specifically defined as herein.

[0065] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Throughout the description, the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0066] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments.

[0067] like Figure 1 As shown, this invention provides a method for verifying operational statistics consistency based on a dynamic data cube, comprising the following steps:

[0068] S1: Obtain raw data including customer information, product information, and transaction records. Then, construct an OLAP cube structure containing time, region, and product category dimensions based on the raw data. Based on the OLAP cube structure, pre-calculate the aggregation results corresponding to the target query scenario. Based on the aggregation results, obtain a multidimensional OLAP data structure.

[0069] This step begins by collecting raw data from various business systems within the enterprise, including key data resources such as basic customer information, product details, and historical transaction records. This raw data is typically scattered across different business systems, exhibiting diverse formats and varying quality. The system systematically cleans, transforms, and standardizes this raw data to ensure data quality and consistency. Then, based on the processed data, the system constructs an OLAP cube structure containing core dimensions such as time, region, and product category. This multidimensional structure allows for analysis of business data from different angles and granularities, meeting diverse analytical needs. Next, the system analyzes historical query patterns and business requirements to identify target query scenarios. Specific target query scenarios can be common queries such as "viewing sales figures for each region by month" or "comparing profit margins for different product categories by quarter." For these target query scenarios, the system pre-calculates the aggregation results for the corresponding dimension combinations, establishing aggregated data views at various levels. Finally, the system efficiently stores these pre-calculated aggregation results, forming a complete multidimensional OLAP data structure. This structure not only includes raw detailed data but also aggregated data at various levels, significantly improving the performance and flexibility of subsequent query analysis.

[0070] S1.1: Perform data cleaning, format conversion, and standardization on the raw data to obtain standardized business data.

[0071] This step begins by collecting raw data from various business units, including customer information (such as customer ID, name, contact information, and region), product information (such as product ID, name, category, and price), and transaction records (such as order ID, transaction time, transaction amount, and transaction product). Then, this raw data undergoes cleaning to remove outliers, duplicate data, and incomplete records; format conversion is performed to transform data from different sources into a unified format; and standardization is applied, such as standardizing date formats and monetary units, ultimately resulting in standardized business data.

[0072] Specifically, in the initial stage of data processing, the raw data collected from various business units undergoes comprehensive cleaning, transformation, and standardization. This is the foundation for building a high-quality OLAP data cube. During the data cleaning phase, various algorithms and rules are used to identify and handle outliers, missing data, and duplicate records. For example, outlier age values ​​in customer information (such as negative numbers or ages exceeding 150) are marked as abnormal and processed according to configured strategies, such as setting them to null values, replacing them with default values, or inferring reasonable values ​​from other relevant fields. For duplicate order numbers in transaction records, timestamps and other attributes are used to determine if they are genuine duplicates, and business rules dictate whether to retain or merge them. In the data format conversion phase, data from different sources is unified into a standard format to ensure data type consistency. For example, different date and time formats (such as "2023-01-01", "01 / 01 / 2023", "January 1, 2023") are uniformly converted to the standard ISO format; monetary data is standardized to the same currency unit and decimal place specifications. In the standardization process, data is standardized at the business level to conform to the company's data standards and business specifications. For example, customer address information is broken down into standard fields such as country, province, city, and street; product categories are remapped and categorized according to the company's standard classification system; and status codes (such as order status) from different business processes are mapped to a unified status definition. Through this series of processes, the original messy data is transformed into standardized business data with a clear structure, uniform format, and consistent semantics, laying a solid foundation for subsequent multidimensional analysis.

[0073] S1.2: Based on the standardized business data, organize it into a multi-dimensional structure according to the dimensions of time, region, and product category to obtain the OLAP cube structure.

[0074] After standardizing the data, we began constructing a multidimensional OLAP cube structure, the core data model supporting complex analytical queries. First, based on business needs and analytical objectives, we determined the key dimensions of the OLAP cube, including but not limited to core business dimensions such as time, region, and product category. For the time dimension, we built a hierarchical time structure, including different granularities such as year, quarter, month, week, and day, and supported specific business time periods (such as fiscal year, promotional season, etc.). Each time unit was assigned rich attributes, such as whether it is a weekday, whether it is a holiday, and the season, making subsequent time analysis richer and more flexible. For the region dimension, we established a geographical hierarchy from country to province to city and county, and associated it with characteristic data such as population and economic development level of each region, providing more dimensions for regional analysis. For the product category dimension, we constructed a product hierarchy from major categories to intermediate categories to minor categories according to the company's product classification system, while also associating it with attributes such as product price range and life cycle stage. In addition to these basic dimensions, we added suitable dimensions based on business characteristics, such as customer type and sales channel. After defining each dimension, the standardized business data was reorganized according to these dimensions to form a multidimensional data structure. Specifically, each business record (such as a sales record) is associated with corresponding dimension members (such as specific time, region, product category, etc.), and the record's metrics (such as sales amount, quantity, etc.) are placed in the "spatial location" determined by these dimensions. In this way, all business data is organized into a multi-dimensional OLAP cube structure, where each dimension represents an analytical perspective, and the intersections of dimensions contain the corresponding metrics. This structure allows users to flexibly analyze data from different angles and at different granular levels, such as slicing along a dimension or drilling down along a dimension, greatly enhancing the capabilities and flexibility of data analysis.

[0075] S1.3: For the target query scenario in the OLAP cube structure, pre-calculate the aggregation results of each dimension combination corresponding to the OLAP cube structure.

[0076] After constructing the OLAP cube structure, the operational statistics system (hereinafter referred to as the system) enters the pre-computation phase, a step crucial for improving query performance. The system first identifies target query scenarios and key business needs by analyzing historical query logs and user behavior patterns. As described earlier, target query scenarios can specifically be common query scenarios; for example, the system might find that "viewing sales figures for each province monthly," "comparing gross profit margins for different product categories quarterly," and "analyzing consumption trends for different customer types annually" are the most common query patterns. For these target query scenarios, the system designs optimized pre-computation strategies, determining the dimension combinations and aggregation levels that need to be pre-computed. The system considers factors such as query frequency, data volume, and update frequency to balance query performance and storage costs, selecting the optimal pre-computation range. Technically, the system employs efficient parallel computing frameworks, such as Apache Spark or a custom distributed computing engine, to perform aggregation calculations on the selected dimension combinations. During the pre-computation process, the system aggregates different levels of each dimension. For example, for the time dimension, the system calculates aggregation results at the daily, weekly, monthly, quarterly, and yearly levels; for the regional dimension, it calculates aggregation results at the city, province, region, and country levels; and for the product dimension, it calculates aggregation results at the single-item, sub-category, medium-category, and major-category levels. The system also calculates various combined aggregations of these dimensions, such as "month × province" or "quarter × product category." For each aggregation level and dimension combination, the system calculates multiple aggregation metrics, such as sales revenue, sales volume, average unit price, and year-on-year growth rate. In aggregation calculations, the system considers the characteristics of different metrics and correctly applies different aggregation functions such as summation, average, maximum / minimum, and count. Furthermore, the system pre-calculates certain complex derived metrics (such as market share and growth contribution rate), which typically involve combined calculations of multiple basic metrics. Through this comprehensive and targeted pre-calculation, the system significantly reduces the computational load during real-time queries, providing strong support for subsequent rapid data analysis.

[0077] S1.4: After caching the aggregation results, a multidimensional OLAP data structure is obtained.

[0078] After pre-calculating the aggregation results for each dimension, the crucial storage optimization phase begins. Efficient caching strategies ensure rapid access to these pre-calculated results. First, a dedicated caching storage architecture is designed, stratifying the aggregation results based on access frequency, importance, and update cycle. The most frequently accessed hot data is placed in an in-memory cache, such as using in-memory databases like Redis or Memcached; data with lower access frequency but still requiring fast response times is stored in an SSD-level cache; and larger-scale but less frequently accessed historical aggregation data is stored on regular disks or cloud storage. For data organization, optimized storage structures are employed, such as columnar storage, storing data of the same dimension or metric contiguously to improve data compression and query efficiency. For sparse data (i.e., data with a large number of null values), a special sparse matrix storage technique is used, recording only non-null values ​​and their dimension coordinates, significantly reducing storage space requirements. To further improve access efficiency, a multi-level index structure is established, including bitmap indexes, inverted indexes, and multi-dimensional indexes (such as R-trees), enabling the query engine to quickly locate and retrieve relevant data. Simultaneously, an intelligent caching management mechanism is implemented, including preloading, expiration, and dynamic adjustment strategies. For example, when frequently accessed aggregate views are detected, they are automatically promoted to a higher-speed cache layer; when peak business periods approach, relevant aggregate data is preloaded into memory. An incremental update mechanism is also designed, so when the underlying detailed data changes, only the affected aggregate results are updated, rather than recalculating all aggregates, greatly improving the efficiency of data updates. Through these optimized storage and caching strategies, the pre-calculated aggregate results are organized into a highly efficient multidimensional OLAP data structure, capable of supporting complex analytical queries at sub-second speeds, significantly improving user experience and performance. This structure not only supports traditional predefined report queries but also meets the needs of ad-hoc queries and interactive analysis, enabling business users to flexibly explore data and discover insights.

[0079] S2: Through the original data verification layer, data integrity and data format verification are performed based on the multidimensional OLAP data structure. Then, in the ETL intermediate layer, business logic verification is performed on the data verified by the original data verification layer. Based on the data verified by the ETL intermediate layer, preset dimension indicators are calculated to obtain indicator data. In the indicator calculation layer, linear congruent check codes and inverted index technology are used to verify the indicator data. Finally, data format verification and business meaning verification are performed in the presentation layer to obtain the verified reliable data.

[0080] This step implements a multi-layered, comprehensive data verification mechanism to ensure the accuracy and consistency of data at each processing stage. First, at the raw data verification layer, the system performs basic checks on the multi-dimensional OLAP data structure, verifying data integrity (existence of necessary fields) and format compliance (whether data types, value ranges, etc., conform to regulations), filtering out obviously unqualified data. Data that passes the initial verification enters the ETL intermediate layer. At this layer, the system performs deeper business logic verification to ensure that the data conforms to business rules and constraints, such as consistency between order amounts and details, and the rationality of inventory changes. This business-level verification can detect potential business anomalies and ensure the correctness of the data's business semantics. Next, based on the data that has passed ETL verification, the system calculates various business indicators, such as sales revenue, profit margin, and customer conversion rate, according to preset calculation formulas and dimension definitions. At the indicator calculation layer, the system innovatively uses linear congruent checksums and inverted index technology to verify the calculation results. Specifically, the system generates a unique checksum for each indicator and establishes a mapping relationship between the checksum and the indicator value. When the same indicator is calculated multiple times under different dimensional combinations, the system verifies the consistency of the calculation results by comparing check codes, effectively solving the problem of inconsistencies in indicators across multi-dimensional reports. Finally, at the presentation layer, the system performs a final check on the data to be presented to the user, ensuring that the data display format meets user expectations and that the business meaning is reasonable, such as whether the growth rate is abnormal and whether the year-on-year change matches business expectations. Through this series of progressive checks, the system ultimately obtains highly reliable data, providing a solid foundation for subsequent analysis and decision-making.

[0081] S2.1: Data integrity and data format are verified through the original data verification layer based on the multidimensional OLAP data structure.

[0082] In this step, the system deploys a validation rule engine at the raw data validation layer to perform preliminary validation of the data in the multidimensional OLAP data structure. Data integrity validation mainly checks the existence of necessary fields, such as key fields like timestamps, IDs, and transaction amounts; data format validation confirms whether the format of each field conforms to preset rules, such as whether the date format is correct and whether the values ​​are within a reasonable range. The system will mark and record data that fails validation and route this data to a correction queue for processing.

[0083] In the data integrity verification phase, the system first scans the entire OLAP cube structure, checking the non-null rate of key dimensions and metric fields. For necessary dimensional attributes (such as year, month, and day in the time dimension, and product ID and product category in the product dimension), the system verifies their integrity to ensure there are no missing or null values. When a key field is found to be missing, the system processes it according to preset rules, such as marking abnormal records, attempting automatic repair, or inferring completion from other relevant fields. The system also checks the integrity between different dimensional levels, ensuring the integrity of hierarchical relationships such as a product belonging to a certain product category or a city belonging to a certain province. In the data format verification phase, the system verifies the format compliance of various types of data. For numerical data (such as sales amount and sales volume), the system checks whether its value range is reasonable and whether there are abnormally large or small values; for date data, it verifies whether its format conforms to the standard and whether the date value is within the valid range; for categorical data (such as product type and customer level), it verifies whether its value is within the predefined enumeration range. The system employs various statistical methods to identify outliers, such as the Z-score method based on the mean and standard deviation, and the box plot method based on quartiles. Different anomaly detection strategies are used for different types of data. For data exceeding a reasonable range, the system marks it and processes it according to configured strategies, such as replacing it with a default value, truncating it to the valid range, or keeping the original value but adding a warning label. Through these rigorous data integrity and format checks, the system ensures that the data entering subsequent processing stages has basic quality assurance, laying a solid foundation for subsequent business logic verification and indicator calculation.

[0084] S2.2: In the ETL intermediate layer, business logic verification is performed on the data that has passed the verification of the original data verification layer.

[0085] At the ETL middleware layer, the system performs deeper business logic validation on data that has passed basic format and integrity checks, ensuring that the data is not only compliant in form but also consistent and reasonable in business logic. The system first loads a predefined business rule base, which is jointly developed by domain experts and data analysts and includes various business constraints and logical relationships. For example, rules in the retail field might include "the quantity of goods sold must be a positive integer," "the total order amount must equal the sum of the amounts of each order item," and "the discount rate during promotions must be within the allowed range." The system has designed dedicated logic validation modules for different business dimensions. For example, the transaction record validation module verifies the timing logic of transactions (e.g., the return date must be later than the purchase date) and the amount logic (e.g., the actual payment amount must not exceed the amount due); the inventory management validation module verifies the reasonableness of inventory changes (e.g., inventory should not suddenly decrease significantly without warning); and the customer behavior validation module verifies the consistency of customer activities (e.g., the same customer should not make transactions in different cities simultaneously). The system also performs cross-dimensional correlation logic validation, such as checking the consistency between sales data and inventory data, and the correlation between marketing activities and sales peaks. To improve verification efficiency, the system adopts rule engine technology, which expresses business rules in a declarative language and stores them in the rule base, enabling rapid execution and evaluation through the rule engine.

[0086] For any business logic anomalies detected, the system handles them in a tiered manner based on the anomaly type and severity: critical errors (such as negative transaction amounts) are rejected from further processing; general anomalies (such as slightly exceeding the normal price range) are marked and recorded but allowed to continue processing; potential anomalies (such as deviations from historical patterns but still within acceptable limits) are marked for manual review. The system also maintains detailed business logic verification logs, recording all verification results, anomaly types, and handling methods for easy tracking and auditing. Through this series of rigorous and comprehensive business logic verifications, the system ensures that the data is not only technically correct but also maintains a high degree of consistency and credibility at the business semantic level.

[0087] S2.3: Calculate preset dimension indicators based on the data verified by the ETL intermediate layer to obtain indicator data.

[0088] After passing the ETL middleware business logic validation, the system enters the indicator calculation stage, a crucial step in transforming raw data to give it business value. The system first loads a pre-defined indicator definition library. These indicators define various business metrics that enterprises care about, such as sales revenue, profit margin, and customer conversion rate. Each indicator has a clear calculation formula, applicable dimensions, and business implications. The system adopts a layered calculation strategy, first calculating basic indicators, and then calculating derived indicators based on these basic indicators. For example, it first calculates basic indicators such as sales volume and sales amount, then calculates derived indicators such as average unit price and gross profit margin; finally, it calculates more complex indicators such as year-on-year growth rate and market share. During the calculation process, the system considers various special cases, such as handling division by zero errors (using default values ​​or special markings when the denominator is zero) and handling missing values ​​(selecting to ignore, interpolate, or use default values ​​according to business needs). For time-related indicators, the system specifically handles the impact of periodic factors (such as seasonality and weekday effects) and special dates (such as holidays and promotional days) to ensure the comparability of indicators. For region-related indicators, the system takes into account the impact of regional differences (such as consumption levels, market size, etc.) on the indicators, providing a comparative perspective.

[0089] The system also optimizes calculation strategies for different granularity levels: a pre-calculation strategy is used for frequently accessed aggregate metrics (such as monthly sales); an on-the-spot calculation strategy is used for less frequently accessed fine-grained metrics (such as sales of a specific product on a specific date); and an approximate calculation algorithm is used for complex metrics in ultra-large-scale datasets to significantly improve computational efficiency within acceptable accuracy ranges. In terms of performance optimization, the system uses multi-threaded parallel computing and in-memory computing technologies to significantly improve the speed of metric calculation. To ensure the traceability of calculations, the system records the calculation path, raw data used, and intermediate results for each metric, forming a complete calculation lineage. Through these carefully designed metric calculation processes, the system transforms verified basic data into a rich variety of business metrics, providing strong data support for enterprise decision-making.

[0090] S2.4: Linear congruent check codes and inverted indexing techniques are used to verify the indicator data at the indicator calculation layer.

[0091] After the indicator calculation is completed, the system implements an innovative data verification mechanism at the indicator calculation layer. It employs linear congruential check codes and inverted index technology to comprehensively verify the calculation results, ensuring the consistency and accuracy of the indicator data. The core idea of ​​linear congruential check code technology is to generate a unique mathematical feature code for each indicator value.

[0092] Specifically, the system selects appropriate prime modulo and multiplier values ​​to map the original indicator values ​​to check codes within a fixed range using a linear congruential formula. These check codes are highly sensitive; even slight changes in the indicator value will result in significantly different check codes, effectively detecting data errors. The system generates check codes for different calculation paths of each indicator (such as the same indicator calculated through different dimension combinations) and compares the consistency of these check codes. For example, monthly sales can be calculated directly from monthly data or by accumulating daily data; the system checks whether the results from these two calculation paths are consistent. When inconsistencies are found, the system flags the anomaly and triggers a correction process.

[0093] Meanwhile, the system utilizes inverted index technology to establish a mapping relationship between indicator values ​​and their data sources. Traditional indexes map records to attributes, while inverted indexes do the opposite, recording all data records corresponding to a specific indicator value. This allows the system to quickly locate the original data that generated the specific indicator value. For example, when an anomaly in an indicator value is detected, the system can immediately query all original records that contributed to that value, facilitating in-depth analysis of the problem's origin.

[0094] The system also combines these two technologies to achieve efficient data tracing and anomaly localization. When a checksum inconsistency is detected, the system uses an inverted index to quickly locate the potential problem data source, analyzes and compares the intermediate results of different calculation paths, and accurately identifies the link and cause of the error. To improve verification efficiency, the system adopts a hierarchical verification strategy: first, key aggregate indicators (such as total sales and total profit) are verified; after a problem is found, it drills down to the subdivided dimension indicators for precise localization. The system also implements intelligent verification threshold management, setting different tolerances for different indicators. For example, financial indicators require 100% accurate matching, while predictive indicators can accept a small range of errors. Through this innovative verification mechanism, the system can not only discover data problems that are difficult to detect with conventional checks, but also accurately locate the source of the problem, greatly improving the efficiency and accuracy of data quality management.

[0095] S2.5: Perform data format validation and business meaning validation at the presentation layer to obtain reliable data after validation.

[0096] Before the data is presented to the end user, the system implements a final rigorous quality check at the presentation layer, ensuring that users are exposed to high-quality and reliable data through data format verification and business meaning verification.

[0097] In the data format validation phase, the system first checks whether the data format conforms to display specifications based on different display scenarios and user roles. For example, for financial statements, it checks whether currency data displays decimal places and currency symbols as required; for sales trend charts, it checks whether time series data conforms to expected regularity and continuity; and for regional distribution maps, it checks whether geographical data is missing or misaligned. The system also validates the data's visualization suitability, such as checking whether the numerical range is suitable for the selected chart type (e.g., minimum values ​​may be difficult to identify in a bar chart) and whether the text length is suitable for displaying completely in a table cell. For any formatting issues found, the system automatically adjusts according to preset rules (e.g., truncating excessively long text, using scientific notation for excessively large or small values), ensuring the data is visually clear and aesthetically pleasing.

[0098] In the business meaning verification stage, the system evaluates the rationality and interpretability of the data from a business semantic perspective. First, the system checks the logical relationships between key indicators, such as whether the mathematical relationship between sales, costs, and profits holds true. Then, it analyzes the time trends of the indicators to check for abnormal fluctuations or changes that do not conform to historical patterns. Next, it checks the consistency of cross-dimensional data, such as whether the values ​​of the same indicator are consistent across different reports. Finally, it incorporates business background knowledge to assess the data's conformity with known business facts, such as whether sales have increased accordingly during major marketing campaigns or whether market share has changed after the launch of a new product.

[0099] The system employs multiple intelligent algorithms to assist in verifying the meaning of business information. For example, a time-series anomaly detection algorithm identifies abnormal fluctuations, a correlation analysis algorithm verifies relationships between indicators, and a predictive model assesses the deviation between actual and expected values. For any anomalies detected, the system generates intelligent explanations and suggestions, such as pointing out possible causes (e.g., "abnormal sales growth may be related to the day's promotional activities"), providing reference points for data interpretation, and marking suspicious points requiring manual review. Through this series of rigorous presentation layer verifications, the system ultimately outputs not only technically correct data but also business-meaningful and reliable information, providing decision-makers with a solid data foundation.

[0100] S3: Based on the verified reliable data, collect metadata information and assign it a unique identifier, parse the SQL statement, extract data relationships based on the parsing results, construct a data lineage graph based on the metadata information and the data relationships using a graph algorithm, and obtain data tracing and impact analysis results based on the data lineage graph.

[0101] This step establishes a complete data lineage tracing system, achieving end-to-end traceability of data from its source to its final presentation. First, based on the reliable data verified in previous steps, the system collects metadata information from each stage through specialized interfaces and tools, including key information such as data structure, business rules, calculation logic, and data flow. A globally unique identifier is assigned to each data element, establishing a data "identity card" system to ensure that each data element can be uniquely identified and tracked. Simultaneously, the system deploys an SQL parsing engine to monitor and capture the SQL statements executed during data processing. Through in-depth analysis of these SQL statements, the system identifies the operation types, tables and fields involved, functions and conditions used, thereby understanding the data's calculation logic and transformation rules. Based on the parsing results, the system identifies the dependencies between data elements, such as which original data fields a certain indicator depends on and what calculation methods are used. Next, the system uses graph algorithms to integrate the collected metadata information and identified data relationships into a dedicated graph database. In this graph structure, data elements, indicators, reports, and other entities act as nodes, and their dependencies act as directed edges, collectively forming a complete data lineage graph. The system applies optimization and validation algorithms to this data lineage map to ensure its structural integrity and consistency. Ultimately, based on the constructed data lineage map, the system provides powerful data tracing and impact analysis capabilities. Users can select any data node to trace its data source upstream to understand the data's generation path; they can also analyze its impact range downstream to assess the potential cascading effects of data changes. This comprehensive lineage analysis capability greatly improves the transparency and controllability of data governance, providing strong support for data quality management and problem localization.

[0102] S3.1: Based on the verified reliable data, collect metadata information and assign it a unique identifier.

[0103] In this step, the system collects metadata information from various data sources and processing stages through pre-defined interfaces and metadata collection tools. The collected metadata includes data structure information (such as table structure and field attributes), business rules, calculation logic, and data flow. For each data element, the system assigns a globally unique identifier using a UUID or other unique identification algorithm. The system associates metadata with the unique identifier to build a metadata index, forming the data's "identity card," providing a foundation for subsequent lineage analysis.

[0104] In the initial stage of the data lineage tracing system, the system conducts comprehensive metadata collection and identification work based on reliable data that has undergone multiple layers of verification, establishing an "identity card" system for data assets. During the metadata collection process, the system first automatically extracts metadata information from various data source systems, ETL tools, data warehouses, and analysis platforms through dedicated metadata crawlers and API interfaces.

[0105] The collected metadata covers multiple levels: technical metadata includes table structure, field types, index information, physical storage location, etc.; business metadata includes data definitions, business rules, explanations of business terminology, etc.; operational metadata includes data update frequency, last update time, data responsible person, etc.; usage metadata includes access permissions, usage scenarios, common query patterns, etc. To ensure the integrity of metadata, the system not only collects existing metadata but also infers missing metadata through data analysis and pattern recognition techniques. For example, it infers data types by analyzing data distribution characteristics, infers business meanings by analyzing field naming rules, and analyzes usage scenarios by analyzing access logs.

[0106] After collecting comprehensive metadata, the system implements an identifier allocation mechanism, assigning a globally unique identifier (UUID) to each data element (from databases, tables, and fields to specific data processing flows and metric definitions). These identifiers employ a multi-layered encoding structure, encompassing multiple dimensions such as data domain, system origin, data type, and version information, ensuring that each data element can be uniquely identified. For example, the identifier for a sales revenue metric might include a combination of "FIN" (financial domain), "DW" (data warehouse), "METRIC" (metric type), "SALES" (sales revenue), and a version number.

[0107] The system also establishes a mapping relationship between identifiers and natural language descriptions, enabling both technical and business personnel to understand the meaning of these identifiers. After identifier allocation, the system embeds identifier tracking mechanisms at each stage of data processing to ensure that identifier information is not lost during data flow. For example, identifier inheritance and propagation rules are added to the ETL process, the identifier mapping relationship between input and output data is maintained during data transformation, and the identifiers of each field involved in the calculation are recorded during indicator calculation. The system also implements an identifier version management mechanism. When the data structure or definition changes, version control maintains the historical record of identifiers, ensuring that data lineage can be correctly tracked even during data evolution. Through this series of metadata collection and identifier allocation tasks, the system lays a solid foundation for subsequent data lineage analysis, giving each data element a traceable "identity card."

[0108] S3.2: Parse the SQL statement and extract data relationships based on the parsing results.

[0109] At the core of data lineage analysis, the system deploys a powerful SQL parsing engine that accurately extracts dependencies between data elements by capturing and analyzing SQL statements during data processing. The system first establishes a comprehensive SQL monitoring network, collecting various SQL statements executed in the enterprise data environment, including data loading, transformation, aggregation, and querying operations, through log capture, query history analysis, and real-time monitoring. The collected SQL statements are then fed into a professional SQL parser capable of handling various SQL dialects (such as MySQL, Oracle, Hive, and SparkSQL) and complex syntactic structures. During parsing, the system first performs lexical and syntactic analysis on the SQL statements to construct an Abstract Syntax Tree (AST); then semantic analysis is performed to identify the tables, fields, functions, and operation types involved in the SQL; finally, data flow analysis is conducted to trace the complete path of data from source to target.

[0110] For simple SELECT statements, the system identifies the source table and fields, as well as the structure of the output result set. For complex queries involving JOINs, the system analyzes the join conditions and association methods between different tables. For statements containing subqueries, views, and common table expressions (CTEs), the system recursively parses each component to build a complete data dependency tree. For statements containing aggregate functions, window functions, and user-defined functions, the system analyzes the input parameters and output results of these functions to identify the data transformation logic. After parsing, the system extracts key data relationship information from the semantic understanding of the SQL.

[0111] For data flow relationships, the system identifies where the data comes from (source table fields) and where it goes (target table fields); for transformation relationships, the system identifies what calculations or processes the data has undergone (such as summation, averaging, joining, filtering, etc.); for derived relationships, the system identifies which new fields are calculated from which source fields, and the specific calculation formulas; for filtering relationships, the system identifies which conditions restrict the range of the data.

[0112] The system also pays special attention to special processing logic in SQL, such as CASE WHEN statements, COALESCE functions, and ISNULL handling. This logic often contains important business rules. To improve parsing accuracy, the system combines metadata information for contextual understanding, such as using table structure information to verify the correctness of field references and using business rules to interpret specific SQL logic. For complex stored procedures and dynamic SQL, the system uses a special logic derivation engine to simulate the SQL execution process and analyze possible data flow paths. Through this series of in-depth SQL parsing and relationship extraction, the system transforms complex code logic into clear data dependencies, providing crucial information for building a complete data lineage graph.

[0113] S3.3: Construct a data lineage graph based on the metadata information and the data relationships using a graph algorithm.

[0114] After acquiring complete metadata and data relationships, the system uses advanced graph algorithm technology to construct a comprehensive and in-depth data lineage graph.

[0115] The system first designed a specialized graph data model, defining various node types (such as data sources, tables, fields, ETL tasks, and metrics) and edge types (such as "originating from," "converted to," "calculated," and "dependent on"), and defining a rich set of attributes for each type. Based on this model, the system maps the metadata and data relationship information collected in the previous steps into nodes and edges of the graph structure. Data elements (such as tables and fields) become nodes in the graph, and the dependencies between them become directed edges connecting the nodes. Each node contains attributes such as its unique identifier, name, description, type, and system affiliation; each edge contains attributes such as relationship type, data flow direction, and transformation logic. During graph construction, the system adopts an incremental construction strategy, first establishing basic table-level lineage relationships, then refining them to field-level lineage relationships, and finally adding the lineage relationships between business metrics and the original data.

[0116] For complex data transformation logic (such as aggregation calculations after multi-table JOINs), the system uses specialized path analysis algorithms to ensure accurate capture of complex relationships between multiple data sources flowing towards the same goal. After constructing the basic graph structure, the system applies graph optimization algorithms to improve graph quality. Graph compression algorithms identify and merge duplicate data paths, reducing graph complexity; graph verification algorithms detect and repair broken lineage links, ensuring the integrity of lineage paths; and graph sorting algorithms optimize node layout, making data flow clearer and more visible.

[0117] The system also employs graph augmentation algorithms to enrich the graph content. Impact analysis algorithms calculate the influence range and importance index of each node, identifying key data nodes; similarity algorithms identify data processing paths with similar functions and discover potential redundant logic; anomaly detection algorithms identify kinship relationships that do not conform to conventional patterns and mark potential data problems. To support efficient management of large-scale kinship graphs, the system adopts a hierarchical storage and indexing strategy. Core business paths are stored in a high-speed cache, supporting real-time queries; complete kinship networks are stored in a dedicated graph database, supporting complex graph traversal and analysis; historical version kinship information is stored in an archive system, supporting time-point retrospective analysis.

[0118] The system also implements a real-time update mechanism for the kinship map. When changes in data structure or processing logic are detected, the corresponding map portion is immediately updated to ensure that kinship information remains synchronized with actual data processing. Through these advanced graph algorithms and technologies, the system ultimately constructs a comprehensive, accurate, and multi-layered data kinship map, providing powerful visualization and tracking capabilities for data management and analysis.

[0119] S3.4: Based on the aforementioned kinship map, obtain the results of data tracing and impact analysis.

[0120] After successfully constructing a data lineage graph, the system developed a series of powerful analytical tools. Based on this graph, it provides in-depth data tracing and impact analysis capabilities to meet various data management and governance needs. The data tracing function allows users to trace upstream from any data node (such as report metrics or analysis results) to explore its complete data source path. Users can select specific business metrics or data fields, and the system will automatically extract all upstream dependency paths of that node from the lineage graph, presenting the complete data flow from the original data source to the target node. This tracing supports multi-level drill-down; users can start from the table-level lineage and gradually delve into the field-level lineage, even viewing specific data transformation logic and SQL code snippets.

[0121] The system also offers intelligent path filtering, helping users find the most critical dependency paths in complex dependency networks, such as the largest contributing data sources and essential transformation nodes. Impact analysis allows users to analyze downwards from any data node, assessing all downstream data and applications that a change at that node might affect. When planning modifications to a data table or ETL process, users can use impact analysis to anticipate the scope of these changes and identify potentially affected reports, metrics, and application systems. The system not only displays the scope of the potential impact but also assesses its severity, providing special alerts for high-risk changes. This proactive analysis significantly reduces unexpected disruptions and errors caused by data changes.

[0122] In addition to basic tracing and impact analysis, the system also provides a variety of advanced analysis functions. The dependency analysis function calculates and visualizes the dependency strength between data elements, helping to identify critical data nodes and potential single points of failure; the change history analysis function records and displays historical changes to data structures and processing logic, supporting lineage status backtracking at any point in time; the abnormal path analysis function automatically identifies data flow paths that do not conform to design specifications, such as circular dependencies, excessively long links, and complex connections that may cause performance problems.

[0123] The system also supports customized analytical views based on business scenarios, such as financial statement lineage views and marketing analysis lineage views, enabling different business users to focus on the data lineage sections relevant to their own needs. To improve the understandability of the analysis results, the system offers various visualization methods, including graphical lineage diagrams, structured lineage trees, and matrix dependency tables, and supports interactive exploration, such as path highlighting, node expansion / collapse, and focus zooming. The system also provides rich export and sharing functions, allowing analysis results to be shared with relevant personnel in the form of reports, images, or interactive documents. Through these comprehensive and in-depth analytical functions, the lineage graph is no longer just static reference information, but has been transformed into a powerful tool that proactively supports data governance and decision-making, providing unprecedented transparency and controllability for enterprise data asset management.

[0124] In a preferred embodiment of the present invention, after obtaining the multidimensional OLAP data structure, the method further includes:

[0125] A1. Receive the user's data query request, parse the time dimension parameter in the data query request, and obtain the time query conditions;

[0126] A2. For the time query conditions, use the time bucketing function to perform bucketing processing, divide the time into different granularities of hour, day, week, and month to obtain the bucketed time dimension;

[0127] A3. Based on the time dimension after bucketing, pre-calculate and store multi-level aggregated views;

[0128] A4. Based on the time range of the data query request, select the most suitable aggregate view, retrieve data from the selected aggregate view, and perform the minimum necessary calculations to obtain the optimized query results.

[0129] In this embodiment, after the query acceleration module is activated, when a user initiates a data query request, the system first parses the time dimension parameters in the request, such as the start and end times of the query and the time granularity. Then, the system uses the `time_bin` function to bucket the time dimension, dividing the time into different time windows such as hours, days, weeks, and months. For example, for hourly queries, the system aggregates data by hour; for daily queries, it aggregates data by day, and so on. Based on these bucketed time dimensions, the system pre-calculates and stores multi-level aggregation views, such as hourly aggregated data and daily aggregated data. When a specific query request is received, the system selects the most suitable aggregation view level based on the query's time range and other conditions. For example, for queries spanning a week, the system might select a daily aggregation view; for queries spanning a year, it might select a monthly aggregation view. The system retrieves data from the selected aggregation view, and if further aggregation or refinement is needed, it performs the minimum necessary calculations, ultimately returning the optimized query results. This approach significantly reduces the amount of data processed during queries and improves query response speed.

[0130] Specifically, in the query optimization phase of the data analysis system, the system implements intelligent bucketing and multi-level aggregation strategies based on the time dimension, significantly improving query performance. When a user submits a data query request, the system first identifies and extracts the time dimension parameters through a dedicated request parser. These parameters may appear in various forms, such as explicit date ranges ("2023-01-01 to 2023-03-31"), relative time expressions ("past 30 days", "this quarter"), or specific business cycles ("2023 fiscal year Q2"). The parser uses semantic understanding algorithms to uniformly convert these different forms of time expressions into standardized time query conditions, clarifying the start and end times of the query and the desired time granularity. After obtaining the time query conditions, the system applies specially designed time bucketing functions to perform hierarchical processing of the time dimension. These bucketing functions divide the continuous time axis into discrete time buckets according to different business scenarios and data characteristics.

[0131] The system supports multiple bucketing granularities: hourly bucketing is suitable for scenarios requiring fine-grained time analysis, such as monitoring system performance fluctuations and analyzing intraday changes in user activity; daily bucketing is the most commonly used granularity, suitable for daily business reports and trend analysis; weekly bucketing specifically considers the weekly effect (such as the difference between weekdays and weekends), suitable for analyzing business indicators with obvious periodicity; and monthly bucketing is suitable for long-term trend analysis and financial reporting scenarios. During the bucketing process, the system also considers factors such as time zone conversion, daylight saving time adjustments, and special holiday markings to ensure semantic consistency of the time dimension. After completing time bucketing, the system constructs and maintains multi-level pre-computed aggregated views based on these different time dimensions. These aggregated views are implemented using materialized view technology, pre-compiling and storing commonly used indicators according to different time granularities. For example, the system maintains hourly, daily, weekly, and monthly aggregated views of total sales, with each view containing the aggregation results at that time granularity. These views form a hierarchical aggregation pyramid, from the finest-grained raw data to the coarsest-grained highly aggregated data, providing optimized paths for different query needs.

[0132] Aggregated views are updated using an incremental calculation strategy. When new data arrives, only the affected time buckets are updated, rather than the entire view being recalculated. When a user's query request is received by the system, the query optimizer analyzes the time range and other conditions in the request, intelligently selecting the most suitable aggregated view level. The system employs the "minimum coverage principle," selecting the aggregated view that best covers the query's time range and has a granularity closest to the query requirements. For example, when a user queries "weekly sales trends over the past two months," the system selects a weekly aggregated view; when querying "hourly order volume today," it selects an hourly aggregated view. In some cases, the system may need to combine multiple aggregated views of different granularities. For instance, querying "average daily sales over the past 45 days" might combine a monthly view (covering the first 30 days) and a daily view (covering the last 15 days).

[0133] After selecting a suitable aggregated view, the system extracts relevant data and performs only necessary computational operations, such as further aggregation, filtering, or association with other dimensions. The system optimizes the execution plan for these operations to minimize data movement and computational load. Finally, the system returns the optimized query results to the user while simultaneously recording query patterns in the background for continuous optimization of the pre-computation strategy. Through this series of time-based optimizations, the system can transform queries that might otherwise require scanning massive amounts of raw data into simple access to pre-computed aggregated views, significantly improving query response speed while maintaining the accuracy and completeness of the results.

[0134] In another preferred embodiment of the present invention, the step of verifying the indicator data using linear congruential check codes and inverted index technology at the indicator calculation layer includes:

[0135] B1. For the aforementioned indicator data, use a linear congruential algorithm to generate the corresponding check code, thus obtaining the indicator check code;

[0136] B2. Construct an inverted index of the indicator check code, establish a mapping relationship between the indicator value and its corresponding indicator check code, and obtain the check code index;

[0137] B3. When the same indicator is calculated multiple times under different dimension combinations, the consistency of the calculation results is verified by the check code index to obtain the consistency verification result;

[0138] B4. If inconsistency is determined based on the consistency verification results, the traceability process is initiated to obtain the corrected consistency index data.

[0139] In this embodiment, after the indicator calculation is completed, the system generates a checksum for each calculation result. Specifically, the system uses a linear congruential algorithm to convert indicator data into a unique checksum. For example, for indicators such as sales revenue and profit margin, the system generates corresponding checksums based on their values ​​and calculation paths. Then, the system constructs an inverted index of these checksums, establishing a mapping relationship between indicator values ​​and their corresponding checksums. This inverted index structure enables the system to quickly find the checksum corresponding to a specific indicator value, or to find the indicator value in reverse using the checksum. When the same indicator is calculated multiple times under different dimensional combinations, such as sales revenue being summarized by region or by product category, the system verifies whether the results calculated by these different paths are consistent through the checksum index. For example, it checks whether the sum of sales revenue in all regions equals the sum of sales revenue in all product categories. If an inconsistency is found, the system initiates a traceability process to find the cause of the inconsistency, such as missing data or incorrect calculation formulas, and corrects it to ensure that consistent indicator data is ultimately obtained.

[0140] It should be noted that in the data quality assurance system, the system has implemented an innovative data consistency verification mechanism based on checksums and inverted indexes to ensure the consistency of key indicators under different computing paths.

[0141] This mechanism first applies a carefully designed linear congruential algorithm to generate unique check codes for various calculated indicator data. The core idea of ​​the linear congruential algorithm is to map the original value X to an integer within a fixed range using the formula: (a×X + c) mod m. Here, a (multiplier), c (increment), and m (modulus) are carefully selected parameters to ensure uniqueness and minimize collision rates. The system configures different parameter combinations for different types of indicators (such as monetary, quantity, and ratio indicators) to adapt to different numerical characteristics. For example, for indicators like sales amount that may have decimals, the system first multiplies the amount by 100 to convert it to an integer before applying the linear congruential algorithm; for order quantity indicators that are already integers, the algorithm is applied directly. The advantages of this algorithm are high computational efficiency, low storage footprint, and extreme sensitivity to minute changes in numerical values, effectively detecting subtle data inconsistencies. After generating a check code for each indicator value, the system constructs a dedicated inverted index structure to establish a mapping relationship between indicator values ​​and check codes. Traditional indexes map record IDs to attribute values, while inverted indexes do the opposite, recording all occurrences of a specific attribute value.

[0142] In this system, the inverted index maps each checksum to all metric calculation instances that generated that checksum, including metric ID, calculation time, dimension combination used, data source, and other information. This structure allows the system to quickly locate all occurrences of the same checksum (i.e., metric values ​​that should theoretically be equal), providing an efficient way for subsequent consistency verification. When business requirements lead to the same metric being calculated multiple times under different dimension combinations (for example, monthly sales can be calculated directly from the monthly table or obtained by summing daily sales), the system automatically performs consistency verification through the checksum index.

[0143] Specifically, the system retrieves indicator instances with the same semantics but different computational paths and compares their checksums. If the checksums match perfectly, it indicates that the results obtained from different computational paths are consistent; if the checksums do not match, it indicates a data inconsistency problem. The system records all verification results and generates a detailed consistency verification report, including the total number of verified indicators, the number of consistent indicators, a list of inconsistent indicators, and the degree of their differences. When an inconsistency is detected, the system immediately initiates an intelligent traceability process to locate and resolve the root cause of the problem. The traceability process first uses a lineage graph (constructed in the previous steps) to trace back the computational paths of all inconsistent indicators to identify the points of divergence.

[0144] The system then performs a fine-grained inspection of the upstream data at the point of divergence, verifying the original data, intermediate results, and computational logic to identify the specific factors causing the inconsistency. Common causes of inconsistency include: differences in data loading timing (e.g., one path uses the latest data while another uses cached data), differences in aggregation logic (e.g., different ways of handling NULL values), and differences in dimension definitions (e.g., differences in date grouping due to different timezone settings). The system determines the "gold standard" based on preset business rules, i.e., which computational path should be considered the correct reference value. For identified inconsistencies, the system implements an automatic or semi-automatic correction process: for clear technical errors (e.g., incorrect calculation formulas), the system automatically corrects and recalculates; for issues requiring business judgment, the system generates correction suggestions and notifies the data administrator for review.

[0145] After the corrections are completed, the system recalculates the relevant metrics and performs consistency verification again to ensure the problem is resolved. All inconsistencies and their resolution processes are recorded in the data quality management system, forming a knowledge base to prevent similar issues from recurring. Through this rigorous set of mechanisms for checksum generation, inverted index construction, consistency verification, and intelligent traceability, the system proactively identifies and resolves inconsistencies in the data calculation process, ensuring the accuracy and reliability of key metrics data upon which enterprise decisions rely, and providing a solid guarantee for data-driven business decisions.

[0146] In another preferred embodiment of the present invention, the step of constructing a data lineage graph based on the metadata information and the data relationships using a graph algorithm includes:

[0147] C1. Based on the metadata and relationship information, initialize the graph database to obtain the graph storage structure;

[0148] C2. Based on the graph storage structure, data elements, indicators, and report-type entities are used as nodes in the graph to obtain lineage nodes;

[0149] C3. Based on the dependencies obtained from the parsing, establish directed edges between related nodes to represent data flow and computational dependencies, thus obtaining the lineage relationship edges;

[0150] C4. Using graph algorithms, optimize and verify the integrity of the graph structure containing the bloodline nodes and bloodline relationship edges to obtain the data bloodline relationship graph.

[0151] In this embodiment, the system first initializes a dedicated graph database based on collected metadata and relationship information, preparing a storage environment for constructing a data lineage graph. The system selects database technologies suitable for graph data storage and retrieval, such as Neo4j and TigerGraph, to ensure efficient graph operation performance. Then, the system transforms various data elements, metrics, reports, and other entities into nodes in the graph. Each node contains rich attribute information, such as node type, name, description, creation time, and update time; these nodes form the foundation of the lineage graph. Next, based on the dependency relationships obtained from parsing the SQL, the system establishes directed edges between related nodes, representing the flow of data and computational dependencies. For example, an edge from a raw data node to a metric node indicates that the metric depends on the raw data; an edge from a metric node to a report node indicates that the report uses the metric. The system applies specialized graph algorithms, such as path optimization and loop detection, to optimize and verify the entire graph structure, ensuring the integrity and consistency of the graph. For example, it can detect whether there are circular dependencies, such as A depending on B, B depending on C, and C depending on A; or it can detect whether there are isolated nodes, i.e., nodes without any incoming or outgoing edges. Through these optimizations and verifications, the system ultimately obtains a complete, consistent, and efficient data lineage graph.

[0152] For example, in constructing an enterprise-level data lineage graph, the system first initializes a dedicated graph database environment based on previously collected metadata and data relationship information, providing an efficient foundation for storing and querying complex lineage relationships. The system selects high-performance native graph database technologies such as Neo4j, TigerGraph, or JanusGraph. These databases are specifically designed for storing and processing large-scale graph structure data and can efficiently handle complex relationships between nodes and multi-level path queries.

[0153] During the graph database initialization phase, the system defines a meticulously designed graph schema, including node types, edge types, attribute definitions, and indexing strategies. Node types cover various entity types such as data source systems (e.g., ERP, CRM), data processing components (e.g., ETL jobs, stored procedures), data storage objects (e.g., tables, views, files), data fields, business metrics, and reports. Edge types define various relationship types, such as "read," "write," "export," "calculate," and "summarize," precisely describing the interaction methods between different entities. The system also configures storage parameters suitable for lineage analysis, such as caching frequently traversed paths, creating indexes for key node attributes, and optimizing the query engine for complex path analysis.

[0154] After initializing the graph storage structure, the system begins transforming various entities into nodes in the graph. First, the system maps all data elements (such as database tables, fields, and files) to basic nodes. Each node contains core attributes such as a unique identifier, name, description, technology type, business category, and system to which it belongs. Then, the system processes business metrics (such as sales revenue and number of customers), mapping them to high-level semantic nodes and adding specific attributes such as metric definition, calculation formula, business definition, and responsible person. Finally, the system processes various reports and data products (such as dashboards, analysis reports, and data services), mapping them to end-application nodes, including attributes such as purpose, audience, and update frequency. For each type of node, the system also adds management attributes such as version information, quality score, and usage frequency, providing a foundation for subsequent data governance.

[0155] After node creation, the system establishes directed edges representing data flow and computational dependencies between related nodes based on the dependencies obtained from previous parsing. These directed edges not only connect nodes but also carry rich relational semantics. For example, transformation relationship edges between fields contain attributes such as transformation rules, SQL fragments, and execution frequency; computational dependency edges contain attributes such as calculation formulas, weight coefficients, and whether they are critical dependencies; data flow edges contain attributes such as data volume, transmission latency, and most recent execution time. The system also specifically handles many-to-many relationships (such as a target field being calculated from multiple source fields, or a source field affecting multiple target fields) and recursive relationships (such as layered dependencies between views) to ensure that complex data dependencies can be fully captured. After establishing the basic lineage nodes and relational edges, the system applies a series of graph algorithms to optimize and verify the integrity of the overall graph structure. The graph compression algorithm identifies and merges semantically repetitive paths, reducing graph redundancy and complexity; the graph completion algorithm detects and repairs broken lineage links, inferring relationships that may exist but are not explicitly recorded. Figure 1 The consistency algorithm verifies whether the attributes of nodes and edges conform to predefined integrity constraints, such as required attributes and value range restrictions.

[0156] The system also employs community detection algorithms to identify tightly connected clusters of nodes in the graph, revealing the natural boundaries of data domains and business processes; centrality analysis algorithms to calculate node influence and dependency, identifying key data assets; and path optimization algorithms to analyze and optimize the performance of commonly used lineage query paths. During graph optimization, the system also performs data augmentation, such as automatically calculating and adding statistical attributes (e.g., the number of upstream and downstream nodes, average path length), derived attributes (e.g., data sensitivity score, change risk score), and temporal attributes (e.g., the establishment time of lineage relationships, the most recent verification time). Through this series of meticulous graph construction and optimization processes, the system ultimately generates a complete, accurate, efficient, and usable data lineage graph. This graph not only intuitively displays the overall flow and use of enterprise data assets but also provides a solid foundation for various business scenarios such as data governance, impact analysis, problem localization, and compliance auditing. After the graph is built, the system also implements an incremental update mechanism, enabling real-time capture of changes in the data environment and corresponding updates to lineage relationships, ensuring that the graph always reflects the latest data dependency status.

[0157] In another preferred embodiment of the present invention, after obtaining the verified reliable data, the method further includes:

[0158] D1. Based on the multidimensional OLAP data structure, establish a temporal logic model for data attributes, transform business rules into LTLf formulas, and obtain temporal attribute definitions;

[0159] D2. Maintain a timestamp version chain for each data element to record its state at different processing stages, thus obtaining a temporal state chain;

[0160] D3. Execute the reactive synthesis algorithm of LTLfMT to automatically generate a monitor program based on the temporal attribute definition, and obtain the temporal monitor;

[0161] D4. Utilize the temporal state chain to perform cross-time point data comparison, verify whether the growth rate or year-on-year change meets business expectations, and obtain the temporal verification result;

[0162] D5. When the temporal monitor detects a violation of temporal attributes, it records the violation information, analyzes the cause of the violation, and performs automatic repair according to a preset repair strategy to obtain the temporally repaired data.

[0163] In this embodiment, the system first establishes a temporal logic model based on a multidimensional OLAP data structure, transforming business rules into linear temporal logic (LTLf) formulas. For example, the rule "the order amount should not decrease during processing" can be expressed as an LTLf formula stating "globally, the order amount at the current time point is not less than the previous time point." The system maintains a timestamp version chain for each data element, recording its state at different processing stages such as data acquisition, cleaning, transformation, and calculation. For example, for the sales revenue indicator, the system records its value at multiple time points, including the raw data stage, the post-cleaning stage, and the post-calculation stage, forming a complete temporal state chain. Then, the system executes the reactive synthesis algorithm of LTLfMT, automatically generating monitor programs based on the defined temporal attributes. These monitors run in real time during the data processing flow, monitoring whether changes in data state conform to the predefined temporal attributes.

[0164] The system utilizes temporal state chains to perform cross-time point data comparisons, paying particular attention to the changing trends of time-series data. For example, it compares whether the month-on-month growth rate of sales is within a reasonable range, verifying whether the time trend of indicators aligns with business expectations. When the monitor detects violations of temporal attributes, the system records detailed violation information, analyzes possible causes such as data anomalies or processing logic errors, and performs automatic repairs according to preset repair strategies. For example, for data with abnormal fluctuations, the system may use historical averages as a substitute or mark it as requiring manual review. In this way, the system ensures the consistency and reasonableness of data across the time dimension.

[0165] In one embodiment, the above steps are specifically described as follows:

[0166] In innovative practices of data quality monitoring and verification, the system introduces an advanced temporal logic framework, enabling accurate modeling and verification of the characteristics of data changes over time.

[0167] First, based on the established multidimensional OLAP data structure, the system builds a temporal logical model of data attributes. This model extends traditional static data rules to the time dimension, enabling it to describe various complex constraints and patterns that data should satisfy over time.

[0168] The system employs Linear Temporal Logic on Finitetraces (LTLf) as its formal language. This logic is particularly well-suited for expressing attributes over finite time series, aligning perfectly with the periodic characteristics of enterprise data. Business analysts and data experts jointly defined a series of key temporal business rules, such as "sales should not decline for three consecutive months," "seasonal fluctuations in customer churn should not exceed 20% of the historical average for the same period," and "market share should steadily increase within six months of a new product launch." These business rules, described in natural language, are processed by the system's rule transformation engine and converted into precise LTLf formulas. For example, "sales should not decline for three consecutive months" is transformed into "G(!(sales_decline∧X(sales_decline)∧ X(X(sales_decline))))", where G represents "global," X represents "the next point in time," and sales_decline represents the atomic proposition of declining sales. Through this transformation, fuzzy business expectations are precisely defined as computable temporal attributes.

[0169] To support the verification of temporal attributes, the system maintains a detailed timestamp version chain for each key data element (such as sales revenue, number of customers, and inventory levels). These version chains not only record the specific values ​​of data elements at different points in time, but also include metadata such as their processing status (e.g., raw, cleaned, aggregated), data source, calculation method, and confidence level. The system employs efficient time-series database technology to store these version chains, supporting rapid time-point queries and interval analysis. To ensure the integrity of the version chains, the system inserts status capture points at each key node of the ETL and data processing flow to record data status changes in real time. For key indicators, the system also implements a multi-version parallel storage strategy, retaining both the original calculation results and the results after each level of correction, facilitating auditing and backtracking analysis.

[0170] With temporal attribute definitions and state version chains established, the system executes the reactive synthesis algorithm of the LTLfMT (LTLf Monitoring Tool) to automatically generate dedicated temporal monitor programs. This process first transforms the LTLf formula into a deterministic finite automaton (DFA), then optimizes the automaton structure, and finally generates efficient monitoring code. The generated monitors can detect violations of temporal attributes in real time during data flow, with low computational complexity, making them suitable for large-scale data environments. The system generates different monitors for different types of temporal attributes, such as periodic check monitors (periodically assessing long-term trends), event-triggered monitors (immediately verifying when critical data is updated), and threshold-sensitive monitors (increasing check frequency when indicators approach warning lines). All monitors are integrated into a unified monitoring framework and coordinated by a scheduling system to ensure comprehensive monitoring coverage while keeping resource consumption under control. The system utilizes the constructed temporal state chains to perform complex cross-time point data comparison analyses. These analyses not only verify simple year-on-year and month-on-month change rates but also apply more complex time-series pattern detection algorithms, such as identifying seasonal patterns, detecting trend change points, and assessing the stability of periodic fluctuations. The system integrates various time-series analysis techniques, such as moving average analysis, exponential smoothing, and ARIMA models, enabling it to accurately distinguish between normal business fluctuations and abnormal changes. For key business indicators, the system also establishes predictive models, comparing actual values ​​with predicted ranges to identify deviations exceeding expectations. These analytical results are integrated into a comprehensive temporal verification report, including time characteristic assessments of various indicators, outlier identification, and risk level evaluation.

[0171] When the temporal monitor detects a violation of predefined temporal attributes, the system initiates a multi-layered response mechanism. First, the system records detailed violation information, including the specific rule violated, the time period involved, the data elements, and the degree of deviation. Then, the system applies a root cause analysis algorithm to trace the possible causes of the violation. This analysis considers various factors, such as data collection anomalies (e.g., sensor malfunction, sampling bias), changes in processing logic (e.g., adjustments to calculation formulas, changes in aggregation rules), and changes in actual business operations (e.g., promotional activities, market competition, seasonal factors). Based on the analysis results and a pre-defined remediation strategy library, the system determines appropriate corrective actions. For confirmed data problems, the system may automatically perform data repair, such as reasonable interpolation based on historical patterns, data adjustment using business rules, or triggering a recalculation process. For complex cases requiring manual judgment, the system generates a detailed problem report and suggested options, notifying the relevant data administrator for decision-making. All violation detection and remediation actions are recorded in the system's audit log, ensuring the traceability and transparency of data changes. Through this advanced data monitoring framework based on temporal logic, the system can not only verify the static correctness of data, but also grasp the dynamic characteristics of data evolution over time, identify time-series anomalies and trend problems that are difficult to detect by traditional methods, greatly improve the depth and breadth of enterprise data quality management, and provide more reliable data assurance for time-series-based business decisions.

[0172] In another preferred embodiment of the present invention, after obtaining the multidimensional OLAP data structure, the method further includes:

[0173] E1. Based on the multidimensional OLAP data structure and CSLPerm rules, assign a score of permissions to each data resource, and adjust the score of permissions according to the type of concurrent operation to generate dynamic permission allocation;

[0174] In high-concurrency environments of multidimensional data processing systems, traditional locking mechanisms often lead to performance bottlenecks. To address this issue, the system introduces an innovative concurrency control mechanism based on fractional permissions. First, based on the existing multidimensional OLAP data structure and combined with Concurrent Separation Logic Permission (CSLPerm) rules, the system achieves fine-grained permission management for data resources. Under this mechanism, the system no longer simply locks data resources with an "all or nothing" approach, but instead adopts the concept of fractional permissions, dividing resource access rights into assignable fractional units. For example, a complete resource has a permission value of "1," a read-only operation might only require "0.25" permissions, while a modify operation requires the full "1" permission.

[0175] E2. Based on the dynamic permission allocation and separation logic strategy, a separation verification mechanism is implemented for the heap memory accessed by concurrent operations;

[0176] Based on the characteristics of the OLAP multidimensional model, the system sets separate permission allocation strategies for data blocks at different dimensions and levels. For dimensions frequently accessed by multiple queries (such as the time dimension), the system tends to assign smaller permission scores, allowing more concurrent reads; for frequently updated dimensions (such as inventory status), stricter permission requirements are set. Each operation type has its corresponding permission requirement template: read-only analysis operations typically receive smaller permission scores, allowing multiple such operations to execute simultaneously; aggregation calculation operations receive medium permission scores, allowing concurrency with a limited number of read operations; data update operations require full or near-full permissions, temporarily excluding other concurrent access. This allocation of permission scores is not static but dynamically adjusted based on system load, operation priority, and resource usage patterns. For example, during peak report generation periods, the system may lower the permission priority of non-critical update operations to ensure that analysis queries receive sufficient resources; while during data loading windows, update operations will receive higher permissions.

[0177] The system also implements an incremental permission acquisition strategy, allowing operations to begin execution with lower permissions and attempt to escalate permissions as needed, reducing unnecessary waiting. Based on this dynamic permission allocation mechanism, the system further incorporates a separation logic strategy to implement an efficient separation verification mechanism for heap memory accessed by concurrent operations. The core idea of ​​separation logic is to divide memory resources into non-overlapping regions, allowing different operations to execute in parallel without interference. The system analyzes the data access pattern of each operation, predicts its "footprint" (i.e., the memory regions it may access), and then uses a complex verification algorithm to determine which operations can be safely executed concurrently. When the footprints of two operations do not overlap, they can run in complete parallelism; when there is partial overlap and both are read-only accesses, they can share this region; when there is overlap and write operations are involved, the system determines whether serialization is required based on permission scores. This separation verification not only logically guarantees the correctness of operations but also significantly reduces unnecessary lock contention and improves system throughput. During implementation, the system tightly integrates separation verification with the memory management system, utilizing hardware-assisted memory protection mechanisms to accelerate the verification process, enabling theoretically complex verification algorithms to execute efficiently in a real-time system.

[0178] E3. When performing a dynamic dimension expansion operation, the CSLPerm rule is used to control concurrent access and lock the affected data area;

[0179] When the system needs to perform dynamic dimension expansion operations (such as adding new dimension members or adjusting the dimension hierarchy), these operations typically affect the structure of the data cube and require careful handling. In this case, the system employs CSLPerm rules to implement precise concurrency control, locking the affected data areas while maximizing accessibility to other areas. Specifically, the system first analyzes the scope of the expansion operation, accurately identifying the data blocks that need modification; then, it acquires full permissions for these blocks while allowing other operations to continue accessing unaffected areas; finally, the system uses a two-phase commit protocol to ensure the atomicity and consistency of the expansion operation. For large-scale dimension expansions, the system implements a gradual strategy, breaking the operation down into a series of smaller steps, releasing the corresponding locks after each step is completed, reducing the blocking time for concurrent operations.

[0180] E4. Based on the residual heap inference algorithm, the heap state changes before and after data operations are automatically analyzed and inferred to track the impact range of each data operation and obtain the heap state change analysis results.

[0181] To accurately track and analyze the impact of each data operation, the system introduces an innovative Residual Heap Inference Algorithm. This algorithm can automatically analyze and infer the changes in the heap state before and after the data operation, generating a detailed "incremental change map".

[0182] When the algorithm is working, it first captures a snapshot of the heap state before the operation is executed, including data values, index structure and metadata information; after the operation is completed, it captures the heap state again, and then uses a complex difference analysis algorithm to calculate the precise change between the two states.

[0183] The system not only identifies changes in values ​​but also infers structural changes (such as the addition or deletion of dimension members) and implicit effects (such as cascading updates of aggregated values). This change information is organized into a structured heap state change report, detailing the scope, intensity, and spillover effects of operations. These analytical results are used not only for operation auditing and fault diagnosis but also fed back to the concurrency control system to optimize permission allocation strategies for similar future operations. By learning the actual impact patterns of historical operations, the system can more accurately predict operational footprints, reduce overly conservative locking, and further improve concurrency performance. Furthermore, residual heap analysis supports advanced data recovery capabilities, allowing the system to precisely roll back the impact of specific operations when needed without affecting the results of other concurrent operations. Through this fractional permission-based concurrency control and heap state analysis framework, the system significantly improves concurrency performance in multidimensional data environments while ensuring data consistency, providing robust support for complex analysis scenarios and large-scale multi-user environments. Tests show that this mechanism can increase throughput by 3-5 times under high concurrency conditions compared to traditional locking mechanisms, especially in read-dominated analytical workloads.

[0184] In another preferred embodiment of the present invention, after obtaining the multidimensional OLAP data structure, the method further includes:

[0185] F1. Collect and analyze historical query logs based on the multidimensional OLAP data structure, and construct the state space of the Markov decision process based on the analysis results;

[0186] F2. Design a multi-dimensional reward function, the optimization objectives of which include query response time, resource consumption and data accuracy, and dynamically adjust the weight of each objective of the reward function according to the business scenario to obtain a multi-objective reward function;

[0187] F3. Using a reinforcement learning algorithm, an optimal query strategy is generated based on the state space and the multi-objective reward function, wherein long-term query behavior patterns are incorporated through ω-regularization attributes;

[0188] F4. Generate a formalized certificate and decision basis record for each query decision. The certificate is used to characterize the optimality of the decision, and the decision basis record is used to record the reasoning basis of the decision.

[0189] F5. Execute the query strategy carrying the certificate and decision basis record, monitor the actual results of the query execution, compare the actual results with the expected results, and automatically adjust the model parameters and reward function of the state space based on the resulting deviation to obtain the optimized query strategy.

[0190] In the core optimization layer of the enterprise data analysis system, a set of intelligent query optimization framework based on reinforcement learning is implemented to provide adaptive performance optimization for complex multidimensional OLAP queries.

[0191] First, the system conducts a comprehensive historical query behavior analysis based on a multidimensional OLAP data structure. This process collects query-related information from multiple data sources: query logs record basic data such as the original SQL statements, execution time, and resource consumption; system monitoring data provides time series data on CPU, memory, and I / O usage during query execution; user interaction logs capture analyst operation patterns, including query sequences, parameter adjustments, and result browsing behavior; and data access statistics record the access frequency and patterns of various dimensions, metrics, and data blocks. The system applies various data mining techniques to process these raw logs: query semantic parsing converts SQL into a structured representation, identifying query dimensions, metrics, and filtering conditions; time pattern analysis reveals periodic characteristics of queries, such as fixed patterns like daily reports, weekly analysis, and monthly comparisons; similarity clustering groups queries into typical query categories, such as drill-down, slice, and year-on-year / month-on-month comparisons; and sequence pattern mining reveals transformation relationships between queries, such as the analysis path from overview to details.

[0192] Based on these analytical results, the system constructs a refined Markov Decision Process (MDP) state-space model. In this model, the state is represented as a multi-dimensional vector, encompassing key factors such as the current system state (e.g., resource load, cached content, number of concurrent queries), query characteristics (e.g., combination of dimensions involved, query complexity, estimated result set size), data characteristics (e.g., cardinality of each dimension, data distribution, update frequency), and time context (e.g., weekday / weekend, peak / off-peak periods). The transition function describes the probability distribution of system state changes after selecting a specific query execution strategy. The system also specifically addresses some observability issues, estimating hidden state variables that cannot be directly measured but are important for decision-making through Bayesian inference. After the state space is constructed, the system designs an innovative multi-dimensional reward function, formalizing the query optimization problem into a reward maximization task. This reward function comprehensively considers several key metrics: query response time, based on user-perceived latency, with higher penalties for long-tail latency; resource consumption, including the weighted sum of various resources such as CPU time, memory usage, network throughput, and disk I / O; data accuracy, measuring the precision and confidence interval of results for scenarios that allow approximate calculations; system fairness, ensuring that different users and task types receive reasonable resource allocation; and long-term optimization goals, such as improving cache hit rate and resource utilization efficiency.

[0193] The system employs an adaptive weighting mechanism, dynamically adjusting the importance of each objective based on different business scenarios. For example, in interactive analysis scenarios, response time receives higher weight; during batch report generation, system fairness and resource efficiency are more important; and in critical decision support scenarios, data accuracy becomes the primary consideration. Weight adjustments are not only based on predefined scenario rules but also continuously optimized using online learning methods based on user feedback and business KPIs. Based on the constructed state space and multi-objective reward function, the system applies advanced reinforcement learning algorithms to generate optimal query strategies. The system uses Deep Q-Network (DQN) as its basic algorithm framework, approximating complex state-action value functions through neural networks. To handle the high-dimensional state space, the system integrates an attention mechanism to automatically identify the most critical features in the current state; to balance exploration and utilization, the system implements a Bayesian exploration strategy based on Thompson sampling, quickly converging to an efficient strategy while maintaining sufficient exploration; to address long-term dependency issues, the system employs an experience replay mechanism and target network separation technology to stabilize the learning process. Most innovatively, the system incorporates long-term query behavior patterns through omega-regular properties. ω-regularity is a powerful tool in formal language theory for describing the behavior of infinite sequences. This system leverages this mathematical foundation to formalize analysts' long-term query patterns (such as the habit of overviewing before drill-down, periodic comparative analysis, etc.) into ω-regular expressions. These expressions are then converted into corresponding Büchi automata and integrated with the MDP through product construction, enabling reinforcement learning algorithms to perceive and optimize strategies targeting these long-term patterns, rather than being limited to optimizing single queries.

[0194] In the actual decision-making process, the system not only generates the optimal execution plan but also produces a formalized certificate and a detailed record of the decision's rationale for each query decision. The certificate is a mathematical construct that formally represents the optimal properties of the decision, including model assumptions, optimality conditions, and proofs. For example, a certificate might state that "under the current system load and query complexity, the selected partition scan strategy theoretically has a response time at least 20% faster than other feasible strategies," along with a mathematical proof supporting this assertion. The decision rationale record is more human-friendly, detailing the factors considered, trade-offs, and reasoning paths during the decision-making process, including feature importance scores, comparisons of alternative strategies, and risk assessments.

[0195] These certificates and decision records not only provide decision transparency, meeting the requirements of explainable AI, but also offer valuable data for subsequent auditing and system improvement. When executing query strategies carrying certificates and decision evidence, the system simultaneously implements comprehensive execution monitoring and adaptive adjustment mechanisms. The real-time monitoring system tracks each stage of query execution, collecting key indicators such as actual execution path, time overhead, resource consumption, and result accuracy. This real-time data is compared with the ideal results predicted by the strategy, calculating deviation values ​​for each dimension. When significant deviations are detected, the system triggers multi-level feedback adjustments: a short-term adjustment mechanism quickly corrects the execution plans of similar queries in the current query queue to prevent the continuous application of erroneous strategies; a medium-term adjustment mechanism updates the state transition model and reward estimation to reflect the latest observed system behavior; and a long-term adjustment mechanism re-evaluates the structure of the state space and feature selection to optimize the overall model architecture.

[0196] The system also implements a case study function, paying particular attention to queries exhibiting abnormal behavior, extracting lessons learned, and strengthening the model's performance in edge scenarios. This multi-layered adaptive optimization forms a closed-loop learning system, enabling query optimization strategies to continuously evolve, adapting to changes in data characteristics, evolving query patterns, and fluctuations in system resources, ultimately providing continuously optimized query performance. Through this comprehensive framework integrating reinforcement learning, formal verification, and adaptive optimization, the system achieves truly intelligent query optimization, significantly surpassing traditional rule-based and cost-based optimizers.

[0197] Experimental evaluations show that the system reduces query response time by an average of 40% and increases resource utilization by 35% in complex OLAP environments, while maintaining strict accuracy standards, providing strong performance assurance for enterprise data analysis.

[0198] In another preferred embodiment of the present invention, after obtaining the multidimensional OLAP data structure, the method further includes:

[0199] G1. Based on the data lineage graph, a Markov decision process model is constructed as a data quality MDP model. The quality indicators of the data quality MDP model include data integrity, accuracy, and consistency.

[0200] G2. Based on the data quality MDP model, define multiple data quality objectives including accuracy, completeness, timeliness, and consistency, set a corresponding reward function for each data quality objective, and determine the optimal data management strategy that matches the multiple data quality objectives through a multi-objective MDP solution algorithm to obtain a multi-objective quality optimization strategy;

[0201] G3. Based on the multi-objective quality optimization strategy, use ω-regular expressions to define long-term behavior norms for data quality, generate behavior trajectories by simulating the execution process of the multi-objective quality optimization strategy, and verify whether the behavior trajectories satisfy the ω-regularity attribute to obtain long-term compliance verification results.

[0202] G4. Integrate the data quality MDP model, the multi-objective quality optimization strategy, the lineage analysis results, and the long-term compliance verification results to construct an intelligent decision support engine. Based on historical data, the current state, and the long-term compliance verification results, adjust the prediction model using the intelligent decision support engine, output decision recommendations through the adjusted prediction model, and generate formalized certificates and decision basis records for each decision recommendation.

[0203] In this embodiment, the system constructs a specialized Markov decision process model based on a data lineage graph, serving as the data quality MDP model. The model's states include quality indicators such as data integrity, accuracy, and consistency, while actions include data cleaning, verification, and repair. The system defines multiple data quality objectives, such as accuracy, integrity, timeliness, and consistency, and sets corresponding reward functions for each objective. Through multi-objective MDP solving algorithms (such as Pareto optimization and weighted sum methods), the system determines the optimal data management strategy that balances these objectives. For example, maximizing processing efficiency while ensuring accuracy, or prioritizing the quality of critical data when resources are limited. The system uses ω-regular expressions to define long-term behavioral norms for data quality, such as "data accuracy should never be lower than the threshold T" and "data consistency checks must be performed regularly." By simulating the execution process of the multi-objective quality optimization strategy, the system generates behavioral trajectories and verifies whether these trajectories satisfy the predefined ω-regularity properties, ensuring that data quality does not degrade during long-term operation. The system integrates the data quality MDP model, multi-objective quality optimization strategies, lineage analysis results, and long-term compliance verification results to build an intelligent decision support engine.

[0204] This engine provides users with data management and business decision-making suggestions based on historical data, current status, and predictive models, such as "Now is the best time to perform data cleaning" and "Data quality issues in this dimension may affect the accuracy of monthly reports." The system generates a formal certificate and decision-making basis record for each suggestion, detailing the basis, factors considered, and expected results. Users can view these certificates to understand the logic and basis of the system's decisions, enhancing their trust in the system's recommendations. Simultaneously, these certificates also serve as the basis for decision auditing, supporting the traceability of decision-making responsibility. In this way, the system not only provides high-quality data but also offers intelligent support for business decisions, significantly improving the value and efficiency of data utilization.

[0205] In advanced practices of enterprise data governance, the system utilizes the previously constructed data lineage graph as a foundation to create an intelligent data quality management framework based on formal methods.

[0206] First, the system constructs a precise Markov Decision Process (MDP) model based on a rich data lineage graph, formalizing data quality management as a sequential decision problem. In this data quality MDP model, the state space is carefully designed as a multi-dimensional vector to comprehensively represent the quality status of data assets. The state includes several key dimensions: real-time measurements of quality indicators for each data entity, such as completeness rate, effective value ratio, and redundancy rate; the operational status of the data processing pipeline, including ETL job status, data flow, and processing latency; the dependency status between data entities, reflecting the impact of upstream changes on downstream entities in the lineage; and data usage, including access frequency, usage scenario distribution, and user feedback. The model's action space covers various intervention measures in data governance, such as data cleaning, missing value imputation, anomaly detection threshold adjustment, data update frequency adjustment, and data quality alarm triggering. The transition function precisely describes the probability distribution of system state evolution after taking a specific action in a specific state. This function is constructed by analyzing historical intervention results and utilizing a propagation model based on data lineage. In particular, the model utilizes the propagation paths in the lineage graph to accurately simulate the mechanism by which quality problems propagate along the data flow, enabling the system to anticipate the chain reaction of intervention measures. Based on the constructed data quality MDP model, the system further defines a multi-dimensional data quality objective system, achieving true multi-objective optimization.

[0207] The system defines a series of interrelated but distinct quality dimensions: data accuracy, focusing on the alignment of data values ​​with real-world entities, assessed through comparison with reference data sources, consistency checks, and anomaly detection; data integrity, focusing on the fill rate of required fields, coverage of key entities, and sufficiency of data granularity; data timeliness, assessing whether data update frequency, processing latency, and real-time performance meet business requirements; and data consistency, verifying whether data across systems and datasets maintains logical consistency to avoid conflicting business views. For each quality objective, the system designs a scientific reward function, transforming abstract quality requirements into quantifiable optimization goals.

[0208] These reward functions are not simple linear functions, but rather take into account complex factors such as diminishing marginal utility of quality improvement, asymmetric penalties for quality decline, and interaction effects between quality metrics. For example, the timeliness reward function is designed with a steeper gradient near the critical threshold of real-time demand, reflecting the business's sensitivity to specific latency limits; the consistency reward function comprehensively considers the severity, scope of impact, and business risks of consistency violations, imposing greater penalties on high-risk inconsistencies.

[0209] To address the trade-offs among multiple objectives, the system employs an advanced multi-objective MDP solution algorithm. First, the system applies Pareto front analysis to identify the non-dominated solution set under different combinations of objectives. Then, considering business priorities, it selects the solution that best meets business requirements from the Pareto front using either a weighted sum method or a lexicographical order method.

[0210] The algorithm further applies constrained reinforcement learning techniques, transforming some quality objectives into hard constraints to ensure the solution meets minimum quality standards. The final generated multi-objective quality optimization strategy is not a static set of rules, but an adaptive policy function capable of dynamically selecting the optimal intervention based on the current data quality state. The system also implements an interpretable representation of the strategy, enabling data managers to understand the decision logic and make manual adjustments when necessary. Based on the generated multi-objective quality optimization strategy, the system further employs formal verification techniques to ensure the strategy consistently meets key quality standards during long-term execution. The system innovatively applies omega-regular expressions to define long-term behavioral norms for data quality. These expressions are powerful tools for describing infinite sequence properties, particularly suitable for expressing temporal properties such as "always" and "infinitely frequent." The system defines several key quality behavioral norms, such as "the accuracy of critical business data will never be below the threshold for three consecutive days," "data processing latency must eventually recover to standard levels after peak load," and "data consistency checks must pass all validation tests infinitely frequently." To verify whether the strategy meets these long-term specifications, the system applies model testing techniques to simulate the execution of the strategy from different initial states, generating a large number of possible behavioral trajectories.

[0211] Each trajectory represents a possible long-term evolution path of the system under the guidance of the strategy, including a state sequence and corresponding changes in quality indicators. The system compares these trajectories with behavioral norms described by ω-regularity attributes and uses formal verification algorithms (such as automata-based model testing) to determine whether the strategy meets the norms on all possible paths. The verification process not only provides a binary conclusion on compliance but also identifies potential violation scenarios and risk points, providing precise guidance for strategy adjustments. Based on the data quality MDP model, multi-objective quality optimization strategies, lineage analysis results, and long-term compliance verification results, the system ultimately constructs a comprehensive intelligent decision support engine. This engine integrates multiple advanced technologies, forming a closed-loop data quality management ecosystem. The core of the decision engine is a set of adaptive prediction models that can predict the possible outcomes of different intervention measures based on historical data and the current state. These models are not static but are continuously adjusted through continuous learning. The system pays particular attention to the discrepancy between the actual and expected effects of historical interventions, continuously optimizing prediction parameters using Bayesian update methods. Simultaneously, it leverages long-term compliance verification results to identify blind spots and weaknesses in the prediction model, especially enhancing its predictive capabilities for high-risk scenarios. Furthermore, it integrates lineage analysis results to optimize the model's accuracy in predicting the propagation paths and impact scope of quality issues. The adjusted prediction model provides data managers with precise decision-making recommendations. These recommendations are not simple operational instructions but detailed intervention plans, including execution steps, expected effects, risk assessments, and alternative options. To ensure the credibility and transparency of decisions, the system generates two key documents for each decision recommendation: a formal certificate and a record of the decision's supporting evidence.

[0212] A formal certificate is a mathematical construct that rigorously proves the optimality or near-optimality of a recommended decision under a given model and assumptions. The certificate includes strategy evaluation metrics, comparison methods, and optimality proofs, enabling objective verification of the decision's quality. The decision justification record, designed for human understanding, details the rationale behind the recommendation, the factors considered, the expected quality improvement, potential side effects, and uncertainties. This record also emphasizes the alignment of the decision with organizational quality goals and compliance requirements, helping managers understand the business value of the recommended decisions. Through this comprehensive framework integrating lineage analysis, formal modeling, multi-objective optimization, and long-term validation, the system achieves truly intelligent data quality management. It not only addresses current quality challenges but also proactively plans long-term quality strategies, providing strong support for the continuous appreciation of organizational data assets. Practical applications show that this framework can improve the average quality metrics of key datasets by over 25% while reducing the human resource costs of quality management by 40%, providing a more reliable foundation for data-driven decision-making.

[0213] In summary, the operational statistics consistency verification method based on dynamic data cubes provided by this invention achieves flexible dimensional expansion and optimized query performance by constructing a dynamically scalable multidimensional data cube; ensures data consistency across all processing stages through a multi-layered intelligent verification mechanism; and enables comprehensive tracking of data sources and impacts through data lineage analysis. These innovations not only solve the technical problems of poor dimensional expansion flexibility, insufficient query performance, difficulty in guaranteeing data consistency, and unclear data lineage in existing technologies, but also provide more efficient and reliable technical support for operational statistics systems, better meeting the needs of enterprises in data analysis and decision support.

[0214] This application also provides an operational statistics consistency verification device based on a dynamic data cube, such as... Figure 2 As shown, it includes:

[0215] The OLAP data structure generation module is used to acquire raw data including customer information, product information, and transaction records, and then construct an OLAP cube structure containing time, region, and product category dimensions based on the raw data. It also pre-calculates the aggregation results corresponding to the target query scenario based on the OLAP cube structure, and obtains a multidimensional OLAP data structure based on the aggregation results.

[0216] The verification data acquisition module is used to perform data integrity verification and data format verification based on the multidimensional OLAP data structure through the original data verification layer, and then perform business logic verification on the data verified by the original data verification layer in the ETL intermediate layer, and calculate the preset dimension indicators based on the data verified by the ETL intermediate layer to obtain indicator data. In the indicator calculation layer, the indicator data is verified using linear congruent check codes and inverted index technology, and the data format verification and business meaning verification are performed in the presentation layer to obtain the verified reliable data.

[0217] The analysis result acquisition module is used to collect metadata information and assign it a unique identifier based on the verified reliable data, parse SQL statements, extract data relationships based on the parsing results, construct a data lineage graph based on the metadata information and the data relationships using a graph algorithm, and obtain data tracing and impact analysis results based on the data lineage graph.

[0218] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for verifying operational statistics consistency based on dynamic data cubes, characterized in that, include: The system acquires raw data including customer information, product information, and transaction records. Based on the raw data, it constructs an OLAP cube structure that includes time, region, and product category dimensions. Based on the OLAP cube structure, it pre-calculates the aggregation results corresponding to the target query scenario. Based on the aggregation results, it obtains a multidimensional OLAP data structure. The data integrity and format are verified through the original data verification layer based on the multidimensional OLAP data structure. Then, in the ETL intermediate layer, business logic verification is performed on the data verified by the original data verification layer. Based on the data verified by the ETL intermediate layer, preset dimension indicators are calculated to obtain indicator data. The indicator data is then verified at the indicator calculation layer using linear congruential checksums and inverted index technology. This includes: generating corresponding checksums for the indicator data using a linear congruential algorithm; constructing an inverted index for the indicator checksums, establishing a mapping relationship between indicator values ​​and their corresponding checksums to obtain a checksum index; verifying the consistency of calculation results when the same indicator is calculated multiple times under different dimension combinations using the checksum index to obtain a consistency verification result; and initiating a traceability process based on the consistency verification result to obtain corrected consistent indicator data. Finally, data format and business meaning verification are performed at the presentation layer to obtain verified reliable data. Based on the verified reliable data, metadata information is collected and assigned a unique identifier. SQL statements are parsed, and data relationships are extracted based on the parsing results. A data lineage graph is constructed based on the metadata information and the data relationships using a graph algorithm. Data tracing and impact analysis results are obtained based on the data lineage graph.

2. The method according to claim 1, characterized in that, The process involves constructing an OLAP cube structure based on the original data, including dimensions of time, region, and product category; pre-calculating aggregation results corresponding to the target query scenario based on the OLAP cube structure; and obtaining a multidimensional OLAP data structure based on the aggregation results, including: The raw data is cleaned, converted in format, and standardized to obtain standardized business data. Based on the standardized business data, a multi-dimensional structure is formed according to the dimensions of time, region, and product category, resulting in an OLAP cube structure. For the target query scenario in the OLAP cube structure, the aggregation result of each dimension combination corresponding to the OLAP cube structure is pre-calculated; After caching the aggregation results, a multidimensional OLAP data structure is obtained.

3. The method according to claim 1, characterized in that, After obtaining the multidimensional OLAP data structure, the process also includes: Receive a user's data query request, parse the time dimension parameter in the data query request, and obtain the time query conditions; For the time query conditions, a time bucketing function is used to perform bucketing, dividing the time into different granularities such as hour, day, week, and month to obtain the bucketed time dimensions; Based on the time dimension after bucketing, multi-level aggregated views are pre-calculated and stored; Based on the time range of the data query request, the most suitable aggregated view is selected, and data is obtained from the selected aggregated view and the minimum necessary calculations are performed to obtain the optimized query results.

4. The method according to claim 1, characterized in that, The method of constructing a data lineage graph based on the metadata information and the data relationships using a graph algorithm includes: Based on the metadata and relational information, the graph database is initialized to obtain the graph storage structure; Based on the graph storage structure, data elements, indicators, and report-type entities are used as nodes in the graph to obtain lineage nodes; Based on the dependencies obtained from the parsing, directed edges representing data flow and computational dependencies are established between related bloodline nodes to obtain bloodline relationship edges; The graph structure containing the bloodline nodes and bloodline relationship edges is optimized and its integrity is verified using graph algorithms to obtain a data bloodline relationship graph.

5. The method according to claim 1, characterized in that, After obtaining the verified reliable data, the process further includes: Based on the aforementioned multidimensional OLAP data structure, a temporal logic model for data attributes is established, and business rules are transformed into LTLf formulas to obtain temporal attribute definitions. Maintain a timestamp version chain for each data element to record its state at different processing stages, thus obtaining a temporal state chain. The reactive synthesis algorithm of LTLfMT is executed to automatically generate a monitor program based on the temporal attribute definition, thus obtaining a temporal monitor. The temporal state chain is used to perform cross-time point data comparison to verify whether the growth rate or year-on-year change meets business expectations and obtain temporal verification results. When the temporal monitor detects a violation of temporal attributes, it records the violation information, analyzes the cause of the violation, and performs automatic repair according to a preset repair strategy to obtain the temporally repaired data.

6. The method according to claim 1, characterized in that, After obtaining the multidimensional OLAP data structure, the process also includes: Based on the multidimensional OLAP data structure and CSLPerm rules, a score-based permission is assigned to each data resource, and the score-based permission is adjusted according to the type of concurrent operation to generate dynamic permission allocation; Based on the dynamic permission allocation and separation logic strategy, a separation verification mechanism is implemented for the heap memory accessed by concurrent operations. When performing a dynamic dimension expansion operation, the CSLPerm rule is used to control concurrent access and lock the affected data area; Based on the residual heap inference algorithm, the heap state changes before and after data operations are automatically analyzed and inferred to track the impact range of each data operation and obtain the heap state change analysis results.

7. The method according to claim 1, characterized in that, After obtaining the multidimensional OLAP data structure, the method further includes: Based on the aforementioned multidimensional OLAP data structure, historical query logs are collected and analyzed, and the state space of a Markov decision process is constructed based on the analysis results. Design a multi-dimensional reward function, the optimization objectives of which include query response time, resource consumption and data accuracy, and dynamically adjust the weight of each objective of the reward function according to the business scenario to obtain a multi-objective reward function; An optimal query strategy is generated based on the state space and the multi-objective reward function using a reinforcement learning algorithm, wherein long-term query behavior patterns are incorporated through ω-regularization. For each query decision, a formalized certificate and decision basis record are generated. The certificate is used to characterize the optimality of the decision, and the decision basis record is used to record the reasoning basis of the decision. The system executes a query strategy that carries certificates and decision-making basis records, monitors the actual results of the query execution, compares the actual results with the expected results, and automatically adjusts the model parameters and reward function of the state space based on the resulting deviations to obtain an optimized query strategy.

8. The method according to claim 1, characterized in that, After obtaining the multidimensional OLAP data structure, the method further includes: A Markov decision process model is constructed based on the data lineage graph as a data quality MDP model. The quality indicators of the data quality MDP model include data integrity, accuracy, and consistency. Based on the data quality MDP model, multiple data quality objectives are defined, including accuracy, completeness, timeliness, and consistency. A corresponding reward function is set for each data quality objective. The optimal data management strategy matching the multiple data quality objectives is determined by the multi-objective MDP solution algorithm, thus obtaining the multi-objective quality optimization strategy. Based on the multi-objective quality optimization strategy, long-term behavior norms for data quality are defined using ω-regular expressions. By simulating the execution process of the multi-objective quality optimization strategy, behavior trajectories are generated, and it is verified whether the behavior trajectories satisfy the ω-regularity attribute to obtain long-term compliance verification results. By integrating the data quality MDP model, the multi-objective quality optimization strategy, the lineage analysis results, and the long-term compliance verification results, an intelligent decision support engine is constructed. Based on historical data, the current state, and the long-term compliance verification results, the intelligent decision support engine adjusts the prediction model and outputs decision recommendations through the adjusted prediction model. For each decision recommendation, a formalized certificate and decision basis record are generated.

9. An operational statistics consistency verification device based on a dynamic data cube, characterized in that, include: The OLAP data structure generation module is used to acquire raw data including customer information, product information, and transaction records, and then construct an OLAP cube structure containing time, region, and product category dimensions based on the raw data. It also pre-calculates the aggregation results corresponding to the target query scenario based on the OLAP cube structure, and obtains a multidimensional OLAP data structure based on the aggregation results. The data verification module is used to perform data integrity and data format verification based on the multidimensional OLAP data structure through the original data verification layer, and then perform business logic verification on the data verified by the original data verification layer in the ETL intermediate layer. Based on the data verified by the ETL intermediate layer, preset dimension indicators are calculated to obtain indicator data. The module also verifies the indicator data using linear congruential checksums and inverted index technology in the indicator calculation layer. This includes: generating corresponding checksums for the indicator data using a linear congruential algorithm; constructing an inverted index of the indicator checksums to establish a mapping relationship between indicator values ​​and their corresponding checksums to obtain a checksum index; verifying the consistency of calculation results when the same indicator is calculated multiple times under different dimension combinations using the checksum index to obtain a consistency verification result; and initiating a traceability process based on the consistency verification result to obtain corrected consistent indicator data, and performing data format and business meaning verification in the presentation layer to obtain verified reliable data. The analysis result acquisition module is used to collect metadata information and assign it a unique identifier based on the verified reliable data, parse SQL statements, extract data relationships based on the parsing results, construct a data lineage graph based on the metadata information and the data relationships using a graph algorithm, and obtain data tracing and impact analysis results based on the data lineage graph.

Images