Data warehouse intelligent construction and management method and system

Abstract

The invention discloses a data warehouse intelligent construction and management method and system, and belongs to the technical field of data warehouse construction and management, and the data warehouse intelligent construction and management method comprises the steps: obtaining an enterprise data source, and extracting the feature parameters of the data source; constructing a data warehouse logic model; self-adaptive ETL configuration parameters are defined; constructing a three-layer physical architecture comprising a data access layer, a data storage layer and a data service layer; intelligent cache strategy configuration is executed based on a deep learning algorithm, a three-level cache architecture is established, and a pre-cache rule and a failure strategy are defined; and running management optimization is implemented through a multi-objective optimization function. According to the method, the construction automation level is remarkably improved, the cache hit rate is increased, the average query response time is shortened, the data consistency and the system performance are effectively balanced, the elastic adaptive capacity of the system is enhanced, the operation and maintenance complexity and cost are reduced, and meanwhile efficient sharing of cache resources is achieved on the premise that data security isolation is guaranteed.

Description

Technical Field

[0001] This invention belongs to the field of data warehouse construction and management technology, and in particular relates to a method and system for intelligent construction and management of data warehouses. Background Technology

[0002] A data warehouse is a subject-oriented, integrated, time-varying, and non-volatile collection of data used to support management decision-making processes. It integrates heterogeneous enterprise data through ETL processes to form a unified analytical environment, serving as the core infrastructure for data-driven decision-making. As data scale and complexity increase, building efficient and scalable data warehouse systems has become a necessity for enterprises, and its management quality directly determines the timeliness and accuracy of data analysis.

[0003] As the key interface directly facing users, the data service and application layer provides data insights through BI tools, ad-hoc queries, and API services. Faced with increasing data scale and query complexity, existing solutions generally employ caching mechanisms to optimize performance. Cloud data warehouses reduce redundant computation overhead through result set caching, materialized views, and shared caching layers; however, these architectures have inherent flaws: predefined caching strategies struggle to adapt to dynamic loads, leading to unstable cache hit rates and unpredictable performance under sudden queries and high concurrency scenarios.

[0004] To address these issues, existing technologies primarily employ cache sharing mechanisms. However, while improving performance, these mechanisms introduce data consistency challenges. During incremental updates, strong consistency requires complex cache invalidation and synchronization mechanisms, significantly diminishing performance gains; while eventual consistency may lead to decisions based on outdated data. Furthermore, security and performance are inherently contradictory: in multi-tenant environments, user-isolated caches create redundancy, while shared caches increase the risk of data leakage. Post-access permission filtering results in a double waste of computational and storage resources.

[0005] The root of these problems lies in the lack of global coordination capabilities in the existing architecture. Static caching strategies are unable to perceive business context, passive optimization struggles to cope with diverse workloads, and security and performance mechanisms are disconnected, forcing enterprises to make difficult trade-offs between performance, consistency, and security. Summary of the Invention

[0006] The purpose of this invention is to provide a method and system for intelligent construction and management of data warehouses, thereby solving the problems existing in the prior art.

[0007] To achieve the above objectives, this invention provides a method for intelligent construction and management of a data warehouse, comprising the following steps: The system acquires the enterprise's data sources and extracts their characteristic parameters by parsing the data format, interface protocol, and update frequency of each data source. These characteristic parameters include data type identifiers, data volume parameters, and timeliness constraints. Based on the data source, the corresponding business terms are matched, semantic similarity is calculated and the subject domains are divided by clustering, and dimensional relationship mapping is constructed through the subject domains to generate a data warehouse logical model; Define data extraction rules, transformation rules, and loading strategies, and generate ETL configuration parameters; The data warehouse physical architecture is constructed based on ETL configuration parameters. The data warehouse physical architecture includes a data access layer, a data storage layer, and a data service layer, and defines the storage format, indexing strategy, and partitioning rules for the data storage layer. Based on the data warehouse physical architecture, execute intelligent caching strategy configuration, define cache level mapping, pre-caching rules and invalidation policies, and generate dynamic cache configuration; The data warehouse operation and management are implemented based on dynamic cache configuration, and management optimization is performed through multi-objective optimization functions to generate management optimization results.

[0008] Furthermore, the data type identifier is generated through a data type mapping table, establishing a correspondence between the source data type and the standard data type, and generating a type identifier that includes precision, length, and constraints. Data volume parameters are calculated through statistical analysis, including the total number of records, average record size, growth rate, and storage usage, using the following formulas: ; in, Indicates the data volume parameter. Indicates the total number of records. Indicates the average record size; Timeliness constraint parameters are determined through data update frequency analysis. The system monitors the change logs of the data source, calculates the data update interval and update mode, and generates timeliness constraint parameters that include the maximum delay tolerance, update frequency level, and data freshness requirements.

[0009] Furthermore, the data warehouse logical model includes: subject domain identifiers, dimension table structure, and fact table structure; The steps for generating a data warehouse logical model include: Build a business dictionary containing commonly used business terms for enterprises; Match field names, comments, and content in the data source with corresponding business terms by identifying business-related roots and affixes in the field names; Calculate the semantic similarity between fields in the data source after matching business terms, and construct a similarity matrix; The K-means clustering algorithm is used, with the similarity matrix as the distance matrix for the clustering algorithm. The number of cluster centers is determined by the elbow rule. Data fields with high semantic similarity are classified into the same topic domain, and each cluster after clustering is called a topic domain. Cluster quality is assessed using the silhouette coefficient, calculated using the following formula: ; in, Represents the profile coefficient. Indicates the total number of data points. Representing data points The average distance to other points in the same cluster, Representing data points The average distance to the nearest heterogeneous cluster; When the silhouette coefficient is greater than 0.5, the subject area division is determined; Dimensional relationship mapping is constructed through hierarchical analysis to identify the relationships between dimension tables and fact tables within a subject domain; the generation of dimension table structures follows the star schema design principle and includes dimension keys, dimension attributes, and hierarchical structure definitions; the fact table structure includes measure fields, dimension foreign keys, and granularity definitions. Generate a logical model of the data warehouse that includes subject domain identifiers, dimension table structures, and fact table structures.

[0010] Furthermore, ETL configuration parameters include data extraction rules, transformation rules, and loading strategies; The steps to generate ETL configuration parameters include: The data extraction rules include extraction strategy and extraction frequency. The extraction strategy adopts a hybrid strategy that combines incremental extraction and full extraction. Incremental extraction is used when the data volume is greater than the set threshold, and full extraction is used otherwise. The extraction frequency is calculated based on the timeliness constraint parameter and data volume parameter of the data source parameters. The formula for calculating the decimation frequency is as follows: ; in, Indicates the sampling frequency. Indicates the time constraint period. Indicates the system's processing capacity. For data volume parameters, It is a complexity factor; The conversion rules include three levels: data type conversion, data format standardization, and application of business rules. Data type conversion is achieved through a type mapping matrix, which establishes a precise mapping relationship between the source system's data types and the target data warehouse's data types; Data format standardization employs regular expression matching and replacement techniques to unify date formats, numerical precision, and character encoding. Business rule applications are executed through the rule engine, including data validation, calculation of derived fields, and business logic validation; The loading strategy combines batch loading and streaming loading. Batch loading is used for the initial loading of historical data, while streaming loading is used for the real-time updating of incremental data. The loading strategy also includes loading priority, which determines which data should be loaded first during the loading process. The loading priority is determined based on the business importance and timeliness requirements of the data, and the calculation formula is as follows: ; in, Indicates loading priority. Indicating business importance index, Indicates timeliness and urgency. , These are the first and second loading priority weight coefficients, respectively.

[0011] Furthermore, after data transformation, a field mapping table is generated, and data quality is checked to see if it is up to standard. If it is not up to standard, data extraction needs to be performed again. Data quality inspection employs a multi-level quality assessment system, considering data completeness, accuracy, and consistency indicators to calculate a comprehensive data quality score. The formula for calculating the comprehensive data quality score is as follows: ; in, This represents the overall score for data quality. Indicates the integrity score. Indicates accuracy score, Indicates consistency score, , , These are the weighting coefficients for the comprehensive score of the first, second, and third data quality, respectively.

[0012] Furthermore, the physical architecture of a data warehouse includes a data access layer, a data storage layer, and a data service layer, specifically: The data access layer is responsible for receiving data streams from the ETL process and uses a message queue buffering mechanism to handle peak loads. The data access layer uses a columnar storage structure, optimized for analytical queries. The buffer size is dynamically adjusted based on data traffic and processing capacity, calculated using the following formula: ; in, Indicates the buffer size. For sampling frequency, For data volume parameters, For the buffer time window, Minimum buffer size; The data storage layer is responsible for storing the field mapping table obtained from the data access layer after processing it according to the transformation rules. The configuration of the data storage layer includes storage format, indexing strategy, and partitioning rules, specifically: The storage format adopts a hybrid storage architecture, including row-based and column-based storage; the selection of the storage format is based on data access analysis; the formula for calculating the weight of the storage format is as follows: ; in, Indicates the storage format weight. Indicates the number of analytical queries. This indicates the total number of queries. Indicates the query performance of columnar storage. Indicates the query performance of row-oriented storage; Columnar storage is selected when the storage format weight is greater than 1; otherwise, row storage is selected. The indexing strategy employs a multi-level index system, including a primary index, secondary indexes, and a clustered index; specifically: the primary index is automatically constructed based on the primary key field; a clustered index is created when no clustered index exists and the clustering effect is greater than 1; a secondary index is created when the index selectivity is greater than 0.1 and no clustered index exists. The partitioning rules adopt a hybrid partitioning strategy that combines range partitioning and hash partitioning. Specifically, the first-level partition is a range partition, and hash partitioning is performed within each first-level partition to locate the second-level partition. The data service layer provides a unified data access interface and adopts a multi-tenant architecture to support concurrent access. The storage format of the data service layer is optimized for query response time, employing a tiered storage mechanism combining in-memory computing and disk storage. Hot data is stored in memory, while cold data is stored on disk. Data temperature is assessed using an access frequency and time decay model, calculated as follows: ; in, Indicates the temperature of the data. Indicates the first Weight of each visit Indicates the first Second visit time. Indicates the current time. This represents the time decay coefficient; data with temperatures above the threshold are stored in memory.

[0013] Furthermore, the cache hierarchy mapping adopts a three-level cache architecture: L1 is a memory result cache, which stores the query results accessed frequently; L2 is an SSD intermediate cache, which stores data blocks accessed at a medium frequency; and L3 is a disk pre-computation cache, which stores the materialized view of complex calculations. The allocation of cache levels is based on access frequency and data value calculation: ; in, Indicates the cache level. Indicates access frequency. Indicates the value score of the data. Indicates the data size. and These represent the L1 and L2 cache access frequency thresholds, respectively. Indicates the data value threshold, Indicates a data size threshold; Data Value Scoring Calculated using the following formula: ; in, Indicates the query complexity. Indicates the calculation time. This indicates the baseline cost calculation.

[0014] Furthermore, the pre-caching rules are as follows: The system first collects query execution logs and extracts query feature vectors. It then uses a cosine similarity algorithm to calculate the similarity between any two query feature vectors. When the similarity score between two queries exceeds a threshold of 0.8, the system classifies them as having the same access pattern. Simultaneously, the system monitors load data, uses a max-min normalization method to transform all load data into a unified range of 0 to 1, and analyzes load characteristics through multi-dimensional indicators. A deep learning algorithm outputs the predicted access probability and predicted access time point for each data item. When the prediction accuracy is greater than 75%, an aggressive pre-caching strategy is enabled; when the prediction accuracy is less than 50%, a conservative pre-caching strategy is switched to; otherwise, the current caching strategy is maintained. The inputs to deep learning algorithms include: a set of historical query feature vectors, query similarity clustering results, time-series data of historical load intensity, and timestamp records of data access; The historical query feature vector set includes a set of historical query feature vectors; The query feature vector includes the query type, the number of tables involved, the data scan range, the execution frequency, and the response time requirements. Load data includes: the number of concurrent users in the current system, the query arrival frequency per unit time, the frequency of data update operations, and the system hardware resource utilization level; The standardization process for load data requires determining the maximum and minimum boundaries of each indicator based on historical monitoring data; The extracted load characteristics include load intensity, which is analyzed through multi-dimensional indicator monitoring, including concurrent users, query arrival rate, data update frequency, and system resource utilization. The formula for calculating load strength is as follows: ; in, Indicates load strength. This represents the normalized value of the number of concurrent users. This represents the normalized query reach value. This represents the normalized value of the data update frequency. Indicates system resource utilization. , , , These are the first, second, third, and fourth load intensity weighting coefficients, respectively.

[0015] Furthermore, the failure strategy is as follows: Based on the data consistency requirements of different business scenarios, the system divides cached data into three levels: strong consistency, weak consistency, and eventual consistency. Strong consistency requires that the synchronization delay between the cache and the source data does not exceed 60 seconds, weak consistency allows a delay of 300 seconds, and eventual consistency can tolerate a delay of 1800 seconds. By analyzing the time intervals between consecutive update operations, the average update frequency of each data item is calculated using the following formula: ; in, Indicates the number of updates within the statistical period. Indicates the first The time interval between the next update and the previous update; Calculate the optimal lifetime for each cached item while ensuring it does not exceed the maximum lifetime allowed by the system capacity limit. The calculation formula is as follows: ; in, Indicates cache capacity. Indicates the average cache item size. Indicates the cache generation rate; Based on the calculated lifetime, the system sets an automatic expiration timer for each cached item. When a cached item reaches its preset lifetime or an update to the source data is detected, the system immediately performs a cache expiration operation. The calculation formula is as follows: ; in, Indicates the cache lifetime. Indicates the consistency tolerance time. Indicates the data update frequency. Indicates the maximum survival time.

[0016] A data warehouse intelligent construction and management system, comprising: Data source acquisition and feature parameter extraction module: Acquires the enterprise's data sources and extracts the data source feature parameters by parsing the data format, interface protocol and update frequency of each data source; Data warehouse logical model generation module: Based on the data source, it matches the corresponding business terms, calculates semantic similarity and clusters subject domains, and constructs dimensional relationship mapping through subject domains to generate the data warehouse logical model; ETL Configuration Parameter Generation Module: Defines data extraction rules, transformation algorithms, and loading strategies, and generates ETL configuration parameters; Data warehouse physical architecture construction module: Constructs the data warehouse physical architecture based on ETL configuration parameters. The data warehouse physical architecture includes a data access layer, a data storage layer, and a data service layer, and defines the storage format, indexing strategy, and partitioning rules for the data storage layer; Dynamic cache configuration generation module: Based on the data warehouse physical architecture, executes intelligent cache strategy configuration, defines cache level mapping, pre-caching rules and invalidation policies, and generates dynamic cache configuration; Data warehouse management module: Implements data warehouse operation management based on dynamic cache configuration and performs management optimization through multi-objective optimization functions to generate management optimization results.

[0017] Therefore, the present invention, employing the above-mentioned intelligent data warehouse construction and management method and system, has the following beneficial effects: By combining an intelligent three-level caching architecture with a predictive pre-caching strategy, the cache hit rate is increased to over 85%, and the average query response time is reduced by 60%, effectively solving the performance bottleneck in high-concurrency scenarios.

[0018] The system intelligently adjusts data synchronization requirements based on business scenarios, ensuring strong consistency for critical business data while allowing eventual consistency for non-critical data, thus achieving an optimal balance between performance and reliability.

[0019] Through adaptive load monitoring and dynamic optimization mechanisms, the system can automatically identify and respond to load changes, automatically expand cache capacity during peak business periods, and reclaim resources during off-peak periods to ensure service quality stability.

[0020] Intelligent configuration management and automated optimization reduce the workload of manual maintenance and lower the professional skill requirements, while optimizing resource allocation reduces hardware investment costs.

[0021] Multi-tenant architecture combined with fine-grained access control enables efficient sharing of cached resources while ensuring data security and isolation, thus resolving the conflict between security and performance in traditional solutions.

[0022] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0023] Figure 1 This is an overall flowchart of a data warehouse intelligent construction and management method according to the present invention; Figure 2 This is a schematic diagram of the structure of a data warehouse intelligent construction and management system according to the present invention. Detailed Implementation

[0024] The following detailed description of embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.

[0025] Please see Figure 1 A method for intelligent construction and management of data warehouses includes the following steps: Step 100: Obtain the enterprise's data sources, and extract the data source characteristic parameters by parsing the data format, interface protocol and update frequency of each data source; This step involves acquiring the enterprise's data sources, including relational databases, NoSQL databases, file systems, web service interfaces, and streaming data sources. The system first establishes a data source connection pool and enables unified access to different data sources by configuring connection parameters.

[0026] Specifically, for relational data sources, the system uses a JDBC connector to parse the database table structure and extract column names, data types, primary key constraints, and foreign key relationships; for NoSQL data sources, the system extracts field hierarchy and data type information through document structure analysis; for file system data sources, the system identifies data format and encoding methods through file header parsing and content sampling; and for Web service interfaces, the system extracts data patterns through API document parsing and response structure analysis.

[0027] Data source characteristic parameters include data type identifiers, data volume parameters, and timeliness constraints. The extraction of data source characteristic parameters adopts a multi-dimensional analysis method: the data type identifier is generated through a data type mapping table to establish the correspondence between the source data type and the standard data type, generating a type identifier that includes precision, length, and constraints; the data volume parameters are calculated through statistical analysis, including the total number of records, average record size, growth rate, and storage usage; the timeliness constraint parameters are determined through data update frequency analysis. The system monitors the change log of the data source, calculates the data update interval and update mode, and generates timeliness constraint parameters that include maximum latency tolerance, update frequency level, and data freshness requirements.

[0028] The data volume parameter is calculated using the following formula: ; in, Indicates the data volume parameter. Indicates the total number of records. This indicates the average record size (in bytes).

[0029] Update frequency levels are divided according to the following rules: real-time level (update interval < 1 second), near real-time level (1 second to 1 minute), timed level (1 minute to 1 hour), and batch processing level (> 1 hour).

[0030] The output data source characteristic parameters are stored in a structured format, including data source identifier, connection configuration, mode definition, magnitude parameters, and time constraints.

[0031] Step 200: Match the corresponding business terms based on the data source, calculate semantic similarity and cluster the subject domains, construct dimensional relationship mapping through the subject domains, and generate the data warehouse logical model; First, a business dictionary is constructed, containing commonly used business terms. Then, the field names, comments, and content in the data source are matched with corresponding business terms by identifying business-related roots and affixes in the field names. Finally, the semantic similarity between fields in the data source after matching business terms is calculated to form a similarity matrix. The topic domains are divided using the K-means clustering algorithm, with the similarity matrix used as the distance matrix for the clustering algorithm. The number of cluster centers is determined by the elbow rule. The quality of the clusters after clustering is evaluated using the silhouette coefficient, and each cluster after clustering is called a topic domain.

[0032] It should be noted that the business dictionary can be constructed by importing industry standard dictionaries, parsing existing enterprise documents, or inputting expert knowledge.

[0033] The formula for calculating the semantic similarity between fields is as follows: ; in, Indicates the overall semantic similarity. Indicates the similarity of field names. Indicates the similarity of field content. Indicates the similarity of field structures. , , These are the weight coefficients for the first, second, and third comprehensive semantic similarity, respectively.

[0034] Field name similarity is calculated using the edit distance algorithm, and the formula is as follows: ; in Indicates field name and Edit distance between and Representing field names and The length of the characters, This indicates the larger of the two lengths.

[0035] Field content similarity is calculated through statistical analysis, using the following formula: ; in, and These represent the value range sets of the two fields, respectively.

[0036] Field structure similarity is calculated based on the matching degree of data types and constraints, using the following formula: ; in, This indicates the data type matching degree (1 for the same, 0.5 for compatible, and 0 for different). Indicates the degree of constraint matching. Indicates the precision matching degree.

[0037] The formula for calculating the constraint matching degree is as follows: ; in, and These represent the sets of constraints for the two fields, including primary key, foreign key, uniqueness, NOT NULL constraints, etc.

[0038] The formula for calculating the accuracy matching degree is as follows: ; in, and These represent the precision (number of decimal places or character length) of the two fields, respectively.

[0039] The formula for calculating the profile coefficient is as follows: ; in, Represents the profile coefficient. Indicates the total number of data points. Representing data points The average distance to other points in the same cluster, Representing data points The average distance to the nearest heterogeneous cluster.

[0040] In one embodiment of the present invention, when the silhouette coefficient is greater than 0.5, the clustering effect is considered to be good, thereby determining the topic domain division.

[0041] Dimensional relationship mapping is constructed through hierarchical analysis to identify the relationships between dimension tables and fact tables within a subject domain. The dimension table structure is generated following the star schema design principle, including dimension keys, dimension attributes, and hierarchical structure definitions. The fact table structure includes measure fields, dimension foreign keys, and granularity definitions. The formula for calculating the depth of the dimension hierarchy is as follows: ; in, Indicates the depth of the dimension hierarchy. Indicates the number of dimension members. This indicates rounding up. This formula ensures the balance of the dimensional hierarchy and optimizes query performance.

[0042] The generated data warehouse logical model comprises three core components: subject domain identifiers, dimension table structures, and fact table structures. Subject domain identifiers are generated using business coding rules, with the format "business domain code-subject code-version number"; the dimension table structure definition includes table name, field list, primary key constraints, and hierarchical relationships; the fact table structure definition includes table name, measure fields, dimension foreign keys, and partitioning strategies, forming a complete data warehouse logical model.

[0043] Step 300: Define data extraction rules, transformation rules, and loading strategies, and generate ETL configuration parameters; ETL configuration parameters include data extraction rules, transformation rules, and loading strategies; Data extraction rules include extraction strategy and extraction frequency. The extraction strategy adopts a hybrid strategy that combines incremental extraction and full extraction. Incremental extraction is used when the data volume is greater than the set threshold, and full extraction is used otherwise. The extraction frequency is calculated based on the timeliness constraint parameter and data volume parameter of the data source parameters.

[0044] The formula for calculating the decimation frequency is as follows: ; in, This indicates the sampling frequency (times / hour). Indicates the time constraint period (in hours). This indicates the system's processing power, calculated using hardware configuration and historical performance data. The data volume parameters calculated in step 100, This is the complexity factor. This formula balances the timeliness requirements of business operations with the system's processing capacity, ensuring the stable execution of the ETL process.

[0045] The data transformation rules consist of three levels: data type transformation, data format standardization, and business rule application. Specifically, data type transformation is achieved through a type mapping matrix, establishing a precise mapping relationship between the data types in the source system and the data types in the data warehouse logical model; data format standardization uses regular expression matching and replacement technology to unify date formats, numerical precision, and character encoding; and business rule application is executed through a rule engine, including data validation, calculation of derived fields, and business logic verification.

[0046] After data transformation, a field mapping table is generated, and the data quality is checked to see if it is up to standard. If it is not up to standard, the data extraction needs to be repeated. Data quality inspection employs a multi-level quality assessment system, considering data completeness, accuracy, and consistency indicators to calculate a comprehensive data quality score. The formula for calculating the comprehensive data quality score is as follows: ; in, This represents the overall score for data quality. Indicates the integrity score. Indicates accuracy score, Indicates consistency score, , , These are the weighting coefficients for the comprehensive score of the first, second, and third data quality, respectively.

[0047] Data quality is considered acceptable when the overall data quality score is greater than the data quality inspection threshold. The data quality inspection threshold is set in advance, and is generally set to 0.8 or higher.

[0048] The formula for calculating the data integrity score is as follows: ; in, Indicates the total number of records. Indicates the number of null records. This indicates the number of empty string records.

[0049] Data accuracy scores are calculated through rule validation, using the following formula: ; in, This indicates the number of records that have passed the business rules verification. The business rules include data type checks, value range verification, and format compliance checks.

[0050] Data consistency score is calculated through cross-table join checks, and the calculation formula is as follows: ; in, This represents the number of records that satisfy the referential integrity constraint. This indicates the number of records that require a reference integrity check.

[0051] The loading strategy combines batch loading and streaming loading. Batch loading is used for the initial loading of historical data, while streaming loading is used for the real-time updating of incremental data.

[0052] The loading strategy also includes loading priority, which determines which data should be loaded first during the loading process. The loading priority is determined based on the business importance and timeliness requirements of the data, and the calculation formula is as follows: ; in, Indicates loading priority. Indicating business importance index, Indicates timeliness and urgency. , These are the first and second priority loading weight coefficients, respectively. Higher priority data is processed first to ensure the timely availability of critical business data.

[0053] The formula for calculating the business importance index is as follows: ; in, Indicate the impact of the data on the business (1-10 points). This indicates the frequency of data usage (times / day). This indicates the criticality coefficient (3 for core business, 2 for important business, and 1 for general business). Indicates the standardization cardinality (default value is 10).

[0054] Timeliness and urgency : ; in, Indicates the current time. Indicates the required completion time. Indicates the maximum tolerable delay time. This represents the latency penalty coefficient, which is determined based on the service type: 5 for real-time services, 3 for near real-time services, and 1 for batch processing services.

[0055] Step 400: Build the data warehouse physical architecture according to the ETL configuration parameters. The data warehouse physical architecture includes a data access layer, a data storage layer, and a data service layer, and defines the storage format, indexing strategy, and partitioning rules for the data storage layer. The data access layer is responsible for receiving data streams from the ETL process, extracting data based on data volume and extraction rules, and using a message queue buffering mechanism to handle peak loads from the data stream input. The buffer size is dynamically adjusted based on data traffic and processing capacity.

[0056] The formula for calculating the buffer size is as follows: ; in, Indicates the buffer size (MB). For sampling frequency, For data volume parameters, For the buffer time window (seconds). This is the minimum buffer size. This formula ensures that the access layer can smoothly handle data flow fluctuations and avoid data loss.

[0057] The data storage layer is responsible for storing the field mapping table obtained by the data access layer after processing the data through transformation rules. The configuration of the data storage layer includes storage format, indexing strategy and partitioning rules.

[0058] The storage format adopts a hybrid storage architecture, including row-based and column-based storage. The choice of storage format is based on data access analysis; the formula for calculating the weight of the storage format is as follows: ; in, Indicates the storage format weight. Indicates the number of analytical queries. This indicates the total number of queries. Indicates the query performance of columnar storage. This indicates the query performance of row-oriented storage.

[0059] Columnar storage is selected when the storage format weight is greater than 1; otherwise, row storage is selected. Columnar storage query performance was calculated using standardized tests, and the formula is as follows: ; in, Indicates the baseline data scan time (default value is 1000ms). Indicates the actual scan time for columnar storage. Indicates columnar storage compression ratio. Indicates the baseline compression ratio (default value is 1).

[0060] The formula for calculating row-oriented storage query performance is as follows: ; in, This represents the baseline random access time (default value is 100ms). This indicates the actual random access time for row-oriented storage. This represents the random access efficiency coefficient. This represents the baseline efficiency coefficient (default value is 1).

[0061] The indexing strategy employs a multi-level index system, including a primary index, secondary indexes, and a clustered index. Specifically: the primary index is automatically built based on the primary key field; a clustered index is created when no clustered index exists and the clustering effect is greater than 1; and a secondary index is created when the index selectivity is greater than 0.1 and no clustered index exists.

[0062] The formula for calculating the clustering effect is as follows: ; in Indicates clustering effect, This indicates the query frequency of the i-th field. This represents the total frequency of all queries. This represents the minimum frequency threshold for clustered indexes.

[0063] ; in Indicates the number of candidate index fields. This represents the frequency multiplier factor, which is typically 2-3.

[0064] The formula for calculating index selectivity is as follows: ; in, Indicates index selectivity. This indicates the number of unique values ​​in the field. This indicates the total number of records.

[0065] The partitioning rules follow data access patterns and storage capacity limits, and adopt a hybrid partitioning strategy that combines range partitioning and hash partitioning.

[0066] In one embodiment of the present invention, the partitioning rule is specifically implemented as follows: Level 1: Range partitioning. First-level partitioning is done according to the time dimension, such as partitioning by month. This allows you to quickly locate the relevant time partition by leveraging the characteristics of time range queries.

[0067] Level 2: Hash partitioning. Within each time partition, hash partitioning is performed again based on user ID or other frequently queried fields, further reducing the amount of data scanned for queries.

[0068] The partitioning rules also include the partition naming format and the number of partitions; Partition naming format: Use the naming format of "table name_time identifier_hash value".

[0069] The formula for calculating the number of partitions is as follows: ; in, Indicates the number of partitions. Indicates the total amount of data. Indicates the maximum capacity of a single partition. This indicates the maximum number of partitions supported by the system. This formula balances query performance with management complexity.

[0070] The data service layer provides a unified data access interface and adopts a multi-tenant architecture to support concurrent access. The service layer's storage format is optimized for query response time, employing a tiered storage mechanism combining in-memory computing and disk storage. Hot data is stored in memory, while cold data is stored on disk. Data temperature is assessed using an access frequency and time decay model, calculated as follows: ; in, Indicates the temperature of the data. Indicates the first Weight of each visit Indicates the first Second visit time. Indicates the current time. This represents the time decay coefficient. Data with a temperature above a threshold is stored in memory to improve access performance.

[0071] Step 500: Configure intelligent caching strategy based on the data warehouse physical architecture, define cache level mapping, pre-caching rules and invalidation policies, and generate dynamic cache configuration; This step, based on the data warehouse physical architecture built in step 400, implements an intelligent caching strategy configuration. This strategy uses deep learning algorithms to analyze query access patterns and load characteristics, dynamically generating the optimal cache configuration and solving the technical challenges of traditional caching mechanisms in terms of performance, consistency, and security.

[0072] The cache hierarchy mapping adopts a three-level cache architecture: L1 is a memory result cache, which stores the query results accessed frequently; L2 is an SSD intermediate cache, which stores data blocks with medium frequency; and L3 is a disk pre-computation cache, which stores the materialized view of complex calculations.

[0073] The allocation of cache levels is based on access frequency and data value calculation: ; in, Indicates the cache level. Indicates access frequency. Indicates the value score of the data. Indicates the data size. and These represent the L1 and L2 cache access frequency thresholds, respectively. Indicates the data value threshold, This indicates the data size threshold.

[0074] Data Value Scoring Calculated using the following formula: ; in, Indicate the query complexity (1-5 points). Indicates the calculation time (in seconds). This represents the baseline cost (a fixed value of 10).

[0075] The system first collects query execution logs and extracts query feature vectors. It then uses a cosine similarity algorithm to calculate the similarity between any two query feature vectors. When the similarity score of two queries exceeds a threshold of 0.8, the system classifies them as having the same access pattern. Simultaneously, the system monitors load data, uses a max-min normalization method to transform all load data into a unified range of 0 to 1, and analyzes load characteristics through multi-dimensional indicators. Finally, it outputs the predicted access probability and predicted access time point for each data item using a deep learning algorithm.

[0076] The inputs to deep learning algorithms include: a set of historical query feature vectors, query similarity clustering results, time-series data of historical load intensity, and timestamp records of data access.

[0077] It should be noted that the deep learning algorithm used can be a long short-term memory network, a recurrent neural network, or a convolutional neural network, etc., and there are no restrictions here; The historical query feature vector set includes a set of historical query feature vectors; The query feature vector includes the query type, the number of tables involved, the data scan range, the execution frequency, and the response time requirements. Load data includes: the number of concurrent users in the current system, the query arrival frequency per unit time, the frequency of data update operations, and the system hardware resource utilization level; The standardization process for load data requires determining the maximum and minimum boundaries of each indicator based on historical monitoring data.

[0078] The extracted load characteristics include load intensity, which is analyzed through multi-dimensional indicator monitoring, including concurrent users, query arrival rate, data update frequency, and system resource utilization.

[0079] The formula for calculating load strength is as follows: ; in, Indicates load strength. This represents the normalized value of the number of concurrent users. This represents the normalized query reach value. This represents the normalized value of the data update frequency. Indicates system resource utilization. , , , These are the first, second, third, and fourth load intensity weighting coefficients, respectively.

[0080] The pre-caching rules use a deep learning algorithm to predict the access probability and access time of each data item in the future. When the prediction accuracy is greater than 75%, an aggressive pre-caching strategy is enabled; when the prediction accuracy is less than 50%, a conservative pre-caching strategy is switched to; otherwise, the current caching strategy is maintained.

[0081] It should be noted that the proactive pre-caching strategy refers to an active caching method used when the prediction accuracy is high. Its characteristics include: increasing the amount of pre-cached data to 1.5-2 times the baseline amount, extending the prediction time window to the next 2-4 hours, synchronously preloading related data, and achieving a cache space utilization rate of 80%-90%. This strategy maximizes the cache hit rate by preloading more potential access data.

[0082] A conservative pre-caching strategy is a cautious caching method used when prediction accuracy is low. Its characteristics include: pre-caching data volume is controlled at 0.5-0.8 times the baseline amount; the prediction time window is limited to the next 0.5-1 hour; only high-deterministic data is cached; and cache space utilization is controlled at 40%-60%. This strategy reduces uncertain predictions, avoids wasting cache resources, and ensures cache quality.

[0083] The failure strategy is as follows: Based on the data consistency requirements of different business scenarios, the system categorizes cached data into three levels: strong consistency, weak consistency, and eventual consistency. Strong consistency requires a synchronization latency of no more than 60 seconds between the cache and the source data; weak consistency allows a latency of 300 seconds; and eventual consistency can tolerate a latency of 1800 seconds. The calculation formula is as follows: ; in Indicates the level of business consistency.

[0084] The system continuously monitors update operations on the source data and statistically analyzes the update frequency patterns of different data items. By analyzing the time intervals between consecutive update operations, the system calculates the average update frequency for each data item, which directly affects the cache lifetime setting. The calculation formula is as follows: ; in, Indicates the number of updates within the statistical period. Indicates the first The time interval (in seconds) between the current update and the previous update.

[0085] The system comprehensively considers data consistency requirements and update frequency, dynamically calculating the optimal lifetime for each cached item. The calculation logic involves dividing the consistency tolerance time by the data update frequency to obtain the theoretically optimal cache lifetime, while ensuring that it does not exceed the maximum lifetime allowed by the system capacity limits. The calculation formula is as follows: ; in, Indicates cache size (MB). This represents the average cache item size (MB). This indicates the cache generation rate (items / second).

[0086] Based on the calculated lifetime, the system sets an automatic expiration timer for each cached item. When a cached item reaches its preset lifetime or an update to the source data is detected, the system immediately performs a cache invalidation operation to ensure the timeliness and consistency of the cached data. The calculation formula is as follows: ; in, Indicates the cache lifetime. Indicates the consistency tolerance time. Indicates the data update frequency. This represents the maximum lifetime. This formula ensures that cache utilization is maximized while meeting consistency requirements.

[0087] The generated dynamic cache configuration includes three core components: cache hierarchy mapping defines the distribution strategy of different types of data in the three-level cache to achieve intelligent tiering; pre-caching rules establish an active caching mechanism based on a prediction model to preload data that may be accessed and improve cache hit rate; and invalidation policy implements fine-grained cache lifecycle management, balances data consistency and cache performance, and solves the inherent defects of traditional caching mechanisms.

[0088] Step 600: Implement data warehouse operation management based on dynamic cache configuration: By monitoring data update frequency, query response time and system resource utilization, dynamically adjust caching strategies and data distribution to generate management optimization results.

[0089] This step, based on the dynamic cache configuration generated in step 500, implements intelligent operation management of the data warehouse. By monitoring key system indicators in real time, it uses adaptive optimization algorithms to dynamically adjust caching strategies and data distribution, ensuring the efficient and stable operation of the data warehouse in complex business environments. Specifically, it includes the following steps: Monitoring System Launch and Deployment: Step 600 first initiates a comprehensive system monitoring mechanism based on the dynamic cache configuration generated in Step 500. This monitoring mechanism is the data foundation for all subsequent optimization decisions, therefore, the real-time, accuracy, and completeness of the monitoring data must be ensured. Layered Implementation of Data Update Frequency Monitoring: The system deploys monitoring points simultaneously at three levels in the data warehouse: table level, partition level, and record level, forming a three-dimensional monitoring network. Table-level monitoring points are responsible for tracking the total amount of data changes in the entire table; partition-level monitoring points focus on the data update activity of each partition; and record-level monitoring points accurately record the modification timestamps of individual records. This layered monitoring design is to meet the data requirements for adjusting caching strategies at different granularities, as invalidation strategies need to be differentiated based on data changes at different levels. Update Frequency Change Detection Mechanism: The system continuously calculates the trend of data update frequency changes using a sliding window statistical method, comparing the update frequency of the current time window with the baseline time window. When the update frequency change rate exceeds a 20% threshold, the system immediately triggers a re-evaluation process for the caching strategy, because significant changes in data update frequency directly affect the freshness requirements of cached data and the rationality of invalidation strategies.

[0090] Distributed performance monitoring agent deployment: Based on the monitoring requirements confirmed in the first phase, the system deploys dedicated monitoring agents on each node of the data service layer. These agents collect execution information for each query in real time, including query start time, end time, amount of data involved, cache level used, and other detailed information. The purpose of distributed deployment is to ensure that the monitoring process itself does not become a bottleneck for system performance, while obtaining a global performance view. It also monitors and calculates the degree of performance degradation and overall resource load.

[0091] Response Time Degradation Assessment: The system uses quantile statistics to assess the trend of query response time changes, focusing on the P95 and P99 quantiles, as these two indicators reflect the system's performance under high load. The current P95 response time is compared with a preset target value to calculate the degree of response time degradation. The calculation formula is as follows: ; in, Indicates the degree of deterioration in response time. This indicates the current P95 response time. This indicates the target P95 response time.

[0092] When the response time deteriorates by more than 30%, it indicates that the current cache configuration can no longer meet the performance requirements, and the performance optimization process is initiated.

[0093] Comprehensive resource load calculation: The system synchronously monitors four core resource indicators: CPU utilization, memory utilization, disk I / O load, and network bandwidth usage. The calculation formula is as follows: ; in, Indicates overall resource load. , , , These represent the load rates of CPU, memory, disk, and network, respectively. , , , These are the weighting coefficients for CPU, memory, disk, and network, respectively.

[0094] Multi-objective optimization problem establishment: Based on the monitoring data collected in the first two stages, the system transforms the caching strategy adjustment into a multi-objective optimization problem. The calculation formula is as follows: ; in, This indicates the target value to be optimized. Indicates cache hit rate. Indicates the average response time. Indicates system resource load. , , The weight coefficients for the first, second, and third optimization objectives are respectively set. The system uses a genetic algorithm to solve for the optimal cache parameter configuration.

[0095] Load imbalance detection: Building upon caching strategy optimization, the system further analyzes the load distribution across storage nodes. By monitoring metrics such as query request volume, data access frequency, and storage capacity utilization on each node, the system identifies problematic nodes with uneven load distribution. Data migration is then performed based on the load of each node.

[0096] The management optimization results are evaluated by calculating a comprehensive performance score, which considers performance improvement indicators, resource utilization efficiency, and system stability indicators. The calculation formula is as follows: ; in, This indicates the overall performance score. , , These represent the target values ​​for cache hit rate, response time, and resource load, respectively.

[0097] Please see Figure 2 A data warehouse intelligent construction and management system, comprising: Data source acquisition and feature parameter extraction module: Acquires the enterprise's data sources and extracts the data source feature parameters by parsing the data format, interface protocol and update frequency of each data source; Data warehouse logical model generation module: Based on the data source, it matches the corresponding business terms, calculates semantic similarity and clusters subject domains, and constructs dimensional relationship mapping through subject domains to generate the data warehouse logical model; ETL Configuration Parameter Generation Module: Defines data extraction rules, transformation algorithms, and loading strategies, and generates ETL configuration parameters; Data warehouse physical architecture construction module: Constructs the data warehouse physical architecture based on ETL configuration parameters. The data warehouse physical architecture includes a data access layer, a data storage layer, and a data service layer, and defines the storage format, indexing strategy, and partitioning rules for the data storage layer; Dynamic cache configuration generation module: Based on the data warehouse physical architecture, executes intelligent cache strategy configuration, defines cache level mapping, pre-caching rules and invalidation policies, and generates dynamic cache configuration; Data warehouse management module: Implements data warehouse operation management based on dynamic cache configuration and performs management optimization through multi-objective optimization functions to generate management optimization results.

[0098] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the technical solutions of the present invention, and these modifications or equivalent substitutions cannot cause the modified technical solutions to deviate from the spirit and scope of the technical solutions of the present invention.

Claims

1. A method for intelligent construction and management of a data warehouse, characterized in that, Includes the following steps: The system acquires the enterprise's data sources and extracts their characteristic parameters by parsing the data format, interface protocol, and update frequency of each data source. These characteristic parameters include data type identifiers, data volume parameters, and timeliness constraints. Based on the data source, the corresponding business terms are matched, semantic similarity is calculated and the subject domains are divided by clustering, and dimensional relationship mapping is constructed through the subject domains to generate a data warehouse logical model; Define data extraction rules, transformation rules, and loading strategies, and generate ETL configuration parameters; The data warehouse physical architecture is constructed based on ETL configuration parameters. The data warehouse physical architecture includes a data access layer, a data storage layer, and a data service layer, and defines the storage format, indexing strategy, and partitioning rules for the data storage layer. Based on the data warehouse physical architecture, execute intelligent caching strategy configuration, define cache level mapping, pre-caching rules and invalidation policies, and generate dynamic cache configuration; The data warehouse operation and management are implemented based on dynamic cache configuration, and management optimization is performed through multi-objective optimization functions to generate management optimization results.

2. The intelligent construction and management method for a data warehouse according to claim 1, characterized in that: Data type identifiers are generated through a data type mapping table, establishing a correspondence between source data types and standard data types, and generating type identifiers that include precision, length, and constraints. Data volume parameters are calculated through statistical analysis, including the total number of records, average record size, growth rate, and storage usage, using the following formulas: ； in, Indicates the data volume parameter. Indicates the total number of records. Indicates the average record size; Timeliness constraint parameters are determined through data update frequency analysis. The system monitors the change logs of the data source, calculates the data update interval and update mode, and generates timeliness constraint parameters that include the maximum delay tolerance, update frequency level, and data freshness requirements.

3. The intelligent construction and management method for a data warehouse according to claim 2, characterized in that, The logical model of a data warehouse includes: subject area identifiers, dimension table structure, and fact table structure; The steps for generating a data warehouse logical model include: Build a business dictionary containing commonly used business terms for enterprises; Match field names, comments, and content in the data source with corresponding business terms by identifying business-related roots and affixes in the field names; The K-means clustering algorithm is used, with the similarity matrix as the distance matrix for the clustering algorithm. The number of cluster centers is determined by the elbow rule. Data fields with high semantic similarity are classified into the same topic domain, and each cluster after clustering is called a topic domain. The silhouette coefficient is used to evaluate the clustering quality, and the calculation formula is as follows: ； in, Represents the profile coefficient. Indicates the total number of data points. Representing data points The average distance to other points in the same cluster, Representing data points The average distance to the nearest heterogeneous cluster; When the silhouette coefficient is greater than 0.5, the subject area division is determined; Dimensional relationship mapping is constructed through hierarchical analysis to identify the relationships between dimension tables and fact tables within a subject domain; the generation of dimension table structures follows the star schema design principle and includes dimension keys, dimension attributes, and hierarchical structure definitions; the fact table structure includes measure fields, dimension foreign keys, and granularity definitions. Generate a logical model of the data warehouse that includes subject domain identifiers, dimension table structures, and fact table structures.

4. The intelligent construction and management method for a data warehouse according to claim 3, characterized in that: ETL configuration parameters include data extraction rules, transformation rules, and loading strategies; The steps to generate ETL configuration parameters include: The data extraction rules include extraction strategy and extraction frequency. The extraction strategy adopts a hybrid strategy that combines incremental extraction and full extraction. Incremental extraction is used when the data volume is greater than the set threshold, and full extraction is used otherwise. The extraction frequency is calculated based on the timeliness constraint parameter and data volume parameter of the data source parameters. The formula for calculating the decimation frequency is as follows: ； in, Indicates the sampling frequency. Indicates the time constraint period. Indicates the system's processing capacity. For data volume parameters, It is a complexity factor; The conversion rules include three levels: data type conversion, data format standardization, and application of business rules. Data type conversion is achieved through a type mapping matrix, which establishes a precise mapping relationship between the source system's data types and the target data warehouse's data types; Data format standardization employs regular expression matching and replacement techniques to unify date formats, numerical precision, and character encoding. Business rule applications are executed through the rule engine, including data validation, calculation of derived fields, and business logic validation; The loading strategy combines batch loading and streaming loading. Batch loading is used for the initial loading of historical data, while streaming loading is used for the real-time updating of incremental data. The loading strategy also includes loading priority, which determines which data should be loaded first during the loading process. The loading priority is determined based on the business importance and timeliness requirements of the data, and the calculation formula is as follows: ； in, Indicates loading priority. Indicating business importance index, Indicates timeliness and urgency. , These are the first and second loading priority weight coefficients, respectively.

5. The intelligent construction and management method for a data warehouse according to claim 4, characterized in that, After data transformation, a field mapping table is generated, and the data quality is checked to see if it is up to standard. If it is not up to standard, the data extraction needs to be repeated. Data quality inspection employs a multi-level quality assessment system, considering data completeness, accuracy, and consistency indicators to calculate a comprehensive data quality score. The formula for calculating the comprehensive data quality score is as follows: ； in, This represents the overall score for data quality. Indicates the integrity score. Indicates accuracy score, Indicates consistency score, , , These are the weighting coefficients for the comprehensive score of the first, second, and third data quality, respectively.

6. The intelligent construction and management method for a data warehouse according to claim 5, characterized in that, The physical architecture of a data warehouse comprises a data access layer, a data storage layer, and a data service layer, specifically: The data access layer is responsible for receiving data streams from the ETL process and uses a message queue buffering mechanism to handle peak loads. The data access layer uses a columnar storage structure, optimized for analytical queries. The buffer size is dynamically adjusted based on data traffic and processing capacity, calculated using the following formula: ； in, Indicates the buffer size. For sampling frequency, For data volume parameters, For the buffer time window, Minimum buffer size; The data storage layer is responsible for storing the field mapping table obtained from the data access layer after processing it according to the transformation rules. The configuration of the data storage layer includes storage format, indexing strategy, and partitioning rules, specifically: The storage format adopts a hybrid storage architecture, including row-based and column-based storage; the selection of the storage format is based on data access analysis; the formula for calculating the weight of the storage format is as follows: ； in, Indicates the storage format weight. Indicates the number of analytical queries. This indicates the total number of queries. Indicates the query performance of columnar storage. Indicates the query performance of row-oriented storage; Columnar storage is selected when the storage format weight is greater than 1; otherwise, row storage is selected. The indexing strategy employs a multi-level index system, including a primary index, secondary indexes, and a clustered index; specifically: the primary index is automatically constructed based on the primary key field; a clustered index is created when no clustered index exists and the clustering effect is greater than 1; a secondary index is created when the index selectivity is greater than 0.1 and no clustered index exists. The partitioning rules adopt a hybrid partitioning strategy that combines range partitioning and hash partitioning. Specifically, the first-level partition is a range partition, and hash partitioning is performed within each first-level partition to locate the second-level partition. The data service layer provides a unified data access interface and adopts a multi-tenant architecture to support concurrent access. The storage format of the data service layer is optimized for query response time, employing a tiered storage mechanism combining in-memory computing and disk storage. Hot data is stored in memory, while cold data is stored on disk. Data temperature is assessed using an access frequency and time decay model, calculated as follows: ； in, Indicates the temperature of the data. Indicates the first Weight of each visit Indicates the first Second visit time. Indicates the current time. This represents the time decay coefficient; data with temperatures above the threshold are stored in memory.

7. The intelligent construction and management method for a data warehouse according to claim 6, characterized in that: The cache hierarchy mapping adopts a three-level cache architecture: L1 is a memory result cache, which stores the query results accessed frequently; L2 is an SSD intermediate cache, which stores data blocks accessed at a medium frequency; and L3 is a disk pre-computation cache, which stores the materialized view of complex calculations. The allocation of cache levels is based on access frequency and data value calculation: ； in, Indicates the cache level. Indicates access frequency. Indicates the value score of the data. Indicates the data size. and These represent the L1 and L2 cache access frequency thresholds, respectively. Indicates the data value threshold, Indicates a data size threshold; Data value scoring Calculated using the following formula: ； in, Indicates the query complexity. Indicates the calculation time. This indicates the baseline cost calculation.

8. The intelligent construction and management method for a data warehouse according to claim 7, characterized in that, The pre-caching rules are as follows: The system first collects query execution logs and extracts query feature vectors. It then uses a cosine similarity algorithm to calculate the similarity between any two query feature vectors. When the similarity score between two queries exceeds a threshold of 0.8, the system classifies them as having the same access pattern. Simultaneously, the system monitors load data, uses a max-min normalization method to transform all load data into a unified range of 0 to 1, and analyzes load characteristics through multi-dimensional metrics. A deep learning algorithm outputs the predicted access probability and predicted access time point for each data item. When the prediction accuracy is greater than 75%, an aggressive pre-caching strategy is activated; when the prediction accuracy is less than 50%, a conservative pre-caching strategy is switched to. Otherwise, maintain the current caching strategy; The inputs to deep learning algorithms include: a set of historical query feature vectors, query similarity clustering results, time-series data of historical load intensity, and timestamp records of data access; The historical query feature vector set includes a set of historical query feature vectors; The query feature vector includes the query type, the number of tables involved, the data scan range, the execution frequency, and the response time requirements. Load data includes: the number of concurrent users in the current system, the query arrival frequency per unit time, the frequency of data update operations, and the system hardware resource utilization level; The standardization process for load data requires determining the maximum and minimum boundaries of each indicator based on historical monitoring data; The extracted load characteristics include load intensity, which is analyzed through multi-dimensional indicator monitoring, including concurrent users, query arrival rate, data update frequency, and system resource utilization. The formula for calculating load strength is as follows: ； in, Indicates load strength. This represents the normalized value of the number of concurrent users. This represents the normalized query reach value. This represents the normalized value of the data update frequency. Indicates system resource utilization. , , , These are the first, second, third, and fourth load intensity weighting coefficients, respectively.

9. The intelligent construction and management method for a data warehouse according to claim 8, characterized in that, The failure strategy is as follows: Based on the data consistency requirements of different business scenarios, the system divides cached data into three levels: strong consistency, weak consistency, and eventual consistency. Strong consistency requires that the synchronization delay between the cache and the source data does not exceed 60 seconds, weak consistency allows a delay of 300 seconds, and eventual consistency can tolerate a delay of 1800 seconds. By analyzing the time intervals between consecutive update operations, the average update frequency of each data item is calculated using the following formula: ； in, Indicates the number of updates within the statistical period. Indicates the first The time interval between the next update and the previous update; Calculate the optimal lifetime for each cached item while ensuring it does not exceed the maximum lifetime allowed by the system capacity limit. The calculation formula is as follows: ； in, Indicates cache capacity. Indicates the average cache item size. Indicates the cache generation rate; Based on the calculated lifetime, the system sets an automatic expiration timer for each cached item. When a cached item reaches its preset lifetime or an update to the source data is detected, the system immediately performs a cache expiration operation. The calculation formula is as follows: ； in, Indicates the cache lifetime. Indicates the consistency tolerance time. Indicates the data update frequency. Indicates the maximum survival time.

10. A data warehouse intelligent construction and management system, characterized in that, A data warehouse intelligent construction and management method for executing any one of claims 1-9 includes: Data source acquisition and feature parameter extraction module: Acquires the enterprise's data sources, and extracts the data source feature parameters by parsing the data format, interface protocol and update frequency of each data source; Data warehouse logical model generation module: Based on the data source, it matches the corresponding business terms, calculates semantic similarity and clusters subject domains, and constructs dimensional relationship mapping through subject domains to generate the data warehouse logical model; ETL Configuration Parameter Generation Module: Defines data extraction rules, transformation algorithms, and loading strategies, and generates ETL configuration parameters; Data warehouse physical architecture construction module: Constructs the data warehouse physical architecture based on ETL configuration parameters. The data warehouse physical architecture includes a data access layer, a data storage layer, and a data service layer, and defines the storage format, indexing strategy, and partitioning rules for the data storage layer; Dynamic cache configuration generation module: Based on the data warehouse physical architecture, executes intelligent cache strategy configuration, defines cache level mapping, pre-caching rules and invalidation policies, and generates dynamic cache configuration; Data warehouse management module: Implements data warehouse operation management based on dynamic cache configuration and performs management optimization through multi-objective optimization functions to generate management optimization results.

Images