Data management method based on real-time data warehouse

Abstract

The invention discloses a data management method based on a real-time data warehouse, and the method comprises the steps: obtaining column type data blocks through vectorization processing according to collected data, dividing data types according to the access frequency of the column type data blocks, and correspondingly setting storage positions; in response to the data request, determining a data access mode according to the cache hit rate of the data request; and according to the real-time monitoring index, a load trend is predicted through a time sequence prediction statistical model, and vectorized nodes and cache capacity are adjusted. Through vectorization processing, cache hit rate analysis and dynamic resource adjustment driven by load prediction, the core problems of low storage efficiency, poor resource utilization rate, high response delay and the like of a traditional data warehouse architecture are solved, and multi-dimensional breakthrough in performance, cost and stability is achieved.

Description

Technical Field

[0001] The present invention belongs to the technical field of data management, and particularly relates to a data management method based on a real-time data warehouse. Background Art

[0002] With the acceleration of digital transformation, the demand for real-time data processing in enterprises has shown an explosive growth. As the core infrastructure supporting scenarios such as financial risk control, industrial Internet of Things, and real-time marketing, a real-time data warehouse requires the system to complete high-concurrency queries and complex analyses of massive data within millisecond-level latency. The traditional data warehouse architecture mainly has the following problems:

[0003] The batch processing architecture (such as Hive) based on the Hadoop ecosystem adopts the T+1 update mode, resulting in business decision-making delays, insufficient data timeliness, and inability to meet the requirements of real-time scenarios. Moreover, the row-based storage structure generates redundant IO operations in complex analyses, and the standard deviation of the access response time of hot data is much larger than the benchmark value, resulting in bottlenecks in storage and computing efficiency.

[0004] Therefore, the data warehouse adopts the traditional B+ tree index technology, an index structure based on disk storage, to achieve single-dimensional query optimization. However, the traditional B+ tree requires multi-layer index scans, and the resulting random IO leads to a decline in query efficiency. Moreover, to support multi-dimensional queries, multiple copies of indexes need to be constructed, increasing the storage overhead.

[0005] In addition, in terms of resource allocation, the adoption of a static resource allocation mechanism results in low average cluster utilization, and scheduling delays are caused by resource contention under burst traffic. Or, a static scaling policy based on thresholds is adopted, and resource scaling is triggered by preset thresholds such as CPU utilization and memory occupancy. It is unable to perceive sudden changes in load in real time, resulting in a high resource mismatch rate, and the resource allocation does not match the actual demand, resulting in a high average idle rate of the cluster.

[0006] Generally speaking, the root causes of these bottleneck problems can be attributed to the lack of a dynamic resource coordination mechanism in the architecture design and the contradiction between the storage model and the query mode mismatch. Summary of the Invention

[0007] The present invention provides a data management method based on a real-time data warehouse to solve the problems of large storage overhead and unbalanced resource allocation caused by the lack of a dynamic resource coordination mechanism in the data warehouse architecture design and the mismatch between the storage model and the query mode.

[0008] The technical solution adopted by the present invention is as follows:

[0009] A data management method based on a real-time data warehouse, comprising:

[0010] According to the collected data, through vectorization processing, columnar data blocks are obtained, and data types are divided according to the access frequencies of the columnar data blocks, and storage locations are correspondingly set;

[0011] In response to a data request, according to the cache hit rate of the data request, a data access mode is determined;

[0012] According to real-time monitoring metrics, the load trend is predicted through a time series prediction statistical model, and the nodes and cache capacity of the vectorization processing are adjusted.

[0013] The data management method based on a real-time data warehouse in the present invention further includes the following additional technical features:

[0014] Dividing data types according to the access frequencies of the columnar data blocks specifically includes:

[0015] Dividing data types according to the historical access frequencies of columnar data blocks, and the data types at least include three types: hot data, warm data, and cold data;

[0016] Among them, the historical access frequencies of the hot data, the warm data, and the cold data decrease in sequence.

[0017] Dividing data types according to the access frequencies of the columnar data blocks and correspondingly setting storage locations specifically includes:

[0018] The hot data is stored in memory, the warm data is stored in local storage, and the cold data is stored in object storage.

[0019] The setting of data caching specifically includes:

[0020] According to a data request, multiple-level caches are set;

[0021] The data access to the hot data within a fixed time period is cached in the first cache layer;

[0022] The data access to the non-decreasing access frequency data in the warm data within a fixed time period is cached in the second cache layer;

[0023] The data access to the data with a decreasing access frequency in the warm data within a fixed time period is cached in the third cache layer.

[0024] Determining the data access mode according to the cache hit rate of the data request specifically includes:

[0025] In response to a data request, data is returned according to the query result of the first cache layer;

[0026] When the first cache layer misses, data is returned according to the query result of the second cache layer;

[0027] When the first and second cache layers miss, return data according to the query result of the third cache layer;

[0028] When the cache misses, perform a data query according to the storage location to return data.

[0029] According to the real-time monitoring metrics, predict the load trend through a time series prediction statistical model, specifically:

[0030] The monitoring metrics at least include any one of CPU utilization, network bandwidth, and queue length;

[0031] Predict the load trend within a future fixed time period according to the monitoring metrics, so as to adjust the nodes and cache capacity of the vectorization processing according to the load trend, where the future fixed time period is 5 minutes.

[0032] Adjust the nodes and cache capacity of the vectorization processing according to the load trend, specifically:

[0033] According to the load trend, set multiple resource allocation schemes through an integer programming model, where the resource allocation schemes meet the resource requirements of vectorization processing and caching under the load trend;

[0034] Obtain the final resource allocation scheme according to the resource costs of multiple resource allocation schemes;

[0035] Adjust the nodes and cache capacity of the vectorization processing according to the final resource allocation scheme.

[0036] Adjust the nodes and cache capacity of the vectorization processing according to the final resource allocation scheme, specifically:

[0037] When the nodes and cache capacity of the vectorization processing in the resource allocation scheme exceed 75% of the current resource allocation value, perform an operation of adding nodes and / or expanding capacity;

[0038] When the nodes and cache capacity of the vectorization processing in the resource allocation scheme are less than 30% of the current resource allocation value, perform an operation of reducing nodes and / or shrinking capacity.

[0039] The present invention also provides a storage medium,

[0040] A computer program is stored on the storage medium, and when the computer program is executed, the steps of the data management method based on the real-time data warehouse are implemented.

[0041] The present invention further provides a processing device, including:

[0042] A memory for storing a computer program;

[0043] A processor, which is used to implement the steps of the data management method based on a real-time data warehouse when executing the computer program.

[0044] Due to the adoption of the above technical solutions, the beneficial effects obtained by the present invention are as follows:

[0045] 1. In the present invention, according to the collected data, columnar data blocks are obtained through vectorization processing, and data types are divided according to the access frequencies of the columnar data blocks, and storage locations are correspondingly set. The vectorization processing of columnar data blocks reduces redundant IO operations of data (such as redundant reads of full-field scans in traditional row-based storage), and reduces the CPU instruction cycles of complex analysis queries.

[0046] In response to a data request, according to the cache hit rate of the data request, a data access mode is determined. By analyzing the cache hit rate in real time, the data access mode (such as concentrated access to hot data or burst access to cold data) can be dynamically identified, so as to adjust the access mode and cache policy. For example, when the cache hits, the requested data is directly called from the cache, reducing the data request response time.

[0047] According to real-time monitoring metrics, the load trend is predicted through a time series prediction statistical model, and the nodes and cache capacity of the vectorization processing are adjusted. Based on the load prediction of the ARIMA model, combined with real-time monitoring metrics (CPU utilization, network bandwidth, queue length), the nodes and cache capacity of the vectorization processing are dynamically adjusted to reduce the resource idle rate. In the case of burst traffic, more resources are allocated according to the prediction results to improve the resource allocation response speed; in the case of reduced access traffic, resource allocation is reduced to improve the average utilization rate of the cluster. Through prediction-driven resource allocation, the hardware cost is reduced, and the waste caused by resource misallocation is avoided.

[0048] Generally speaking, through vectorization processing, cache hit rate analysis and dynamically adjusting resources driven by load prediction, the present invention solves the core problems of low storage efficiency, poor resource utilization rate and high response latency in the traditional data warehouse architecture, and realizes multi-dimensional breakthroughs in performance, cost and stability. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] The drawings described herein are used to provide a further understanding of the present invention, and constitute a part of the present invention. The schematic embodiments of the present invention and their descriptions are used to explain the present invention, and do not constitute an improper limitation to the present invention. In the drawings:

[0050] Figure 1 It is a schematic flowchart of the data management method based on a real-time data warehouse under an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0051] To more clearly illustrate the overall concept of the present invention, the following provides a detailed description by way of example in conjunction with the accompanying drawings of the specification.

[0052] In the following description, many specific details are set forth in order to provide a thorough understanding of the present invention. However, the present invention may be practiced in other ways different from those described herein. Therefore, the scope of protection of the present invention is not limited by the specific embodiments disclosed below.

[0053] As Figure 1 shown, a data management method based on a real-time data warehouse includes:

[0054] S100: According to the collected data, through vectorization processing, obtain columnar data blocks, divide the data types according to the access frequencies of the columnar data blocks, and correspondingly set the storage locations.

[0055] The core objective of this step is to convert the original data into columnar data blocks for efficient storage and access through vectorization processing and access frequency analysis, and dynamically allocate storage locations according to the data access heat, thereby reducing redundant IO operations, lowering storage costs, optimizing storage efficiency, and reducing access latency.

[0056] It can be understood that in a columnar data block, the data is a structured unit stored by column rather than by row, supporting efficient aggregation calculations (such as summation, grouping), and reducing the scanning range of complex queries.

[0057] Vectorization processing is a technology that stores data by column and performs batch calculations. In this step, vectorization processing generates columnar data blocks, converting the original data (such as a table stored in row format) into columnar data blocks through a vectorization engine, that is, storing data by column instead of the traditional row storage.

[0058] Columnar storage reduces the full-field scanning of complex queries and reduces redundant IO operations. It can be understood that for columnar storage data blocks, batch calculations can be accelerated through the SIMD instruction set, reducing the CPU instruction cycle.

[0059] Divide the data types according to the access frequencies of the columnar data blocks. Based on the historical access records of the columnar data blocks, divide the data types into multiple categories. By dynamically distinguishing the data heat, avoid resource waste caused by a "one-size-fits-all" storage strategy.

[0060] Divide the data types according to the access frequencies of the columnar data blocks, and correspondingly set the storage locations. For example, for columnar data blocks with higher access frequencies, they can be stored in memory to shorten the access path of hot data, thereby reducing the access latency of hot data. In addition, for columnar data blocks with lower access frequencies, they can be stored in external storage to avoid using high-cost memory for all data, which can reduce storage costs.

[0061] Generally speaking, this step reduces redundant I / O through columnar data blocks, lowers storage costs, and improves storage efficiency. The storage location is set according to the access frequency of the columnar data blocks, which facilitates shortening the access path for hot data and optimizing access latency. In addition, the hierarchical storage strategy based on access frequency improves the average utilization rate of the cluster.

[0062] S200: In response to a data request, determine the data access mode according to the cache hit rate of the data request.

[0063] The core objective of this step is to dynamically identify the data access mode (such as hot access, cold data access, or mixed mode) through cache hit rate analysis, so as to optimize the data access path and resource allocation strategy.

[0064] Specifically, this step quickly responds to high-frequency requests through the cache hierarchy, reduces direct access to the underlying storage, and reduces access latency.

[0065] Among them, a multi-level cache system is set in the present invention, which refers to a hierarchical storage structure composed of three layers of caches, namely L1 - L3. In order from high to low in terms of access speed and capacity, they are in-process cache (memory), Redis cluster (memory + disk), and Alluxio (memory acceleration).

[0066] The cache hit rate refers to the proportion of successful hits in cache requests. Generally speaking, the L1 cache has the highest hit rate, indicating that hot data can be quickly responded to without accessing the underlying storage.

[0067] It should be noted that according to the key values of the data request (such as primary key, timestamp), the L1, L2, and L3 cache layers are queried in sequence.

[0068] L1 cache query: The high hit rate of the in-process cache (such as Caffeine) can quickly respond to hot data requests; L2 cache query: The Redis cluster processes sub-hot data and supports cross-node sharing; L3 cache query: Alluxio caches warm data with a decreasing cache frequency through memory acceleration; cache miss, perform underlying data query according to the storage location (memory / local storage / object storage).

[0069] This step reduces the metadata access latency and shortens the cold data access path.

[0070] According to the cache hit results, the data access mode is divided into the following four categories: Hot access mode (L1 hit) is high-frequency access data, and the cache priority needs to be maintained; Warm data access mode (L2 hit) is medium-frequency access data, and the cache and storage costs need to be balanced; Pre-warmed data access mode (L3 hit) is medium-frequency access data with a decreasing frequency, and the cache lifecycle needs to be controlled; Cold data access mode (miss): Low-frequency and non-pre-warmed data, and the storage location needs to be directly accessed.

[0071] In this step, cache eviction policies can be optimized to reduce cache penetration (such as BloomFilter filtering invalid requests). This step not only shortens the access path for hot data but also improves the efficiency of cold data preloading.

[0072] S300: According to real-time monitoring metrics, predict the load trend through a time series prediction and statistical model, and adjust the nodes for vectorized processing and cache capacity.

[0073] The core objective of this step is to dynamically predict the future load trend through real-time monitoring metric analysis and time series prediction models, and optimize the resource allocation plan based on the prediction results through an integer programming model, so as to reduce the resource idle rate and improve resource utilization through prediction-driven resource adjustment; avoid waste caused by resource mismatch, reduce hardware costs, and optimize costs; quickly respond to sudden load changes, improve processing capabilities, and enhance system elasticity.

[0074] Among them, the time series prediction model predicts the future load trend by analyzing the autoregressive (AR), differencing (I), and moving average (MA) characteristics of historical data.

[0075] Real-time monitoring metric collection: Collect real-time metrics of system operation through a monitoring module, including CPU utilization, network bandwidth, queue length, etc.

[0076] This step provides high-precision input data for load prediction and reduces the prediction error rate; the real-time metric collection frequency is at the second level to ensure that the prediction results are highly synchronized with the actual load.

[0077] Load trend prediction using the time series prediction model (ARIMA): Input the collected real-time metrics into the ARIMA model to predict the load trend for a future period. Improve the load prediction accuracy and support early resource allocation for sudden traffic scenarios; generally, the prediction time granularity is 5 minutes, which matches the resource adjustment cycle to avoid lagging prediction results.

[0078] In addition, according to the load trend prediction, dynamically adjust the nodes for vectorized processing and cache capacity. Dynamic adjustment can avoid over-response, reduce the resource mismatch rate, reduce hardware costs, and improve the average cluster utilization rate.

[0079] It can be understood that the resource adjustment plan in this step takes into account both cost and performance, reduces the resource idle rate; in the case of sudden traffic scenarios, improves the resource allocation response speed and the cluster processing ability.

[0080] As a preferred implementation manner of the present invention, divide data types according to the access frequency of the columnar data blocks, specifically:

[0081] Divide data types according to the historical access frequency of columnar data blocks, and the data types include at least three types: hot data, warm data, and cold data;

[0082] Among them, the historical access frequencies of the hot data, the warm data, and the cold data decrease in turn.

[0083] The core objective of this embodiment is to divide columnar data blocks into three categories: hot data, warm data, and cold data through historical access frequency analysis, and allocate different storage locations (memory, local storage, object storage) according to the high and low access frequencies.

[0084] In this way, redundant IO operations are reduced, storage costs are lowered, and storage efficiency is optimized; the access path of hot data is shortened, the preloading efficiency of cold data is improved, and access latency is reduced; the storage medium matches the data access mode, resource waste is avoided, and resource utilization is improved.

[0085] Collect the historical access records of columnar data blocks from the real-time data warehouse (such as the access logs in the past 24 hours). Provide accurate historical access data for subsequent classification and reduce the classification error rate.

[0086] Define the thresholds for hot, warm, and cold data according to the historical access frequency. The access frequency ≥ 100 times / minute is hot data; the access frequency is between 10 - 100 times / minute is warm data; the access frequency < 10 times / minute is cold data.

[0087] Step dynamically distinguishes the data heat, avoids resource waste caused by the "one-size-fits-all" storage strategy, and improves the classification accuracy of this method.

[0088] It should be noted that the present invention does not limit the thresholds for hot, warm, and cold data. The above 100 times / minute and 10 times / minute are only specific embodiments. The thresholds of other values can also be dynamically set according to the historical access frequency, as long as the historical access frequencies of the hot data, the warm data, and the cold data decrease in turn.

[0089] As a preferred embodiment of this embodiment, divide data types according to the access frequency of the columnar data block, and set the storage location correspondingly, specifically:

[0090] The hot data is stored in memory, the warm data is stored in local storage, and the cold data is stored in object storage.

[0091] The core objective of this embodiment is to achieve precise mapping between data types and storage media. Store the hot data with high-frequency access in memory to reduce access latency; store the cold data with low-frequency access in object storage to reduce storage costs; match the data access mode through a hierarchical storage strategy to avoid resource waste and improve resource utilization.

[0092] It is understandable that local storage (SSD) is a solid-state drive-based storage medium with an access speed between that of memory and object storage, suitable for medium-frequency data storage.

[0093] Object storage (S3 / OSS) is a cloud-based object storage service that provides high-capacity and low-cost storage, but has a relatively high access latency and is suitable for cold data storage.

[0094] In this embodiment, storage media are allocated according to data types. Hot data resides in memory (DRAM), supporting nanosecond-level access to reduce the access latency of hot data; warm data is stored in local SSDs, taking into account both capacity and access speed; cold data is stored in object storage (such as S3 / OSS) and loaded on demand to reduce the cold data storage cost.

[0095] It should be noted that the present invention also supports dynamically adjusting the data storage location according to changes in access frequency. Hot data → warm data: If the access frequency is lower than 100 times per minute for 24 consecutive hours, it is migrated to local SSDs. Warm data → cold data: If the access frequency is lower than 10 times per minute for 7 consecutive days, it is migrated to object storage.

[0096] Through dynamic storage adjustment in this embodiment, the utilization rate of cluster resources is improved; the long-term occupation of memory / SSD resources by cold data is avoided, and high-speed storage space is released.

[0097] It is understandable that an access path is selected according to the storage location of the data request. For memory access, data is directly returned through an in-process cache (such as Caffeine); for local SSD access, it is quickly read through a file system (such as EXT4); for object storage access, it is loaded on demand through a distributed storage service (such as MinIO).

[0098] Through the execution of this embodiment, storage layer optimization is achieved. Hot data resides in memory, shortening the access path; cold data uses object storage to reduce the storage cost. The access latency is controlled in layers, reducing the access latency of hot data. The resource utilization rate is improved, with memory only used for hot data, releasing high-speed storage resources; the mixed use of SSDs and object storage reduces the storage cost.

[0099] This embodiment solves the problem of the mismatch between the storage medium and the data access mode in the traditional architecture through the precise mapping of data types and storage media, achieving a balance among storage efficiency, access latency, and cost.

[0100] As another embodiment under this implementation manner, the setting of data caching is specifically as follows:

[0101] Set multiple levels of caches according to data requests;

[0102] Cache the data access to the hot data within a fixed time period in the first cache layer;

[0103] Cache the data access to the non-decreasing access frequency data in the warm data within a fixed time period in the second cache layer;

[0104] Cache the data access to the data with decreasing access frequency in the warm data within a fixed time period in the third cache layer.

[0105] The core objective of this embodiment is to implement a multi-level cache hierarchical strategy. By hierarchical caching, it reduces the direct access to the underlying storage, shortens the response time of hot data, and reduces access latency; it dynamically allocates cache levels according to the data access pattern, avoids invalid caches occupying resources, and optimizes the utilization rate of cache resources.

[0106] In this embodiment, a multi-level cache hierarchical design is provided. According to the data type and access pattern, the cache is divided into three layers:

[0107] The first cache layer (L1), in-process cache (such as Caffeine), is used for high-frequency access to hot data; the second cache layer (L2), Redis cluster, is used for access to warm data with stable access frequency; the third cache layer (L3), Alluxio memory acceleration layer, is used for access to warm data with decreasing access frequency but requiring preheating.

[0108] It can be understood that the L1 cache has the highest hit rate, can quickly respond to hot data requests, reduce metadata access latency, and reduce the direct access overhead to memory; the hierarchical design of L2 and L3 shortens the warm data access path and improves the cold data preloading efficiency.

[0109] For warm data, a dynamic allocation strategy is implemented. According to the change of the access frequency of warm data, it is dynamically allocated to the L2 or L3 layer. Warm data with stable access frequency reduces SSD access through the L2 cache (latency reduced by 90%); warm data with decreasing access frequency is preheated through the L3 cache to avoid direct access to object storage (latency reduced by 75%).

[0110] Through the execution of this embodiment, the cache efficiency is improved, the multi-level cache hierarchy improves the overall hit rate and reduces metadata latency; in addition, the L3 cache preheating mechanism shortens the access path of cold warm data; the dynamic mechanism of warm data in the L2 and L3 layers reduces cache pollution and releases invalid cache space.

[0111] This embodiment solves the problems of low cache hit rate and resource waste in the traditional architecture through multi-level cache hierarchy and dynamic frequency analysis.

[0112] Specifically, according to the cache hit rate of the data request, determine the data access pattern as follows:

[0113] In response to a data request, data is returned according to the query result of the first cache layer;

[0114] When the first cache layer misses, data is returned according to the query result of the second cache layer;

[0115] When both the first and second cache layers miss, data is returned according to the query result of the third cache layer;

[0116] When the cache misses, a data query is performed according to the storage location to return data.

[0117] The core objective of this embodiment is to dynamically identify data access patterns (such as hot access, warm data access, cold data access) according to the cache hit rate through a three-level cache hierarchical query mechanism, so as to optimize the data access path and resource allocation strategy.

[0118] This embodiment reduces the direct access to the underlying storage through multiple-level caches, shortens the response time of hot data, and reduces the access latency. The cache levels are dynamically allocated according to the data access pattern, avoiding the waste of resources by invalid caches, and optimizing the utilization rate of cache resources.

[0119] The cache level query process is to sequentially query the L1, L2, and L3 cache layers according to the key value of the data request, and return data according to the hit result.

[0120] In this way, the access latency is optimized. The high hit rate of the L1 cache can quickly respond to hot data requests and reduce the metadata access latency; the microsecond-level access speed of the L2 cache (Redis) reduces the warm data latency; the preheating mechanism of the L3 cache (Alluxio) shortens the access path of cold warm data. Generally speaking, the three-level cache hierarchy improves the overall cache hit rate and enhances the resource utilization efficiency.

[0121] According to the cache query result, the data access patterns are divided into four categories, and the corresponding processing processes are triggered. L1 hit (hot access pattern), shortening the hot data access path and reducing the latency.

[0122] L2 hit (warm data access pattern), compared with SSD access, reduces the warm data access latency.

[0123] L3 hit (preheated warm data access pattern), improving the preloading efficiency of cold warm data. For warm data with reduced frequency, adjust it to enter the L3 cache layer to prevent cold warm data from occupying high-priority cache resources for a long time.

[0124] Cache miss (cold data access pattern), compared with directly retrieving from the object storage, reduces the cold data access latency.

[0125] This embodiment solves the problems of low cache hit rate and resource waste in the traditional architecture through the three-level cache hierarchical access mechanism.

[0126] As a preferred embodiment of the present invention, according to real-time monitoring metrics, the load trend is predicted through a time series prediction statistical model, specifically as follows:

[0127] The monitoring metrics at least include any one of CPU utilization, network bandwidth, and queue length;

[0128] Predict the load trend within a fixed future time period according to the monitoring metrics, so as to adjust the nodes and cache capacity of the vectorization processing according to the load trend, where the fixed future time period is 5 minutes.

[0129] The core objective of this embodiment is to dynamically predict the load trend within the next 5 minutes through real-time monitoring metric analysis and time series prediction models, so as to optimize the number of nodes and cache capacity allocation of vectorization processing based on the prediction results.

[0130] Collect real-time metrics of system operation through a monitoring module, including but not limited to CPU utilization, network bandwidth, queue length, etc. It should be noted that the real-time metric collection frequency is at the second level to ensure that the prediction results are highly synchronized with the actual load; smooth the noise through a sliding window (such as 1 minute) to reduce the prediction error rate.

[0131] Predict the load trend using a time series prediction model (ARIMA). Input the collected monitoring metrics into the ARIMA model to predict the load trend in the next 5 minutes.

[0132] The prediction accuracy of the ARIMA model can reach 92%, supporting the early allocation of resources for burst traffic scenarios; the prediction time granularity is 5 minutes, which matches the resource adjustment cycle to avoid lag. It can be understood that choosing 5 minutes as the prediction window balances the prediction accuracy and the real-time nature of resource adjustment. The 5-minute prediction can cover the peak of burst traffic and avoid prediction instability caused by too short a window.

[0133] As a preferred example of this embodiment, adjust the nodes and cache capacity of the vectorization processing according to the load trend, specifically as follows:

[0134] According to the load trend, set multiple resource allocation plans through an integer programming model, where the resource allocation plans meet the resource requirements for vectorization processing and caching under the load trend;

[0135] Obtain the final resource allocation plan according to the resource costs of multiple resource allocation plans;

[0136] Adjust the nodes and cache capacity of the vectorization processing according to the final resource allocation plan.

[0137] The core objective of this embodiment is to generate a multi-resource allocation plan through an integer programming model and select the optimal plan based on cost optimization. Ensure that the vectorized processing nodes and cache capacity meet the load requirements, avoid resource waste, and maximize resource utilization; select the resource allocation plan with the lowest cost under the premise of meeting the load requirements to minimize costs.

[0138] According to the predicted load trend, generate multiple resource allocation plans through an integer programming model. The constraint conditions of the resource allocation plan are as follows

[0139]

[0140] Among them, t∈[1,300] represents the time range, and the time interval is from t = 1 to t = 300 (5 minutes). Each time point within this time period satisfies the constraint conditions.

[0141] It means that at time t, all decision variables x

[0146] , , ,

[0143] , m , , , i , , , ij , , , ,

[0144] ,

[0145] , v ,

[0148] , , i , , j , ,

[0142] ,

[0147] ,

[0140] ,

[0141] The sum of the products with the coefficient a ij Needs to Ensure that at each time point t, the resource requirements or limitations of the i-th constraint condition are met.

[0142] According to the resource costs of multiple said resource allocation plans, obtain the final resource allocation plan, and the objective function is

[0143]

[0144] Among them, v i Represents the number of virtual nodes of the i-th node; m i Represents the cache capacity (unit: GB) of the i-th node; c v , c m Represents the unit costs of virtual nodes and caches.

[0145] According to the resource costs of each plan, select the final plan with the lowest cost. The cost evaluation mechanism ensures that the final plan reduces the hardware cost under the premise of meeting the load requirements; generating multiple plans avoids the risk of single-point optimization. For example, an extreme cost plan may lead to performance bottlenecks.

[0146] Dynamically adjust the vectorized processing nodes and cache capacity according to the final plan.

[0147] Specifically, adjust the nodes and cache capacity of the vectorized processing according to the final said resource allocation plan. Specifically:

[0148] When the nodes and cache capacity of the vectorized processing in the resource allocation plan exceed 75% of the current resource allocation value, perform the operation of adding nodes and / or expanding capacity;

[0149] When the nodes and cache capacity for vectorized processing in the resource allocation scheme are less than 30% of the current resource allocation value, a node reduction and / or capacity reduction operation is performed.

[0150] The core objective of this embodiment is to trigger the expansion or reduction operations of nodes and cache capacity based on the comparison between the final resource allocation scheme and the current resources through a threshold mechanism for dynamic resource adjustment. In this way, the resource utilization rate is optimized, and performance bottlenecks caused by over-allocation or insufficient resources are avoided; resource waste is reduced through the threshold trigger mechanism, and hardware costs are lowered; system jitter caused by frequent expansion and contraction is avoided through progressive adjustment, ensuring system stability.

[0151] Obtain the comparison data between the current resource allocation value (number of nodes, cache capacity) and the final resource allocation scheme. In this step, the ratio calculation is accurate to two decimal places to avoid misjudgment due to floating-point errors.

[0152] Based on the resource change ratio, determine whether to trigger the expansion or reduction operation. The 75% threshold avoids performance degradation caused by a sharp increase in resource requirements (such as in the case of burst traffic scenarios); the 30% threshold prevents waste caused by long-term resource idleness (such as during off-peak periods); progressive adjustment avoids frequent operations caused by overly tight thresholds (such as the difference between 75% and 100%).

[0153] Dynamically adjust the number of nodes and cache capacity according to the trigger conditions. It should be noted that through elastic space reservation, 10% buffer is reserved during expansion to avoid insufficient resources due to prediction errors; a conservative strategy for capacity reduction is executed, reducing the capacity to 110% of the target value to prevent service interruption due to insufficient resources; fine-grained adjustment (such as cache priority) is performed, and the lower-level caches (such as L3 cache) are preferentially adjusted to reduce the resource mismatch rate.

[0154] Through the execution of this embodiment, the maximum resource utilization rate is achieved, the resource idle rate is reduced, and the processing ability in burst traffic scenarios is improved. Moreover, the cost is significantly reduced, saving the hardware cost. This embodiment solves the problems of lagging resource adjustment and high cost in the traditional architecture through the threshold trigger mechanism.

[0155] The present invention also provides a storage medium,

[0156] wherein a computer program is stored on the storage medium, and when the computer program is executed, the steps of the data management method based on a real-time data warehouse are implemented.

[0157] Therefore, any effects of the data management method based on a real-time data warehouse can be achieved, which will not be elaborated here.

[0158] The present invention further provides a processing device, including:

[0159] A memory for storing a computer program;

[0160] A processor for implementing the steps of the data management method based on a real-time data warehouse when executing the computer program.

[0161] Therefore, any effects of the data management method based on a real-time data warehouse can be achieved, which will not be elaborated here.

[0162] In the present invention, what has not been described can be implemented by adopting or referring to the existing technologies.

[0163] Each embodiment in this specification is described in a progressive manner. For the same or similar parts among the embodiments, reference can be made to each other. Each embodiment focuses on the differences from other embodiments.

[0164] The above are only the embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, various modifications and changes can be made to the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included within the scope of the claims of the present invention.

Claims

1. A data management method based on a real-time data warehouse, characterized in that, Including: Based on the collected data, through vectorization processing, columnar data blocks are obtained, and data types are divided according to the access frequencies of the columnar data blocks, and storage locations are correspondingly set. In response to a data request, according to the cache hit rate of the data request, a data access mode is determined. According to real-time monitoring metrics, the load trend is predicted through a time series prediction statistical model, and the nodes and cache capacity of the vectorization processing are adjusted.

2. The data management method based on a real-time data warehouse according to claim 1, wherein Dividing data types according to the access frequencies of the columnar data blocks specifically includes: Dividing data types according to the historical access frequencies of columnar data blocks, and the data types at least include three types: hot data, warm data, and cold data. Among them, the historical access frequencies of the hot data, the warm data, and the cold data decrease in sequence.

3. The data management method based on a real-time data warehouse according to claim 2, wherein Dividing data types according to the access frequencies of the columnar data blocks and correspondingly setting storage locations specifically includes: The hot data is stored in memory, the warm data is stored in local storage, and the cold data is stored in object storage.

4. The data management method based on a real-time data warehouse according to claim 2, wherein The setting of data caching specifically includes: According to data requests, multiple-level caches are set. Within a fixed time period, the data access to the hot data is cached in the first cache layer. Within a fixed time period, the data access to the non-decreasing access frequency data in the warm data is cached in the second cache layer. Within a fixed time period, the data access to the data with decreasing access frequency in the warm data is cached in the third cache layer.

5. The data management method based on a real-time data warehouse according to claim 4, wherein, Determining the data access mode according to the cache hit rate of the data request specifically includes: In response to a data request, data is returned according to the query result of the first cache layer. When the first cache layer misses, data is returned according to the query result of the second cache layer. When the first and second cache layers miss, data is returned according to the query result of the third cache layer. When the cache misses, data queries are executed according to the storage locations to return data.

6. The data management method based on a real-time data warehouse according to claim 1, characterized in that Predicting the load trend through a time series prediction statistical model according to real-time monitoring metrics specifically includes: The monitoring metrics at least include any one of CPU utilization rate, network bandwidth, and queue length. Predict the load trend within a future fixed time period according to the monitoring metrics, so as to adjust the nodes and cache capacity of the vectorization processing according to the load trend, where the future fixed time period is 5 minutes.

7. The data management method based on a real-time data warehouse according to claim 6, wherein Adjusting the nodes and cache capacity of the vectorization processing according to the load trend specifically includes: According to the load trend, through an integer programming model, multiple resource allocation schemes are set, where the resource allocation schemes meet the resource requirements of vectorization processing and caching under the load trend. According to the resource costs of multiple resource allocation schemes, a final resource allocation scheme is obtained. Adjust the nodes and cache capacity of the vectorization processing according to the final resource allocation scheme.

8. The data management method based on a real-time data warehouse according to claim 7, characterized in that Adjusting the nodes and cache capacity of the vectorization processing according to the final resource allocation scheme specifically includes: When the nodes and cache capacity of the vectorization processing in the resource allocation scheme exceed 75% of the current resource allocation value, then perform operations such as adding nodes and / or expanding capacity. When the nodes and cache capacity for vectorized processing in the resource allocation scheme are less than 30% of the current resource allocation value, a node reduction and / or capacity reduction operation is performed.

9. A storage medium, characterized in that, a computer program is stored on the storage medium, and when the computer program is executed, the steps of the data management method based on a real-time data warehouse according to any one of claims 1 to 8 are implemented.

10. A processing device, characterized in that, Comprising: a memory for storing a computer program; a processor for implementing the steps of the data management method based on a real-time data warehouse according to any one of claims 1 to 8 when executing the computer program.

Images