A method for predicting and monitoring data volume of a data warehouse
By using a multi-objective particle swarm optimization algorithm and an LSTM network model in the data warehouse, the problems of inaccurate data volume prediction and monitoring lag were solved, achieving high-precision data volume prediction and automated anomaly detection.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- HANGZHOU EBOYLAMP ELECTRONICS CO LTD
- Filing Date
- 2023-02-22
- Publication Date
- 2026-06-02
AI Technical Summary
Existing technologies suffer from low accuracy and lagging monitoring mechanisms in predicting data volume in data warehouses, resulting in a large workload for manual intervention and untimely adjustments.
A multi-objective particle swarm optimization algorithm and an LSTM network model are used to establish a prediction network model by layering the data in the data warehouse, obtaining the optimal time period and prediction step size. The data volume is predicted by combining data characteristics and time granularity, and anomaly detection is performed by setting a volatility range during monitoring.
It improved the accuracy of data volume prediction and monitoring in the data warehouse, reduced manual intervention, and enabled automated anomaly detection and information push.
Smart Images
Figure CN116204381B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of data prediction and monitoring, and specifically relates to a method for predicting and monitoring the amount of data in a data warehouse. Background Technology
[0002] In the field of big data, data warehouse technology is constantly evolving, and one of its important functions is to better organize and store data. In practical applications, as businesses grow, the amount of data generated increases, leading to a greater demand for storage space. Analyzing and monitoring the volume of data in a data warehouse is a crucial part of storage management and optimization.
[0003] Currently, monitoring typically involves comparing and statistically analyzing the volume of newly added data in the data warehouse with historical data. When the difference between the volume of newly added periodic data and historical periodic data exceeds a preset range, it is considered abnormal data. Statistical indicators such as the mean, quantiles, and variance of historical data are analyzed and monitored to identify anomalies. The approximate range of future data volume is estimated through manual, experience-based assessment.
[0004] The above methods are based on analysis and monitoring of the existing storage situation and cannot perceive the future data storage situation. In terms of data warehouse data volume prediction, developers generally make range estimates based on experience, without specific numerical estimates, resulting in low accuracy. In terms of setting up monitoring mechanisms, manual configuration is generally used, which not only increases the workload of personnel, but also cannot be dynamically adjusted according to the actual situation, and manual adjustments are lagging. Summary of the Invention
[0005] The purpose of this invention is to address the problems raised in the background art by proposing a method for predicting and monitoring the amount of data in a data warehouse.
[0006] To achieve the above objectives, the technical solution adopted by the present invention is as follows:
[0007] The present invention proposes a method for predicting and monitoring the amount of data in a data warehouse, which includes obtaining a time series set of each data layer of the data warehouse based on selected data characteristics and time granularity.
[0008] Initialize particle position and velocity, obtain the initial number of input nodes, hidden layer nodes and output nodes of the prediction network model based on the initial particle position, and initialize the prediction network model for each data layer.
[0009] After reconstructing the time series sets of each data layer, the data is input into the prediction network model for iterative training. The training error of the prediction network model is used as the fitness value of the particles, and the individual extreme value and the population extreme value, as well as the position and velocity of the particles, are updated until the number of iterations is reached.
[0010] The particle position corresponding to the optimal particle fitness value of each data layer is obtained, and then the optimal time period and prediction step size are obtained.
[0011] Then, the prediction network model predicts the data based on the optimal time period and prediction step size, obtains the predicted data volume of each data layer, and then obtains the predicted total data volume of the data warehouse.
[0012] During monitoring, set the volatility range of data volume for each data layer and the total data volume, obtain the prediction results of the current time prediction network model, and obtain the current actual data. Compare the prediction results with the current actual data to determine whether the difference is within the volatility range.
[0013] Preferably, the prediction network model is an LSTM network model.
[0014] Preferably, the data layers of the data warehouse include an operational data layer, a data detail layer, a summary data layer, an application data layer, and a dimension table layer.
[0015] Preferably, the time series sets of each data stratification are standardized:
[0016]
[0017] T = {T1, ..., T} j , ..., T J}
[0018] Where T represents the time series set of each data stratification, T′ represents the standardization of the time series set of each data stratification, σ represents the feature mean, u represents the feature standard deviation, and J represents the total length of the time series.
[0019] Preferably, the particle position is represented as X. id =(x i1 x i2 x i3 The speed is expressed as V. id =(v i1 v i2 v i3 ), and x i1 x is the number of input nodes and equal to the time period. i2 x is the number of hidden layer nodes. i3 The output number of nodes is equal to the prediction step size.
[0020] Preferably, when reconstructing the time series set, a sliding time window method is used to transform the time series set into the following structure: train
[0021]
[0022] Where N represents the size of the sliding window.
[0023] Preferably, the particle's fitness value, individual extreme value, and population extreme value, as well as the particle's position and velocity, satisfy the following formula:
[0024]
[0025]
[0026] d = 1, 2, ..., D
[0027] i = 1, 2, ..., sizepop
[0028] V id ∈[v d，min v d，max ]
[0029] X id ∈[x d，min x d，max ]
[0030] Wherein, the individual extreme value corresponding to the i-th particle is represented as Population extreme values are represented as θ represents the inertia weight, sizepop represents the population size, k represents the number of iterations, and when k represents the current generation, then k+1 represents the next generation, c1 and c2 both represent learning factors, λ represents the shrinkage factor, r1 and r2 both represent random numbers between [0, 1], d represents the particle dimension, and round represents the rounding function.
[0031] Preferably, when the difference between the predicted result and the current actual data is outside the volatility range, the current data is set as problematic data, triggering an alarm mechanism and pushing abnormal information.
[0032] Compared with the prior art, the beneficial effects of the present invention are as follows:
[0033] 1. The data volume prediction and monitoring method of this data warehouse establishes different prediction network models for each data layer of the data warehouse, obtains the optimal time period and prediction step size for each data layer, obtains the predicted data volume value for each data layer, and then obtains the predicted value of the total data volume of the data warehouse. This improves the accuracy of prediction and solves the problem of large workload and inaccurate prediction in the existing technology by making predictions manually.
[0034] 2. The data volume prediction and monitoring method of this data warehouse compares the prediction results with the current actual data during monitoring. When the difference is not within the fluctuation range, the current data is set as problem data, triggering the alarm mechanism and pushing abnormal information, which improves the accuracy of monitoring. Attached Figure Description
[0035] Figure 1 This is a flowchart illustrating the prediction process of the data warehouse data volume prediction and monitoring method of the present invention.
[0036] Figure 2 This is a flowchart illustrating the monitoring process of the data warehouse data volume prediction and monitoring method of the present invention. Detailed Implementation
[0037] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0038] It should be noted that when a component is referred to as being "connected" to another component, it can be directly connected to the other component or there may be an intervening component. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein in the specification of this application is for the purpose of describing particular embodiments only and is not intended to limit the application.
[0039] like Figure 1-2 As shown, a method for predicting and monitoring the amount of data in a data warehouse includes:
[0040] Step S1: Based on the selected data characteristics and time granularity, obtain the time series set of each data layer in the data warehouse.
[0041] Specifically, the data layers of a data warehouse include the Operational Data Layer (ODS), the Data Detail Layer (DWD), the Summary Data Layer (DWS), the Application Data Layer (ADS), and the Dimension Table Layer (DIM).
[0042] Data warehouse data can be viewed as temporal data with time attributes. Time granularity refers to the aggregation over a period of time. The longer the time period, the higher the time granularity. Commonly known time granularities such as months, quarters, and years are considered coarse time granularities, while hours, days, and weeks are considered fine time granularities. Data characteristics refer to the data volume of the operational data layer, the data volume of the detailed data layer, the data volume of the summary data layer, the data volume of the application data layer, and the data volume of the dimension table layer. Different time granularities can be used to divide the data to present different patterns. The time granularity should be selected according to the data characteristics and analysis needs.
[0043] To eliminate the dimensional relationships between data, the time series sets of each data stratification are standardized:
[0044]
[0045] T = {T1, ..., T} j ,…,T J}
[0046] Where T represents the time series set for each data stratification, T' represents the standardization of the time series set for each data stratification, σ represents the feature mean, u represents the feature standard deviation, and J represents the total length of the time series.
[0047] Step S2: Initialize particle position and velocity. Based on the initial particle position, obtain the initial number of input nodes, hidden layer nodes, and output nodes of the prediction network model, and initialize the prediction network model for each data layer.
[0048] Specifically, it should be noted that this method employs a multi-objective particle swarm optimization algorithm, starting randomly to find the optimal solution. Each particle represents an individual, and each particle has two attributes: position and velocity. The particle position is represented by X. id =(x i1 x i2 x i3 The speed is expressed as V. id =(v i1 v i2 v i3 ), and x i1 x is the number of input nodes and equal to the time period. i2 x is the number of hidden layer nodes. i3 The number of output nodes is equal to the prediction step size. For example, if the selected time granularity is days, use the data from the first N days as the input number of nodes, and the data from the N+1 to N+M days as the output number of nodes, where N is the time period to be determined, and M is the prediction step size.
[0049] The prediction network model is an LSTM network model.
[0050] Based on experience, set the number of input nodes [N] min N max The range of the number of output nodes is [5, 30]. min M max The range of the number of hidden layer nodes is [1, 5], and the range of the number of hidden layer nodes is [10, 100].
[0051] Step S3: After reconstructing the time series sets of each data layer, input them into the prediction network model for iterative training. Use the training error of the prediction network model as the fitness value of the particle, and update the individual extreme value and the population extreme value, as well as the position and velocity of the particle, until the number of iterations is reached.
[0052] Specifically, it should be noted that during the optimization process, by updating the position and attributes of the particles, the training error function of the prediction network model is determined as the fitness function of the multi-objective particle swarm optimization algorithm. The fitness function determines the fitness value of the particles, which represents the quality of the particles. In the process of continuous particle evolution, the individual extreme value and the population extreme value are tracked. The individual extreme value is the position with the best fitness value among the positions that the particle has experienced, and the population extreme value refers to the position with the best fitness value searched by all particles in the population.
[0053] When reconstructing the time series dataset, the sliding time window method is used to transform the time series dataset into the following structure: train:
[0054]
[0055] Where N represents the size of the sliding window.
[0056] The particle's fitness value, individual extreme value, and population extreme value, as well as the particle's position and velocity, satisfy the following formula, and the particle's velocity and position are updated according to the following formula:
[0057]
[0058]
[0059] d = 1, 2, ..., D
[0060] i = 1, 2, ..., sizepop
[0061] V id ∈[v d，min v d，max ]
[0062] X id ∈[x d，min x d，max ]
[0063] Wherein, the individual extreme value corresponding to the i-th particle is represented as Population extreme values are represented as θ represents the inertia weight, sizepop represents the population size, k represents the number of iterations, and when k represents the current generation, then k+1 represents the next generation, c1 and c2 both represent learning factors, λ represents the shrinkage factor, r1 and r2 both represent random numbers between [0, 1], d represents the particle dimension, and round represents the rounding function. If a certain dimension V id X id If the range exceeds the respective limits, the boundary value is taken.
[0064] Step S4: Obtain the particle position corresponding to the optimal particle fitness value of each data layer, and then obtain the optimal time period and prediction step size.
[0065] Specifically, the particle position corresponding to the optimal particle fitness: X best =(x best1 x best2 x best3 ), where x best1 x is the number of input nodes (time period). best2 x is the number of hidden layer nodes. best3 This is the number of output nodes (prediction step size).
[0066] Step S5: Then, the prediction network model predicts the data based on the optimal time period and prediction step size to obtain the predicted data volume of each data layer, and then obtains the predicted total data volume of the data warehouse.
[0067] Specifically, after obtaining the predicted data volume for each data layer, the predicted data volume for all five layers is aggregated to obtain the final predicted total data volume for the data warehouse.
[0068] During monitoring, set the volatility range of data volume for each data layer and the total data volume, obtain the prediction results of the current time prediction network model, and obtain the current actual data. Compare the prediction results with the current actual data to determine whether the difference is within the volatility range.
[0069] Specifically, the volatility of the data volume of each data layer and the total data volume can be selected from 0% to 100%, and an appropriate volatility can be set according to actual needs; when the difference between the prediction result and the current actual data is not within the volatility range, the current data is set as problem data, triggering an alarm mechanism and pushing abnormal information.
[0070] At the same time, the data volume of each data layer, the total data volume, and the problematic data are displayed in the form of visual charts to help data users discover data patterns, correlations between data, and data anomalies in a more intuitive way.
[0071] This data warehouse data volume prediction and monitoring method establishes different prediction network models for each data layer of the data warehouse, obtains the optimal time period and prediction step size for each data layer, obtains the predicted data volume value for each data layer, and then obtains the predicted value of the total data volume of the data warehouse. This improves the accuracy of prediction and solves the problem of large workload and inaccurate prediction in existing technologies that rely on manual prediction. During monitoring, this data warehouse data volume prediction and monitoring method compares the prediction results with the current actual data. When the difference is not within the fluctuation range, the current data is set as problem data, triggering an alarm mechanism and pushing abnormal information, which improves the accuracy of monitoring.
[0072] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0073] The embodiments described above are merely specific and detailed examples of the embodiments described in this application, and should not be construed as limiting the scope of the patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the scope of protection of this application. Therefore, the scope of protection of this patent application should be determined by the appended claims.
Claims
1. A method for predicting and monitoring the amount of data in a data warehouse, characterized in that: The method for predicting and monitoring the data volume of the data warehouse includes: Based on the selected data characteristics and time granularity, obtain the time series sets of each data layer in the data warehouse; Initialize particle position and velocity, obtain the initial number of input nodes, hidden layer nodes and output nodes of the prediction network model based on the initial particle position, and initialize the prediction network model of each data layer; After reconstructing the time series sets of each data layer, the data is input into the prediction network model for iterative training. The training error of the prediction network model is used as the fitness value of the particle, and the individual extreme value and the population extreme value, as well as the position and velocity of the particle, are updated until the number of iterations is reached. The optimal particle fitness value for each data layer is obtained, and the optimal time period and prediction step size are obtained. Then, the prediction network model predicts the data based on the optimal time period and prediction step size to obtain the predicted data volume of each data layer, and then obtains the predicted total data volume of the data warehouse. During monitoring, set the volatility range of data volume for each data layer and the total data volume, obtain the prediction results of the current time prediction network model, and obtain the current actual data. Compare the prediction results with the current actual data to determine whether the difference is within the volatility range. Wherein, the particle position is represented as Speed is expressed as ,and The number of input nodes must equal the time period. This represents the number of hidden layer nodes. The output number of nodes is equal to the prediction step size; The fitness value, individual extreme value, and population extreme value of the particle, as well as the particle's position and velocity, satisfy the following formula: Among them, the The individual extreme values corresponding to each particle are represented as follows: Population extreme values are represented as , Indicates inertia weight, Indicates population size, Indicates the number of iterations, and when To indicate the contemporary era, Representing the next generation, All represent learning factors. Indicates the contractility factor. and All indicate Random numbers between Represents the dimension of a particle. This represents the floor function.
2. The method for predicting and monitoring data volume in a data warehouse as described in claim 1, characterized in that: The prediction network model is an LSTM network model.
3. The method for predicting and monitoring data volume in a data warehouse as described in claim 1, characterized in that: The data warehouse is layered into operational data layer, detailed data layer, summary data layer, application data layer, and dimension table layer.
4. The method for predicting and monitoring data volume in a data warehouse as described in claim 1, characterized in that: Standardize the time series sets for each data stratification: in, This represents a collection of time series data strata. This indicates the standardization of time series sets stratified by data level. Represents the characteristic mean. Indicates the characteristic standard deviation, This indicates the total length of the time series.
5. The method for predicting and monitoring data volume in a data warehouse as described in claim 4, characterized in that: When reconstructing a time series dataset, the sliding time window method is used to transform the dataset into the following structure. : in, This indicates the size of the sliding window.
6. The method for predicting and monitoring data volume in a data warehouse as described in claim 1, characterized in that: When the difference between the predicted result and the current actual data is outside the volatility range, the current data is set as problematic data, triggering an alarm mechanism and pushing abnormal information.