A real-time data warehouse management method and system

By using dynamic cache and principal component analysis methods in real-time data warehouses to extract feature values ​​and monitor the number of requests thresholds, the problem of low performance and throughput when processing large amounts of data in real-time data warehouses is solved, and efficient data storage and reading is achieved.

CN118332054BActive Publication Date: 2024-11-22SHENZHENSHI LUTEJIACHENG NETWORK TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410531262.4
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-04-29
Publication Date
2024-11-22
Estimated Expiration
2044-04-29

AI Technical Summary

Technical Problem

When real-time data warehouses process large amounts of real-time data, it is difficult to ensure high performance and high throughput, resulting in high query and analysis latency.

Method used

By extracting the eigenvalues ​​of the target data and storing them into the dynamic cache, the corresponding target data is retrieved according to the eigenvalues ​​in the cache for storage and reading. The eigenvalues ​​are obtained using principal component analysis method (PCA), and the processing is monitored in real time. The request threshold determines whether to perform storage or read operations to avoid overloading of data into the channel.

Benefits of technology

It significantly improves the running speed and throughput of real-time data warehouses, ensures the efficiency of data storage and reading processes, and keeps data channels unobstructed.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118332054B_ABST
    Figure CN118332054B_ABST
Patent Text Reader

Abstract

The present invention discloses a real-time data warehouse management method and system. When storing data, the characteristic value of the target data group is obtained and a copy is created and first stored in the dynamic cache of the real-time data warehouse. Then, according to the threshold judgment result set for the real-time number of requests processed per second and the number of requests processed per second of the data warehouse at this time, the corresponding target data is retrieved for data storage; when reading data, the corresponding target data is first retrieved for reading according to the characteristic value stored in the dynamic cache. At this time, the threshold judgment result is set for the real-time number of requests processed per second and the number of requests processed per second of the data warehouse. With the method of the present specification, the operation speed of the real-time data warehouse can be greatly improved, and the throughput can be improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a real-time data warehouse management method and system. Background Art

[0002] A real-time data warehouse is a data warehouse system used to store and manage real-time data. Unlike traditional data warehouse systems, real-time data warehouses focus on processing real-time data streams, which are usually continuously generated and updated, such as sensor data, log data, financial transaction data, etc. Real-time data warehouses usually need to quickly process large amounts of real-time data, capture and process data updates in a timely manner, and support real-time data queries. This requires real-time data warehouse systems to have high performance and high throughput, be able to quickly process and store data, and provide low-latency query and analysis capabilities. Summary of the invention

[0003] This specification provides a real-time data warehouse management method and system. When storing data, the characteristic value of the target data group is obtained and a copy is created and first stored in the dynamic cache of the real-time data warehouse. Then, according to the characteristic value stored in the dynamic cache, the corresponding target data is retrieved for data storage; when reading data, the corresponding target data is first retrieved according to the characteristic value stored in the dynamic cache for reading, and then stored or read based on the real-time data throughput. This can greatly improve the running speed of the real-time data warehouse and improve the throughput.

[0004] To achieve the above object, the present invention provides a real-time data warehouse management method, comprising the following steps:

[0005] Step S1: extracting n characteristic values ​​of N groups of target data to be stored;

[0006] Step S2: copying the n characteristic values ​​and storing them in the dynamic cache of the real-time data warehouse;

[0007] Step S3: Retrieve the real-time number of requests processed per second (QPS) A of the data warehouse and the threshold value X for the number of requests processed per second, and determine:

[0008] When A≤X,

[0009] Then, according to the n characteristic values ​​in the dynamic cache, the corresponding N groups of target data are written into the data warehouse.

[0010] When A>X,

[0011] Then suspend data writing until A≤X;

[0012] The reading process T includes the following steps:

[0013] Step T1: acquiring m characteristic values ​​of M groups of target data to be read in the dynamic cache of the data warehouse at real time;

[0014] Step T2: reading corresponding M groups of target data from the data warehouse according to the m characteristic values;

[0015] Step T3: Retrieve the real-time number of requests processed per second (QPS) A of the data warehouse and the threshold value X for the number of requests processed per second, and determine:

[0016] When A≤X,

[0017] Then, according to the m characteristic values ​​in the dynamic cache, the corresponding M groups of target data are read from the data warehouse.

[0018] When A>X,

[0019] Then data reading is suspended until A≤X.

[0020] In the real-time data warehouse management method, preferably, the n eigenvalues ​​of the N groups of target data in step S are acquired using principal component analysis (PCA).

[0021] In the real-time data warehouse management method, preferably, the process of using principal component analysis (PCA) to obtain the characteristic value z of the Zth group of data in the N groups of data is:

[0022] Get the covariance matrix of the matrix Z composed of the Z-th group of data arrangements:

[0023] ∑(Z)=(S ij )p×p,

[0024] in, x is the characteristic vector of the Zth group of data,

[0025] Get the eigenvalues ​​of the covariance matrix, take the eigenvectors corresponding to the first q larger eigenvalues ​​to form a new matrix P, and calculate:

[0026] z=P*Z,

[0027] That is, the characteristic value of the Zth group of data.

[0028] In the real-time data warehouse management method, preferably, the cumulative variance contribution rate G(q) of the first q larger eigenvalues ​​is greater than 88%, wherein:

[0029]

[0030] In the real-time data warehouse management method, preferably, the first q larger eigenvalues ​​correspond to at least the state information, field information, degree of freedom information and timestamp information of the target data.

[0031] In the real-time data warehouse management method, preferably, the maximum number of requests processed per second by the real-time data warehouse is Y, and the number of requests processed per second is set to a threshold value

[0032] In the real-time data warehouse management method, preferably, when there are storage and reading requirements at the same time, the storage process S is executed first.

[0033] In the real-time data warehouse management method, preferably, when there are storage and reading requirements at the same time, the reading process T is executed first.

[0034] In the real-time data warehouse management method, preferably, the optimization objectives include:

[0035] The real-time data warehouse is always in a state of maximum data throughput based on the amount of data stored and read in real time.

[0036] In another aspect, the present invention provides a real-time data warehouse management system, on which a computer program is stored. When the computer program is executed by a processor, the steps of the real-time data warehouse management method as described above are implemented.

[0037] The real-time data warehouse management method and system of the present invention include execution logic of two dimensions: target data storage and target data reading. The principal component analysis (PCA) method is used to obtain the characteristic values ​​of the target data, and then a copy of the characteristic values ​​of the obtained target data is stored in the dynamic cache and then distributed, rather than directly storing the target data in the real-time data warehouse, which can avoid a large amount of data from pouring into the storage channel at the same time. Similarly, in terms of data reading, the characteristic values ​​of the target data stored in the dynamic cache are first retrieved, and then the corresponding target data is retrieved based on the characteristic values, which can avoid a large amount of data from pouring into the reading channel at the same time. The use of the real-time data warehouse management method and system shown in the present invention can greatly improve the operating speed of the real-time data warehouse and improve throughput. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the technical solutions in the embodiments of this specification, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0039] Figure 1A schematic diagram showing the interaction between a user and a real-time data warehouse provided according to an embodiment of the present application is shown;

[0040] Figure 2 A flow chart of storing target data in a real-time data warehouse according to an embodiment of the present application is shown;

[0041] Figure 3 A flowchart of reading target data from a real-time data warehouse provided according to an embodiment of the present application is shown;

[0042] Figure 4 A flow chart of obtaining characteristic values ​​of target data using principal component analysis (PCA) according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0043] The following description provides specific application scenarios and requirements of this specification, with the purpose of enabling those skilled in the art to make and use the contents of this specification. Various local modifications to the disclosed embodiments will be apparent to those skilled in the art, and the general principles defined herein may be applied to other embodiments and applications without departing from the spirit and scope of this specification. Therefore, this specification is not limited to the embodiments shown, but to the widest scope consistent with the claims.

[0044] The terms used herein are only used for the purpose of describing specific example embodiments and are not restrictive. For example, unless the context clearly indicates otherwise, as used herein, the singular forms "a", "an" and "the" may also include plural forms. When used in this specification, the terms "include", "comprise" and / or "contain" mean that the associated integers, steps, operations, elements and / or components exist, but do not exclude the existence of one or more other features, integers, steps, operations, elements, components and / or groups or that other features, integers, steps, operations, elements, components and / or groups may be added in the system / method.

[0045] These and other features of the present specification, as well as the operation and function of the related elements of the structure, and the economy of combination and manufacture of the parts can be significantly improved in view of the following description. Reference is made to the accompanying drawings, all of which form a part of this specification. However, it should be clearly understood that the drawings are for illustration and description purposes only and are not intended to limit the scope of this specification.

[0046] The flowcharts used in this specification illustrate the operations implemented by the system according to some embodiments in this specification. It should be clearly understood that the operations of the flowcharts may not be implemented in sequence. On the contrary, the operations may be implemented in reverse order or simultaneously. In addition, one or more other operations may be added to the flowchart. One or more operations may be removed from the flowchart.

[0047] Figure 1 A schematic diagram of the interaction between a user and a real-time data warehouse 1 provided according to an embodiment of the present application is shown. The real-time data warehouse 1 has a dynamic cache 11. When a user 2 stores or reads data in the real-time data warehouse 1, it is in direct contact with the dynamic cache, and then the dynamic cache 11 exchanges data and information with the inside of the real-time data warehouse 1. In some embodiments, the dynamic cache can be an LRU (Least Recently Used) / LFU (Least Frequently Used) cache. This type of cache eliminates data in the cache based on the most recent access time or the least number of accesses to the data, and retains the most commonly used data; in some embodiments, the dynamic cache can be an ARC (Adaptive Replacement Cache) cache. This type of cache dynamically adjusts the weights of the LRU and LFU algorithms based on the access frequency and recent usage of the data; the dynamic cache type can also be an RC cache (Adaptive Replacement Cache). This type of cache dynamically adjusts the weights of the aforementioned LRU and LFU algorithms based on the access frequency and recent usage of the data, making cache elimination more intelligent.

[0048] Figure 2 A schematic diagram P100 of a process of storing target data in a real-time data warehouse 1 according to an embodiment of the present application is shown, including:

[0049] S110: Extracting feature values ​​of target data.

[0050] In the present application, in the storage stage, the target data is the data that the user needs to store in the real-time data warehouse 1. In some embodiments, at the same time, N groups of different target data need to be written. For example, in a real-time data warehouse serving a financial trading system, the stock information of thousands of listed companies needs to be written at the same time. In some embodiments of the present application, feature value extraction is performed on the N groups of target data that need to be stored in the real-time data warehouse 1 to obtain n feature values ​​corresponding to each group of feature data. In some embodiments, the extraction of feature values ​​can adopt principal component analysis (PCA); in other embodiments, the extraction of feature values ​​can also adopt linear discriminant analysis, statistical feature method, etc.

[0051] S120: Copy the characteristic value and store it in the dynamic cache.

[0052] In the embodiment of the present application, after extracting the characteristic values ​​of the N groups of target data, the characteristic values ​​are copied and stored in the dynamic cache 11 and are in a state of being available for recall at any time. The original characteristic value data will follow the target data 11. The advantage of this is that compared with directly storing the characteristic value data in the dynamic cache 11, the loss of characteristic value data after the cache data is eliminated can be prevented.

[0053] S130: Retrieve the real-time number of requests per second (QPS) and set a threshold for processing requests per second.

[0054] The number of requests per second (QPS) is one of the important indicators for measuring the performance of a data warehouse system, indicating the number of queries that a data warehouse can process per second. Under the limitation of hardware conditions, in order to improve the operating speed and throughput of a real-time data warehouse, a QPS upper limit is often set for the real-time data warehouse. In some embodiments of the present application, the storage process of real-time data will retrieve the QPS parameters of the real-time data warehouse 1 and the QPS setting threshold pre-stored in the real-time data warehouse 1 for subsequent logical judgment.

[0055] S140: Perform threshold determination on the real-time number of requests per second (QPS).

[0056] In the embodiment of the present application, let the real-time number of requests per second be A, and the real-time number of requests per second setting threshold be X, then:

[0057] When A≤X,

[0058] According to the n characteristic values ​​in the dynamic cache, the corresponding N groups of target data are written into the data warehouse 1

[0059] When A>X,

[0060] Pause data writing until A≤X; what is actually limited here is the maximum QPS that the real-time data warehouse 1 can bear during the storage process. It can be understood that in order to take into account the smooth flow of both storage and reading channels, the real-time QPS during the storage process should not occupy too many channels. In some embodiments, assuming that the maximum number of requests processed per second by the real-time data warehouse is Y, the threshold value of the number of requests processed per second during the storage process can be set

[0061] S150: Writing target data into the real-time data warehouse.

[0062] In the present application, under the premise of the aforementioned steps, the real-time data warehouse 1 can choose to store the target data corresponding to the characteristic value stored in the dynamic cache 11 into the real-time data warehouse 1 when the storage channel bandwidth meets the requirements based on the judgment result of step S140. The storage can be timely or the storage operation can be performed after the bandwidth meets the requirements.

[0063] Figure 3 The flowchart P200 of reading target data from the real-time data warehouse 1 according to an embodiment of the present application is shown, including:

[0064] T210: Extract the feature values ​​of the target data.

[0065] In the present application, the data stored in the real-time data warehouse 1 is regarded as being stored in the aforementioned storage process, so the corresponding feature values ​​that have been stored in the real-time data warehouse 1 along with the target data can be directly called during the reading process without recalculating the feature values.

[0066] T220: Store the feature value into the dynamic cache.

[0067] In the embodiment of the present application, during the reading process, after extracting the characteristic values ​​of the N groups of target data, the characteristic values ​​are stored in the dynamic cache 11 and are in a state where they can be called at any time.

[0068] T230: Retrieve the real-time number of requests per second (QPS) and set the threshold for requests processed per second.

[0069] In some embodiments of the present application, the real-time data reading process will retrieve the QPS parameters of the real-time data warehouse 1 and the QPS setting threshold pre-stored in the real-time data warehouse 1 for subsequent logical judgment.

[0070] T240: Performs threshold judgment on the real-time number of requests per second (QPS).

[0071] In the embodiment of the present application, during the reading process, let the real-time number of requests per second be A, and the real-time number of requests per second setting threshold be X, then:

[0072] The real-time number of requests processed per second (QPS) A of the data warehouse 1 and the threshold value X of the number of requests processed per second are retrieved to determine:

[0073] When A≤X,

[0074] Then, according to the m characteristic values ​​in the dynamic cache, the corresponding M groups of target data are read from the data warehouse 1.

[0075] When A>X,

[0076] Then suspend data reading until A≤X;

[0077] Similarly, what is actually limited here is the maximum QPS that the real-time data warehouse 1 can bear during the reading process. It can be understood that in order to take into account the smooth flow of both storage and reading channels, the real-time QPS during the reading process should not occupy too many channels. In some embodiments, assuming that the maximum number of requests processed per second of the real-time data warehouse is Y, the threshold value of the number of requests processed per second during the reading process can be set

[0078] S150: Read target data from the real-time data warehouse.

[0079] In the present application, during the data reading process, under the premise of the aforementioned steps, the real-time data warehouse 1 can choose to read the target data corresponding to the features stored in the dynamic cache 11 from the real-time data warehouse 1 when the storage channel bandwidth meets the judgment result of step T240. The reading can be timely, or the reading operation can be performed after the bandwidth meets the conditions.

[0080] In some embodiments of the present application, for example, when the real-time data warehouse 1 is mainly used for storage-intensive tasks such as monitoring system monitoring or log recording, it can be set that when the storage process and the reading process are in the queue to be executed at the same time, the storage process is executed first.

[0081] In other embodiments, for example, when the real-time data warehouse 1 is mainly used for read-intensive tasks such as data analysis and report generation, it can be set that when the storage process and the read process are in the queue to be executed at the same time, the read process is executed first.

[0082] The following is an example of the feature value extraction method involved in this application. It should be pointed out that the feature value extraction is essentially a compression process of the target data, extracting the most representative parameters in the target data and corresponding to the target data one by one, and using feature parameters with small sample capacity and low memory consumption to represent the group of target data for pre-caching operations, which can greatly improve the speed of real-time data warehouse storage and reading.

[0083] In some embodiments of the present application, principal component analysis (PCA) is used to extract the eigenvalues ​​of target data. Principal component analysis is essentially a data dimensionality reduction algorithm, which can convert high-dimensional data sets into low-dimensional data sets. Low-dimensional data sets usually require fewer computing resources and can be trained and predicted faster, thereby improving the efficiency and performance of the model, while retaining the most important information in the data, thereby reducing computing and storage costs, and helping to reduce overfitting problems. On the other hand, PCA is actually finding the main features in the data by finding the direction of the maximum variance in the data, thereby removing the correlation between different features. This helps to eliminate noise and redundant information in the data and improves the effect of subsequent analysis. Based on the above characteristics, PCA is more suitable for target data eigenvalue extraction in real-time data warehouses than other eigenvalue extraction algorithms.

[0084] Please refer to Figure 4 In the present application, the process P300 of obtaining the characteristic value of the target data by using PCA includes:

[0085] S310: Data standardization.

[0086] In an embodiment of the present application, the target data to be stored in the real-time data warehouse 1 often include indicators of multiple dimensions. For example, the target data reflecting the degree of innovation and development in a region may include the number of patent applications (x1), the proportion of scientific and technological expenditures in fiscal expenditures (x2), R&D funds of industrial enterprises above designated size (x3), and technology market turnover (x4). Due to the different dimensions and magnitudes of the basic indicators, we first need to standardize all the original indicators. Here, some data processing software such as the PCA data preprocessing command in Stata can be used to standardize the data vectors x of different dimensions. In some embodiments, other data processing models can also be used to perform the above operations, such as models such as minimum-maximum normalization or Z-score normalization. Of course, when the dimensions of each data vector in the target data are consistent, there is no need to perform this conversion.

[0087] S320: Calculate the covariance matrix.

[0088] The covariance matrix describes the relationship between the various features in the data. In an embodiment of the present application, the mathematical expression of the covariance matrix of the Zth group of data in the N groups of data can be:

[0089] ∑(Z)=(S ij )p×p,

[0090] in, x is the characteristic vector of the Zth group of data.

[0091] S330: Solve the eigenvalues ​​and eigenvectors of the covariance matrix and select the first q larger eigenvalues.

[0092] In the embodiment of the present application, the data vectors in the target data corresponding to the first q largest eigenvalues ​​at least include the state information, field information, degree of freedom information and timestamp information of the target data. Generally speaking, the eigenvalues ​​of the above four groups of data vectors can basically reflect most of the information of the target data. For the purpose of optimization, the cumulative variance contribution rate G(q) of the first q largest eigenvalues ​​is required to be greater than 88%, where:

[0093]

[0094] S340: Constructing the characteristic value of target data

[0095] Take the eigenvectors corresponding to the first q larger eigenvalues ​​to form a new matrix P. In this application, calculate,

[0096] z=P*Z,

[0097] That is, the characteristic value of the Zth group of data.

[0098] In some embodiments, the method of determining the characteristic value of the target data may also include: descriptive statistics method, linear discriminant analysis method, t-SNE (t-distributed Stochastic Neighbor Embedding) method, etc., which will not be described in detail here.

[0099] On the other hand, the present specification provides a non-transitory storage medium, including a computer program, wherein the computer program stores at least one set of executable instructions for signal processing. When the executable instructions are executed by the computer program, the executable instructions instruct the processor to implement the real-time data management methods P100 and P200 described in the present specification. In some possible implementations, various aspects of the present specification can also be implemented in the form of a program product, which includes a program code. When the program product is run on a computer device, the program code is used to enable the computer program to execute the sales forecasting method real-time data management method P100 and P200 described in the present specification. The program product for implementing the above method can use a portable compact disk read-only memory (CD-ROM) to include program code and can be run on a computer device or a server device. However, the program product of the present specification is not limited to this. In the present specification, the non-transitory storage medium can be any tangible medium containing or storing a program, which can be used by an instruction execution system or used in combination with it. The program product can use any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage chip. The readable storage medium may be, for example but not limited to, an electronic, magnetic, optical, electromagnetic, infrared or semiconductor system, apparatus or device, or any combination of the above.

[0100] The basic concepts have been described above. Obviously, for those skilled in the art, the above detailed disclosure is only for example and does not constitute a limitation of this specification. Although not explicitly stated here, those skilled in the art may make various modifications, improvements and corrections to this specification. Such modifications, improvements and corrections are suggested in this specification, so such modifications, improvements and corrections still belong to the spirit and scope of the exemplary embodiments of this specification.

Claims

1. A real-time data warehouse management method, It is characterized in that For optimization purposes, the stored procedure S includes the following steps: Step S1: extracting n characteristic values ​​of N groups of target data to be stored; Step S2: copying the n eigenvalues ​​and storing them in the dynamic cache of the real-time data warehouse, and the original eigenvalue data of the n eigenvalues ​​and / or the N groups of target data are not stored in the dynamic cache; Step S3: Retrieve the real-time number of requests processed per second (QPS) A of the data warehouse and the threshold value X for the number of requests processed per second, and determine: When A≤X, Then, according to the n characteristic values ​​in the dynamic cache, the corresponding N groups of target data are written into the data warehouse. When A>X, Then suspend data writing until A≤X; The reading process T includes the following steps: Step T1: acquiring m characteristic values ​​of M groups of target data to be read in the real-time data warehouse and storing them in a dynamic cache; Step T2: reading corresponding M groups of target data from the data warehouse according to the m characteristic values; Step T3: Retrieve the real-time number of requests processed per second (QPS) A of the data warehouse and the threshold value X for the number of requests processed per second, and determine: When A≤X, Then, according to the m characteristic values ​​in the dynamic cache, the corresponding M groups of target data are read from the data warehouse. When A>X, Then data reading is suspended until A≤X.

2. The real-time data warehouse management method according to claim 1, It is characterized in that The n eigenvalues ​​of the N groups of target data in the step S are obtained using principal component analysis (PCA).

3. The real-time data warehouse management method according to claim 2, It is characterized in that The process of using principal component analysis (PCA) to obtain the characteristic value z of the Zth group of data in the N groups of data is: Get the covariance matrix of the matrix Z composed of the Z-th group of data arrangements: ∑=(S ij )p×pwhere, x is the characteristic vector of the Zth group of data, Get the eigenvalues ​​of the covariance matrix, take the eigenvectors corresponding to the eigenvalues ​​corresponding to the first q larger eigenvalues ​​to form a new matrix P, and calculate: z=P*Z, That is, the characteristic value of the Zth group of data.

4. The real-time data warehouse management method according to claim 3, It is characterized in that The cumulative variance contribution rate G(q) of the first q larger eigenvalues ​​is greater than 88%, where 5. The real-time data warehouse management method according to claim 4, It is characterized in that The first q larger eigenvalues ​​correspond to at least the state information, field information, degree of freedom information and timestamp information of the target data.

6. The real-time data warehouse management method according to claim 1, It is characterized in that The maximum number of requests processed per second by the real-time data warehouse is Y, and the number of requests processed per second sets a threshold 7. The real-time data warehouse management method according to claim 1, It is characterized in that When there are storage and reading requirements at the same time, the stored procedure S is executed first.

8. The real-time data warehouse management method according to claim 1, It is characterized in that When there are storage and reading requirements at the same time, the reading process T is executed first.

9. The real-time data warehouse management method according to claim 1, It is characterized in that The optimization purpose includes: ensuring that the real-time data warehouse is always in a state of maximum data throughput based on the amount of data stored and read in real time.

10. A real-time data warehouse management system having a computer program stored thereon, It is characterized in that When the computer program is executed by a processor, the steps of the real-time data warehouse management method according to any one of claims 1 to 9 are implemented.