Method for improving metadata storage and query efficiency

Abstract

The invention relates to the technical field of metadata management, and discloses a method for improving metadata storage and query efficiency, comprising: establishing a meta-model of each type of metadata, the meta-model comprising an object, an attribute and a relationship of the metadata; obtaining a heat value of each meta-model based on historical information of the meta-models, performing dynamic heat partitioning on the meta-models according to the heat values, and adjusting an index strategy, a storage strategy and a cache strategy according to the dynamic heat partitioning of the meta-models; the storage and query process is monitored in real time, index data monitored in real time in different dynamic popularity partitions are analyzed, and the meta-model is adjusted according to the analysis result. According to the method, the corresponding management mode can be dynamically adjusted according to the popularity state of the meta-model, so that the metadata storage and query efficiency is improved on the whole; meanwhile, real-time optimization of the meta-model is achieved by monitoring the storage and query processes in real time, and the metadata storage and query efficiency is improved.

Description

Technical Field

[0001] The present application relates to the technical field of metadata management, and in particular to a method for improving metadata storage and query efficiency. Background Art

[0002] With the rapid development of big data technology, various advanced data processing technologies and components have been rapidly introduced, system architectures have become increasingly complex, and the difficulty of data governance has also increased significantly. The most important of these is the storage and query of various types of metadata. Traditional data governance tools store metadata based on metadata type and store and query in separate tables. This method is simple and fast to implement, but difficult to expand. When meeting the storage requirements of new types of metadata, the database design and program interface logic must be adjusted simultaneously, which makes the expansion cost extremely high.

[0003] In the existing metadata management process, various metadata objects, attributes, and relationships can be abstracted and generalized through a common warehouse metamodel, and the function of carrying unlimited types of metadata attributes and relationship information can be realized through a limited number of data tables. This method makes the storage, management, and query of metadata more efficient and consistent, and can greatly simplify the metadata integration and consistency management between different data processing tools and information systems.

[0004] However, as the amount of metadata increases and the frequency of use of different metadata varies, the storage and query efficiency of metadata will gradually decrease. Therefore, how to improve the storage and query efficiency in a large-scale metadata general warehouse is the fundamental problem to be solved by the present invention. Summary of the Invention

[0005] In order to improve storage and query efficiency in a large-scale metadata general warehouse, the present application provides a method for improving metadata storage and query efficiency.

[0006] In a first aspect, the present application provides a method for improving metadata storage and query efficiency, using the following technical solutions:

[0007] A method for improving metadata storage and query efficiency, comprising:

[0008] Establishing a meta-model for each type of metadata, wherein the meta-model includes objects, attributes, and relationships of the metadata;

[0009] Obtain the popularity value of each metamodel based on the historical information of the metamodel, dynamically partition the metamodel based on the popularity value, and adjust the indexing strategy, storage strategy, and cache strategy based on the dynamic popularity partitioning of the metamodel;

[0010] Monitor the storage and query process in real time, analyze the indicator data of different dynamic heat partitions in real time, and adjust the meta-model based on the analysis results.

[0011] By adopting the above technical solution, the popularity status of each metamodel is judged according to the category and historical information of the metamodel and measured by the popularity value, the metamodel is dynamically partitioned according to the popularity value, and the index strategy, storage strategy and cache strategy are adjusted according to the dynamic popularity partition of the metamodel. The corresponding management method can be dynamically adjusted according to the popularity status of the metamodel, thereby improving the overall efficiency of metadata storage and query; at the same time, by real-time monitoring of the storage and query process, the indicator data of real-time monitoring of different dynamic popularity partitions is analyzed, and the metamodel is adjusted according to the analysis results. The relevant parameters of each metamodel can be dynamically adjusted, thereby realizing real-time optimization of the metamodel and improving the efficiency of metadata storage and query.

[0012] Optionally, the process of obtaining the heat value of each meta-model includes:

[0013] Obtain the historical data of each meta-model and obtain the heat value H(t0) at time point t0 in the historical data, t-t0=Δt, t is the current time point, Δt is a fixed value;

[0014] Obtain the periodicity coefficient p of the meta-model based on historical data;

[0015] By formula Calculate the heat value H(t) at the current time point;

[0016] Among them, T() is the natural decay function of heat, p is the periodic coefficient, q(t) is the visit curve of the meta-model, is the reference value of visit volume, k is the overall slope of q(t) in the period t0~t, is the reference value of the visit volume change rate. and is the adjustment factor.

[0017] By adopting the above technical solution, it is possible to accurately judge the popularity status and popularity trend of the metamodel at the current time point, and then realize accurate dynamic popularity partitioning of the metamodel. By executing different strategies for different dynamic popularity partitions, the efficiency of metadata storage and query is improved.

[0018] Optionally, the process of obtaining the periodic coefficient includes:

[0019] Get the time point corresponding to the peak of q(t) in the preset historical period, and get the time difference between adjacent time points , through the formula Calculate the time difference consistency coefficient S, where m is the number of time differences, i∈[1,m], is the mean of all time differences;

[0020] Compare the time difference consistency coefficient S with the preset threshold Sthr:

[0021] If S≥Sthr, the metamodel is judged to be a non-periodic metamodel, and p=1;

[0022] If S < Sthr, the metamodel is judged to be a periodic metamodel, and whether the current time point t is Within the range:

[0023] If yes, let p=p1, where p1 is the first fixed periodic coefficient and p1>1;

[0024] If not, let p=p2, where p2 is the second fixed period coefficient and 0<p1<1;

[0025] in, is the peak time point closest to the current time point, is a fixed time error value.

[0026] By adopting the above technical solution, the size of the heat value can be accurately adjusted according to the periodic state of the metamodel call, thereby improving the accuracy of the heat value calculation and the accuracy of the dynamic heat partitioning.

[0027] Optionally, the process of dynamically partitioning the metamodel based on the heat value includes:

[0028] The heat value H(t) at the current time point is compared with the preset threshold interval, and the corresponding heat partition is determined according to the threshold interval where the heat value H(t) lies.

[0029] By adopting the above technical solution, the corresponding heat partition can be determined according to the threshold range of the heat value, and the corresponding management method can be dynamically adjusted according to the heat status of the metamodel, thereby improving the overall efficiency of metadata storage and query.

[0030] Optionally, the process of adjusting the indexing strategy according to the dynamic popularity partition of the metamodel includes:

[0031] The order priority of the metadata index is determined based on the dynamic heat partition where the metamodel is located. The higher the heat of the dynamic heat partition where the metamodel is located, the higher the order priority of the metamodel index.

[0032] The process of adjusting the storage strategy based on the dynamic heat partitioning of the metamodel includes:

[0033] The level of the metadata storage medium is determined according to the dynamic heat partition where the metamodel is located. The higher the heat of the dynamic heat partition where the metamodel is located, the higher the level of the metamodel storage medium.

[0034] The process of adjusting the cache strategy based on the dynamic popularity partitioning of the metamodel includes:

[0035] The cache priority of the metadata is determined according to the dynamic heat partition where the metamodel is located. The higher the heat of the dynamic heat partition where the metamodel is located, the higher the cache priority of the metamodel.

[0036] By adopting the above technical solutions, the order priority of the metamodel index of the metamodel with higher popularity is increased, which can improve the overall query efficiency of the metadata; the metamodel storage medium level of the metamodel with higher popularity is increased, which can improve the overall storage efficiency of the metadata; and the metamodel cache priority of the metamodel with higher popularity is increased, which can improve the overall query efficiency of the metadata.

[0037] Optionally, the process of analyzing the real-time monitoring indicator data of different dynamic heat partitions includes:

[0038] Monitor the query latency and indexing efficiency of different dynamic hot partition metamodels, determine whether there are abnormal metamodels based on the monitoring results, and optimize the model if there are abnormal metamodels.

[0039] By adopting the above technical solution, on the basis of partitioning the metamodel, through the monitoring process of each dynamic heat partition metamodel, it is possible to accurately judge the efficiency anomalies of the metamodel during the actual call process, and then optimize it in time to improve the efficiency of metadata query.

[0040] Optionally, the process of determining whether an abnormal meta-model exists according to the monitoring result includes:

[0041] Obtain the performance coefficient F of the metamodel based on query latency and index efficiency, ;

[0042] Where L is the query delay, is the indexing efficiency, u is the system hardware capability constant, C is the maximum total system resources, The system resource usage;

[0043] The performance coefficient F of the metamodel is compared with the threshold Ft of its corresponding dynamic heat partition. When F≤Ft, the metamodel is judged to be an abnormal metamodel.

[0044] By adopting the above technical solution, the abnormal meta-model is obtained, which provides an accurate reference basis for the subsequent optimization process. The process of optimizing the abnormal meta-model includes adjusting the index and reducing redundant indexes.

[0045] Optionally, the process of optimizing the abnormal meta-model includes adjusting indexes and reducing redundant indexes.

[0046] By adopting the above technical solution, the efficiency of metamodel calling can be improved, that is, the efficiency of metadata query can be improved.

[0047] In summary, this application includes at least one of the following beneficial technical effects:

[0048] The present invention judges the popularity of each metamodel according to the category and historical information of the metamodel and measures it through the popularity value, dynamically partitions the metamodel according to the popularity value, adjusts the index strategy, storage strategy and cache strategy according to the dynamic popularity partition of the metamodel, and can dynamically adjust its corresponding management method according to the popularity status of the metamodel, thereby improving the efficiency of metadata storage and query as a whole; at the same time, by real-time monitoring of the storage and query process, analyzing the indicator data of real-time monitoring of different dynamic popularity partitions, and adjusting the metamodel according to the analysis results, it can dynamically adjust the relevant parameters of each metamodel, thereby realizing real-time optimization of the metamodel and improving the efficiency of metadata storage and query. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 The present invention is a flowchart of the steps of a method for improving metadata storage and query efficiency. DETAILED DESCRIPTION

[0050] Embodiments of the present application are described in detail below, examples of which are illustrated in the accompanying drawings.

[0051] Throughout this specification, reference to the terms "certain embodiments," "one embodiment," "some embodiments," "illustrative embodiments," "examples," "specific examples," or "some examples" means that a specific feature, structure, material, or characteristic described in conjunction with the embodiment or example is included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

[0052] The present application discloses a method for improving metadata storage and query efficiency. Figure 1, including: establishing a metamodel for each type of metadata, wherein the metamodel includes objects, attributes and relationships of the metadata; obtaining the heat value of each metamodel based on the historical information of the metamodel, dynamically partitioning the metamodel according to the heat value, and adjusting the index strategy, storage strategy and cache strategy according to the dynamic heat partition of the metamodel; monitoring the storage and query process in real time, analyzing the indicator data of different dynamic heat partitions in real time, and adjusting the metamodel according to the analysis results. In the above embodiment, first, based on the Common Warehouse Metamodel Metamodel) abstracts and generalizes various metadata objects, attributes and relationships, thereby improving the efficiency of metadata management. At the same time, by judging the popularity of each metamodel based on the category and historical information of the metamodel and measuring it through the popularity value, the metamodel is dynamically partitioned according to the popularity value, and the index strategy, storage strategy and cache strategy are adjusted according to the dynamic popularity partition of the metamodel. Through the above process, the corresponding management method can be dynamically adjusted according to the popularity status of the metamodel, thereby improving the efficiency of metadata storage and query as a whole. At the same time, this embodiment also monitors the storage and query process in real time, analyzes the indicator data of different dynamic popularity partitions monitored in real time, and adjusts the metamodel according to the analysis results. Through the above method, the relevant parameters of each metamodel can be dynamically adjusted, thereby realizing real-time optimization of the metamodel, which also improves the efficiency of metadata storage and query.

[0053] It should be noted that the Common Warehouse Metamodel in the above embodiment is a metadata management standard for the data warehouse and business intelligence fields. Its specific process of abstracting and generalizing metadata objects, attributes and relationships belongs to the existing technology and will not be further described here; in addition, although the process of partitioning the metamodel according to the heat value in this embodiment is dynamic, in actual operation, the partition will only be adjusted when there is a large change in the metamodel heat value.

[0054] In one embodiment, a heat value acquisition process for each metamodel is provided, including: first, acquiring historical data of each metamodel, and acquiring the heat value H(t0) at time point t0 in the historical data, wherein time point t0 is the time point after Δt before the current time point, that is, t-t0=Δt, t is the current time point, and Δt is a fixed value, wherein the selection of Δt is set according to the demand state. In this embodiment, Δt is selected as 24 minutes, and then the periodic coefficient p of the metamodel is acquired according to the historical data. The periodic coefficient p reflects the periodic state of the metamodel. Since the calling demand of some metamodels appears regularly, while the calling demand of some metamodels appears irregularly, the periodic coefficient p can be used to judge when the calling demand of the metamodel is regular, thereby improving the accuracy of the heat value judgment at the current time point, wherein the heat value H(t) at the current time point is acquired through the formula Calculated, where T() is the natural decay function of heat, which can determine the remaining proportion after decay according to the time of heat decay. The natural decay function of heat is set after integrating the heat decay algorithm in the prior art and will not be further detailed here. p is the periodic coefficient, and q(t) is the access curve of the metamodel. is the reference value of visit volume, which is set according to the average visit data of the meta-model, k is the overall slope of q(t) in the period t0~t, is the reference value of the visit volume change rate, which is mainly used to adjust the dimension of the overall slope k. Its corresponding value is 1. In addition, and It is the adjustment coefficient, and its specific value is set after fitting the test data. It is used to adjust the influence weight of the metamodel access volume and change trend on the popularity judgment during the period t0~t. Therefore, through the above-mentioned heat value calculation process, the popularity status and popularity trend of the metamodel at the current time point can be accurately judged, and then the metamodel can be accurately dynamically partitioned. By executing different strategies for different dynamic heat partitions, the efficiency of metadata storage and query is improved.

[0055] In one embodiment, the process of obtaining the periodic coefficient includes: first obtaining the time point corresponding to the peak of q(t) in the preset historical period. This process can be directly implemented by existing technology and is not limited here. Then, the time difference between adjacent time points is obtained. , through the formula Calculate the time difference consistency coefficient S, where m is the number of time differences, i∈[1,m], is the mean of all time differences. Obviously, when the value of the time difference consistency coefficient S is small, it means that the consistency state of obtaining the time difference of adjacent time points is higher. When the consistency state value is high, it means that the calling demand of the metamodel is periodic. Therefore, the size of the heat value can be adjusted by the time difference consistency coefficient S to improve the accuracy of the heat value judgment. On the contrary, when the consistency state value is low, it means that the calling demand of the metamodel is not periodic. Therefore, the influence of the periodic factor is not considered. In the judgment process, the time difference consistency coefficient S is compared with the preset threshold Sthr. The preset threshold Sthr is set after fitting the test data and is used to judge whether the time difference is periodically distributed. Therefore, if S≥Sthr, the metamodel is judged to be a non-periodic metamodel. At the same time, let p=1, that is, the influence of the periodic factor is not considered. If S<Sthr, the metamodel is judged to be a periodic metamodel. Therefore, considering the influence of the periodic factor, it is judged whether the current time point t is in In the interval, is the peak time point closest to the current time point, is a fixed time error value. The size of the fixed time error value is set according to the average value of the deviation range data of the periodic historical data after integration. Therefore, the current time point t is When the time interval is within the range, it means that the probability of the meta-model being called is high in the time interval. Therefore, let p=p1, p1 is the first fixed period coefficient, and its size is set according to the empirical data to satisfy p1>1. The fixed period coefficient p1 is used to increase the size of the heat value at the current time point. On the contrary, the current time point t is not within the range. When it is within the interval, it means that the probability of the metamodel being called within the time interval is low, so let p=p2, p2 is the second fixed periodic coefficient, its size is set according to empirical data, and it satisfies 0<p1<1. The fixed periodic coefficient p1 is used to reduce the size of the heat value at the current time point. Therefore, through the above-mentioned periodic coefficient acquisition process, the size of the heat value can be accurately adjusted according to the periodic state of the metamodel call, thereby improving the accuracy of the heat value calculation and the accuracy of the dynamic heat partitioning.

[0056] In addition, the process of dynamically partitioning the metamodel based on the heat value includes: comparing the heat value H(t) at the current time point with the preset threshold interval, the preset threshold interval is divided according to the numerical distribution range of the heat values ​​of all metamodels, and then the corresponding heat partition is determined according to the threshold interval where the heat value H(t) is located. Subsequently, the corresponding management method can be dynamically adjusted according to the heat status of the metamodel, thereby improving the overall efficiency of metadata storage and query.

[0057] In one embodiment, the process of adjusting the indexing strategy according to the dynamic heat partition of the metamodel includes: first, determining the order priority of the metadata index according to the dynamic heat partition where the metamodel is located, the higher the heat of the dynamic heat partition where the metamodel is located, the higher the order priority of the metamodel index, and by increasing the order priority of the metamodel index of the metamodel with higher heat, the overall query efficiency of the metadata can be improved; at the same time, the process of adjusting the storage strategy according to the dynamic heat partition of the metamodel includes: determining the level of the metadata storage medium according to the dynamic heat partition where the metamodel is located, the higher the heat of the dynamic heat partition where the metamodel is located, the higher the level of the metamodel storage medium, wherein the levels of the storage medium include in-memory database (Redis), memory cache, SSD, mechanical hard disk (HDD), object storage, archive storage, etc. High-level storage media include in-memory database (Redis), memory cache, mid-level storage media include SSD, and low-level storage media include mechanical hard disk (HDD), object storage and archive storage. Through the above process, the metamodel storage medium level of the metamodel with higher popularity can be increased, thereby improving the overall storage efficiency of the metadata; in addition, the process of adjusting the cache strategy according to the dynamic heat partition of the metamodel includes: determining the cache priority of the metadata according to the dynamic heat partition where the metamodel is located, the higher the heat of the dynamic heat partition where the metamodel is located, the higher the cache priority of the metamodel, and by increasing the metamodel cache priority of the metamodel with higher heat, the overall query efficiency of the metadata can be improved.

[0058] In one embodiment, the process of analyzing the indicator data of real-time monitoring of different dynamic heat partitions includes: monitoring the query delay and index efficiency of different dynamic heat partition metamodels, judging whether there is an abnormal metamodel based on the monitoring results, and optimizing the model when an abnormal metamodel exists. Through the above process, on the basis of partitioning the metamodel, through the monitoring process of each dynamic heat partition metamodel, the efficiency anomaly of the metamodel in the actual calling process can be accurately judged, and then it can be optimized in time to improve the efficiency of metadata query.

[0059] The process of determining whether there is an abnormal meta-model based on the monitoring results includes: obtaining the performance coefficient F of the meta-model based on the query delay and index efficiency, ; where L is the query delay, is the indexing efficiency, u is the system hardware capability constant, which is related to system CPU main frequency, disk IOPS and other parameters. It is set according to the test data under the same hardware state. C is the maximum total system resources, which is obtained according to the system hardware parameters. It is the system resource occupancy, which is obtained based on the real-time running parameters of the current coefficient. Therefore, the efficiency status of the metamodel calling process can be judged by the size of the performance coefficient F, and the performance coefficient F of the metamodel is compared with the threshold Ft of its corresponding dynamic heat partition. The threshold Ft is set according to the fitting of multiple groups of test data under the dynamic heat partition. Therefore, when F≤Ft, it means that the calling state of the metamodel is less efficient, and the metamodel is judged to be an abnormal metamodel. By obtaining the abnormal metamodel, an accurate reference basis is provided in the subsequent optimization process. Among them, the process of model optimization of the abnormal metamodel includes adjusting the index and reducing redundant indexes. The above method can improve the efficiency of metamodel calling, that is, improve the efficiency of metadata query.

[0060] It should be noted that the process of optimizing the abnormal metamodel is mainly achieved through manual adjustment. The identification of abnormal metamodels can effectively assist managers in making judgments.

[0061] Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limitations on the present application. Ordinary technicians in this field can change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. A method for improving metadata storage and query efficiency, characterized in that: include: Establishing a meta-model for each type of metadata, wherein the meta-model includes objects, attributes, and relationships of the metadata; Obtain the popularity value of each metamodel based on the historical information of the metamodel, dynamically partition the metamodel based on the popularity value, and adjust the indexing strategy, storage strategy, and cache strategy based on the dynamic popularity partitioning of the metamodel; Monitor the storage and query process in real time, analyze the indicator data of different dynamic heat partitions in real time, and adjust the meta-model based on the analysis results.

2. A method for improving metadata storage and query efficiency according to claim 1, characterized in that: The process of obtaining the heat value of each metamodel includes: Obtain the historical data of each meta-model and obtain the heat value H(t0) at time point t0 in the historical data, t-t0=Δt, t is the current time point, Δt is a fixed value; Obtain the periodicity coefficient p of the meta-model based on historical data; By formula Calculate the heat value H(t) at the current time point; Among them, T() is the natural decay function of heat, p is the periodic coefficient, q(t) is the visit curve of the meta-model, is the reference value of visit volume, k is the overall slope of q(t) in the period t0~t, is the reference value of the visit volume change rate. and is the adjustment factor.

3. The method for improving metadata storage and query efficiency according to claim 2, characterized in that: The process of obtaining the periodic coefficient includes: Get the time point corresponding to the peak of q(t) in the preset historical period, and get the time difference between adjacent time points , through the formula Calculate the time difference consistency coefficient S, where m is the number of time differences, i∈[1,m], is the mean of all time differences; Compare the time difference consistency coefficient S with the preset threshold Sthr: If S≥Sthr, the metamodel is judged to be a non-periodic metamodel, and p=1; If S < Sthr, the metamodel is judged to be a periodic metamodel, and whether the current time point t is Within the range: If yes, let p=p1, where p1 is the first fixed periodic coefficient and p1>1; If not, let p=p2, where p2 is the second fixed period coefficient and 0<p1<1; in, is the peak time point closest to the current time point, is a fixed time error value.

4. The method for improving metadata storage and query efficiency according to claim 3, characterized in that: The process of dynamically partitioning the metamodel based on heat values ​​includes: The heat value H(t) at the current time point is compared with the preset threshold interval, and the corresponding heat partition is determined according to the threshold interval in which the heat value H(t) lies.

5. The method for improving metadata storage and query efficiency according to claim 4, characterized in that: The process of adjusting the indexing strategy based on the dynamic popularity partitioning of the metamodel includes: The order priority of the metadata index is determined based on the dynamic heat partition where the metamodel is located. The higher the heat of the dynamic heat partition where the metamodel is located, the higher the order priority of the metamodel index. The process of adjusting the storage strategy based on the dynamic heat partitioning of the metamodel includes: The level of the metadata storage medium is determined according to the dynamic heat partition where the metamodel is located. The higher the heat of the dynamic heat partition where the metamodel is located, the higher the level of the metamodel storage medium. The process of adjusting the cache strategy based on the dynamic popularity partitioning of the metamodel includes: The cache priority of the metadata is determined according to the dynamic heat partition where the metamodel is located. The higher the heat of the dynamic heat partition where the metamodel is located, the higher the cache priority of the metamodel.

6. The method for improving metadata storage and query efficiency according to claim 1, characterized in that: The process of analyzing the real-time monitoring indicator data of different dynamic heat partitions includes: Monitor the query latency and indexing efficiency of different dynamic hot partition metamodels, determine whether there are abnormal metamodels based on the monitoring results, and optimize the model if there are abnormal metamodels.

7. The method for improving metadata storage and query efficiency according to claim 6, characterized in that: The process of determining whether there is an abnormal meta-model based on the monitoring results includes: Obtain the performance coefficient F of the metamodel based on query latency and index efficiency, ; Where L is the query delay, is the indexing efficiency, u is the system hardware capability constant, C is the maximum total system resources, The system resource usage; The performance coefficient F of the metamodel is compared with the threshold Ft of its corresponding dynamic heat partition. When F≤Ft, the metamodel is judged to be an abnormal metamodel.

8. The method for improving metadata storage and query efficiency according to claim 7, characterized in that: The process of model optimization for the abnormal meta-model includes adjusting indexes and reducing redundant indexes.

Images