A metadata storage query efficiency monitoring optimization method

Abstract

The application relates to the technical field of metadata query optimization, and discloses a metadata storage query efficiency monitoring optimization method, which comprises the following steps: step one, obtaining historical access data of metadata, matching the metadata with a preset data feature model according to the historical access data, and obtaining a feature model corresponding to the metadata; judging the cold and hot degree of the metadata according to the feature model and data type information of the metadata; step two, setting a corresponding storage strategy for each metadata according to the cold and hot degree; step three, in the metadata query process, the response data and performance indexes are monitored in real time, the real-time monitoring result is compared with a monitoring threshold corresponding to the cold and hot degree of the metadata, and directional optimization is carried out according to the comparison result. According to the preset data feature model, the access trend of the metadata is judged, and a relatively accurate reference basis can be provided for the cold and hot division of the metadata.

Description

Technical Field

[0001] This application relates to the technical field of metadata query optimization, and in particular to a method for monitoring and optimizing metadata storage query efficiency. Background Technology

[0002] With the explosive growth of data volume, from TB to PB / EB, data lakes, IoT, and log data have led to a surge in the number of data objects (files, tables, models). Traditional centralized metadata databases (such as single-machine MySQL) suffer from index expansion, connection bottlenecks, and limited storage capacity, which slows down even simple queries. Therefore, monitoring and optimizing the metadata storage and query process is crucial for metadata management.

[0003] Improving metadata query efficiency is a key system optimization issue, especially in scenarios with huge data volumes and frequent access. Physically separating frequently accessed recent metadata (hot data) from historical metadata (cold data) is an effective way to improve metadata storage and query efficiency. By analyzing the recent access volume of metadata and its own attributes, we can classify metadata as hot or cold. Different storage and query strategies can be adopted according to the degree of hotness or coldness of the metadata, thereby improving the efficiency of metadata storage and query.

[0004] However, different metadata access patterns vary greatly. For example, some metadata is frequently accessed on weekdays / during working hours, which leads to a significant deviation in the accuracy of existing metadata hot and cold classification results. Therefore, how to improve the accuracy of metadata hot and cold classification is the fundamental problem that this invention aims to solve. Summary of the Invention

[0005] To improve the accuracy of hot / cold metadata classification, this application provides a method for monitoring and optimizing metadata storage and query efficiency, employing the following technical solution:

[0006] A method for monitoring and optimizing metadata storage and query efficiency includes:

[0007] Step 1: Obtain historical access data for metadata; match the metadata with a preset data feature model based on the historical access data to obtain the feature model corresponding to the metadata; determine the popularity of the metadata based on the feature model and the data type information of the metadata.

[0008] Step 2: Set corresponding storage strategies for each piece of metadata based on its popularity level;

[0009] Step 3: During the metadata query process, monitor the response data and performance indicators in real time, compare the real-time monitoring results with the monitoring threshold corresponding to the hotness or coldness of the metadata, and perform targeted optimization based on the comparison results;

[0010] The process of determining the "hotness" or "coldness" of metadata includes:

[0011] The data access popularity value Hr of this metadata is obtained based on the access popularity factor in the feature model;

[0012] The adjustment coefficient for real-time access popularity is obtained based on the access type factor in the feature model. , ;

[0013] Based on the data importance, user activity, and data status in the data type information, the first data popularity value H1, the second data popularity value H2, and the third data popularity value H3 are obtained respectively.

[0014] right Sum H1, H2, and H3 to obtain the data popularity value H. The popularity of the data is determined based on the magnitude of the data popularity value.

[0015] The process of step two includes:

[0016] Based on the range of the data popularity value H, the corresponding storage strategy for this metadata is obtained.

[0017] Optionally, the process of matching data feature models includes:

[0018] Fit a real-time curve of historical access volume based on historical access data, obtain the access volume value corresponding to the preset time point, sum the access volume values ​​corresponding to all preset time points, obtain the ratio of the access volume value of each preset time point to the sum of the access volume values, sort the ratios according to the time order of the preset time points, and use the sorted ratios as the access data model of the metadata.

[0019] The access data model of the metadata is compared with the preset data feature model, and the feature model corresponding to the metadata is established based on the comparison results.

[0020] Optionally, the process of comparing the access data model of the metadata with the preset data feature model includes:

[0021] Through mathematical models Calculate the similarity between the access data model of this metadata and the x-th preset data feature model. n is the number of preset time points. This is the ratio of the access data model for this metadata at the t-th preset time point. This represents the ratio of the x-th preset data feature model to the t-th preset time point.

[0022] Get similarity The maximum value will be the similarity. The maximum value is compared with a preset similar value:

[0023] If similarity If the maximum value is less than the preset similarity value, the ratio of the sum of the access volume values ​​to the preset access volume is obtained. The weighted sum of this ratio and the slope K value of the access volume change is used as the access popularity factor of the metadata. The feature model corresponding to the metadata is established based on the access popularity factor.

[0024] Otherwise, the similarity The preset data feature model corresponding to the maximum value is used as the access type factor of the metadata; the ratio of the sum of access volume values ​​to the sum of access volume values ​​corresponding to the preset data feature model is obtained, and this ratio is used as the access popularity factor of the metadata; the feature model corresponding to the metadata is established based on the access type factor and the access popularity factor.

[0025] Optionally, the parameters of the response data include QPS and total latency;

[0026] The performance metrics include the real-time CPU utilization, memory utilization, and disk I / O response time of each service node.

[0027] The process of real-time monitoring of response data and performance metrics includes:

[0028] The parameters of the response data are compared with the corresponding alarm thresholds, which are set according to the hotness or coldness of the metadata.

[0029] If the response data contains parameters that exceed the corresponding alarm threshold, then a root cause analysis and diagnosis process will be performed based on the parameters of the performance indicators.

[0030] If no parameter exceeds the corresponding alarm threshold in the response data, performance analysis is performed based on the parameters of the performance indicators. If the performance analysis result is abnormal, root cause analysis and diagnosis are performed.

[0031] Optionally, the performance analysis process includes:

[0032] Set corresponding alarm thresholds for CPU utilization, memory utilization, and disk I / O response time;

[0033] If any of the parameters in CPU utilization, memory utilization, or disk I / O response time exceed the corresponding alarm threshold, the performance analysis result is judged as abnormal.

[0034] If none of the parameters—CPU utilization, memory utilization, and disk I / O response time—exceed the corresponding alarm threshold, integrate the real-time CPU and memory utilization of each service node to obtain the cumulative CPU and memory usage for each service node during a single query. ;

[0035] Based on the metadata storage policy, obtain the typical cumulative CPU and memory usage of the service node; and store the data for each service node. The cumulative CPU and memory usage of the service node are compared with the typical cumulative usage, and the absolute difference of the cumulative CPU and memory usage of the service node is obtained. When the absolute difference of the cumulative CPU and memory usage of all service nodes is less than the preset threshold, the performance analysis result is judged to be normal; otherwise, the performance analysis result is judged to be abnormal.

[0036] Optionally, the root cause analysis and diagnostic process includes:

[0037] Determine if performance issues are occurring on all service nodes:

[0038] If so, it is determined to be a global problem, and fault analysis is performed based on the parameters corresponding to the performance anomaly.

[0039] If not, then perform fault analysis on the service nodes that have experienced performance issues.

[0040] In summary, this application includes at least one of the following beneficial technical effects:

[0041] This invention matches the metadata with a preset data feature model based on historical access data. The preset data feature model is established based on historical access data of metadata of different modes, and it represents the access volume change characteristics of this type of metadata at different time points. Therefore, by matching the preset data feature model, the closest preset data feature model can be selected. Based on the preset data feature model, the access trend of the metadata can be judged, which can provide a more accurate reference for the hot and cold classification of the metadata. Attached Figure Description

[0042] Figure 1 This is a flowchart illustrating the steps involved in monitoring and optimizing the efficiency of metadata storage and query. Detailed Implementation

[0043] The embodiments of this application are described in detail below, and examples of the embodiments are shown in the accompanying drawings.

[0044] In the description of this specification, the references to "certain embodiments," "one embodiment," "some embodiments," "illustrative embodiment," "example," "specific example," or "some examples" refer to specific features, structures, materials, or characteristics described in connection with the described embodiment or example, which are included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0045] This application discloses a method for monitoring and optimizing metadata storage and query efficiency, referring to... Figure 1 The process includes the following steps: Step 1: Obtain historical access data for metadata, match the metadata with a preset data feature model based on the historical access data, and obtain the feature model corresponding to the metadata; determine the hotness or coldness of the metadata based on the feature model and the data type information of the metadata; Step 2: Set a corresponding storage strategy for each metadata based on its hotness or coldness; Step 3: During the metadata query process, monitor the response data and performance indicators in real time, compare the real-time monitoring results with the monitoring threshold corresponding to the hotness or coldness of the metadata, and perform targeted optimization based on the comparison results.

[0046] In this embodiment, when determining the hotness or coldness of metadata, the metadata is first matched with a preset data feature model based on historical access data. The preset data feature model is established based on historical access data of metadata of different modes, which represents the access volume change characteristics of this type of metadata at different points in time. Therefore, by matching the preset data feature model, the closest preset data feature model can be selected. Based on the access trend of the metadata according to the preset data feature model, a more accurate reference basis can be provided for the hotness or coldness of the metadata.

[0047] It should be noted that the preset data feature model includes a set of metadata preset data feature models for relatively disordered access change trends. This preset data feature model only considers the access popularity factor and does not consider the access type factor.

[0048] In one embodiment, the process of matching the data feature model includes: fitting a real-time curve of historical access volume based on historical access data. Since the real-time curve of historical access volume contains too much information, this embodiment only obtains the access volume value corresponding to a preset time point. The preset time point is selected as the median value of different access periods, and the historical access data is at least more than one week's worth of data. Therefore, the access characteristics of the metadata can be judged through historical access data. Then, the access volume value corresponding to the preset time point is obtained, and the access volume values ​​corresponding to all preset time points are summed to obtain the ratio of the access volume value of each preset time point to the sum of the access volume values. Obviously, the ratio of different preset time points reflects the access status characteristics of this type of metadata. The ratios are sorted according to the time order of the preset time points, and the sorted ratio results are used as the access data model of the metadata. Therefore, each access data model reflects the change characteristics of each type of metadata in access time. The access data model of the metadata is compared with the preset data feature model, and the feature model corresponding to the metadata is established based on the comparison results.

[0049] The comparison process includes: firstly, using a mathematical model... Calculate the similarity between the access data model of this metadata and the x-th preset data feature model. Where n is the number of preset time points. This is the ratio of the access data model for this metadata at the t-th preset time point. This represents the ratio of the x-th preset data feature model to the t-th preset time point; therefore, the closer the change characteristic of a certain metadata is to the ratio of a certain preset data feature model at each time point, the better. and The smaller the absolute value of the difference, the higher the similarity. The larger the value, the higher the similarity score. The maximum value will be the similarity. The maximum value is compared with a preset similarity value, which reflects the critical state of similarity. This preset similarity value is set based on the test data; if the similarity... If the maximum value is less than the preset similarity value, it indicates that the access change data of this metadata is relatively disordered and does not conform to the preset data feature model. In this case, only the access volume is used as a feature of this metadata. Therefore, the ratio of the sum of access volume values ​​to the preset access volume is obtained. The preset access volume is a fixed standard value set by the user based on the overall access popularity of the metadata. The weighted sum of this ratio and the slope K value of the access volume change is used as the access popularity factor of this metadata. K value is the slope of the access volume change. Therefore, the larger the ratio and the positive K value, the higher the popularity of the metadata. It should be noted that in the process of calculating the access popularity factor, the slope K value of the access volume change is normalized to remove the dimension, and the corresponding weight is set according to the test data. Therefore, the size of the access popularity factor can be used to judge the access popularity of the metadata, and a feature model corresponding to the metadata can be built based on the access popularity factor. Otherwise, the similarity is used. The preset data feature model corresponding to the maximum value is used as the access type factor of the metadata; the ratio of the sum of access volume values ​​to the sum of access volume values ​​corresponding to the preset data feature model is obtained. The access volume corresponding to the preset data feature model is set when the preset data feature model is established, so this parameter is only used as a reference for comparison. This ratio is used as the access popularity factor of the metadata; a feature model corresponding to the metadata is established based on the access type factor and the access popularity factor. Therefore, the feature model reflects the access volume change characteristics and access volume popularity status of the metadata. In addition, the feature model can be used to predict the popularity of metadata, providing an accurate reference for the division of the popularity of metadata.

[0050] In one embodiment, the process of determining the hotness or coldness of metadata includes: obtaining the data access heat value Hr of the metadata based on the access heat factor in the feature model. In this embodiment, different data access heat values ​​Hr are set according to different ranges of the access heat factor; the larger the access heat factor, the higher the corresponding heat value. Then, an adjustment coefficient for the real-time access heat value is obtained based on the access volume change characteristics of the access type factor in the feature model. , Real-time access popularity adjustment coefficient The percentage of visits to this preset data feature model at the current time point is set; the higher the percentage, the higher the access popularity at that time point. A value greater than 1 indicates a lower percentage, which in turn indicates lower access activity at that time. The value is less than 1; it should be noted that in similarity... When the maximum value is less than the preset similarity value, the real-time access popularity value adjustment coefficient is adjusted. The initial value is 1. Then, based on the data importance, user activity, and data status in the data type information, the first data popularity value H1, the second data popularity value H2, and the third data popularity value H3 are obtained respectively. Data importance includes important and unimportant data; user activity includes active and inactive user data; and data status includes data of ongoing and non-ongoing projects. Based on the data type information, the first data popularity value H1, the second data popularity value H2, and the third data popularity value H3 are set for the metadata. When the metadata is unimportant data, inactive user data, or non-ongoing project data, the corresponding first data popularity value H1, the second data popularity value H2, and the third data popularity value H3 are all 0. Sum H1, H2, and H3 to obtain the data popularity value H. Based on the magnitude of the data popularity value, determine the popularity of the metadata. Pre-divide several data popularity value H intervals, and each data popularity value H interval corresponds to a storage strategy. Based on the interval in which the data popularity value H value is located, obtain the corresponding storage strategy for the metadata.

[0051] In one embodiment, the parameters of the response data include QPS and total latency; the parameters of the performance metrics include the real-time CPU utilization, memory utilization, and disk I / O response time of each service node; the process of real-time monitoring of the response data and performance metrics includes: firstly, comparing the parameters of the response data with the corresponding alarm thresholds, the corresponding alarm thresholds being set according to the popularity of the metadata; the higher the popularity of the metadata, the higher the alarm threshold corresponding to QPS, and the lower the alarm threshold corresponding to total latency; when there are parameters in the response data that exceed the corresponding alarm thresholds, then root cause analysis and diagnosis are performed based on the parameters of the performance metrics; when there are no parameters in the response data that exceed the corresponding alarm thresholds, then performance analysis is performed based on the parameters of the performance metrics; when the performance analysis result is abnormal, root cause analysis and diagnosis are performed.

[0052] The performance analysis process includes: setting corresponding alarm thresholds for CPU utilization, memory utilization, and disk I / O response time. This setting can be done using empirical data. If any parameter in these parameters exceeds its corresponding alarm threshold, the performance analysis result is considered abnormal. If no parameter in these parameters exceeds its corresponding alarm threshold, the real-time CPU and memory utilization of each service node are integrated to obtain the cumulative CPU and memory usage for each service node during a single query. The service nodes include API gateway nodes, application service nodes, cache nodes, search service nodes, etc. Each node has different cumulative CPU and memory usage during runtime. Therefore, based on different storage strategies, the parameter changes of the service nodes are queried during metadata storage, and different typical cumulative usage values ​​are set. Thus, the typical cumulative CPU and memory usage of the service node is obtained according to the metadata storage strategy. The typical cumulative CPU and memory usage of each service node is then calculated. The cumulative CPU and memory usage of the service node are compared with the typical cumulative usage to obtain the absolute difference of the cumulative CPU and memory usage of the service node. When the absolute difference of the cumulative usage exceeds a certain level, it indicates that the operation of the service node is abnormal. Therefore, when the absolute difference of the cumulative CPU and memory usage of all service nodes is less than the preset threshold, the performance analysis result is judged to be normal; otherwise, the performance analysis result is judged to be abnormal. The preset threshold is set according to the test data.

[0053] In addition, the root cause analysis and diagnosis process includes: determining whether each service node has a performance abnormality problem; if so, it is determined to be a global problem, and fault analysis is performed based on the parameters corresponding to the performance abnormality problem; if not, fault analysis is performed on the service node with the performance abnormality problem. Through the above process, the metadata storage and query process can be optimized, and the metadata storage and query efficiency can be improved.

[0054] Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of this application.

Claims

1. A method for monitoring and optimizing metadata storage and query efficiency, characterized in that, include: Step 1: Obtain historical access data for metadata; match the metadata with a preset data feature model based on the historical access data to obtain the feature model corresponding to the metadata. The popularity of the metadata is determined based on the feature model and the data type information of the metadata. Step 2: Set corresponding storage strategies for each piece of metadata based on its popularity level; Step 3: During the metadata query process, monitor the response data and performance indicators in real time, compare the real-time monitoring results with the monitoring threshold corresponding to the hotness or coldness of the metadata, and perform targeted optimization based on the comparison results; The process of determining the "hotness" or "coldness" of metadata includes: The data access popularity value Hr of this metadata is obtained based on the access popularity factor in the feature model; The adjustment coefficient for real-time access popularity is obtained based on the access type factor in the feature model. , ; Based on the data importance, user activity, and data status in the data type information, the first data popularity value H1, the second data popularity value H2, and the third data popularity value H3 are obtained respectively. right Sum H1, H2, and H3 to obtain the data popularity value H. The popularity of the data is determined based on the magnitude of the data popularity value. The process of step two includes: Based on the range of the data popularity value H, the corresponding storage strategy for this metadata is obtained; The process of matching data feature models includes: Fit a real-time curve of historical access volume based on historical access data, obtain the access volume value corresponding to the preset time point, sum the access volume values ​​corresponding to all preset time points, obtain the ratio of the access volume value of each preset time point to the sum of the access volume values, sort the ratios according to the time order of the preset time points, and use the sorted ratios as the access data model of the metadata. The access data model of the metadata is compared with the preset data feature model, and the feature model corresponding to the metadata is established based on the comparison results. The process of comparing the access data model of this metadata with the preset data feature model includes: Through mathematical models Calculate the similarity between the access data model of this metadata and the x-th preset data feature model. n is the number of preset time points. This is the ratio of the access data model for this metadata at the t-th preset time point. This represents the ratio of the x-th preset data feature model to the t-th preset time point. Get similarity The maximum value will be the similarity. The maximum value is compared with a preset similar value: If similarity If the maximum value is less than the preset similarity value, the ratio of the sum of the access volume values ​​to the preset access volume is obtained. The weighted sum of this ratio and the slope K value of the access volume change is used as the access popularity factor of the metadata. The feature model corresponding to the metadata is established based on the access popularity factor. Otherwise, the similarity The preset data feature model corresponding to the maximum value is used as the access type factor of the metadata; the ratio of the sum of access volume values ​​to the sum of access volume values ​​corresponding to the preset data feature model is obtained, and this ratio is used as the access popularity factor of the metadata; the feature model corresponding to the metadata is established based on the access type factor and the access popularity factor.

2. The method for monitoring and optimizing metadata storage and query efficiency according to claim 1, characterized in that, The parameters of the response data include QPS and total latency; The performance metrics include the real-time CPU utilization, memory utilization, and disk I / O response time of each service node. The process of real-time monitoring of response data and performance metrics includes: The parameters of the response data are compared with the corresponding alarm thresholds, which are set according to the hotness or coldness of the metadata. If the response data contains parameters that exceed the corresponding alarm threshold, then a root cause analysis and diagnosis process will be performed based on the parameters of the performance indicators. If no parameter exceeds the corresponding alarm threshold in the response data, performance analysis is performed based on the parameters of the performance indicators. If the performance analysis result is abnormal, root cause analysis and diagnosis are performed.

3. The method for monitoring and optimizing metadata storage and query efficiency according to claim 2, characterized in that, The performance analysis process includes: Set corresponding alarm thresholds for CPU utilization, memory utilization, and disk I / O response time; If any of the parameters in CPU utilization, memory utilization, or disk I / O response time exceed the corresponding alarm threshold, the performance analysis result is judged as abnormal. If none of the parameters—CPU utilization, memory utilization, and disk I / O response time—exceed the corresponding alarm threshold, integrate the real-time CPU and memory utilization of each service node to obtain the cumulative CPU and memory usage for each service node during a single query. ; Based on the metadata storage policy, obtain the typical cumulative CPU and memory usage of the service node; and store the data for each service node. The cumulative CPU and memory usage of the service node are compared with the typical cumulative usage, and the absolute difference of the cumulative CPU and memory usage of the service node is obtained. When the absolute difference of the cumulative CPU and memory usage of all service nodes is less than the preset threshold, the performance analysis result is judged to be normal; otherwise, the performance analysis result is judged to be abnormal.

4. The method for monitoring and optimizing metadata storage and query efficiency according to claim 3, characterized in that, The root cause analysis and diagnostic process includes: Determine if performance issues are occurring on all service nodes: If so, it is determined to be a global problem, and fault analysis is performed based on the parameters corresponding to the performance anomaly. If not, then perform fault analysis on the service nodes that have experienced performance issues.

Images