A method and system for processing operation index data

Abstract

The present invention provides a method and system for processing operation indicator data. According to this solution, time intervals are divided according to a specified time granularity, and a corresponding storage solution is constructed for each time interval, wherein a new storage solution is constructed based on the storage solution of the historical time interval and its corresponding index set. Based on the storage solution, an index set containing at least one index is created in a specified database, and each index has a specified capacity. When the operation indicator data to be stored is obtained, the corresponding storage solution is determined according to the collection time of the operation indicator data to be stored, and then the obtained operation indicator data is stored in the index corresponding to the determined storage solution. The operation indicator data is distributedly stored to improve the efficiency and timeliness of querying the operation indicator data.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular to a method and system for processing operation index data. Background Art

[0002] In cloud operation and maintenance work, it is necessary to periodically collect a large amount of operating indicator data of monitored objects, store the collected operating indicator data on a medium that is easy to query, and use the operating indicator data to monitor the operating status of each monitored object.

[0003] Currently, a single dataset is typically used to store the operational metrics of each monitored object. However, with the development of cloud computing technology and the industry, the scale of clouds has grown, and the single dataset used to store operational metrics data has become increasingly bloated. This results in low efficiency and poor timeliness when querying operational metrics data from a single dataset. Summary of the Invention

[0004] In view of this, an embodiment of the present invention provides a method and system for processing operation index data to solve the problems of low efficiency and poor timeliness of querying operation index data in the existing method of storing operation index data.

[0005] To achieve the above objectives, the embodiments of the present invention provide the following technical solutions:

[0006] A first aspect of an embodiment of the present invention discloses a method for processing operating indicator data, the method comprising:

[0007] When first operating indicator data to be stored is collected, a target time interval within which the collection time of the first operating indicator data falls is determined, where the target time interval is a time interval divided according to a specified time granularity;

[0008] If a first storage scheme corresponding to the target time interval does not exist in the storage component, determining the first storage scheme as a default storage scheme, and determining a second storage scheme for a time interval next to the target time interval as the default storage scheme, wherein the storage scheme is used to create an index set including at least one index in a specified database, the index having a specified capacity, and the index being used to store operating indicator data of a preset indicator type of a specified monitoring object;

[0009] Saving the first storage solution and the second storage solution in the storage component, and storing the first operating indicator data in an index of the index set corresponding to the first storage solution;

[0010] If a first storage solution corresponding to the target time interval exists in the storage component, the first operating indicator data is stored in an index of an index set corresponding to the first storage solution, and the following steps are performed:

[0011] The following steps include:

[0012] If the second storage scheme corresponding to the next time interval of the target time interval does not exist in the storage component, the second storage scheme is constructed and stored in the storage component using the third storage scheme of the previous time interval of the target time interval and the index set corresponding to the third storage scheme.

[0013] Preferably, the storage component includes a disk and a cache;

[0014] The process of saving the first storage scheme and the second storage scheme in the storage component includes:

[0015] Persistently save the first storage solution and the second storage solution in a disk, and back up the first storage solution and the second storage solution in a cache in the form of a hash table;

[0016] The keys in the hash table are monitoring objects and indicator types, and the values in the hash table are indexes.

[0017] Preferably, the process of storing the first operating indicator data in the index of the index set corresponding to the first storage scheme includes:

[0018] Using the monitoring object and indicator type corresponding to the first operating indicator data as keys, query the indexes contained in the index set corresponding to the first storage solution in the cache to determine and obtain the corresponding first target index;

[0019] The first operating indicator data is stored in the first target index in the designated database.

[0020] Preferably, if the storage component does not have a second storage solution corresponding to the next time interval of the target time interval, using the third storage solution of the previous time interval of the target time interval and the index set corresponding to the third storage solution to construct the second storage solution and store it in the storage component, including:

[0021] If the second storage solution corresponding to the next time interval of the target time interval does not exist in the storage component, the number of indexes to be created in the next time interval is determined by using the total amount of data stored in the index set corresponding to the third storage solution in the previous time interval of the target time interval in combination with the specified capacity;

[0022] Analyzing the distribution of the operating indicator data stored in the index set corresponding to the third storage solution;

[0023] The second storage solution is constructed by using the distribution and the number of indexes to be created in the next time interval and stored in the storage component.

[0024] Preferably, the method further comprises:

[0025] In the storage component, query the storage solution whose construction time is greater than the preset time length;

[0026] The storage solutions whose construction time is greater than the preset time length are deleted from the storage component, and the index sets corresponding to the storage solutions whose construction time is greater than the preset time length are deleted from the designated database.

[0027] Preferably, the method further comprises:

[0028] Upon receiving the first query request, parsing the first query request to obtain at least the monitoring object to be queried, the indicator type to be queried, and the time range to be queried;

[0029] If a fourth storage solution corresponding to at least one time interval covered by the time range to be queried is found in the disk, the index set corresponding to the fourth storage solution in the cache is queried using the monitoring object to be queried and the indicator type to be queried as a key to determine a corresponding second target index;

[0030] A second query request is sent to the second target index in the specified database to obtain and output indicator data results, wherein the indicator data results at least include: the second operating indicator data of the monitored object to be queried within the time range to be queried, and the indicator type of the second operating indicator data is the indicator type to be queried.

[0031] A second aspect of an embodiment of the present invention discloses a system for processing operating indicator data, the system comprising:

[0032] a first determining unit, configured to, when first operating indicator data to be stored is collected, determine a target time interval within which the collection time of the first operating indicator data falls, the target time interval being a time interval divided according to a specified time granularity;

[0033] a second determining unit, configured to, if the first storage scheme corresponding to the target time interval does not exist in the storage component, determine the first storage scheme as a default storage scheme, and determine a second storage scheme for a time interval subsequent to the target time interval as the default storage scheme, wherein the default storage scheme is used to create an index set comprising at least one index in a specified database, the index having a specified capacity, and the index being used to store operating indicator data of a preset indicator type of a specified monitoring object;

[0034] a storage unit, configured to save the first storage scheme and the second storage scheme into the storage component, and store the first operating indicator data into an index of the index set corresponding to the first storage scheme;

[0035] The storage unit is further configured to: if a first storage scheme corresponding to the target time interval exists in the storage component, store the first operating indicator data in an index of an index set corresponding to the first storage scheme, and execute the construction unit;

[0036] The construction unit is used to construct the second storage scheme and store it in the storage component by using the third storage scheme of the previous time interval of the target time interval and the index set corresponding to the third storage scheme if the second storage scheme corresponding to the next time interval of the target time interval does not exist in the storage component.

[0037] Preferably, the storage component includes a disk and a cache;

[0038] The storage unit for saving the first storage scheme and the second storage scheme in the storage component is specifically used to: persist the first storage scheme and the second storage scheme on the disk, and back up the first storage scheme and the second storage scheme in the storage component cache in the form of a hash table; wherein the keys in the hash table are monitoring objects and indicator types, and the values in the hash table are indexes.

[0039] Preferably, the storage unit for storing the first operating indicator data in the index of the index set corresponding to the first storage scheme is specifically used to: use the monitoring object and indicator type corresponding to the first operating indicator data as keys to query the index contained in the index set corresponding to the first storage scheme in the cache to determine the corresponding first target index; and store the first operating indicator data in the first target index in the designated database.

[0040] Preferably, the building block comprises:

[0041] a determination module configured to determine, if the second storage solution corresponding to the next time interval of the target time interval does not exist in the storage component, the number of indexes to be created in the next time interval by using the total amount of data stored in the index set corresponding to the third storage solution of the previous time interval of the target time interval and the specified capacity;

[0042] An analysis module, configured to analyze the distribution of the operating indicator data stored in the index set corresponding to the third storage solution;

[0043] A construction module is used to construct the second storage solution by using the distribution and the number of indexes to be created in the next time interval and store the solution in the storage component.

[0044] Based on the above-mentioned embodiment of the present invention, a method and system for processing operation indicator data is provided, which comprises: when first operation indicator data to be stored is collected, determining the target time interval in which the collection time of the first operation indicator data falls; if the first storage scheme corresponding to the target time interval does not exist in the storage component, determining the first storage scheme as the default storage scheme, and determining the second storage scheme for the next time interval of the target time interval as the default storage scheme; saving the first storage scheme and the second storage scheme in the storage component, and storing the first operation indicator data in the index of the index set corresponding to the first storage scheme; if the first storage scheme exists in the storage component, storing the first operation indicator data in the index of the index set corresponding to the first storage scheme; if the second storage scheme corresponding to the next time interval of the target time interval does not exist in the storage component, using the third storage scheme of the previous time interval of the target time interval and the index set corresponding to the third storage scheme, constructing the second storage scheme and storing it in the storage component. Using this scheme, the time interval is divided according to the specified time granularity, and a corresponding storage scheme is constructed for each time interval, wherein a new storage scheme is constructed based on the storage scheme of the historical time interval and its corresponding index set. Based on the storage plan, an index set containing at least one index is created in a specified database, with each index having a specified capacity. When the operating indicator data to be stored is obtained, the corresponding storage plan is determined based on the collection time, and the obtained operating indicator data is then stored in the index corresponding to the determined storage plan. This distributed storage of operating indicator data improves the efficiency and timeliness of operating indicator data queries. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0046] Figure 1 A flowchart of a method for processing operating indicator data provided by an embodiment of the present invention;

[0047] Figure 2 A flowchart of querying operation indicator data provided by an embodiment of the present invention;

[0048] Figure 3 A schematic diagram of a method for processing operating indicator data provided by an embodiment of the present invention;

[0049] Figure 4 This is a structural block diagram of a system for processing operation indicator data provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0050] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0051] In this application, the terms "comprises," "comprising," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or apparatus that includes a list of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not preclude the presence of additional identical elements in the process, method, article, or apparatus that includes the element.

[0052] As we know from the background, cloud operations and maintenance require periodic collection of operational indicator data for a large number of monitored objects. Currently, a single dataset is typically used to store the operational indicator data for each monitored object. However, as the scale of the cloud grows, the single dataset used to store operational indicator data is becoming increasingly bloated, resulting in low efficiency and poor timeliness when querying operational indicator data from this single dataset.

[0053] Therefore, an embodiment of the present invention provides a method and system for processing operation indicator data, which divides the time interval according to a specified time granularity, and constructs a corresponding storage scheme for each time interval, wherein a new storage scheme is constructed based on the storage scheme of the historical time interval and its corresponding index set. Based on the storage scheme, an index set containing at least one index is created in a specified database, and each index has a specified capacity. When the operation indicator data to be stored is obtained, the corresponding storage scheme is determined according to the collection time of the operation indicator data to be stored, and then the obtained operation indicator data is stored in the index corresponding to the determined storage scheme. The operation indicator data is distributedly stored to improve the efficiency and timeliness of querying the operation indicator data.

[0054] See also Figure 1 , which shows a flow chart of a method for processing operating indicator data provided by an embodiment of the present invention, the processing method includes:

[0055] Step S101: when first operating indicator data to be stored is collected, a target time interval in which the collection time of the first operating indicator data falls is determined.

[0056] It should be noted that a specified time granularity is pre-determined and then divided into multiple time intervals based on it. This specified time granularity can be seconds, minutes, hours, or days. In this solution, there is no specific limit on the specified time granularity and it can be set according to actual needs. The start and end points of the time interval are both timestamps. The target time interval is the time interval divided according to the specified time granularity.

[0057] In the specific implementation of step S101, when the first operating indicator data to be stored is collected, the collection time of the first operating indicator data is determined according to the collection time field in the first operating indicator data; then, it is determined in which time interval the collection time is located, and the time interval in which the collection time is located is the target time interval.

[0058] It can be understood that the first operating indicator data is operating indicator data of a certain indicator type collected from a certain monitoring object.

[0059] Step S102: Determine whether a first storage solution corresponding to the target time interval exists in the storage component. If the first storage solution corresponding to the target time interval does not exist in the storage component, steps S103 and S104 are executed. If the first storage solution corresponding to the target time interval exists in the storage component, step S105 is executed, and steps S106 and S107 are executed.

[0060] It should be noted that, as can be seen from the above, each time interval is pre-divided into specified time intervals, and a corresponding storage solution is required for each time interval. In other words, a storage solution is required for each storage solution. A storage solution is used to create an index set in a specified database, containing at least one index with a specified capacity. This index is used to store operational indicator data of a preset indicator type for a specified monitoring object.

[0061] It is understandable that, since the operating indicator data has a time series characteristic, the time intervals are pre-divided according to the specified time granularity. For each time interval, a corresponding storage scheme needs to be constructed and an index set is created in the specified database based on the storage scheme. For a certain time interval, the operating indicator data collected within the time interval is stored in the index set corresponding to the time interval; a new index set needs to be created for each time interval (based on the constructed storage scheme); for example: assuming the current time interval is S n , S n The previous time interval is S n-1 , S n The next time interval is S n+1 , S n 、S n-1 and S n An index set must be created.

[0062] To better understand the following step S102 and the contents of the following steps, the specific contents of the aforementioned designated database, index, and storage solution are first explained.

[0063] Notes on specifying databases and indexes:

[0064] The designated database can be a time series database, such as ElasticSearch (ES) or InfluxDB. Indexes in the designated database can be used to store operational indicator data. If the designated database is ES, the index can be an ES index. An index set consists of at least one index, each of which is used to store operational indicator data of a preset indicator type for a specified monitoring object. In other words, each index in an index set can be used to store operational indicator data within a specific range, meaning each index can store operational indicator data of certain indicator types for certain monitoring objects.

[0065] It is understandable that in order to ensure the efficiency of querying operating indicator data from the index, it is necessary to stipulate that the capacity of each index is a specified capacity. The capacity of the index specifically refers to: the upper limit of the amount of data that the index can store. For example: assuming that the index has a higher query efficiency when storing less than 100 million operating indicator data, the index capacity can be set to C = 100000000, where C is the specified capacity mentioned above. In actual applications, the specific value of the specified capacity can be set according to actual needs and is not limited here.

[0066] Notes on storage solutions:

[0067] The storage plan is used to create an index set in a specified database; the storage plan defines the number of indexes contained in the index set, and defines which indicator types of operating indicator data of which monitoring objects each index in the index set is used to store; that is, after the index set is created according to the storage plan, each index in the index set can be used to store the operating indicator data of the preset indicator types of the specified monitoring object.

[0068] The specific content of the storage solution is: using the backpack algorithm (just as an example), the operating indicator data of different monitored objects are allocated to I indexes with a capacity of C (i.e., the specified capacity) for storage. Among them, the operating indicator data from the same monitored object are preferentially allocated to the same index for storage. This allocation method can effectively reduce the number of queries when querying the operating indicator data.

[0069] Since the monitoring objects and the operating indicator data to be collected will change with the actual needs (such as the actual needs of the monitoring system), before creating a new index set each time, you need to build a storage plan first, and then create an index set based on the storage plan.

[0070] It's important to note that all constructed storage solutions must be saved to the storage component, which consists of at least a disk and a cache, which can be either an in-memory cache or a sidecar cache component (such as Redis). Specifically, the constructed storage solution is persisted to disk and backed up in the cache in the form of a hash table (key-value hash table).

[0071] It can be understood that since the storage scheme defines which index in the index set is used to store which monitoring objects and which indicator types of operating indicator data, for the hash table corresponding to the storage scheme backed up in the cache, the key in the hash table is the monitoring object and indicator type, and the value in the hash table is the index. In other words, for a certain operating indicator data to be stored, after determining the storage scheme corresponding to the time interval in which the collection time is located, the monitoring object and indicator type corresponding to the operating indicator data are used as the key to query the index corresponding to the storage scheme in the cache, and the value corresponding to the key can be queried; the operating indicator data is stored in the specified database at the index corresponding to the queried value. By saving the storage scheme in the cache in the form of a hash table, the scope of the query index can be narrowed, and it can be quickly and efficiently determined in which index the operating indicator data needs to be stored.

[0072] It should be noted that the storage scheme defines which index in the index set is used to store which indicator type of operating indicator data of which monitoring objects. However, in actual applications, on the one hand, when configuring the collection of operating indicator data of a new indicator type for a certain monitoring object, it is necessary to update the storage scheme stored in the storage component and store the operating indicator data of the new indicator type in the index corresponding to the monitoring object. After updating the storage scheme, it can be determined in which index the operating indicator data of the new indicator type is stored; on the other hand, an index is reserved in the index set corresponding to the storage scheme. When configuring the collection of operating indicator data of a new monitoring object, it is necessary to update the storage scheme stored in the storage component and store the operating indicator data of the new monitoring object in the reserved index.

[0073] The above content is about specifying the database, index, and storage scheme.

[0074] During the specific implementation of step S102, a query is performed in the storage component to determine whether a first storage solution corresponding to the target time interval exists in the storage component. If no first storage solution corresponding to the target time interval is found in the storage component, it indicates that the initialization operation phase is currently in progress, i.e., the target time interval is the first time interval divided, and steps S103 and S104 are executed. If a first storage solution corresponding to the target time interval is found in the storage component, step S105 is executed, and steps S106 and S107 are executed.

[0075] Step S103: if the first storage solution corresponding to the target time interval does not exist in the storage component, determine the first storage solution as the default storage solution, and determine the second storage solution for the next time interval of the target time interval as the default storage solution.

[0076] During the specific implementation of step S103, if the storage component does not have a first storage solution corresponding to the target time interval, a pre-established default storage solution is used as the first storage solution corresponding to the target time interval, i.e., the first storage solution is determined to be the default storage solution. Simultaneously, a second storage solution for the time interval immediately following the target time interval is determined to be the default storage solution, which is equivalent to pre-establishing the second storage solution. Subsequently, when operating indicator data is collected within the time interval immediately following the target time interval, the collected operating indicator data can be directly stored using the second storage solution to avoid affecting normal data writing.

[0077] In some specific embodiments, an index set containing at least one index can be created in a specified database based on a default storage scheme; it should be noted that, since the amount of operating indicator data collected during the initial operation phase is usually less than the specified capacity of an index, the index set created based on the default storage scheme may contain only one index (for example only). When the index set created based on the default storage scheme contains only one index, after determining that the first storage scheme is the default storage scheme, all monitored objects can be used as new monitored objects, and at this time all collected operating indicator data are stored in an index corresponding to the first storage scheme. It should be noted that the number of indexes in the index set created based on the default storage scheme mentioned above is only for example, and the number of indexes in the index set created based on the default storage scheme can also be multiple, which is not specifically limited here.

[0078] For example: assuming that the first storage scheme is P0 and the second storage scheme is P1, if P0 does not exist in the storage component, P0 is determined as the default storage scheme, and P1 is determined as the default storage scheme. The number of indexes in the index set created based on the default storage scheme is I=1.

[0079] Step S104: Save the first storage scheme and the second storage scheme into the storage component, and store the first operating indicator data into the index of the index set corresponding to the first storage scheme.

[0080] From the above content, it can be seen that after constructing the storage plan, the constructed storage plan needs to be saved in the storage component. In the specific implementation process of step S104, the storage component includes a disk and a cache, and the first storage plan and the second storage plan are persisted on the disk, and the first storage plan and the second storage plan are backed up in the cache in the form of a hash table. The key in the hash table is the monitoring object and indicator type, and the value in the hash table is the index.

[0081] Using the monitoring object and indicator type corresponding to the first operating indicator data as the key, query the index contained in the index set corresponding to the first storage solution in the cache to determine the corresponding first target index (or obtain the name of the first target index). That is, using the monitoring object and indicator type corresponding to the first operating indicator data as the key, query the index contained in the index set corresponding to the first storage solution in the cache to query the value corresponding to the key. The index corresponding to the queried value is the first target index. Store the first operating indicator data in the first target index of the specified database.

[0082] Step S105: when there is a first storage solution corresponding to the target time interval in the storage component, the first operation indicator data is stored in the index of the index set corresponding to the first storage solution.

[0083] In the specific implementation of step S105, when there is a first storage solution corresponding to the target time interval in the storage component, the monitoring object and indicator type corresponding to the first operating indicator data are used as keys to query the index contained in the index set corresponding to the first storage solution in the cache to determine the corresponding first target index (or obtain the name of the first target index). That is, the monitoring object and indicator type corresponding to the first operating indicator data are used as keys to query the index contained in the index set corresponding to the first storage solution in the cache, and the value corresponding to the key is queried. The index corresponding to the queried value is the first target index. The first operating indicator data is stored in the first target index of the specified database.

[0084] Step S106: Determine whether the storage component has a second storage solution corresponding to the time interval immediately following the target time interval. If the storage component does not have a second storage solution corresponding to the time interval immediately following the target time interval, proceed to step S107. If the storage component has a second storage solution corresponding to the time interval immediately following the target time interval, the process ends.

[0085] It should be noted that when storing the first operating indicator data in the index of the index set corresponding to the first storage scheme, in order to ensure the normal writing of data in the subsequent process, it is necessary to pre-build a second storage scheme corresponding to the next time interval of the target time interval. In the process of specifically implementing step S106, it is determined whether the second storage scheme corresponding to the next time interval of the target time interval exists in the storage component. If the second storage scheme does not exist in the storage component, step S107 is executed to build the second storage scheme. If the second storage scheme exists in the storage component, it means that the second storage scheme has been pre-built, and the process ends.

[0086] It is understood that, because the construction process of the storage solution may lag behind the writing of the operational indicator data, the aforementioned determination of whether the second storage solution exists in the storage component specifically refers to: determining whether the second storage solution has already been generated in the storage component, or determining whether the second storage solution is currently being generated in the storage component. When the second storage solution already exists in the storage component or the second storage solution is currently being generated, it indicates that the second storage solution exists in the storage component.

[0087] Step S107: construct a second storage solution by using the third storage solution of the previous time interval of the target time interval and the index set corresponding to the third storage solution, and store it in the storage component.

[0088] In the process of implementing step S107, if the second storage scheme corresponding to the next time interval of the target time interval does not exist in the storage component, the total amount of data stored in the index set corresponding to the third storage scheme of the previous time interval of the target time interval is used, combined with the specified capacity of the index, to determine the number of indexes to be created in the next time interval of the target time interval.

[0089] Specifically, assuming the target time interval is S n , the previous time interval of the target time interval is S n-1 , the next time interval of the target time interval is S n+1 ; The first storage scheme is P n , the third storage solution is P n-1 , the second storage scheme is P n+1 ; Use formula (1) to determine S n+1 The number of indexes to create I n+1 .

[0090] I n+1 =T n-1 / C+1(1)

[0091] In formula (1), T n-1 For the specified database and P n-1 The total amount of data stored in the corresponding index set, where C is the specified capacity of the index.

[0092] After determining the number of indexes that need to be created in the next time interval of the target time interval, the distribution of the operating indicator data stored in the index set corresponding to the third storage solution is analyzed from the three dimensions of collection time, monitoring object, and operating indicator data, and the distribution is corrected using the backpack algorithm; the second storage solution P is constructed based on the distribution after correction and the number of indexes that need to be created in the next time interval of the target time interval. n+1 And P n+1It should be noted that the manner of storing the second storage solution in the storage component can be found in the above step S104 and will not be described in detail here.

[0093] It should be further explained that, since the operating indicator data stored in the target time interval may not be complete, and the index involved in the first storage scheme is in the state of writing a large amount of operating indicator data, the operating indicator data stored in the target time interval is insufficient for analysis to construct a second storage scheme for the next time interval; therefore, the operating indicator data stored in the index set corresponding to the third storage scheme of the previous time interval of the target time interval is used as the basis for constructing the second storage scheme; here, the operating indicator data stored in the index set corresponding to the third storage scheme is equivalent to historical data.

[0094] As can be seen from the above, the storage component (disk and cache) stores storage solutions corresponding to various time intervals. To conserve storage component and designated database resources, data in these components and designated databases needs to be periodically cleared. Specifically, in the storage component, query for storage solutions whose build time exceeds the preset duration; delete storage solutions whose build time exceeds the preset duration from the storage component; and delete the index sets corresponding to storage solutions whose build time exceeds the preset duration from the designated database.

[0095] It should be noted that storage solutions with a construction time greater than the preset time and their corresponding index sets can be considered expired data, and expired data usually no longer has business significance; therefore, each storage solution needs to be managed; specifically, storage solutions with a construction time greater than the preset time in the storage component should be regularly cleared, and index sets corresponding to storage solutions with a construction time greater than the preset time in the specified database should be regularly cleared.

[0096] For example: Assuming that the designated database is ES, determine the storage plan whose time from the current time is greater than the preset length; send a delete request to ES to delete the index involved in the storage plan whose time from the current time is greater than the preset length; and delete the storage plan whose time from the current time is greater than the preset length from the disk and memory cache.

[0097] In an embodiment of the present invention, time intervals are divided according to a specified time granularity, and a corresponding storage scheme is constructed for each time interval, wherein a new storage scheme is constructed based on the storage scheme of the historical time interval and its corresponding index set. Based on the storage scheme, an index set containing at least one index is created in a specified database, and each index has a specified capacity. When the operating indicator data to be stored is obtained, the corresponding storage scheme is determined according to the collection time of the operating indicator data to be stored, and then the obtained operating indicator data is stored in the index corresponding to the determined storage scheme. The operating indicator data is distributedly stored to improve the efficiency and timeliness of querying the operating indicator data.

[0098] The above content is about the relevant content of storing the operation index data. Accordingly, the embodiment of the present invention also provides the corresponding content of querying the operation index data. Please refer to Figure 2 , shows a flowchart of querying operation indicator data provided by an embodiment of the present invention, including the following steps:

[0099] Step S201: When a first query request is received, the first query request is parsed to obtain at least the monitoring object to be queried, the indicator type to be queried, and the time range to be queried.

[0100] It should be noted that the query request for querying operational indicator data carries the monitored object, indicator type, and time range. In the specific implementation of step S201, upon receiving a first query request via the indicator data query interface (API), the first query request is parsed to obtain at least the monitored object to be queried, the indicator type to be queried, and the time range to be queried. The first query request can be sent by a client.

[0101] It should be noted that the time range to be queried covers at least one pre-divided time interval, that is, the operating indicator data of a certain indicator type (i.e., the indicator type to be queried) of a certain monitoring object (i.e., the monitoring object to be queried) collected within one or more time intervals can be queried.

[0102] Step S202: Determine whether a fourth storage solution corresponding to at least one time interval covered by the query time range can be found on the disk. If the fourth storage solution cannot be found on the disk, the process ends. If the fourth storage solution can be found on the disk, steps S203 and S204 are executed.

[0103] During the specific implementation of step S202, at least one time interval covered by the time range to be queried is determined, and the disk is searched for a fourth storage solution corresponding to the time interval covered by the time range to be queried. If the fourth storage solution cannot be found on the disk, it indicates that the fourth storage solution has expired or has not yet been constructed. If the fourth storage solution can be found on the disk, steps S203 and S204 are executed.

[0104] It should be noted that the fourth storage solution has expired specifically means that the construction time of the fourth storage solution is greater than the preset time, and the fourth storage solution has been deleted from the disk and the cache.

[0105] Step S203: using the monitored object to be queried and the indicator type to be queried as keys, query the indexes contained in the index set corresponding to the fourth storage solution in the cache to determine and obtain the corresponding second target index.

[0106] As can be seen from the above, the time range to be queried covers at least one pre-divided time interval. If the fourth storage scheme can be queried on the disk, at least one fourth storage scheme can be queried at this time. In the specific implementation of step S203, for each fourth storage scheme queried, the index set corresponding to the fourth storage scheme in the cache is queried using the monitored object to be queried and the type of indicator to be queried as the key to determine the corresponding second target index (or obtain the name of the second target index). For each fourth storage scheme queried, the corresponding second target index can be determined.

[0107] It can be understood that all the determined second target indexes are indexes involved in the operating indicator data to be queried by the first query request.

[0108] Step S204: Send a second query request to the second target index in the specified database to obtain and output the indicator data results.

[0109] In the process of specifically implementing step S204, a second query request is sent to the second target index in the specified database to obtain the indicator data result, which at least includes: the second operating indicator data (such as the indicator value) of the monitored object to be queried within the time range to be queried, and the indicator type of the second operating indicator data is the indicator type to be queried.

[0110] For example: assuming that the first query request needs to query the CPU usage of a virtual machine in the past 30 minutes, and assuming that the time interval is divided into minutes, the monitoring object to be queried is the virtual machine, the indicator type to be queried is CPU usage, and the time range to be queried is 30 minutes; through the above steps S201 to S204, 30 operating indicator data indicating CPU usage can be queried from the specified database.

[0111] After obtaining the indicator data results, the indicator data results can be output in a preset manner, for example: the indicator data results are fed back to the client for display.

[0112] In this embodiment of the present invention, upon receiving a first query request, a query is performed on the index corresponding to the fourth storage scheme in the cache using the monitored object to be queried and the metric type to be queried as keys, thereby quickly determining the second target index associated with the desired operational metric data. A second query request is then sent to the second target index in the designated database to retrieve and output the metric data results, thereby improving the efficiency and timeliness of querying operational metric data.

[0113] Combine Figure 1 and Figure 2 For the contents in Figure 3 , Figure 3 A principle architecture diagram of a method for processing operating indicator data provided by an embodiment of the present invention; wherein, the indicator data collection 100 is used to collect operating indicator data, and calls the "data write" function in the data access layer 300 to write the collected operating indicator data into the index of the corresponding index set in the specified database 400, and the specified database 400 is ElasticSearch; when querying the operating indicator data, the indicator data display 200 calls the "data query" function in the data access layer 300 to query the specified database 400 to obtain the indicator data results and display the indicator data results, and the function includes at least multiple index sets from the specified database 400; the specific implementation method of the "storage solution generation and maintenance" in the data access layer 300 can be found in the above-mentioned embodiment of the present invention. Figure 1 The content related to building and managing storage solutions in the previous section will not be repeated here.

[0114] Corresponding to the method for processing operation index data provided by the above embodiment of the present invention, see Figure 4 , an embodiment of the present invention also provides a structural block diagram of a system for processing operation index data, the processing system comprising: a first determining unit 401, a second determining unit 402, a storage unit 403 and a construction unit 404;

[0115] The first determining unit 401 is configured to determine, when first operating indicator data to be stored is collected, a target time interval in which the collection time of the first operating indicator data falls, where the target time interval is a time interval divided according to a specified time granularity.

[0116] The second determination unit 402 is used to determine the first storage scheme as the default storage scheme when the first storage scheme corresponding to the target time interval does not exist in the storage component, and to determine the second storage scheme for the next time interval of the target time interval as the default storage scheme, wherein the default storage scheme is used to create an index set containing at least one index in a specified database, the index has a specified capacity, and the index is used to store operating indicator data of a preset indicator type of a specified monitoring object.

[0117] The storage unit 403 is configured to save the first storage scheme and the second storage scheme into a storage component, and store the first operating indicator data into an index of an index set corresponding to the first storage scheme.

[0118] In a specific implementation, the storage component includes a disk and a cache; the storage unit 403 for saving the first storage scheme and the second storage scheme in the storage component is specifically used to: persist the first storage scheme and the second storage scheme on the disk, and back up the first storage scheme and the second storage scheme in the storage component cache in the form of a hash table; wherein the key in the hash table is the monitoring object and indicator type, and the value in the hash table is the index.

[0119] The storage unit 403 is further configured to: if there is a first storage solution corresponding to the target time interval in the storage component, store the first operating indicator data in an index of an index set corresponding to the first storage solution, and execute the construction unit 404 .

[0120] In a specific implementation, the storage unit 403 for storing the first operating indicator data in the index of the index set corresponding to the first storage scheme is specifically used to: use the monitoring object and indicator type corresponding to the first operating indicator data as keys to query the index contained in the index set corresponding to the first storage scheme in the cache to determine the corresponding first target index; and store the first operating indicator data in the first target index in the specified database.

[0121] Construction unit 404 is used to construct a second storage scheme and store it in the storage component using the third storage scheme of the previous time interval of the target time interval and the index set corresponding to the third storage scheme if the second storage scheme corresponding to the next time interval of the target time interval does not exist in the storage component.

[0122] Preferably, combined Figure 4 The content shown, the construction unit 404 includes: a determination module, a parsing module and a construction module, and the execution principle of each module is as follows:

[0123] A determination module is used to determine the number of indexes to be created in the next time interval if there is no second storage solution corresponding to the next time interval of the target time interval in the storage component, using the total amount of data stored in the index set corresponding to the third storage solution of the previous time interval of the target time interval, combined with the specified capacity.

[0124] The parsing module is used to parse the distribution of the operating indicator data stored in the index set corresponding to the third storage solution.

[0125] The building module is configured to build a second storage solution by using the distribution situation and the number of indexes to be created in the next time interval, and store the solution in the storage component.

[0126] Preferably, combined Figure 4 The processing system further includes:

[0127] A first query unit is used to query the storage component for storage solutions whose construction time is greater than a preset time length;

[0128] The deletion unit is used to delete the storage solution whose construction time is greater than the preset time from the storage component, and delete the index set corresponding to the storage solution whose construction time is greater than the preset time from the specified database.

[0129] In an embodiment of the present invention, time intervals are divided according to a specified time granularity, and a corresponding storage scheme is constructed for each time interval, wherein a new storage scheme is constructed based on the storage scheme of the historical time interval and its corresponding index set. Based on the storage scheme, an index set containing at least one index is created in a specified database, and each index has a specified capacity. When the operating indicator data to be stored is obtained, the corresponding storage scheme is determined according to the collection time, and then the obtained operating indicator data is stored in the index corresponding to the determined storage scheme. The operating indicator data is distributedly stored to improve the efficiency and timeliness of querying the operating indicator data.

[0130] Preferably, combined Figure 4 The processing system further includes:

[0131] The parsing unit is configured to, upon receiving the first query request, parse the first query request to obtain at least the monitoring object to be queried, the indicator type to be queried, and the time range to be queried.

[0132] The second query unit is configured to query the index set corresponding to the fourth storage scheme in the cache using the monitored object to be queried and the indicator type to be queried as a key, if a fourth storage scheme corresponding to at least one time interval covered by the time range to be queried is found in the disk, to determine a corresponding second target index;

[0133] The request unit is used to send a second query request to the second target index in the specified database, obtain and output the indicator data results, and the indicator data results at least include: the second operating indicator data of the monitored object to be queried within the time range to be queried, and the indicator type of the second operating indicator data is the indicator type to be queried.

[0134] In this embodiment of the present invention, upon receiving a first query request, a query is performed on the index corresponding to the fourth storage scheme in the cache using the monitored object to be queried and the metric type to be queried as keys, thereby quickly determining the second target index associated with the desired operational metric data. A second query request is then sent to the second target index in the designated database to retrieve and output the metric data results, thereby improving the efficiency and timeliness of querying operational metric data.

[0135] In summary, an embodiment of the present invention provides a method and system for processing operation indicator data. By adopting this solution, time intervals are divided according to a specified time granularity, and a corresponding storage solution is constructed for each time interval, wherein a new storage solution is constructed based on the storage solution of the historical time interval and its corresponding index set. Based on the storage solution, an index set containing at least one index is created in a specified database, and each index has a specified capacity. When the operation indicator data to be stored is obtained, the corresponding storage solution is determined according to the collection time, and then the obtained operation indicator data is stored in the index corresponding to the determined storage solution. The operation indicator data is distributedly stored to improve the efficiency and timeliness of querying the operation indicator data.

[0136] Each embodiment in this specification is described in a progressive manner. The same or similar parts between the embodiments can be referred to each other. Each embodiment focuses on the differences from other embodiments. In particular, for system or system embodiments, since they are basically similar to method embodiments, the description is relatively simple. For relevant parts, refer to the partial description of the method embodiment. The system and system embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. A person of ordinary skill in the art can understand and implement it without expending creative work.

[0137] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the components and steps of each example according to their functions. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present invention.

[0138] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the scope of the present invention. Therefore, the present invention is not limited to the embodiments shown herein, but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for processing operating index data, characterized in that: The method comprises: When first operating indicator data to be stored is collected, a target time interval within which the collection time of the first operating indicator data falls is determined, where the target time interval is a time interval divided according to a specified time granularity; In a case where the first storage scheme corresponding to the target time interval does not exist in the storage component, the first storage scheme is determined to be the default storage scheme, and the second storage scheme for the next time interval of the target time interval is determined to be the default storage scheme, wherein the storage scheme defines the number of indexes contained in the index set, and defines the monitoring object and indicator type corresponding to the operating indicator data stored in each index in the index set; the storage scheme is used to create an index set containing at least one index in a specified database, the index having a specified capacity, and the index being used to store operating indicator data of a preset indicator type for a specified monitoring object; Saving the first storage solution and the second storage solution in the storage component, and storing the first operating indicator data in an index of the index set corresponding to the first storage solution; If a first storage solution corresponding to the target time interval exists in the storage component, the first operating indicator data is stored in an index of an index set corresponding to the first storage solution, and the following steps are performed: The following steps include: If the second storage scheme corresponding to the next time interval of the target time interval does not exist in the storage component, the second storage scheme is constructed and stored in the storage component using the third storage scheme of the previous time interval of the target time interval and the index set corresponding to the third storage scheme.

2. The method according to claim 1, characterized in that The storage component includes disk and cache; The process of saving the first storage scheme and the second storage scheme in the storage component includes: Persistently save the first storage solution and the second storage solution in a disk, and back up the first storage solution and the second storage solution in a cache in the form of a hash table; The keys in the hash table are monitoring objects and indicator types, and the values in the hash table are indexes.

3. The method according to claim 2, characterized in that The process of storing the first operating indicator data in the index of the index set corresponding to the first storage solution includes: Using the monitoring object and indicator type corresponding to the first operating indicator data as keys, query the indexes contained in the index set corresponding to the first storage solution in the cache to determine and obtain the corresponding first target index; The first operating indicator data is stored in the first target index in the designated database.

4. The method according to claim 1, wherein If the storage component does not have a second storage solution corresponding to a time interval next to the target time interval, using a third storage solution for a time interval previous to the target time interval and an index set corresponding to the third storage solution, constructing the second storage solution and storing it in the storage component includes: If the second storage solution corresponding to the next time interval of the target time interval does not exist in the storage component, the number of indexes to be created in the next time interval is determined by using the total amount of data stored in the index set corresponding to the third storage solution in the previous time interval of the target time interval in combination with the specified capacity; Analyzing the distribution of the operating indicator data stored in the index set corresponding to the third storage solution; The second storage solution is constructed by using the distribution and the number of indexes to be created in the next time interval and stored in the storage component.

5. The method according to any one of claims 1 to 4, characterized in that: The method further comprises: In the storage component, query the storage solution whose construction time is greater than the preset time length; The storage solutions whose construction time is greater than the preset time length are deleted from the storage component, and the index sets corresponding to the storage solutions whose construction time is greater than the preset time length are deleted from the designated database.

6. The method according to claim 2 or 3, characterized in that The method further comprises: Upon receiving the first query request, parsing the first query request to obtain at least the monitoring object to be queried, the indicator type to be queried, and the time range to be queried; If a fourth storage solution corresponding to at least one time interval covered by the time range to be queried is found in the disk, the index set corresponding to the fourth storage solution in the cache is queried using the monitoring object to be queried and the indicator type to be queried as a key to determine a corresponding second target index; A second query request is sent to the second target index in the specified database to obtain and output indicator data results, wherein the indicator data results at least include: the second operating indicator data of the monitored object to be queried within the time range to be queried, and the indicator type of the second operating indicator data is the indicator type to be queried.

7. A system for processing operating index data, characterized in that: The system comprises: a first determining unit, configured to, when first operating indicator data to be stored is collected, determine a target time interval within which the collection time of the first operating indicator data falls, the target time interval being a time interval divided according to a specified time granularity; a second determining unit, configured to determine, when the first storage scheme corresponding to the target time interval does not exist in the storage component, that the first storage scheme is a default storage scheme, and to determine, as the default storage scheme, a second storage scheme for a time interval following the target time interval, wherein the storage scheme defines the number of indexes included in the index set, and defines the monitoring objects and indicator types corresponding to the operating indicator data stored in each index in the index set; the default storage scheme is used to create an index set containing at least one index in a specified database, the index having a specified capacity, and the index being used to store operating indicator data of a preset indicator type for a specified monitoring object; a storage unit, configured to save the first storage scheme and the second storage scheme into the storage component, and store the first operating indicator data into an index of the index set corresponding to the first storage scheme; The storage unit is further configured to: if a first storage scheme corresponding to the target time interval exists in the storage component, store the first operating indicator data in an index of an index set corresponding to the first storage scheme, and execute the construction unit; The construction unit is used to construct the second storage scheme and store it in the storage component by using the third storage scheme of the previous time interval of the target time interval and the index set corresponding to the third storage scheme if the second storage scheme corresponding to the next time interval of the target time interval does not exist in the storage component.

8. The system according to claim 7, characterized in that The storage component includes disk and cache; The storage unit for saving the first storage scheme and the second storage scheme in the storage component is specifically used to: persist the first storage scheme and the second storage scheme on the disk, and back up the first storage scheme and the second storage scheme in the storage component cache in the form of a hash table; wherein the keys in the hash table are monitoring objects and indicator types, and the values in the hash table are indexes.

9. The system according to claim 8, characterized in that The storage unit for storing the first operating indicator data in the index of the index set corresponding to the first storage scheme is specifically used to: use the monitoring object and indicator type corresponding to the first operating indicator data as keys, query the index contained in the index set corresponding to the first storage scheme in the cache, and determine the corresponding first target index; store the first operating indicator data in the first target index in the designated database.

10. The system according to claim 7, wherein: The building block comprises: a determination module configured to determine, if the second storage solution corresponding to the next time interval of the target time interval does not exist in the storage component, the number of indexes to be created in the next time interval by using the total amount of data stored in the index set corresponding to the third storage solution of the previous time interval of the target time interval and the specified capacity; An analysis module, configured to analyze the distribution of the operating indicator data stored in the index set corresponding to the third storage solution; A construction module is used to construct the second storage solution by using the distribution and the number of indexes to be created in the next time interval and store the solution in the storage component.

Images