Monitoring data storage and query method and device

Abstract

The present invention discloses a storage and query method and device for monitoring data, and relates to the field of computer technology. A specific implementation of the storage method includes: encoding the monitoring data to be stored, and obtaining the encoded monitoring data to be stored; when the time dimension information of the monitoring data to be stored is at the second level, storing the encoded monitoring data to be stored in a cache and a database; when the time dimension information of the monitoring data to be stored is not at the second level, storing the encoded monitoring data to be stored in a database. This implementation method performs encoding processing on the monitoring data, which can significantly reduce the disk space for data storage, and stores the second-level monitoring data in the cache and the database respectively, which can meet the efficiency of different types of queries and ensure the query speed of recent data in the second-level monitoring data.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a method and device for storing and querying monitoring data. Background Art

[0002] In large-scale, high-concurrency Internet applications, when the system has functional or performance problems, it is necessary to be able to quickly and accurately locate the problem. An efficient monitoring system plays a key role in problem location. Currently, system monitoring has entered the era of second-level monitoring from the past minute-level monitoring. The nearly 100-fold increase in data volume and concurrency has posed unprecedented challenges to the entire monitoring system, especially the core data storage system.

[0003] At present, the commonly used solutions for time series data storage such as monitoring data are: (1) using non-relational databases (i.e. NoSQL databases, such as Hbase and Cassandra), concatenating metrics and timestamps as the row key, and storing the data at each time point as a record; (2) using a developed time series database (i.e. OpenTSDB), with the design idea of ​​designing the row key as metric_timestamp_[tag1_val1…_tagn_valn] or merging multiple rows of data into a single row with multiple columns, and then further merging multiple columns into a single column; (3) solutions based on native sequence data storage (such as Prometheus), which are redesigned from the storage layer.

[0004] In the process of implementing the present invention, the inventors found that there are at least the following problems in the prior art: First, the existing non-relational database is a monitoring data storage solution for minutes. Faced with the magnitude and concurrency of second-level data, its data writing or data query performance cannot meet business needs; Second, OpenTSDB needs to put the source data into the memory, and each time a range of data is queried, a large range of data needs to be fetched into the memory and then filtered, which has a large memory demand and is very likely to cause memory overflow; Third, the solution based on native time series data storage has weak support for data replication and linear expansion in design, which is unacceptable for large-scale monitoring applications with rapidly growing data. Summary of the invention

[0005] In view of this, an embodiment of the present invention provides a method and device for storing and querying monitoring data, which can significantly reduce the disk space for data storage, meet the efficiency of different types of queries, and ensure the query speed of recent data in second-level monitoring data.

[0006] To achieve the above objective, according to a first aspect of an embodiment of the present invention, a method for storing monitoring data is provided.

[0007] A method for storing monitoring data in an embodiment of the present invention includes: encoding monitoring data to be stored to obtain the encoded monitoring data to be stored; when the time dimension information of the monitoring data to be stored is at the second level, storing the encoded monitoring data to be stored in a cache and a database; when the time dimension information of the monitoring data to be stored is not at the second level, storing the encoded monitoring data to be stored in a database.

[0008] Optionally, the encoding processing of the monitoring data to be stored includes: obtaining the string data in the monitoring data to be stored; determining whether a string identifier corresponding to the string data exists in the memory, cache or database; if so, obtaining the string identifier, and replacing the string data in the monitoring data to be stored with the string identifier; if not, calling the encoding service, generating a string identifier corresponding to the string data, and replacing the string data in the monitoring data to be stored with the generated string identifier.

[0009] Optionally, storing the encoded monitoring data to be stored in a cache includes: storing the encoded monitoring data to be stored in a cache according to a cache storage period and a cache storage format.

[0010] Optionally, storing the encoded monitoring data to be stored in a database includes: determining a first storage table corresponding to the monitoring data to be stored according to time dimension information and condition information of the monitoring data to be stored, the time dimension information including: second level, grade level, hour level and day level, the condition information including: monitoring indicator conditions and multi-dimensional combination conditions, the first storage table is located in the database; storing the encoded monitoring data to be stored in the first storage table according to the storage period and storage format corresponding to the first storage table.

[0011] Optionally, the method further includes: if the condition information of the monitoring data to be stored is a multi-dimensional combination condition, calling a coding service to generate a multi-dimensional combination identifier corresponding to the multi-dimensional combination condition of the monitoring data to be stored.

[0012] To achieve the above objective, according to a second aspect of an embodiment of the present invention, a method for querying monitoring data is provided.

[0013] A method for querying monitoring data according to an embodiment of the present invention includes: determining time dimension information and condition information of the monitoring data to be queried according to a received query request, the time dimension information including: second level, grade level, hour level and day level, and the condition information including: monitoring indicator condition and multi-dimensional combination condition; judging whether the time dimension information of the monitoring data to be queried is at the second level, and whether the condition information of the monitoring data to be queried is a monitoring indicator condition; if so, querying the data to be queried in a cache or a database, and if not, querying the data to be queried in a database.

[0014] Optionally, querying the data to be queried in the cache or database includes: judging whether the data to be queried is stored in the cache according to the query time information in the query request; if so, querying the data to be queried in the cache, otherwise, querying the data to be queried in the database.

[0015] Optionally, querying the data to be queried in the database includes: determining a second storage table corresponding to the monitoring data to be queried according to time dimension information and condition information of the monitoring data to be queried, and querying the data to be queried from the second storage table, where the second storage table is in the database.

[0016] Optionally, after determining the time dimension information and condition information of the monitoring data to be queried, the method further includes: if the condition information of the monitoring data to be queried is a multi-dimensional combination condition, obtaining a multi-dimensional combination identifier corresponding to the multi-dimensional combination condition of the monitoring data to be queried.

[0017] To achieve the above objective, according to a third aspect of an embodiment of the present invention, a storage device for monitoring data is provided.

[0018] A storage device for monitoring data according to an embodiment of the present invention is characterized in that it includes: an encoding module, which is used to encode the monitoring data to be stored and obtain the encoded monitoring data to be stored; a first storage module, which is used to store the encoded monitoring data to be stored in a cache and a database when the time dimension information of the monitoring data to be stored is in the second level; and a second storage module, which is used to store the encoded monitoring data to be stored in a database when the time dimension information of the monitoring data to be stored is not in the second level.

[0019] Optionally, the encoding module is also used to: obtain the string data in the monitoring data to be stored; determine whether there is a string identifier corresponding to the string data in the memory, cache or database; if so, obtain the string identifier, and replace the string data in the monitoring data to be stored with the string identifier; if not, call the encoding service, generate a string identifier corresponding to the string data, and replace the string data in the monitoring data to be stored with the generated string identifier.

[0020] Optionally, the first storage module is further used to: store the encoded monitoring data to be stored in the cache according to the cache storage period and cache storage format.

[0021] Optionally, the second storage module is also used to: determine the first storage table corresponding to the monitoring data to be stored according to the time dimension information and condition information of the monitoring data to be stored, the time dimension information includes: second level, grade level, hour level and day level, the condition information includes: monitoring indicator conditions and multi-dimensional combination conditions, the first storage table is located in the database; according to the storage period and storage format corresponding to the first storage table, the encoded monitoring data to be stored is stored in the first storage table.

[0022] Optionally, the encoding module is further used to: if the condition information of the monitoring data to be stored is a multi-dimensional combination condition, call the encoding service to generate a multi-dimensional combination identifier corresponding to the multi-dimensional combination condition of the monitoring data to be stored.

[0023] To achieve the above objective, according to a fourth aspect of an embodiment of the present invention, a monitoring data query device is provided.

[0024] A monitoring data query device according to an embodiment of the present invention includes: a determination module, used to determine the time dimension information and condition information of the monitoring data to be queried according to a received query request, the time dimension information including: second level, grade level, hour level and day level, the condition information including: monitoring indicator condition and multi-dimensional combination condition; a judgment module, used to judge whether the time dimension information of the monitoring data to be queried is at the second level, and whether the condition information of the monitoring data to be queried is a monitoring indicator condition; a query module, used to query the data to be queried in a cache or a database if yes, and if not, query the data to be queried in a database.

[0025] Optionally, the query module is further used to: determine whether the data to be queried is stored in the cache according to the query time information in the query request; if so, query the data to be queried in the cache, otherwise, query the data to be queried in the database.

[0026] Optionally, the query module is also used to: determine a second storage table corresponding to the monitoring data to be queried according to time dimension information and condition information of the monitoring data to be queried, and query the data to be queried from the second storage table, where the second storage table is in a database.

[0027] Optionally, the determination module is further used to: if the condition information of the monitoring data to be queried is a multi-dimensional combination condition, obtain a multi-dimensional combination identifier corresponding to the multi-dimensional combination condition of the monitoring data to be queried.

[0028] To achieve the above objective, according to a fifth aspect of an embodiment of the present invention, an electronic device is provided.

[0029] An electronic device according to an embodiment of the present invention includes: one or more processors; a storage device for storing one or more programs, and when the one or more programs are executed by the one or more processors, the one or more processors implement the monitoring data storage and / or query method according to the embodiment of the present invention.

[0030] To achieve the above objective, according to a sixth aspect of an embodiment of the present invention, a computer-readable medium is provided.

[0031] A computer-readable medium according to an embodiment of the present invention stores a computer program, and when the program is executed by a processor, the method for storing and / or querying monitoring data according to an embodiment of the present invention is implemented.

[0032] One embodiment of the above invention has the following advantages or beneficial effects:

[0033] (1) In terms of spatial dimension design, the strings in the monitoring data are converted into string identifiers, which significantly reduces the disk space for data storage; the second-level monitoring data is stored in the cache and database respectively, which can meet the efficiency of different types of queries; the multi-dimensional combined query conditions are converted into identifiers, which reduces the storage space and greatly improves the data query speed;

[0034] (2) In terms of time dimension design, when monitoring data is stored in the database, monitoring data of different time granularities are stored in different storage tables. Each data table is specially designed for rows and columns to control the amount of data in a single table and ensure the speed of single table storage and retrieval. When data is stored in the cache, second-level data is recorded at a granularity of minutes to ensure the query speed of the latest monitoring data in seconds.

[0035] (3) In terms of overall dimensional design, the encoding service is deployed separately, which can be linearly expanded according to the concurrency and data volume to ensure the millisecond-level response of the encoding service; monitoring data is diverted according to the time dimension and the data stream is processed in streaming batch mode to ensure high-speed data processing and avoid data backlog.

[0036] The further effects of the above-mentioned non-conventional optional manner will be described below in conjunction with the specific implementation manner. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The accompanying drawings are used to better understand the present invention and do not constitute an improper limitation of the present invention.

[0038] Figure 1 is a schematic diagram of main steps of a method for storing monitoring data according to an embodiment of the present invention;

[0039] Figure 2 is a message format diagram of a KAFKA queue according to an embodiment of the present invention;

[0040] Figure 3 is a schematic diagram of a Redis cache data format design according to an embodiment of the present invention;

[0041] Figure 4 is a schematic diagram of the design of an HBase database table according to an embodiment of the present invention;

[0042] Figure 5 is a schematic diagram of the main flow of the method for storing monitoring data according to an embodiment of the present invention;

[0043] Figure 6 is a schematic diagram of the main steps of the monitoring data query method according to an embodiment of the present invention;

[0044] Figure 7 This is a schematic diagram of the main flow of a method for querying monitoring data according to an embodiment of the present invention;

[0045] Figure 8 is a schematic diagram of the architecture of a monitoring data storage and monitoring system according to an embodiment of the present invention;

[0046] Fig. 9 is a schematic diagram of the architecture of a coding service according to an embodiment of the present invention;

[0047] Fig.10 is a schematic diagram of the architecture of a storage service according to an embodiment of the present invention;

[0048] Fig.11 is a schematic diagram of the architecture of a query service according to an embodiment of the present invention;

[0049] Fig.12 is a schematic diagram of main modules of a storage device for monitoring data according to an embodiment of the present invention;

[0050] Fig.13 is a schematic diagram of main modules of a monitoring data query device according to an embodiment of the present invention;

[0051] Fig.14is an exemplary system architecture diagram to which embodiments of the present invention may be applied;

[0052] Fig.15 It is a schematic diagram of the structure of a computer system of a terminal device or a server suitable for implementing an embodiment of the present invention. DETAILED DESCRIPTION

[0053] The following is a description of exemplary embodiments of the present invention in conjunction with the accompanying drawings, including various details of the embodiments of the present invention to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0054] At present, system monitoring has entered the era of second-level monitoring from the past minute-level monitoring, that is, the original 1.5 million records per second at the minute level will become 90 million records per second now, but the existing storage technology means are powerless for monitoring data of this magnitude and concurrency. Therefore, the present invention provides a storage and query method for monitoring data. Figure 1 Schematic diagram of the main steps of the method for storing monitoring data according to an embodiment of the present invention. As a reference embodiment of the present invention, Figure 1 As shown, the main steps of the monitoring data storage method of the embodiment of the present invention may include:

[0055] Step S101: Encoding the monitoring data to be stored to obtain the encoded monitoring data to be stored;

[0056] Step S102: When the time dimension information of the monitoring data to be stored is at the second level, the encoded monitoring data to be stored is stored in the cache and the database;

[0057] Step S103: When the time dimension information of the monitoring data to be stored is not at the second level, the encoded monitoring data to be stored is stored in the database.

[0058] Since there are a lot of string data in the monitoring data, such as Metric key (monitoring indicator key, used to indicate a certain indicator of a certain business, such as the number of records written per second in the log database, or the memory occupancy rate of the virtual machine, etc.), Host (host key, used to indicate which host a certain indicator value at a certain moment comes from), Group (group key, used to indicate which group it comes from, group refers to the logical grouping of the business, for example, terminal business may be divided into mobile phones, desktops, etc.), etc., these strings will take up a lot of disk or memory space when stored, and the efficiency of matching and retrieving the strings is not high. Therefore, in the present invention, these strings can be encoded to obtain a unique identifier of the string, for example, the string is converted into a long integer value.

[0059] In summary, the present invention can not only greatly reduce the storage space but also improve the query speed by encoding the monitoring data to be stored. As another reference embodiment of the present invention, the specific method of programming the monitoring data to be stored may include:

[0060] Step S1011: Obtaining character string data in the monitoring data to be stored;

[0061] Step S1012: Determine whether there is a string identifier corresponding to the string data in the memory, cache or database;

[0062] Step S1013: If yes, obtain the string identifier, and replace the string data in the monitoring data to be stored with the string identifier;

[0063] Step S1014: If not, call the encoding service to generate a string identifier corresponding to the string data, and replace the string data in the monitoring data to be stored with the generated string identifier.

[0064] It is worth noting that the string data is obtained in step S1011, so the present invention can set which strings need to be encoded and which do not need to be encoded according to business needs, thereby improving the practicality of the present solution. Considering the magnitude and concurrency of the monitoring data, the response time of the encoding in the present invention should be in milliseconds. In actual application, the fastest query speed in the memory, cache and database is in the memory, followed by the cache, and then the database. Therefore, in step S1012, the optimal judgment order is to first query whether there is a string identifier corresponding to the string data in the memory. If not, query whether there is a required string identifier in the cache. If it does not exist in the cache, it is necessary to query whether the required string identifier is stored in the database. Of course, the present invention can also follow other judgment orders, and there is no restriction on this. In addition, it is mentioned in step S1014 that if the required string identifier does not exist in the memory, cache and database, it is necessary to call the encoding service to generate the string identifier, thereby completing the encoding of the query data.

[0065] After encoding the monitoring data to be stored in step S101, the present invention divides the monitoring data into two categories for storage processing according to the time dimension information of the monitoring data. In step S102, it is mentioned that if the time dimension information of the monitoring data to be stored is at the second level, the monitoring data needs to be stored in both the cache and the database. In the present invention, in order to meet the query of second-level monitoring data, the second-level monitoring data is put into the cache and the database at the same time, and the data of other time dimensions is directly put into the database. In order to coordinate the speed between the inflow of monitoring data and data storage (cache and database), two thread groups and two data buffer queues can be designed for second-level monitoring data, one for processing data written to the cache, and the other for processing data written to the database. Such a design ensures that each thread can process a single type of data stream in batches, greatly speeding up the speed of data storage.

[0066] In contrast, it is mentioned in step S103 that if the time dimension information of the monitoring data to be stored is not at the second level, the monitoring data is directly stored in the database. The time dimension information of the present invention refers to the time dimension of the monitoring data inflow, which may include: second level, graded level (i.e. minute level), hour level (i.e. hour level) and day level (i.e. day level), etc. In the present invention, when storing the second-level monitoring data in the cache, the storage cycle and storage format requirements of the cache storage need to be met. In the present invention, the monitoring data may be a KAFKA queue (i.e., a high-throughput message queue) or monitoring data sent by a third-party application, and then after encoding the monitoring data, the encoded monitoring data is stored in the cache or database according to the rules. In the present invention, the cache may be a Redis (i.e., a key-value-based distributed cache database suitable for fast retrieval of small-scale data) cache, and the database may be an HBase (i.e., a NoSQL database designed based on rowkey and column family, suitable for storage and retrieval of large-scale data). Taking the storage of the KAFKA queue in Redis as an example, Figure 2 4 is a message format diagram of a KAFKA queue according to an embodiment of the present invention. Figure 2 In the metric key, the attribute name is a unique identifier after encoding; the attribute value is the calculated value, such as TP99, MAX_VALUE, etc.; the attribute granularity is the time dimension, indicating whether the monitoring data is second-level data, minute-level data, or hour-level data; tags indicate other dimensional information of the monitoring data, such as host, group, computer room, etc. For reference, Figure 2 The message text in the upper right corner is the message data printed in the actual test.

[0067] Figure 3 Schematic diagram of the Redis cache data format design according to an embodiment of the present invention. Figure 3 It can be seen that one piece of data is stored every minute, and the value of each piece of data is a hash table, which contains 60 pieces of data from 0 to 59 seconds. The purpose of this design is to reduce the number of keys in the cache and facilitate data queries at the minute level. For example, the most common query requirement is to view application performance statistics for the last 10 minutes. This design makes the access efficiency of cached data the highest.

[0068] The above embodiments explain how to store monitoring data in the cache. Here, the specific process of storing the encoded monitoring data to be stored in the database is described. Since the speed of storing data in the database is much slower than the speed of the cache and the message queue, it is the place where bottlenecks are most likely to occur. Therefore, the design of the storage table in the database is one of the key links of the present invention. The present invention may include storing monitoring data in the database: determining the first storage table corresponding to the monitoring data to be stored according to the time dimension information and condition information of the monitoring data to be stored; storing the encoded monitoring data to be stored in the first storage table according to the storage period and storage format corresponding to the first storage table.

[0069] Among them, the time dimension information has been explained above, which refers to the time dimension of the inflow of monitoring data; the condition information refers to the storage dimension conditions when the monitoring data is stored, which may include monitoring indicator conditions (i.e., storage is performed only based on monitoring indicator conditions) and multi-dimensional combination conditions (i.e., storage is performed based on a combination of other dimensional information of the monitoring data, such as combined queries based on multiple conditions such as hosts, groups, computer rooms, etc.). Figure 4 Schematic diagram of the design of the HBase database table according to an embodiment of the present invention. Figure 4 It can be seen that the second-level data table and the non-second-level data table adopt different designs. The data of each time dimension is divided into a data table based on monitoring indicators (ump_xxx) and a data table based on multi-dimensional combinations (ump_tags_xxx).

[0070] The database storage in the present invention is comprehensively considered from the time dimension, space dimension and data processing dimension, which disperses the pressure of data processing and ensures the data storage capacity under large-scale and high-concurrency conditions as a whole.

[0071] (1) From the time dimension, monitoring data flows at different levels, such as second level, classification level, hour level, and day level, are processed separately. The monitoring data flows into different queues, and the processing logic is processed by different threads. Finally, the data is written into different database storage tables. The purpose of this design is to ensure that each link of monitoring data processing can be processed in batch streaming at a very fast speed.

[0072] (2) From the spatial dimension, different types of monitoring data are written into different tables, which not only reduces the amount of data in each table, but also allows the table format to be designed according to each different type of data, thereby improving the access speed of a single table. For monitoring data with minute and hour granularity, since the total amount is within the acceptable range of a single table, a single table can be used. If the efficiency of a single table decreases due to the rapid growth of the amount of monitoring data, a weekly or daily table building method can also be used. For monitoring data with second-level granularity, since the amount of data is too large (trillions per day), a daily table building method is required. In addition, the design idea of ​​one record per hour can be adopted for monitoring data at the second level, which further reduces the number of records in each table, greatly alleviates the data access pressure of a single table, and significantly improves the speed of data writing and retrieval.

[0073] (3) From the data processing dimension, different thread pools and local cache queues are used to process different types of monitoring data. For example, data written to the cache and written to the database are processed separately to avoid delays caused by differences in data processing speed.

[0074] It should be noted that, from the perspective of large-scale data processing, monitoring data storage and monitoring data query belong to a system concept, so the monitoring data storage method can also include: if the condition information of the monitoring data to be stored is a multi-dimensional combination condition, call the encoding service to generate a multi-dimensional combination identifier corresponding to the multi-dimensional combination condition of the monitoring data to be stored. The advantage of this is that for queries based on multi-dimensional combination conditions, such as combined queries based on Metric key, Host or multiple conditions, the encoding service can also be called to encode the combined conditions, so that all combinations of query conditions can be avoided from being written into the rowkey, greatly improving query efficiency.

[0075] Figure 5 FIG. 2 is a schematic diagram of the main flow of the method for storing monitoring data according to an embodiment of the present invention. Figure 5 As shown, the main process of the monitoring data storage method of the present invention may include:

[0076] Step S501: Obtaining character string data in the monitoring data to be stored;

[0077] Step S502: Determine whether there is a string identifier corresponding to the string data in the memory, cache or database. If so, execute step S503; otherwise, execute step S504;

[0078] Step S503: Obtain a string identifier, and replace the string data in the monitoring data to be stored with the string identifier;

[0079] Step S504: calling the encoding service to generate a string identifier corresponding to the string data, and replacing the string data in the monitoring data to be stored with the generated string identifier;

[0080] Step S505: determine whether the time dimension information of the monitoring data to be stored is seconds, if so, execute step S506, otherwise, execute step S507;

[0081] Step S506: storing the encoded monitoring data to be stored in the cache according to the cache storage period and cache storage format;

[0082] Step S507: Obtaining time dimension information and condition information of the monitoring data to be stored, wherein the time dimension information may include: second level, level, hour level and day level, and the condition information may include: monitoring indicator condition and multi-dimensional combination condition;

[0083] Step S508: determining whether the condition information of the monitoring data to be stored is a multi-dimensional combination condition, if so, executing step S509;

[0084] Step S509: calling the encoding service to generate a multi-dimensional combination identifier corresponding to the multi-dimensional combination condition of the monitoring data to be stored;

[0085] Step S510: determining a first storage table corresponding to the monitoring data to be stored according to the time dimension information and condition information of the monitoring data to be stored, where the first storage table is located in a database;

[0086] Step S511: storing the encoded monitoring data to be stored in the first storage table according to the storage period and storage format corresponding to the first storage table.

[0087] It is worth noting that the judgment order of step S502 has been specifically introduced in step S1012 and will not be repeated here. The execution order of steps S508, S509, S510 and S511 can be adjusted according to actual conditions, as long as step S508 is executed before step S509 and step S510 is executed before step S511.

[0088] Compared with storage, the pressure of high concurrency of query requests is not great. For monitoring data at the second level, the main pressure of query is how to quickly find monitoring data of a period of time (for example, 20 minutes) near a certain time point in the trillion-level monitoring data. Since the monitoring data is evenly distributed from multiple dimensions when it is stored, different query requests will fall on different storage systems (cache or database) or storage tables. Figure 6 Schematic diagram of the main steps of the method for querying monitoring data according to an embodiment of the present invention. As a reference embodiment of the present invention, Figure 6As shown, the main steps of the monitoring data query method of the embodiment of the present invention may include:

[0089] Step S601: determining the time dimension information and condition information of the monitoring data to be queried according to the received query request, wherein the time dimension information and condition information have been explained in the above embodiment and will not be described again here;

[0090] Step S602: determining whether the time dimension information of the monitoring data to be queried is at the second level, and whether the condition information of the monitoring data to be queried is a monitoring indicator condition;

[0091] Step S603: If yes, query the cache or database for the data to be queried; if no, query the database for the data to be queried.

[0092] The query request of the present invention includes the time dimension level of the monitoring data to be queried and the condition information. If the monitoring data to be queried meets the following two conditions: the time dimension information is at the second level and the condition information is a monitoring indicator condition, it means that the data to be queried may be stored in the cache or in the database. Therefore, it is necessary to judge whether there is monitoring data to be queried in the cache according to the query time information in the query request. The query time information refers to the generation time of the monitoring data, and the cache stores recent monitoring data. For example, the cache only stores second-level monitoring data for nearly half an hour, while the database can store historical monitoring data.

[0093] Therefore, as another reference embodiment of the present invention, querying the data to be queried in the cache or database may include: judging whether the data to be queried is stored in the cache according to the query time information in the query request; if so, querying the data to be queried in the cache, otherwise, querying the data to be queried in the database. For example, for the second-level monitoring data of the most recent time (for example, within 20 minutes from now), it is necessary to ensure the real-time nature and fast response of the query, so this type of query will retrieve data from the cache. For historical data, the query is retrieved from the database. Although the amount of data of the second-level monitoring data is extremely large, the present invention can adopt a mechanism of building tables by day to control the amount of data in a single table to less than one trillion, ensuring the retrieval speed of a single table. In addition, the second-level monitoring data is stored at an hourly granularity, one record per hour, so that each query request only needs to locate one (query within a single hour) or two (query across an hourly time range) records, so the query speed is extremely fast.

[0094] The above embodiment introduces how to query data in the cache, and as another reference embodiment of the present invention, querying the data to be queried in the database may include: determining the second storage table corresponding to the monitoring data to be queried according to the time dimension information and condition information of the monitoring data to be queried, and querying the data to be queried from the second storage table, and the second storage table is in the database. In the present invention, the monitoring data query based on the non-second time dimension and the multi-dimensional combination query request are all executed in the database, because the amount of data of this type is too large to be suitable for storage in the cache.

[0095] In addition, for requests for multi-dimensional combination queries, since the multi-dimensional combination has been converted into an identifier through the encoding service when the monitoring data is stored, the query speed is almost the same as the single condition query (the only time difference is the difference in querying the identifier from the encoding service, but since the encoding service response time is in milliseconds, the difference can be ignored). Moreover, since the multi-dimensional combination has been converted into an identifier through the encoding service when the monitoring data is stored, after determining the condition information of the monitoring data to be queried, the monitoring data query method of the present invention may also include: if the condition information of the monitoring data to be queried is a multi-dimensional combination condition, obtaining the multi-dimensional combination identifier corresponding to the multi-dimensional combination condition of the monitoring data to be queried.

[0096] Figure 7 FIG. 1 is a schematic diagram of the main flow of the method for querying monitoring data according to an embodiment of the present invention. Figure 7 As shown, the main process of the monitoring data query method of the present invention may include:

[0097] Step 701: Determine the time dimension information and condition information of the monitoring data to be queried according to the received query request;

[0098] Step 702: determine whether the condition information of the monitoring data to be queried is a multi-dimensional combination condition, if so, execute step S703;

[0099] Step S703: obtaining a multi-dimensional combination identifier corresponding to the multi-dimensional combination condition of the monitoring data to be queried;

[0100] Step S704: determine whether the time dimension information of the monitoring data to be queried is at the second level, if so, execute step S705, otherwise, execute step S708;

[0101] Step S705: determine whether the condition information of the monitoring data to be queried is a monitoring indicator condition, if so, execute step S706, otherwise, execute step S708;

[0102] Step S706: judging whether the cache stores the data to be queried according to the query time information in the query request, if so, executing step S707, otherwise, executing step S708;

[0103] Step S707: Query the cache for the data to be queried;

[0104] Step S708: Determine the second storage table corresponding to the monitoring data to be queried according to the time dimension information and condition information of the monitoring data to be queried.

[0105] It is worth noting that the determination order of steps S704, S705 and S706 can be adjusted according to actual needs and will not be elaborated on in detail here.

[0106] The storage and query technical solution of monitoring data according to the present invention is comprehensively designed from the spatial dimension, time dimension and overall dimension, which not only ensures the rapid storage of large-scale data, but also ensures the data query speed and query flexibility, and also supports the linear expansion capability of the system, which is specifically manifested as follows:

[0107] (1) In terms of spatial dimension design, the strings in the monitoring data are converted into string identifiers, which significantly reduces the disk space for data storage; the second-level monitoring data is stored in the cache and database respectively, which can meet the efficiency of different types of queries; the multi-dimensional combined query conditions are converted into identifiers, which reduces the storage space and greatly improves the data query speed;

[0108] (2) In terms of time dimension design, when monitoring data is stored in the database, monitoring data of different time granularities are stored in different storage tables. Each data table is specially designed for rows and columns to control the amount of data in a single table and ensure the speed of single table storage and retrieval. When data is stored in the cache, second-level data is recorded at a granularity of minutes to ensure the query speed of the latest monitoring data in seconds.

[0109] (3) In terms of overall dimensional design, the encoding service is deployed separately, which can be linearly expanded according to the concurrency and data volume to ensure the millisecond-level response of the encoding service; monitoring data is diverted according to the time dimension and the data stream is processed in streaming batch mode to ensure high-speed data processing and avoid data backlog.

[0110] For ease of understanding, in the subsequent description of the embodiments of the present invention, the "monitoring data" is taken as the "KAFKA queue", the "cache" is taken as the "Redis", and the "database" is taken as the "HBase" for detailed description. Of course, "KAFKA queue", "Redis" and "HBase" are not used to limit the protection scope of the technical solution of the present invention. The specific values ​​in the present invention can be adaptively adjusted according to the specific business scenarios. Figure 8FIG. 1 is a schematic diagram of the architecture of a monitoring data storage and monitoring system according to an embodiment of the present invention. Figure 8 As shown, the present invention adopts an architecture design based on distributed services, which mainly includes three services: code service (Code Service), storage service (Storage Service) and query service (QueryService). Each service is deployed as an independent cluster, which can be linearly expanded or reduced according to business needs, and the services collaborate through RPC (Remote Procedure Call). In addition, the User Query in the figure refers to the user query request; Computation Service refers to the computing service, which is used to calculate and process the monitoring data, such as calculating the values ​​of various monitoring indicators, such as TP99, MAX_VALUE, AVG_VALUE, etc. The results will be sent to the data storage service through the message queue KAFKA for unified storage; DataStore Application refers to the monitoring data sent by third-party applications.

[0111] Fig. 9 : is a schematic diagram of the architecture of the encoding service according to an embodiment of the present invention. The computing service (for example, the flink streaming computing cluster) and the data storage service will call the encoding service to convert the string process that needs to be converted. The query service will call the encoding service, first convert the string in the user query request, and then query the cache or database according to the conversion result. After the encoding service receives the transcoding request, it will write the newly generated code to Redis and HBase at the same time. There are two points that need to be explained in particular: one is that when storing the encoded data, it is directly stored in the Redis cache, and then the result is returned immediately. The operation of writing to HBase is an asynchronous operation; the other point is that the query will directly query the cache data, and only when the cache is invalid will it query HBase. These two points ensure that the encoding service can complete the response instantly (1 millisecond level). In the second-level monitoring system, the ultra-large-scale data volume will rapidly amplify the system response time, so the instantaneous response of the encoding service is one of the key links to ensure the high performance of the entire data storage system. Among them, asynchronous operation means that the system main thread will not wait for the data to be stored in HBase before returning, but send the data to a queue, which will be read and stored by another thread. The main thread will directly return the encoded identifier to the caller to reduce the response speed.

[0112] Currently, the idea of ​​converting strings into codes has also been adopted in OpenTSDB, but these databases use codes as internal implementations, which require both transcoding and data storage queries. In this way, the database itself will face great pressure for calculation and storage, and the efficiency will be low. However, the solution proposed in the present invention extracts the encoding function and makes it a separate service, which can not only reduce the pressure on database storage, but also determine which content to encode according to business needs. In addition, the separately deployed service can be flexibly expanded according to the actual data volume and concurrency, providing strong support for large-scale data storage.

[0113] Fig.10 The following is a schematic diagram of the storage service architecture according to an embodiment of the present invention. The storage service is the core service of the system. Its basic task is to receive monitoring data from the KAFKA queue or third-party applications and store it in the Redis cache or HBase database according to the rules. The storage module disperses the pressure of data processing from multiple dimensions and ensures the ability to store data in large-scale and high-concurrency situations as a whole. Fig.10 As can be seen in the figure, the storage service receives the KAFKA message queue according to different time dimensions (seconds, minutes, hours, etc.), and then uses corresponding logic to process the data of different dimensions. After that, the storage service calls the getOrCreateXXX interface of the encoding service to generate the corresponding data record. Finally, according to the processing rules of data of different dimensions, the data is stored in the cache and database. Fig.11 : is a schematic diagram of the query service architecture according to an embodiment of the present invention. As can be seen from the figure, for second-level monitoring data, if you query the second-level monitoring data of the past 30 minutes, you will query from Redis; if you query the second-level monitoring data of more than 30 minutes, you will query from HBase.

[0114] Fig.12 FIG. 1 is a schematic diagram of the main modules of the storage device for monitoring data according to an embodiment of the present invention. Fig.12 As shown, the monitoring data storage device 1200 of the embodiment of the present invention mainly includes the following modules: an encoding module 1201, a first storage module 1202 and a second storage module 1203.

[0115] The encoding module 1201 can be used to: encode the monitoring data to be stored, and obtain the encoded monitoring data to be stored;

[0116] The first storage module 1202 may be used to: when the time dimension information of the monitoring data to be stored is at the second level, store the encoded monitoring data to be stored into the cache and the database;

[0117] The second storage module 1203 can be used to: when the time dimension information of the monitoring data to be stored is not at the second level, store the encoded monitoring data to be stored in the database.

[0118] In an embodiment of the present invention, the encoding module 1201 can also be used to: obtain string data in the monitoring data to be stored; determine whether there is a string identifier corresponding to the string data in the memory, cache or database; if so, obtain the string identifier, and replace the string data in the monitoring data to be stored with the string identifier; if not, call the encoding service, generate a string identifier corresponding to the string data, and replace the string data in the monitoring data to be stored with the generated string identifier.

[0119] In the embodiment of the present invention, the first storage module 1202 may also be used to: store the encoded monitoring data to be stored in the cache according to the cache storage period and the cache storage format.

[0120] In an embodiment of the present invention, the second storage module 1203 can also be used to: determine the first storage table corresponding to the monitoring data to be stored according to the time dimension information and condition information of the monitoring data to be stored, the time dimension information includes: second level, grade level, hour level and day level, the condition information includes: monitoring indicator conditions and multi-dimensional combination conditions, and the first storage table is located in the database; according to the storage period and storage format corresponding to the first storage table, the encoded monitoring data to be stored is stored in the first storage table.

[0121] In the embodiment of the present invention, the encoding module 1201 may also be used to: if the condition information of the monitoring data to be stored is a multi-dimensional combination condition, call the encoding service to generate a multi-dimensional combination identifier corresponding to the multi-dimensional combination condition of the monitoring data to be stored.

[0122] Fig.13 FIG. 1 is a schematic diagram of the main modules of the monitoring data query device according to an embodiment of the present invention. Fig.13 As shown, the monitoring data query device 1300 of the embodiment of the present invention mainly includes the following modules: a determination module 1301, a judgment module 1302 and a query module 1303.

[0123] The determination module 1301 may be used to: determine the time dimension information and condition information of the monitoring data to be queried according to the received query request, the time dimension information includes: second level, level, hour level and day level, and the condition information includes: monitoring indicator condition and multi-dimensional combination condition;

[0124] The judgment module 1302 may be used to: judge whether the time dimension information of the monitoring data to be queried is at the second level, and whether the condition information of the monitoring data to be queried is a monitoring indicator condition;

[0125] The query module 1303 may be used to: if yes, query the data to be queried in the cache or database; if no, query the data to be queried in the database.

[0126] In the embodiment of the present invention, the query module 1303 can also be used to: determine whether the data to be queried is stored in the cache according to the query time information in the query request; if so, query the data to be queried in the cache, otherwise, query the data to be queried in the database.

[0127] In an embodiment of the present invention, the query module 1303 can also be used to: determine the second storage table corresponding to the monitoring data to be queried according to the time dimension information and condition information of the monitoring data to be queried, and query the data to be queried from the second storage table, where the second storage table is in the database.

[0128] In the embodiment of the present invention, the determination module 1301 may also be used to: if the condition information of the monitoring data to be queried is a multi-dimensional combination condition, obtain a multi-dimensional combination identifier corresponding to the multi-dimensional combination condition of the monitoring data to be queried.

[0129] From the above description, it can be seen that the storage and query device of monitoring data of the present invention is comprehensively designed from the spatial dimension, the temporal dimension and the overall dimension, which not only ensures the rapid storage of large-scale data, but also ensures the data query speed and query flexibility, and also supports the linear expansion capability of the system, which is specifically manifested as follows:

[0130] (1) In terms of spatial dimension design, the strings in the monitoring data are converted into string identifiers, which significantly reduces the disk space for data storage; the second-level monitoring data is stored in the cache and database respectively, which can meet the efficiency of different types of queries; the multi-dimensional combined query conditions are converted into identifiers, which reduces the storage space and greatly improves the data query speed;

[0131] (2) In terms of time dimension design, when monitoring data is stored in the database, monitoring data of different time granularities are stored in different storage tables. Each data table is specially designed for rows and columns to control the amount of data in a single table and ensure the speed of single table storage and retrieval. When data is stored in the cache, second-level data is recorded at a granularity of minutes to ensure the query speed of the latest monitoring data in seconds.

[0132] (3) In terms of overall dimensional design, the encoding service is deployed separately, which can be linearly expanded according to the concurrency and data volume to ensure the millisecond-level response of the encoding service; monitoring data is diverted according to the time dimension and the data stream is processed in streaming batch mode to ensure high-speed data processing and avoid data backlog.

[0133] Fig.14 An exemplary system architecture 1400 is shown to which the monitoring data storage and query method or monitoring data storage and query device according to the embodiment of the present invention can be applied.

[0134] like Fig.14 As shown, system architecture 1400 may include terminal devices 1401, 1402, 1403, a network 1404, and a server 1405. Network 1404 is used to provide a medium for communication links between terminal devices 1401, 1402, 1403 and server 1405. Network 1404 may include various connection types, such as wired, wireless communication links, or optical fiber cables, etc.

[0135] Users can use terminal devices 1401, 1402, 1403 to interact with server 1405 through network 1404 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 1401, 1402, 1403, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only as examples).

[0136] The terminal devices 1401 , 1402 , and 1403 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, and desktop computers, etc.

[0137] The server 1405 may be a server that provides various services, such as a backend management server (only an example) that provides support for shopping websites browsed by users using the terminal devices 1401, 1402, and 1403. The backend management server may analyze and process the received data such as product information query requests, and feed back the processing results (such as target push information, product information - only an example) to the terminal device.

[0138] It should be noted that the monitoring data storage and query method provided in the embodiment of the present invention is generally executed by the server 1405 , and accordingly, the monitoring data storage and query device is generally set in the server 1405 .

[0139] It should be understood that Fig.14 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0140] Reference below Fig.15 , which shows a schematic diagram of the structure of a computer system 1500 of a terminal device suitable for implementing an embodiment of the present invention. Fig.15 The terminal device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0141] like Fig.15As shown, the computer system 1500 includes a central processing unit (CPU) 1501, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1502 or a program loaded from a storage part 1508 into a random access memory (RAM) 1503. In the RAM 1503, various programs and data required for the operation of the system 1500 are also stored. The CPU 1501, the ROM 1502, and the RAM 1503 are connected to each other through a bus 1504. An input / output (I / O) interface 1505 is also connected to the bus 1504.

[0142] The following components are connected to the I / O interface 1505: an input section 1506 including a keyboard, a mouse, etc.; an output section 1507 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 1508 including a hard disk, etc.; and a communication section 1509 including a network interface card such as a LAN card, a modem, etc. The communication section 1509 performs communication processing via a network such as the Internet. A drive 1510 is also connected to the I / O interface 1505 as needed. A removable medium 1511, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 1510 as needed, so that a computer program read therefrom is installed into the storage section 1508 as needed.

[0143] In particular, according to the embodiments disclosed in the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present invention include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 1509, and / or installed from the removable medium 1511. When the computer program is executed by the central processing unit (CPU) 1501, the above-mentioned functions defined in the system of the present invention are executed.

[0144] It should be noted that the computer-readable medium shown in the present invention may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present invention, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0145] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0146] The modules involved in the embodiments of the present invention may be implemented in software or hardware. The modules described may also be arranged in a processor, for example, they may be described as: a processor includes a coding module, a first storage module, and a second storage module. The names of these modules do not constitute a limitation on the modules themselves in some cases, for example, the coding module may also be described as "a module for encoding monitoring data to be stored and obtaining the encoded monitoring data to be stored".

[0147] As another aspect, the present invention also provides a computer-readable medium, which may be included in the device described in the above embodiment; or it may exist independently without being assembled into the device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by a device, the device includes: encoding the monitoring data to be stored, obtaining the encoded monitoring data to be stored; when the time dimension information of the monitoring data to be stored is at the second level, storing the encoded monitoring data to be stored in the cache and the database; when the time dimension information of the monitoring data to be stored is not at the second level, storing the encoded monitoring data to be stored in the database.

[0148] The technical solution according to the embodiment of the present invention is comprehensively designed from the spatial dimension, the temporal dimension and the overall dimension, which not only ensures the rapid storage of large-scale data, but also ensures the data query speed and query flexibility, and also supports the linear expansion capability of the system, which is specifically manifested as follows:

[0149] (1) In terms of spatial dimension design, the strings in the monitoring data are converted into string identifiers, which significantly reduces the disk space for data storage; the second-level monitoring data is stored in the cache and database respectively, which can meet the efficiency of different types of queries; the multi-dimensional combined query conditions are converted into identifiers, which reduces the storage space and greatly improves the data query speed;

[0150] (2) In terms of time dimension design, when monitoring data is stored in the database, monitoring data of different time granularities are stored in different storage tables. Each data table is specially designed for rows and columns to control the amount of data in a single table and ensure the speed of single table storage and retrieval. When data is stored in the cache, second-level data is recorded at a granularity of minutes to ensure the query speed of the latest monitoring data in seconds.

[0151] (3) In terms of overall dimensional design, the encoding service is deployed separately, which can be linearly expanded according to the concurrency and data volume to ensure the millisecond-level response of the encoding service; monitoring data is diverted according to the time dimension and the data stream is processed in streaming batch mode to ensure high-speed data processing and avoid data backlog.

[0152] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for storing monitoring data, characterized in that: include: Encoding the monitoring data to be stored, and obtaining the encoded monitoring data to be stored; When the time dimension information of the monitoring data to be stored is at the second level, storing the encoded monitoring data to be stored in the cache according to the cache storage period and the cache storage format; The encoded monitoring data to be stored is stored in a database; When the time dimension information of the monitoring data to be stored is not at the second level, storing the encoded monitoring data to be stored in the database; Among them, storing the encoded monitoring data to be stored in a database includes: determining a first storage table corresponding to the monitoring data to be stored according to the time dimension information and condition information of the monitoring data to be stored; the time dimension information includes: second level, grade level, hour level and day level, the condition information includes: monitoring indicator conditions and multi-dimensional combination conditions, and the first storage table is located in the database; according to the storage period and storage format corresponding to the first storage table, storing the encoded monitoring data to be stored in the first storage table.

2. The storage method according to claim 1, characterized in that: The encoding process of the monitoring data to be stored includes: Obtaining character string data in the monitoring data to be stored; Determine whether there is a string identifier corresponding to the string data in the memory, cache or database; If yes, then obtaining the string identifier, and replacing the string data in the monitoring data to be stored with the string identifier; If not, the encoding service is called to generate a string identifier corresponding to the string data, and the string data in the monitoring data to be stored is replaced with the generated string identifier.

3. The storage method according to claim 1, characterized in that: The method further comprises: If the condition information of the monitoring data to be stored is a multi-dimensional combination condition, the encoding service is called to generate a multi-dimensional combination identifier corresponding to the multi-dimensional combination condition of the monitoring data to be stored.

4. A method for querying monitoring data, characterized in that: include: Determine, according to the received query request, the time dimension information and condition information of the monitoring data to be queried, wherein the time dimension information includes: second level, level, hour level and day level, and the condition information includes: monitoring indicator condition and multi-dimensional combination condition; Determine whether the time dimension information of the monitoring data to be queried is at the second level, and whether the condition information of the monitoring data to be queried is a monitoring indicator condition; If yes, query the data to be queried in the cache or database; if no, query the data to be queried in the database; the cache stores the encoded monitoring data to be stored according to the cache storage period and cache storage format; Wherein, querying the data to be queried in the database includes: determining a second storage table corresponding to the monitoring data to be queried from the first storage table according to the time dimension information and condition information of the monitoring data to be queried, and querying the data to be queried from the second storage table, wherein the second storage table is in the database; The first storage table is stored in the following manner: according to the time dimension information and condition information of the monitoring data to be stored, the first storage table corresponding to the monitoring data to be stored is determined, the time dimension information includes: second level, grade level, hour level and day level, the condition information includes: monitoring indicator conditions and multi-dimensional combination conditions, and the first storage table is located in a database; according to the storage period and storage format corresponding to the first storage table, the encoded monitoring data to be stored is stored in the first storage table.

5. The query method according to claim 4, characterized in that: The querying the to-be-queried data in the cache or the database includes: Determining whether the to-be-queried data is stored in the cache according to the query time information in the query request; If so, the cache is queried for the data to be queried; otherwise, the database is queried for the data to be queried.

6. The query method according to claim 4, characterized in that: After determining the time dimension information and condition information of the monitoring data to be queried, the method further includes: If the condition information of the monitoring data to be queried is a multi-dimensional combination condition, a multi-dimensional combination identifier corresponding to the multi-dimensional combination condition of the monitoring data to be queried is obtained.

7. A storage device for monitoring data, characterized in that: include: An encoding module, used for encoding the monitoring data to be stored, and obtaining the encoded monitoring data to be stored; A first storage module, used for storing the encoded monitoring data to be stored in a cache according to a cache storage period and a cache storage format when the time dimension information of the monitoring data to be stored is at the second level; The encoded monitoring data to be stored is stored in a database; A second storage module is used to store the encoded monitoring data to be stored in a database when the time dimension information of the monitoring data to be stored is not at the second level; Among them, storing the encoded monitoring data to be stored in a database includes: determining a first storage table corresponding to the monitoring data to be stored according to the time dimension information and condition information of the monitoring data to be stored, the time dimension information includes: second level, grade level, hour level and day level, the condition information includes: monitoring indicator conditions and multi-dimensional combination conditions, and the first storage table is located in the database; storing the encoded monitoring data to be stored in the first storage table according to the storage period and storage format corresponding to the first storage table.

8. The storage device according to claim 7, characterized in that: The encoding module is also used for: Obtaining character string data in the monitoring data to be stored; Determine whether there is a string identifier corresponding to the string data in the memory, cache or database; If yes, then obtaining the string identifier, and replacing the string data in the monitoring data to be stored with the string identifier; If not, the encoding service is called to generate a string identifier corresponding to the string data, and the string data in the monitoring data to be stored is replaced with the generated string identifier.

9. The storage device according to claim 7, characterized in that: The encoding module is also used for: If the condition information of the monitoring data to be stored is a multi-dimensional combination condition, the encoding service is called to generate a multi-dimensional combination identifier corresponding to the multi-dimensional combination condition of the monitoring data to be stored.

10. A monitoring data query device, characterized in that: include: A determination module, configured to determine the time dimension information and condition information of the monitoring data to be queried according to the received query request, wherein the time dimension information includes: second level, level, hour level and day level, and the condition information includes: monitoring indicator condition and multi-dimensional combination condition; A judgment module, used to judge whether the time dimension information of the monitoring data to be queried is at the second level, and whether the condition information of the monitoring data to be queried is a monitoring indicator condition; A query module, for querying the data to be queried in the cache or database if yes, and querying the data to be queried in the database if no; the cache stores the encoded monitoring data to be stored according to the cache storage period and cache storage format; Wherein, querying the data to be queried in the database includes: determining a second storage table corresponding to the monitoring data to be queried from the first storage table according to the time dimension information and condition information of the monitoring data to be queried, and querying the data to be queried from the second storage table, wherein the second storage table is in the database; The first storage table is stored in the following manner: according to the time dimension information and condition information of the monitoring data to be stored, the first storage table corresponding to the monitoring data to be stored is determined, the time dimension information includes: second level, grade level, hour level and day level, the condition information includes: monitoring indicator conditions and multi-dimensional combination conditions, and the first storage table is located in a database; according to the storage period and storage format corresponding to the first storage table, the encoded monitoring data to be stored is stored in the first storage table.

11. The query device according to claim 10, characterized in that: The query module is also used for: Determining whether the to-be-queried data is stored in the cache according to the query time information in the query request; If so, the cache is queried for the data to be queried; otherwise, the database is queried for the data to be queried.

12. The query device according to claim 10, characterized in that: The determining module is also used for: If the condition information of the monitoring data to be queried is a multi-dimensional combination condition, a multi-dimensional combination identifier corresponding to the multi-dimensional combination condition of the monitoring data to be queried is obtained.

13. An electronic device, characterized in that: include: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 6.

14. A computer readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images