Data processing method and device, electronic equipment, storage medium and program

Abstract

The embodiment of the invention provides a data processing method and device, electronic equipment, a storage medium and a program, and the method comprises the steps: responding to a data query instruction, and obtaining global index information; obtaining target time sequence information from a preset time sequence database according to the data block index; and obtaining target sampling data according to the target data block and the position index information. By creating global index information, index index data capable of being reused by sampled time series data are unified and integrated, and only index data changing in corresponding time needs to be stored in each data block based on local index information, so that when the index data are archived and compressed, the index data written into a disk can be reduced, and the efficiency of archiving the index data is improved. Storage control is saved, index data read from a disk can be reduced during data query, and the purpose of improving query efficiency is achieved.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a data processing method, a data processing device, an electronic device, a computer-readable storage medium, and a computer program product. Background Art

[0002] Prometheus is a general monitoring solution in cloud-native scenarios. Its built-in TSDB (Time Series Database) service is mainly responsible for storing the time series information of monitoring data, including indicator data, label information, and timestamps. Specifically, the aforementioned TSDB service can periodically archive and compress indicator data to generate data blocks (i.e., Data Blocks) for the corresponding time period.

[0003] Among them, each data block can contain independent indicator index data. For long-term stable monitoring data, the data index information will exist in multiple data blocks, and as the time span increases, the duplicate data index information will increase, resulting in the need to reference the indicator index data multiple times when querying data values ​​within a longer period of time. Summary of the invention

[0004] The purpose of the embodiments of the present invention is to provide a data processing method, a data processing device, an electronic device, a computer-readable storage medium, and a computer program product to optimize data indexes. The specific technical solutions are as follows:

[0005] In a first aspect of the present invention, a data processing method is provided, the method comprising:

[0006] In response to the data query instruction, global index information is obtained; the global index information is data index information used to identify indicator index data obtained based on time series data, and the data index information is used to indicate a data block index;

[0007] According to the data block index, target time series information is obtained from a preset time series database; the target time series information is determined based on the target data block of the corresponding time period, wherein the local index information contained in each data block is used to store the position index information when the numerical position of the time series data of the corresponding time period is offset;

[0008] Target sampling data is obtained according to the target data block and the position index information.

[0009] In a second aspect of the present invention, there is also provided a data processing device, the device comprising:

[0010] A global index information acquisition module, used to respond to a data query instruction and acquire global index information; the global index information is data index information used to identify indicator index data obtained based on time series data, and the data index information is used to indicate a data block index;

[0011] A target time series information acquisition module is used to acquire target time series information from a preset time series database according to the data block index; the target time series information is determined based on the target data block of the corresponding time period, wherein the local index information contained in each data block is used to store the position index information when the numerical position of the time series data of the corresponding time period is offset;

[0012] The target sampling data determination module is used to obtain the target sampling data according to the target data block and the position index information.

[0013] In another aspect of the implementation of the present invention, there is also provided an electronic device, including a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus; the memory is used to store computer programs; and the processor is used to implement any of the data processing methods when executing the programs stored in the memory.

[0014] In another aspect of the present invention, a computer-readable storage medium is provided, wherein instructions are stored in the computer-readable storage medium, and when the computer-readable storage medium is run on a computer, the computer executes any of the above-mentioned data processing methods.

[0015] In yet another aspect of the present invention, a computer program product comprising instructions is provided, which, when executed on a computer, enables the computer to execute any of the above-mentioned data processing methods.

[0016] The data processing method, device, electronic device and readable storage medium provided by the embodiments of the present invention, when querying data, use the data index information of the indicator index data identified by the global index information to indicate the data block index of the corresponding time series data, and then obtain the target time series information from the preset time series database based on the obtained data block index. The target time series information can be determined based on the target data block. At this time, the target sampling data can be obtained based on the target data block and the local index information containing the position index information contained in the data block. The created global index is used to identify the data index information of the time series data, and the local index information contained in the data block only stores the index information when the numerical position of the time series data in the corresponding time period is offset. The data index information identified by the global index is unique, and the local index within each data block only records the data changes within the data block, ensuring the uniqueness of the local index information. When the user performs a data query, the global index is first used to locate the data block containing the required data, and then the local index is used to quickly locate the specific data change point to query the target sampling data. The aforementioned unique index structure can avoid duplication of data index information in each data block, thereby reducing disk space usage and reducing the reading of duplicate index information during data query, thereby improving query efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art are briefly introduced below.

[0018] Figure 1 A flowchart of a data processing method according to an embodiment of the present invention;

[0019] Figure 2 is a flowchart of another data processing method embodiment in an embodiment of the present invention;

[0020] Figure 3 A data schematic diagram of local index information provided by an embodiment of the present invention;

[0021] Figure 4 A data schematic diagram of global index information provided by an embodiment of the present invention;

[0022] Figure 5 A schematic diagram of generating a data block provided by an embodiment of the present invention;

[0023] Figure 6 A schematic diagram of a data query process provided by an embodiment of the present invention;

[0024] Figure 7A schematic structural diagram of a data processing device embodiment of the present invention;

[0025] Figure 8 The figure is a schematic diagram of the structure of an electronic device in an embodiment of the present invention. DETAILED DESCRIPTION

[0026] The technical solutions in the embodiments of the present invention will be described below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0027] Prometheus is a general monitoring solution for cloud-native scenarios. Its built-in TSDB service can periodically archive and compress indicator data to generate data blocks (i.e., Data Blocks) for the corresponding time period. For example, a data block is generated every 2 hours by default, so 12 data blocks can be generated within 24 hours.

[0028] Archival compression refers to compressing data to reduce storage space and storing it in long-term storage media, such as tapes, optical disks, cloud storage, etc., so that data can be quickly retrieved and restored when needed in the future. The archived compressed data is usually historical data or backup data that needs to be retained for a long time. During the archive compression process, indexes and metadata are usually generated. Through the indexes and metadata, the compressed data can be quickly located and retrieved.

[0029] Specifically, Prometheus's TSDB can adopt a block-based storage mechanism to divide the monitoring data into different blocks for storage according to time. Each block can contain all time series data within a fixed time period, and each block is independent and contains its own index data. When you need to query data values ​​within a longer period of time, you usually need to find the data block corresponding to the time period. Since each data block contains independent indicator index data, during the query process, you need to perform an independent index query on each data block involved, and then query the corresponding data value from the data block, and merge the queried data values ​​to return the merged data. This means that when querying data values ​​for a longer period of time, you need to perform an independent index query on each data block involved during the query process, and you need to perform the indicator index data of multiple data blocks. Multiple references are made to obtain the data in each data block for merging the returned data. Since each Block contains all time series data within a fixed time period, assuming that a data block is generated every 2 hours by default, the data block generated in the next 2 hours can contain the data index information of the data block generated in the previous 2 hours, that is, the data index information contained in the data block is full information. For long-term and stable monitoring data, its data will span multiple time periods. As the time span increases, the data blocks generated in the later time periods will also contain the corresponding data index information after the monitoring data is archived and compressed based on the historical time period, that is, the data index information corresponding to the monitoring data of a certain historical time period will exist in multiple data blocks, and as the time span increases, the duplicate data index information will increase, resulting in the need to reference the indicator index data multiple times when querying data values ​​within a longer time period.

[0030] In the embodiment of the present invention, global index information is created to uniformly integrate the index data of the sampled time series data that can be reused, and each data block only needs to save the index data that changes within the corresponding time. This allows the amount of index data written to disk to be reduced when the index data is archived and compressed, saving storage control. It can also reduce the amount of index data read from disk when querying data, thereby achieving the purpose of improving query efficiency.

[0031] Reference Figure 1 , shows a flow chart of steps of a data processing method embodiment of the present invention, which may specifically include the following steps:

[0032] Step 101, responding to a data query instruction and obtaining global index information;

[0033] In an embodiment of the present invention, the data used for storage may be monitoring data, such as the CPU (Central Processing Unit) load of the machine, memory usage, etc. Monitoring data is typical time series data, and it is usually necessary to sample and store the values ​​at each time point.

[0034] When using Prometheus, its built-in TSDB service will generate multiple data blocks according to the time dimension, and each data block can contain all the data index information within the time period. However, for monitoring data, the corresponding indicator index data does not change frequently, which means that most of the indicator index data in multiple data blocks are the same, and can even be reused. In one embodiment of the present invention, global index information can be created. The global index information can be used to identify the data index information obtained based on the time series data, which is unique. In an embodiment of the present invention, the created global indicator index data can unify and integrate the indicator index data that can be reused, and then save the indicator index data that has not changed for a long time.

[0035] Among them, the global index information may refer to the data index information for the indicator index data when the time series data is archived and compressed into data blocks. Specifically, the time series data may be divided into multiple data blocks. While the data blocks are generated by archiving and compression, the indicator index data may be extracted from the time series data, and data index information may be generated for the indicator index data, which is then used to indicate the data blocks of the corresponding time series data. At this time, the data index information may refer to the monitoring data information index.

[0036] In the specific implementation, the time series obtained by dividing according to the time dimension can be a unique time series (i.e., time series) composed of a set of indicator names (i.e., metrics), a set of label names (i.e., labels), and label values. Each time series can be uniquely determined by a metric and a set of labels (i.e., labels). That is, through the combination of indicators and labels, Prometheus can uniquely identify and query specific monitoring data; and for monitoring data, its indicator index data can refer to the indicator name (i.e., metric) and label name (i.e., labels). These two types of data do not change frequently. Among them, the indicator (i.e., metric) refers to the name of the monitoring data, which can be used to describe a specific measurement of the monitored object, such as CPU usage, memory usage, etc., while the label (i.e., labels) can be a set of key-value pairs, which can be mainly used to further describe and distinguish different time series. The index of the indicator name and label in the monitoring data information index is actually an indexed representation of these metric and label data. When storing time series data, Prometheus can establish corresponding indexes to quickly retrieve and locate data matching specific indicators and labels. These indexes allow Prometheus to efficiently perform complex query operations, such as label-based filtering and aggregation. The global index information created in this embodiment of the present invention is mainly to reuse the above two types of data.

[0037] In a preferred embodiment of the present invention, when newly generated monitoring data is detected, the newly generated monitoring data can be written into a preset log file, at which time the data index information of the newly generated monitoring data can be obtained, and the data index information of the newly generated monitoring data can be updated to the time series data obtained according to the preset period, so as to incrementally update the local index information in the corresponding data block. In addition, the preset log file can be processed to obtain a data block according to the preset sampling time period, and the specific processing method can refer to archiving compression, so as to realize the built-in TSDB service of Prometheus to periodically perform archiving compression of indicator data.

[0038] Step 102, obtaining target time series information from a preset time series database according to the data block index;

[0039] In practical applications, unique data index information can be used to indicate a data block index, such as a data block ID. At this time, target time series information can be obtained from a preset time series database according to the data block index.

[0040] Specifically, the target time series information can be determined based on the target data block of the corresponding time period. Optionally, the data query instruction can carry the query time, in which case at least one time series information matching the query time can be obtained from a preset time series database, and then the target time series information can be determined from each data block based on the data block index.

[0041] The local index information contained in each data block is used to store data changes that occur in a corresponding time period, such as position index information when the numerical position of time series data is offset.

[0042] Step 103, obtaining target sampling data according to the target data block and the position index information.

[0043] Since the target time series information can usually be stored in the target data block of the corresponding time period, and each data block can contain local index information, when acquiring the target sampling data, the target sampling data can be queried from the target data block based on the local index information storing the position index information to complete the data query.

[0044] In an embodiment of the present invention, when querying data, the data index information of the indicator index data identified by the global index information is used to indicate the data block index of the corresponding time series data, and then the target time series information is obtained from a preset time series database based on the obtained data block index. The target time series information can be determined based on the target data block. At this time, the target sampling data can be obtained based on the target data block and the local index information containing the position index information contained in the data block. The created global index is used to identify the data index information of the time series data, and the local index information contained in the data block only stores the index information when the numerical position of the time series data in the corresponding time period is offset. The data index information identified by the global index is unique, and the local index within each data block only records the data changes within the data block, ensuring the uniqueness of the local index information. When the user performs a data query, the global index is first used to locate the data block containing the required data, and then the local index is used to quickly locate the specific data change point to query the target sampling data. The aforementioned unique index structure can avoid duplication of data index information in each data block, thereby reducing disk space usage and reducing the reading of duplicate index information during data query, thereby improving query efficiency.

[0045] Reference Figure 2 , shows a flowchart of another data processing method embodiment in an embodiment of the present invention, which may specifically include the following steps:

[0046] Step 201, when newly generated monitoring data is detected according to a preset period, the newly generated monitoring data is written into a preset log file;

[0047] In an embodiment of the present invention, the data used for storage may be monitoring data, such as the CPU load of a machine, memory usage, etc. Monitoring data is typical time series data, and it is usually necessary to sample and store the values ​​at each time point. Specifically, the TSDB service built into the Prometheus monitoring solution may periodically archive and compress the indicator data to generate data blocks for the corresponding time period.

[0048] In Prometheus, in order to quickly retrieve and query data, each data block can have independent index information to locate the data in the block. The index information contained in each data block can be called local index information (ie, Block Index).

[0049] like Figure 3 As shown in the figure, the Prometheus TSDB data index can mainly include the symbol table, data index information and location index information. Among them, the symbol table (i.e., Symbol Table) may contain string information such as all indicator names, label names, label values, etc., and these non-repeating strings may correspond to a certain digital ID respectively; the data index information (i.e., SeriesTable) may refer to the monitoring data information index, which may usually include the index of the indicator name (i.e., metric) and the label. The label index may include the label name (i.e., labels) and the label value. Usually, a unique time series consisting of a set of indicator names and a set of label names and label values ​​may be used, that is, the corresponding time series data may be uniquely determined based on the monitoring data information index; the position index information (i.e., Pos i tings) may refer to the monitoring value position index, which specifically involves the position information of the time series data in the storage. Specifically, the sample data of each time series will be organized into a series of chunks. Chunks are the basic units for Prometheus to store data. Specifically, after the monitoring data is separated into different blocks according to time, each block will be further divided into multiple chunks. These chunks contain timestamps and sample values, and the monitoring value position index can be used to indicate the location where the actual sample data containing the time series is stored, that is, the offset position of the chunk.

[0050] In one embodiment of the present invention, within a preset period, such as 2 hours, since business may go online or offline during the current time period, when newly generated monitoring data is detected, the newly generated monitoring data can be written into a preset log file to facilitate periodic compression of the data that will be subsequently written into the preset log file.

[0051] In actual applications, when periodic compression is performed subsequently, the preset log file can be mainly used to indicate the time series data obtained according to the preset period. In order to improve the index information of the data block, the data index information of the newly generated monitoring data can be obtained at this time, and the data index information can be updated to the time series data obtained according to the preset period, so as to incrementally update the local index information in the corresponding data block.

[0052] Step 202, according to a preset sampling time period, the preset log file is archived and compressed to obtain a data block;

[0053] In an embodiment of the present invention, the data used for storage may be monitoring data. For the monitoring data, the indicator index data corresponding to the data does not change frequently, which means that most of the indicator index data in multiple data blocks are the same and can even be reused. In one embodiment of the present invention, global index information can be created, that is, global indicator index data is created to unify and integrate the reusable indicator index data, thereby preserving the unchanged indicator index data for a long time.

[0054] When compressing the preset log file, when the preset sampling time period is greater than the preset time threshold, the data index information of the time series data obtained according to the preset period can be updated to the global index information, and the local index information in the data block can be updated so that the local index information in the data block can be fully updated.

[0055] The global index information (i.e., Global Index) may refer to the data index information of the indicator index data in the time series data, and specifically may refer to the data index information when the time series data is archived and compressed into data blocks. At this time, the data index information may refer to the monitoring data information index. Figure 4 As shown in the figure, for monitoring data, the corresponding indicator index data does not change frequently. Its indicator index data can refer to the indicator name (ie, metric) and label name (ie, labels). The global index information can unify the data index information, that is, the monitoring data information index, to achieve reuse in subsequent data queries.

[0056] In addition to reusing the indicator name and the label name, the symbol table can also be reused. The symbol table can contain string information such as all indicator names, label names, label values, and these non-repeating strings can correspond to a certain digital ID. In practical applications, the created global index information can mainly be the data index information obtained by simplifying the aforementioned string. The simplified data index information can be used to indicate a data block index, such as a data block ID, which is not limited by the embodiment of the present invention.

[0057] It should be noted that the location index information can be mainly used to indicate the location where the actual sample data of the time series is stored. Since the time series data is a set of values ​​generated over time, its storage method needs to be able to efficiently handle this dynamic change. As time goes by, some old data may no longer need to be retained, or in some cases, Prometheus may reorganize the data according to the storage strategy or performance requirements, and the actual sample storage location cannot remain unchanged, such as Figure 4 As shown, the created global index information does not include the location index information data.

[0058] For the archiving compression process of data blocks, such as Figure 5As shown in the figure, it can be implemented through WAL (Write-AheadLogging, which is an important component in Prometheus TSDB). Specifically, WAL can mainly write the data in the form of logs to a pre-write log file before writing the data to the chunks on the disk. In an asynchronous process, the data will be compressed and stored in the chunks; when the WAL archive is compressed into data blocks, it is usually manifested as periodic compression. For example, a data block is generated every 2 hours by default, so 12 data blocks can be generated within 24 hours. In the process of archive compression, the sampling time span of the time series data can be used to determine whether to update its corresponding index information to the global index. If the preset sampling time period is greater than the preset time threshold, it means that the sampling time span is too large relative to the preset period. At this time, the number of data blocks will increase significantly, which may easily lead to duplication of index information and waste of storage space. The embodiment of the present invention can update the data index information of the time series data obtained according to the preset period to the global index information, so as to update the data block metadata in the global index information, such as the start time, end time, and unique identifier of the data block, and update the local index information in the data block, so as to ensure that only the index information when the data changes is retained, and avoid duplication of index information, so that when the data needs to be accessed later, the global index information can be used to quickly locate the data block containing the required data, and then the local index information can be used to quickly locate the specific data change point, so as to improve the query efficiency and reduce the query time on the basis of reducing the duplication of index information.

[0059] Exemplarily, assuming that the time span for writing index data into the global index is set to 7 days, when a new data block is generated, the time series information can be queried from the global index first. If it exists, it is used directly. If it does not exist, it is necessary to consider whether the acquisition time span of the time series exceeds 7 days. If it exceeds, it is written to the global index and the index in the existing historical data block is updated at the same time. Otherwise, only the index in the data block is written. This is not limited in this embodiment of the present invention.

[0060] Optionally, after periodically performing archive compression to generate data blocks, a chunk directory can also be generated and a local index can be created to help optimize storage and query performance, especially when processing large-scale time series data. Specifically, after generating data blocks, a chunk directory can usually be generated to record metadata information of each data block, and the chunk directory can help quickly locate and access data blocks, wherein the created chunk directory can generate a unique identifier (such as chunk ID) for each data block, and record its metadata information, such as time range, label, data block size, etc. Specifically, the chunk directory can be stored in persistent storage, usually in the same location as the data block or in an associated metadata storage; and, in order to further improve query performance, a local index can be created in each data block, and the local index can help quickly locate time series information in the data block, wherein a local index can be created in each data block to record metadata information of the time series, such as time series ID, label, time range, etc., and the local index is stored together with the data block, usually embedded in the data block file or stored as a separate index file.

[0061] In a preferred embodiment of the present invention, when determining the target time series information from each data block, it can also rely on the chunk directory and local index created above to make the determination. For example, it can be determined which data blocks contain the required sampling data based on the metadata information and chunk directory of the time series, and then the local index in the data block can be read to obtain the time series information contained in the data block. This embodiment of the present invention is not limited to this.

[0062] Step 203, responding to the data query instruction and obtaining global index information;

[0063] When executing a query, Prometheus can determine which data blocks need to be scanned based on the time range of the query, and use the indexes of these data blocks to quickly locate the required data.

[0064] Step 204, obtaining target time series information from a preset time series database according to the data block index;

[0065] In practical applications, the simplified data index information can be used to indicate a data block index, such as a data block ID. At this time, the target time series information can be obtained from a preset time series database according to the data block index.

[0066] Specifically, the data query instruction may carry a query time. At this time, at least one time series information matching the query time may be obtained from a preset time series database. At this time, at least one time series information matching the query time may be obtained from a preset time series database, and the target time series information may be determined from each data block based on the data block index, so as to first query the relevant time series information from the global index.

[0067] In a specific implementation, the target data block of the corresponding time period can be determined from each data block based on the data block index, and the target time series information stored in the target data block can be obtained. Figure 6 As shown, after constructing a query statement based on the query conditions provided by the user, such as labels, time ranges, indicator names, etc., the constructed query conditions can be used to query in the global index to obtain the target data block corresponding to the time series list that matches the conditions, assuming it is data block 2. At this time, the matching information returned is the time series information corresponding to data block 2.

[0068] Step 205: Query and obtain target sample data from the target data block based on the position index information.

[0069] The local index information contained in each data block is used to store data changes that occur in a corresponding time period, such as position index information when the numerical position of time series data is offset.

[0070] In practical applications, the location index information contained in the local index information can indicate the location where the actual sample data of the time series is stored, such as Figure 6 As shown, in the process of data query, after querying the relevant time series information from the global index, specific sampling data can be queried from each data block, and specifically, the target sampling data can be obtained based on the location index information.

[0071] In a specific implementation, the target location for storing data in the target data block may be determined based on the location index information, and then the target sampling data stored in the target location in the target data block may be directly acquired.

[0072] It should be noted that the data processing solution provided in the embodiment of the present invention can mainly be used for Prometheus-related monitoring services and other services that use Prometheus TSDB as a time series data storage engine, so as to improve data query efficiency while saving disk space. It is suitable for directly or indirectly using Prometheus monitoring solutions, and the embodiment of the present invention is not limited to this.

[0073] In an embodiment of the present invention, by creating global index information, the index data of the sampled time series data that can be reused is uniformly integrated, and each data block only needs to save the index data that changes within the corresponding time. This allows the amount of index data written to the disk to be reduced when the index data is archived and compressed, saving storage control. It can also reduce the amount of index data read from the disk when querying data, thereby achieving the purpose of improving query efficiency.

[0074] It should be noted that, for the sake of simplicity, the method embodiments are described as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.

[0075] Reference Figure 7 , shows a structural block diagram of a data processing device embodiment of the present invention, which may specifically include the following modules:

[0076] The global index information acquisition module 701 is used to respond to the data query instruction and acquire the global index information; the global index information is data index information used to identify the index data obtained based on the time series data, and the data index information is used to indicate the data block index;

[0077] The target time series information acquisition module 702 is used to acquire the target time series information from the preset time series database according to the data block index; the target time series information is determined based on the target data block of the corresponding time period, wherein the local index information contained in each data block is used to store the position index information when the numerical position of the time series data of the corresponding time period is offset;

[0078] The target sampling data determination module 703 is used to obtain the target sampling data according to the target data block and the position index information.

[0079] In an embodiment of the present invention, the data query instruction carries the query time; the target time series information acquisition module 702 may include the following submodules:

[0080] The target time series information acquisition submodule is used to obtain at least one time series information matching the query time from a preset time series database; wherein each time series information is stored in each data block of the corresponding time period; based on the data block index, the target data block of the corresponding time period is determined from the each data block, and the target time series information stored on the target data block is obtained.

[0081] In one embodiment of the present invention, the target sampling data determination module 703 may include the following submodules:

[0082] The target sampling data determination submodule is used to obtain the target sampling data from the target data block based on the position index information.

[0083] In one embodiment of the present invention, the target sampling data determination submodule may include the following units:

[0084] The target sampling data determining unit is used to determine the target location for data storage in the target data block based on the location index information; and collect the target sampling data stored in the target location in the target data block.

[0085] In an embodiment of the present invention, the device provided by the embodiment of the present invention may further include the following modules:

[0086] An index information incremental update module is used to write newly generated monitoring data into a preset log file when newly generated monitoring data is detected according to a preset period; the preset log file is used to store time series data obtained according to a preset period; obtain data index information of the newly generated monitoring data, and update the data index information of the newly generated monitoring data to the time series data obtained according to the preset period, so as to incrementally update the local index information in the corresponding data block.

[0087] In an embodiment of the present invention, the device provided by the embodiment of the present invention may further include the following modules:

[0088] The data block generation module is used to archive and compress the preset log file to obtain data blocks according to a preset sampling time period.

[0089] In an embodiment of the present invention, in the process of archiving and compressing the preset log file to obtain a data block according to the preset sampling time period, the device provided by the embodiment of the present invention may further include the following modules:

[0090] The global index information updating module is used to update the data index information of the time series data obtained according to the preset period to the global index information when the preset sampling time period is greater than the preset time threshold, and to update the local index information in the data block.

[0091] In an embodiment of the present invention, when querying data, the data processing device embodiment provided by the embodiment of the present invention uses the data index information of the indicator index data identified by the global index information to indicate the data block index of the corresponding time series data, and then obtains the target time series information from the preset time series database based on the obtained data block index. The target time series information can be determined based on the target data block. At this time, the target sampling data can be obtained based on the target data block and the local index information containing the position index information contained in the data block. The created global index is used to identify the data index information of the time series data, and the local index information contained in the data block only stores the index information when the numerical position of the time series data in the corresponding time period is offset. The data index information identified by the global index is unique, and the local index within each data block only records the data changes within the data block, ensuring the uniqueness of the local index information. When the user performs a data query, the global index is first used to locate the data block containing the required data, and then the local index is used to quickly locate the specific data change point to query the target sampling data. The aforementioned unique index structure can avoid duplication of data index information in each data block, thereby reducing disk space usage and reducing the reading of duplicate index information during data query, thereby improving query efficiency.

[0092] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0093] The embodiment of the present invention further provides an electronic device, such as Figure 8 As shown, it includes a processor 801, a communication interface 802, a memory 803 and a communication bus 804, wherein the processor 801, the communication interface 802, and the memory 803 communicate with each other through the communication bus 804.

[0094] Memory 803, used for storing computer programs;

[0095] The processor 801 is used to execute the program stored in the memory 803, and implements the following steps:

[0096] In response to the data query instruction, global index information is obtained; the global index information is data index information used to identify indicator index data obtained based on time series data, and the data index information is used to indicate a data block index;

[0097] According to the data block index, target time series information is obtained from a preset time series database; the target time series information is determined based on the target data block of the corresponding time period, wherein the local index information contained in each data block is used to store the position index information when the numerical position of the time series data of the corresponding time period is offset;

[0098] Target sampling data is obtained according to the target data block and the position index information.

[0099] In an embodiment of the present invention, when querying data, the electronic device provided by the embodiment of the present invention uses the data index information of the indicator index data identified by the global index information to indicate the data block index of the corresponding time series data, and then obtains the target time series information from the preset time series database based on the obtained data block index. The target time series information can be determined based on the target data block. At this time, the target sampling data can be obtained based on the target data block and the local index information containing the position index information contained in the data block. The created global index is used to identify the data index information of the time series data, and the local index information contained in the data block only stores the index information when the numerical position of the time series data in the corresponding time period is offset. The data index information identified by the global index is unique, and the local index within each data block only records the data changes within the data block, ensuring the uniqueness of the local index information. When the user performs a data query, the global index is first used to locate the data block containing the required data, and then the local index is used to quickly locate the specific data change point to query the target sampling data. The aforementioned unique index structure can avoid duplication of data index information in each data block, thereby reducing disk space usage and reducing the reading of duplicate index information during data query, thereby improving query efficiency.

[0100] The communication bus mentioned in the above terminal can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0101] The communication interface is used for communication between the above terminal and other devices.

[0102] The memory may include a random access memory (RAM) or a non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0103] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0104] In another embodiment of the present invention, a computer-readable storage medium is provided. The computer-readable storage medium stores instructions. When the instructions are executed on a computer, the computer executes the data processing method described in any one of the above embodiments.

[0105] In another embodiment of the present invention, a computer program product including instructions is provided. When the computer program product is run on a computer, the computer executes the data processing method described in any one of the above embodiments.

[0106] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (Digital Subscriber Line, referred to as DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrations. The available medium may be a magnetic medium (eg, a floppy disk, a hard disk, a magnetic tape), an optical medium (eg, a DVD), or a semiconductor medium, such as a solid state drive (SSD for short).

[0107] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0108] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0109] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims

1. A data processing method, characterized in that: The method comprises: In response to the data query instruction, global index information is obtained; the global index information is data index information used to identify indicator index data obtained based on time series data, and the data index information is used to indicate a data block index; According to the data block index, target time series information is obtained from a preset time series database; the target time series information is determined based on the target data block of the corresponding time period, wherein the local index information contained in each data block is used to store the position index information when the numerical position of the time series data of the corresponding time period is offset; Target sampling data is obtained according to the target data block and the position index information.

2. The method according to claim 1, characterized in that: The data query instruction carries the query time; The step of acquiring target time series information from a preset time series database according to the data block index includes: Obtaining at least one time series information matching the query time from a preset time series database; wherein each time series information is stored in each data block of a corresponding time period; Based on the data block index, a target data block of a corresponding time period is determined from each data block, and target time series information stored in the target data block is acquired.

3. The method according to claim 1 or 2, characterized in that: The step of obtaining target sampling data according to the target data block and the position index information includes: Target sampling data is obtained from the target data block based on the position index information.

4. The method according to claim 3, characterized in that The step of obtaining target sampled data from the target data block based on the position index information includes: Determining a target location for data storage in the target data block based on the location index information; The target sampling data stored at the target position in the target data block is collected.

5. The method according to claim 1, characterized in that Also includes: According to a preset period, when newly generated monitoring data is detected, the newly generated monitoring data is written into a preset log file; The preset log file is used to store time series data obtained according to a preset period; The data index information of the newly generated monitoring data is obtained, and the data index information of the newly generated monitoring data is updated into the time series data obtained according to the preset period, so as to incrementally update the local index information in the corresponding data block.

6. The method according to claim 5, characterized in that Also includes: According to a preset sampling time period, the preset log file is archived and compressed to obtain a data block.

7. The method according to claim 6, characterized in that The process of archiving and compressing the preset log file to obtain a data block according to the preset sampling time period also includes: If the preset sampling time period is greater than a preset time threshold, the data index information of the time series data obtained according to the preset period is updated to the global index information, and the local index information in the data block is updated.

8. A data processing device, characterized in that: The device comprises: A global index information acquisition module, used to respond to a data query instruction and acquire global index information; the global index information is data index information used to identify indicator index data obtained based on time series data, and the data index information is used to indicate a data block index; A target time series information acquisition module is used to acquire target time series information from a preset time series database according to the data block index; the target time series information is determined based on the target data block of the corresponding time period, wherein the local index information contained in each data block is used to store the position index information when the numerical position of the time series data of the corresponding time period is offset; The target sampling data determination module is used to obtain the target sampling data according to the target data block and the position index information.

9. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, for implementing the data processing method according to any one of claims 1 to 7 when executing a program stored in a memory.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the data processing method as described in any one of claims 1 to 7 is implemented.

11. A computer program product comprising instructions, characterized in that When the computer program product is run on a computer, the computer is enabled to execute the data processing method according to any one of claims 1 to 7.

Images