A method for data storage engine to process time series data and system thereof

Abstract

The application discloses a method and system for data storage engine to process time series data, and belongs to the field of data storage engine.The system is mainly composed of a preprocessing module, a transfer storage module, a query module, a deletion module and a recovery module, and meanwhile, data saving and processing of each processing module on an external storage device is realized through a preprocessing data area, a metadata storage area, an active data area and a compressed data area and the like.The method and system for data storage engine to process time series data can improve data memory management utilization, save data processing operation amount, reduce system CPU usage, and improve data processing operation speed.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of data storage engine, and in particular to a method for processing time series data by a data storage engine and a system thereof. BACKGROUND

[0002] Time series data is time series data, which represents a data column recorded in time order according to the same index. Analyzing time series data is helpful to find out the statistical characteristics and development law of sample time series. The amount of time series data will increase linearly with time, and general data storage systems cannot handle time series data well, so special time series databases are needed. In the time series database, tag sets represent some basic attributes of a collection point, such as device ID, device name and other information; field represents a characteristic value, such as the value of a certain register.

[0003] In a general time series processing system, data is divided into independent shards by time region. And the user can specify the tag by himself, and store the data with the tag set plus a data field as a key, and the data in the time line of the field is stored. Therefore, when the data usage of a certain tag needs to be counted, the information of different shards needs to be scanned and summarized. Similarly, the deletion of time series data also needs to scan the data block, and only a mark is made for fake deletion, and the space will be released only when the recycling is triggered.

[0004] Chinese patent document publication number CN 108073625 A discloses a system and method for metadata information management. The system includes: an acquisition module for acquiring metadata change information in real time; a processing module for real-time stream processing of the metadata change information to obtain a processing result; and a storage module for storing the processing result. The embodiment can process metadata change information in real time and update metadata information in the metadata database in real time. In order to quickly realize data query and management, in the system for metadata information management, the metadata of each data block in the SSTable stored persistently to the disk is acquired and stored normally in the memory, so as to facilitate the use of these metadata to assist the execution of data retrieval query. However, with the increase of data volume, these resident memory metadata occupy a large amount of memory, becoming a bottleneck of memory performance and management.

[0005] With the linear growth of the number of data blocks, there are a large number of metadata, and the normal occupation of the memory becomes a bottleneck of memory management. However, when the data is stored, the data is only stored in the memory, and the normal occupation of the memory will cause the utilization rate of the data memory management to be reduced. At the same time, the system needs to call a large amount of metadata when querying data information, deleting corresponding data and recycling, which leads to large operation amount of the query, deletion and recycling operations, and it is difficult to allocate data storage space, the space utilization rate is low, and the CPU usage rate is high, which affects the data processing speed and reduces the response speed of the data query. SUMMARY

[0006] In order to overcome the defects of the prior art, the technical problem to be solved by the present application is to provide a method for processing time series data by a data storage engine, improve the utilization rate of data memory management, save data processing operation amount, reduce system CPU usage rate and improve data processing speed.

[0007] Correspondingly, another technical problem to be solved by the present application is to provide a system for processing time series data by a data storage engine, which can realize data transfer, query, deletion and recycling operation, and ensure that the data allocation and management process is more efficient and reasonable.

[0008] To achieve the above purpose, the present application adopts the following technical solutions:

[0009] The method for processing time series data by a data storage engine provided by the present application puts the original data into a preprocessing data area, and transfers the preprocessing data to the corresponding positions of a metadata storage area, an active data area and a compressed data area through a transfer operation.

[0010] The active data area is used for storing active data, and the active data includes time series data distinguished according to device ID.

[0011] The compressed data area is used for merging and compressing the active data to release the active data storage space.

[0012] The metadata storage area is used for storing metadata, and the metadata includes various index information data pointing to the active data and the compressed data.

[0013] The transfer operation includes: after scanning the preprocessing data, the data of each device is sorted by time and inserted into the corresponding active data block of the active data area, and the index information data is updated and recorded in the metadata storage area.

[0014] It is judged whether the number of active data blocks after the transfer operation exceeds a preset number threshold.

[0015] If yes, a compression operation is performed: a compressed data area is obtained, data of an active data block is extracted, compressed data is formed after being combined into the same array, the compressed data is stored into a compressed data block in the compressed data area, the active data block is released and a bitmap occupation mark of the active data block is cleared, index information data is updated and recorded into a metadata storage area, and the like.

[0016] If no, the copy operation is continued, and the data is inserted into the active data block.

[0017] The preferred technical scheme of the present application is to insert the data into the active data block, which comprises:

[0018] It is judged whether the storage space of the active data block exceeds a preset storage space threshold value.

[0019] If yes, a new active data block obtaining operation is performed: new active data is stored into a new active data block, index information data is updated and recorded into a metadata storage area, and the like.

[0020] If no, the copy operation is continued, and the data is inserted into the active data block.

[0021] The preferred technical scheme of the present application is that the new active data block obtaining operation comprises:

[0022] It is judged whether there is a free new active data block in the active data area of the device.

[0023] If yes, a new active data is stored into a new active data block, and a bitmap mark is updated.

[0024] If no, a corresponding new active data block space and a corresponding bitmap space are pre-allocated, a new active data block is created, a first available new active data block is obtained, a new active data is stored into the new active data block, and a bitmap mark is updated.

[0025] The preferred technical scheme of the present application is that the compression operation comprises:

[0026] It is judged whether there is a free compressed data block in the compressed data area of the device.

[0027] If yes, a compressed data block with a length meeting a use requirement is selected, the compressed data block is removed from a free compressed data block queue, and space information of the compressed data block is removed from a free compressed storage index area.

[0028] If no, a new compressed data block is added in the compressed data area, and compressed data is stored in the new compressed data block.

[0029] The preferred technical scheme of the present application is that if the time point of the active data block when the data is inserted has been archived, then the following is performed: the corresponding compressed data of the compressed data block is extracted, enough active data blocks are obtained for decompression data, the corresponding compressed data area of the compressed data is released, the corresponding index information data is updated and recorded in the metadata storage area.

[0030] The preferred technical scheme of the present application is that the data appending and recording operation is performed in the pre-processing data area, including:

[0031] It is judged whether the number of data records exceeds the preset record number threshold value;

[0032] If yes, a new file is created for storing the collected data of the device;

[0033] If no, the collected data of the device is stored in the created file.

[0034] The preferred technical scheme of the present application is that the data is queried according to the device ID and the time stamp range, the corresponding device index data is found through the index area of the metadata storage area, the query operation is performed, and includes:

[0035] It is judged whether the data set needs to be returned;

[0036] If yes, the data of the corresponding active data block or compressed data block is extracted and returned;

[0037] If no, the corresponding statistical information data is returned.

[0038] The preferred technical scheme of the present application is that the data is deleted according to the device ID and the time point, the corresponding index is found through the device ID, the data block to be deleted is found according to the time point, the data type of the data block to be deleted is judged, and the data block deletion operation is performed, including:

[0039] If the data block to be deleted is an active data block, then the following is performed: the active data block is released, the corresponding index information data is updated and recorded in the metadata storage area;

[0040] If the data block to be deleted is a compressed data block, it is judged whether the compressed data block contains data that needs to be retained, and the compressed data block deletion operation is performed, including:

[0041] If yes, the following is performed: the data of the compressed data block is decompressed, the data that does not need to be retained is deleted, and the data that needs to be retained is saved to the active data block, the corresponding index information data is updated and recorded in the metadata storage area;

[0042] If no, the compressed data block is released, the corresponding index information data is updated and recorded in the metadata storage area;

[0043] determining whether there is still data block to be deleted;

[0044] if yes, continuing the data block deletion operation;

[0045] if no, ending the task triggering process.

[0046] The preferred technical solution of the present application is that when the proportion of the idle space to the entire compressed data area exceeds a preset proportion threshold, a recycling operation is triggered, the index area is scanned to find out the data block to be recycled corresponding to each device, the data type of the data block to be recycled is determined, and the data block recycling operation is performed, including:

[0047] if the data block to be recycled is a compressed data block, copying the compressed data block to a new compressed data area is performed;

[0048] if the data block to be recycled is an active data block, merging the continuous active data blocks and compressing and storing to a new compressed data area are performed;

[0049] determining whether there is still device in the index area that has not been subjected to the data block recycling processing;

[0050] if yes, continuing the data block recycling operation for the device;

[0051] if no, ending the task triggering process.

[0052] The present application provides a system for processing time sequence data by a data storage engine, comprising a preprocessing module, a storing module, a querying module, a deleting module and a recycling module.

[0053] The preprocessing module is used for putting data into a preprocessing data area.

[0054] The storing module is used for storing the data in the preprocessing data area to corresponding positions in a metadata storage area, an active data area and a compressed data area.

[0055] The querying module is used for extracting corresponding data from the active data area or the compressed data area and returning, and extracting corresponding statistical information data and returning.

[0056] The deleting module is used for performing corresponding data block deletion processing on the metadata storage area, the active data area and the compressed data area.

[0057] The recycling module is used for performing corresponding data block recycling processing on the metadata storage area, the active data area and the compressed data area.

[0058] The present application has the following beneficial effects:

[0059] This invention provides a method and system for processing time-series data in a data storage engine. The system mainly consists of a preprocessing module, a transfer module, a query module, a deletion module, and a recycling module. Simultaneously, it utilizes data areas such as a preprocessed data area, a metadata storage area, an active data area, and a compressed data area to enable data storage and processing by each processing module on external storage devices. Specifically, the preprocessed data area contains the raw data written to the data system; the active data area contains the latest time-series data obtained from the preprocessed data area by device; the compressed data area contains data obtained by compressing and merging multiple active data sets; and the metadata storage area contains various indexes pointing to the active and compressed data. Using the system and data processing method of this invention can improve data memory management utilization, reduce data processing computation, decrease system CPU utilization, and increase data processing speed. Attached Figure Description

[0060] Figure 1 This is a schematic diagram of the module control connection of a system for processing time-series data by a data storage engine, provided in a specific embodiment of the present invention.

[0061] Figure 2 This is a schematic diagram of the preprocessing operation flow of a method for processing time-series data in a data storage engine, provided in a specific embodiment of the present invention.

[0062] Figure 3 This is a schematic diagram of the transfer and compression operation process of the method for processing time-series data by a data storage engine provided in a specific embodiment of the present invention.

[0063] Figure 4 This is an index pointing diagram of the method for processing time-series data by a data storage engine provided in a specific embodiment of the present invention.

[0064] Figure 5 This is a schematic diagram of the index structure of a method for processing time-series data in a data storage engine, provided in a specific embodiment of the present invention.

[0065] Figure 6 This is a schematic diagram of device entries for a method for processing time-series data in a data storage engine, provided in a specific embodiment of the present invention.

[0066] Figure 7 This is a flowchart illustrating the process of obtaining new active data blocks in a method for processing time-series data in a data storage engine, as provided in a specific embodiment of the present invention.

[0067] Figure 8 This is a flowchart illustrating the method for releasing active data blocks in a data storage engine for processing time-series data, as provided in a specific embodiment of the present invention.

[0068] Figure 9is a compression operation flowchart of the method for processing time-series data by a data storage engine provided in the embodiment of the present application.

[0069] Figure 10 is a flowchart of releasing a compressed data area of the method for processing time-series data by a data storage engine provided in the embodiment of the present application.

[0070] Figure 11 is a data query operation flowchart of the method for processing time-series data by a data storage engine provided in the embodiment of the present application.

[0071] Figure 12 is a data deletion operation flowchart of the method for processing time-series data by a data storage engine provided in the embodiment of the present application.

[0072] Figure 13 is a data recycling operation flowchart of the method for processing time-series data by a data storage engine provided in the embodiment of the present application.

[0073] Figure 14 is a time-series data organization manner of the method for processing time-series data by a data storage engine provided in the embodiment of the present application.

[0074] Figure 15 is an internal data organization manner of an active data block of the method for processing time-series data by a data storage engine provided in the embodiment of the present application.

[0075] In the figure:

[0076] Preprocessing module 1; transfer module 2; query module 3; deletion module 4; recycling module 5. EMBODIMENT

[0077] The technical solutions of the present application will be further illustrated by specific embodiments in combination with the accompanying drawings.

[0078] Embodiment one

[0079] As Figures 1 to 15As shown, in order to improve the utilization rate of data memory management, save data processing operation amount, reduce system CPU usage, and improve data processing speed, further, the method for processing time series data provided in the embodiment is used to put the original data into the pre-processing data area, and through the transfer operation, the pre-processing data is respectively transferred to the corresponding positions of the metadata storage area, the active data area, and the compressed data area; the active data area is used to store active data, and the active data includes time series data distinguished according to device ID; the compressed data area is used to merge and compress the active data to release the active data storage space; and the metadata storage area is used to store metadata, and the metadata includes various index information data pointing to the active data and the compressed data.

[0080] The transfer operation (S1) includes: after scanning the pre-processing data, the data of each device is respectively sorted by time and inserted into the corresponding active data block of the active data area, and the index information data is updated and recorded in the metadata storage area.

[0081] It is judged whether the number of active data blocks after the transfer operation (S1) exceeds a preset number threshold.

[0082] If yes, the compression operation (S2) is performed: the compressed data area is obtained, the data of an active data block is extracted, merged into the same array, compressed to form compressed data, and stored in the compressed data block in the compressed data area, the active data block is released and the bitmap occupation flag of the active data block is cleared, and the index information data is updated and recorded in the metadata storage area.

[0083] If no, the transfer operation (S1) is continued, and the data is inserted into the active data block.

[0084] In order to simplify the design of the application, each storage node has a limit of a maximum number of service devices (such as 20K), and the index adopts a pre-allocated manner. Each index is 1K, and the storage node needs to pre-allocate index nodes. A single storage needs to pre-allocate 20M (20K*1K) of index space. The necessary tag of the application is only the device ID, and other tags can be embodied in the metadata area, but the uniqueness is not guaranteed, so random tag query is not supported.

[0085] The active data area is composed of data blocks of a fixed size. In the embodiment, the size of each data block is 16K, and the preset number threshold of the active data block is 50. The data transfer is first transferred to the active data block, the device index needs to point to the corresponding active data block, and then each device has at most 50 active data blocks, and when 50 active data blocks are reached, the compressed data needs to be merged.

[0086] The index structure includes a description of the query point, the number of stored entries, a data block pointer, a collection point, a collection record number, a collection record timestamp, a file type, a file offset, and a file length, as shown in Figure 5

[0087] The migration system scans the preprocessed file, extracts the collection point that needs to be supported for query, inserts the data node into the corresponding position, and updates the current processed offset to the metadata area. Through the offset and the inserted node position of the data, the required data can be found in the file.

[0088] As shown in Figures 14 to 15 , the time series data is represented by a matrix of f1~fn by t1~tm, and the active data block is stored in time units.

[0089] To facilitate subsequent data query, the data in a single data block is sorted by time, and the inserted data needs to be sorted. Data is generally processed in chronological order, so there will be a large number of append operations for single device data storage. If the data time is more than a day greater than the current time, it is discarded directly.

[0090] When the number of active data blocks exceeds a preset threshold (such as 50), the device triggers to compress data, extracts the data of the earlier active data block, merges it into the same array, and then compresses and updates the index. The compressed data of the active data block is stored in the compressed data area to form a compressed data block, and the file offset and file length in the data node linked list in the device index record are added to the compressed data area, and the active data block is released. Through the file offset and file length, the required data can be found in the file, the memory occupancy rate is reduced through the compressed data, the memory space is released to store new data, and the memory space is reasonably allocated and efficiently utilized.

[0091] Through the above process, the data memory management utilization rate is improved, the data processing operation amount is saved, the system CPU usage rate is reduced, and the data processing running speed is improved.

[0092] To facilitate subsequent data acquisition of storage space, the limited memory space is reasonably allocated and utilized, and further, the data is inserted into the active data block, including:

[0093] determining whether the storage space of the active data block exceeds a preset storage space threshold;

[0094] If yes, a new active data block operation is performed: storing new active data into a new active data block, updating index information data and recording to the metadata storage area;

[0095] If no, continue the migration operation (S1) to insert the data into the active data block. ​

[0096] In this embodiment, the maximum threshold of each data block is set to 16K, and when the threshold is exceeded, a new active data block needs to be obtained, new data is stored in the new active data block, and the index record of the device is updated. The data is stored in the specified storage space, the limited memory space is reasonably allocated and utilized, and the utilization rate of the data memory space is improved.

[0097] In order to reasonably allocate memory space for active data, further, the operation of obtaining a new active data block includes:

[0098] determining whether there is a free new active data block in the active data area of the device;

[0099] If yes, then execute: store new active data in the new active data block, and update the bitmap flag;

[0100] If no, then execute: pre-allocate corresponding new active data block space and corresponding bitmap space, create a new active data block, then obtain the first available new active data block, store new active data in the new active data block, and update the bitmap flag.

[0101] Since each data block is of a fixed size (for example, 16K), the bitmap method is used to manage whether the data block has been allocated and used. If there is no free data block, 16K*1K*8=128M data blocks need to be pre-allocated as storage space for new data, and corresponding bitmap (1K) space needs to be allocated, then the first available data block is obtained, and the corresponding bitmap flag is set. If the active data block needs to be released, only the corresponding active data block bitmap occupation flag needs to be cleared, and it can be reused subsequently. Through the above process, the memory space is reasonably allocated and efficiently utilized.

[0102] In order to reasonably allocate memory space for compressed data, further, the compression operation (S2) includes: determining whether there is a free compressed data block in the compressed data area of the device;

[0103] If yes, then execute: select a compressed data block with a length that meets the use requirement, remove the compressed data block from the free compressed data block queue, and remove the space information of the compressed data block from the free compressed storage index area;

[0104] If no, then execute: append a new compressed data block to the compressed data area, and store compressed data in the new compressed data block.

[0105] Through the above process, it is known that obtaining available compressed data area only needs to scan the sequence and return the first compressed data block with sufficient length, and then remove the compressed data block from the free compressed storage index area, to realize the storage of compressed data. If it is necessary to release the storage space of the compressed storage area, the compressed data block needs to be removed from the metadata storage index area, the compressed data block is removed from the metadata, and the information of the released compressed data area is added in the free compressed storage index area.

[0106] In order to ensure the reliability of data transfer and avoid the waste of memory resources caused by repeated storage, further, if the time point of the active data block when the data is inserted has been archived, the following is performed: extracting the corresponding compressed data of the compressed data block, obtaining sufficient active data blocks for decompressing data, releasing the corresponding compressed data area of the compressed data, updating the corresponding index information data and recording it in the metadata storage area.

[0107] If the time point of the inserted data belongs to the archived, the corresponding compressed data needs to be taken out, sufficient data blocks are obtained for decompressing data, the corresponding compressed storage area is released, and the corresponding index is updated. After inserting the corresponding data, it is temporarily stored in the working area. Through the above process, the reliability of data transfer is ensured, and the waste of memory resources caused by repeated storage is avoided.

[0108] In order to realize the collection and recording of data, further, data appending and collection and recording operation are performed in the preprocessing data area, including:

[0109] Judging whether the number of data records exceeds the preset record number threshold;

[0110] If yes, a new file is created for storing the collection data of the device;

[0111] If no, the collection data of the device is stored in the created file.

[0112] The data is preprocessed (S0). First, it is ensured that the data is not lost after entering the system and stored. It also plays a role in peak clipping and valley filling. Generally, after the data is successfully processed in the preprocessing module, it is considered to be solidified. In this case, even if an extreme situation (such as power failure) occurs, it will not be lost. If further data integrity and security are required, duplication can be performed, which is not discussed in the scope of the method.

[0113] The preprocessing operation (S0) adopts a WAL similar to a general database system, records the acquisition record length and the acquisition record itself. The acquisition record contains the device ID, the acquisition point, the acquisition tag, the data type, the acquisition timestamp, and the actual data. When the number of acquisition reaches a preset threshold (for example, 100000), a new file is created for storing the acquisition data of the new device. The file is named by a number (for example, 0000...9999), and the data can be compressed or deleted regularly according to actual needs. All the data in the metadata storage area, the active data area, and the compressed data area are converted and stored from the preprocessing data area, so the distributed purpose can be achieved by directly copying the preprocessing data area.

[0114] Through the above process, the data is preprocessed, and the acquisition record effective information is collected, so as to facilitate subsequent operation and processing of the data.

[0115] In order to facilitate data query and improve the response speed of data query, further, the data is queried according to the device ID and the timestamp range of the data, the corresponding device index data is found in the index area of the metadata storage area, and the query operation (S3) is performed, including:

[0116] It is judged whether the data set needs to be returned;

[0117] If yes, the data of the corresponding active data block or compressed data block is extracted and returned;

[0118] If no, the corresponding statistical information data is returned.

[0119] The application supports query by device ID and timestamp range, and does not support whole library scanning query. Through the device ID and the data time range, the corresponding device index data (binary search) is found in the index area, if only the basic information such as usage is queried, the corresponding statistical information data can be directly returned. For the specific device data usage, only the relevant information of the corresponding device in the metadata storage area needs to be read and added, and the data block does not need to be scanned. Otherwise, the data of the corresponding active data block or compressed data block is extracted and returned. The index records the current time range and the corresponding position, and the data block supported by a single data query is not more than 2 compressed data blocks or 50 active data blocks in principle. If the application end has a large data return demand (such as export), multiple times of acquisition are required.

[0120] Through the above process, the data of a single device is concentrated together, data query is facilitated, and according to the query demand, the basic information such as the number of data and the usage can be directly returned without scanning the device data block, the response speed of data query is improved, the CPU occupancy rate is effectively reduced, and the system running speed is improved.

[0121] In order to facilitate data deletion, ensure the accuracy of data deletion, avoid data deletion operation, further, according to the device ID and time point for data deletion, find the corresponding index through the device ID, find the data block to be deleted according to the time point, judge the data type of the data block to be deleted, execute the data block deletion operation (S4), including:

[0122] If the data block to be deleted is an active data block, execute: release the active data block, update the corresponding index information data and record it in the metadata storage area;

[0123] If the data block to be deleted is a compressed data block, it is necessary to judge whether the compressed data block contains data that needs to be retained, and execute the compressed data block deletion operation, including:

[0124] If yes, execute: data decompression is performed on the compressed data block, the data that does not need to be retained is deleted, and the data that needs to be retained is saved to the active data block, the corresponding index information data is updated and recorded in the metadata storage area;

[0125] If no, execute: release the compressed data block, update the corresponding index information data and record it in the metadata storage area;

[0126] Judge whether there is still a data block to be deleted;

[0127] If yes, continue the data block deletion operation;

[0128] If no, end the task triggering process.

[0129] The present application only supports deleting data before a certain time point through the device ID and finding the corresponding index according to the time point to find the data block to be deleted. If it is an active data block, the active data block is directly released, and then the corresponding index is updated. If it is a compressed data block, the compressed data block is released, and the corresponding index is updated. If the compressed data block contains a part that is not deleted, the data that needs to be retained is stored in the active data block after being decompressed, and then the index is updated. For deletion, the present application will release the corresponding compressed data block, release the storage space, realize the reasonable allocation and efficient utilization of space, and the compressed data block can be reused.

[0130] Through the above process, the data deletion process does not need to scan the actual data information of the data block, but finds the data block index to be deleted through the device ID and the time point and realizes the data block deletion, saves the data processing operation amount, reduces the system CPU usage rate, and improves the deletion data processing running speed. At the same time, the data that needs to be retained in the compressed data block can be decompressed and saved, avoiding data deletion operation, ensuring the accuracy of data deletion.

[0131] To reduce the memory occupation of the recycled data and reasonably use the memory space, further, when the proportion of the idle space to the entire compressed data area exceeds a preset proportion threshold, a recycling operation is triggered, the index area is scanned, the to-be-recycled data block corresponding to each device is found, the data type of the to-be-recycled data block is judged, and a data block recycling operation (S5) is performed, including:

[0132] If the to-be-recycled data block is a compressed data block, the following is performed: copying the compressed data block to a new compressed data area;

[0133] If the to-be-recycled data block is an active data block, the following is performed: merging and compressing continuous active data blocks and storing them in the new compressed data area;

[0134] It is judged whether there is still a device in the index area that has not undergone the data block recycling processing;

[0135] If yes, the data block recycling operation continues for the device;

[0136] If no, the task triggering process is ended.

[0137] When the idle space occupies the entire compressed data area by a certain proportion, a recycling is triggered. The recycling process is as follows: the index area is scanned, the compressed data block corresponding to each device is found, the data block is copied to a new compressed storage area. If there is an active data block, the active data block can be merged and compressed and stored in the new compressed storage area. Until all devices are completed. In the recycling process, the query of the old data has no influence, and the writing has no influence, because the new data can be directly written to the preprocessing data area. Since the new data only exists in the preprocessing data area, the query service cannot be directly provided.

[0138] Through the above process, the deleted data block can be quickly recycled and utilized, at the same time, by reasonably using the memory idle space, the memory occupation of the recycled data is reduced, and the efficient management and utilization of the memory space are realized.

[0139] Embodiment Two

[0140] As Figure 1As shown, in order to realize the data transfer, query, deletion and recovery operation, improve the data memory management utilization rate, save the data processing operation amount, reduce the system CPU usage rate, improve the data processing running speed, further, the system for processing time series data of data storage engine provided in the embodiment, including preprocessing module 1, transfer module 2, query module 3, deletion module 4 and recovery module 5; preprocessing module 1 is used for putting data into preprocessing data area; transfer module 2 is used for transferring the data of preprocessing data area to the corresponding position of metadata storage area, active data area and compressed data area; query module 3 is used for extracting corresponding data from active data area or compressed data area and returning, and extracting corresponding statistical information data and returning; deletion module 4 is used for corresponding data block deletion processing to metadata storage area, active data area and compressed data area; recovery module 5 is used for corresponding data block recovery processing to metadata storage area, active data area and compressed data area. Through the above system module and combined with the data processing method of embodiment one, the data transfer, query, deletion and recovery operation are realized, the data memory management utilization rate is improved, the data processing operation amount is saved, the system CPU usage rate is reduced, and the data processing running speed is improved.

[0141] The application is described by preferred embodiments, and those skilled in the art know that various changes or equivalent replacements can be made to the features and embodiments without departing from the spirit and scope of the application. The application is not limited by the specific embodiments disclosed herein, and other embodiments falling within the scope of the claims of the application are within the scope of the application.

[0142] It should be noted that in the claims, any reference signs placed between parentheses shall not be construed as limiting the claim. The word 'comprising' does not exclude the presence of elements or steps not listed in a claim. The word 'a' or 'an' preceding an element does not exclude the presence of a plurality of such elements. The application can be implemented by means of hardware comprising several distinct elements, and by means of a suitably programmed computer. In the unitary claim, several apparatuses can be listed with corresponding sub-claims in the dependent claims. Individual features mentioned in different dependent claims, can be combined and the application can also be implemented by combining the features of the dependent claims and those of the independent claims. The use of relative terms in the claims (for example, 'about') does not exclude the equivalent embodiments falling within the scope of the application.

[0143] Those skilled in the art will appreciate that embodiments of the application can be devised for use with systems other than the application described herein. Embodiments of the application can be embodied in a variety of ways, including as a method, as a system, or as a computer program product. Accordingly, the application can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the application can take the form of a computer program product on one or more computer-usable storage media (including, but not limited to, magnetic disks, CD-ROMs, optical storage media, and the like) embodying computer program code thereon for use with any suitable processor or processors.

[0144] The present application is described in reference to the flowchart and / or block diagram of the method, apparatus (system) and computer program product according to an embodiment of the present application. It will be understood that each block of the flowchart and / or block diagram, and combinations of blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart and / or block diagram block or blocks. Figure 1 one or more flowcharts and / or blocks Figure 1 means for carrying out the function specified by the block or blocks.

[0145] These computer program instructions can also be stored in a computer- readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instructions which implement the flowchart and / or block diagram block or blocks. Figure 1 one or more flowcharts and / or blocks Figure 1 means for carrying out the function specified by the block or blocks.

[0146] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the flowchart and / or block diagram block or blocks. Figure 1 one or more flowcharts and / or blocks ​ means for carrying out the function specified by the block or blocks.

[0147] While the preferred embodiments of the application have been described, additional variations and modifications can be made to the embodiments by those of skill in the art once they have the benefit of the present disclosure without departing from the spirit and scope of the application. Accordingly, the attached claims are intended to cover all such variations and modifications as falling within the scope of the application.

[0148] Obviously, numerous modifications and variations of the present application are possible in light of the above teachings. It is therefore to be understood that within the scope of the appended claims and their equivalents, the application can be practiced otherwise than as specifically described.

[0149] In the description of the specification, the description of the terms "one embodiment", "some embodiments", "an example", "a specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In the specification, the illustrative description of the above terms should not be understood as necessarily referring to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any suitable manner in any one or more embodiments or examples. In addition, the person skilled in the art can combine and combine the different embodiments or examples described in the specification and the features of the different embodiments or examples without contradiction.

[0150] The above description is only the preferred embodiment of the present application, and any equivalent changes and modifications made within the scope of the patent application of the present application shall belong to the scope of the present application.

Claims

1. A method for data storage engine processing time series data, raw data is put into a pre-processing data area to obtain pre-processed data, and the pre-processed data is respectively copied to corresponding positions of a metadata storage area, an active data area and a compressed data area through a copy operation, characterized in that: the active data area is used to store active data, the active data includes time series data distinguished by device ID, and the time series data is a data column representing data recorded in time sequence for the same index; the compressed data area is used to merge and compress the active data to release active data storage space; the metadata storage area is used to store metadata, the metadata includes various index information data pointing to the active data and the compressed data; the copy operation includes: after scanning the pre-processed data, the data of each device is respectively sorted by time and inserted into the corresponding active data block of the active data area, and the index information data is updated and recorded in the metadata storage area; it is judged whether the number of active data blocks after the copy operation exceeds a preset number threshold; if yes, a compression operation is performed: the compressed data area is obtained, data of a plurality of active data blocks located in a previous position in time sequence is extracted, the data is merged into an array for compression to form compressed data, the compressed data is stored in a compressed data block in the compressed data area, the active data block is released and the bitmap occupation flag of the active data block is cleared, and the index information data is updated and recorded in the metadata storage area; if no, the copy operation is continued, and the data is inserted into the active data block. 2.The method for data storage engine processing time series data according to claim 1, characterized in that: the data is inserted into the active data block, including: it is judged whether the storage space of the active data block exceeds a preset storage space threshold; if yes, a new active data block operation is performed: new active data is stored in a new active data block, and the index information data is updated and recorded in the metadata storage area; if no, the copy operation is continued, and the data is inserted into the active data block. 3.The method for data storage engine processing time series data according to claim 2, characterized in that: the new active data block operation includes: it is judged whether there is an idle new active data block in the active data area of the device; if yes, new active data is stored in a new active data block, and the bitmap flag is updated; if no, corresponding new active data block space and corresponding bitmap space are pre-allocated, a new active data block is created, a first available new active data block is obtained, new active data is stored in the new active data block, and the bitmap flag is updated. 4.The method for data storage engine processing time series data according to claim 1, characterized in that: the compression operation includes: it is judged whether there is an idle compressed data block in the compressed data area of the device; ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ If yes, then execute: select a compressed data block with a length satisfying the use requirement, remove the compressed data block from the free compressed data block queue, and remove the space information of the compressed data block from the free compressed storage index area; If no, then execute: append a new compressed data block to the compressed data area, and store the new compressed data block in the compressed data.

5. The method for processing time-series data by a data storage engine according to claim 1, wherein: If the time point of the active data block when the data is inserted has been archived, then execute: extract the corresponding compressed data of the compressed data block, obtain enough active data blocks for decompression, release the corresponding compressed data area of the compressed data, update the corresponding index information data and record it in the metadata storage area.

6. The method for processing time-series data by a data storage engine according to claim 1, wherein: perform data appending and data collection operations in the preprocessed data area, including: determine whether the number of data records exceeds the preset record threshold; If yes, then execute: create a new file for storing the collected data of the device; If no, then execute: store the collected data of the device in the created file.

7. The method for processing time-series data by a data storage engine according to claim 1, wherein: query data according to the device ID and the timestamp range of the data, find the corresponding device index data through the index area of the metadata storage area, and perform the query operation, including: determine whether to return the data set; If yes, then execute: extract the data of the corresponding active data block or compressed data block and return it; If no, then execute: return the corresponding statistical information data.

8. The method for processing time-series data by a data storage engine according to claim 1, wherein: delete data according to the device ID and the time point, find the corresponding index through the device ID, find the data block to be deleted according to the time point, determine the data type of the data block to be deleted, and perform the data block deletion operation, including: If the data block to be deleted is an active data block, then execute: release the active data block, update the corresponding index information data and record it in the metadata storage area; If the data block to be deleted is a compressed data block, then determine whether the compressed data block contains data that needs to be retained, and perform the compressed data block deletion operation, including: If yes, then execute: decompress the compressed data block, delete the data that does not need to be retained, save the data that needs to be retained to the active data block, update the corresponding index information data and record it in the metadata storage area; If no, then execute: release the compressed data block, update the corresponding index information data and record it in the metadata storage area; determine whether there are still data blocks to be deleted; If yes, then continue the data block deletion operation; If no, then end the task triggering process.

9. The method for processing time-series data by a data storage engine according to claim 1, wherein: When the proportion of the idle space to the whole compressed data area exceeds a preset proportion threshold, a recovery operation is triggered, the index area is scanned to find out the data block to be recovered corresponding to each device, the data type of the data block to be recovered is judged, and the data block recovery operation is executed, including: If the data block to be recovered is a compressed data block, then the following is executed: copying the compressed data block to a new compressed data area; If the data block to be recovered is an active data block, then the following is executed: merging and compressing the continuous active data blocks and storing them to a new compressed data area; Judging whether there is still a device in the index area that has not undergone the data block recovery processing; If yes, then the data block recovery operation is continued for the device; If no, then the task triggering process is ended.

10. A system for processing time series data by a data storage engine, adopting the method for processing time series data by a data storage engine according to any one of claims 1 to 9, characterized in that: The system for processing time series data by a data storage engine comprises a preprocessing module (1), a storing module (2), a querying module (3), a deleting module (4) and a recycling module (5); The preprocessing module (1) is used for putting data into the preprocessing data area; The storing module (2) is used for storing the data in the preprocessing data area to the corresponding positions of the metadata storage area, the active data area and the compressed data area; The querying module (3) is used for extracting and returning the corresponding data from the active data area or the compressed data area, and extracting and returning the corresponding statistical information data; The deleting module (4) is used for performing corresponding data block deletion processing on the metadata storage area, the active data area and the compressed data area; The recycling module (5) is used for performing corresponding data block recycling processing on the metadata storage area, the active data area and the compressed data area.

Images