Time series data storage method and device, electronic equipment and storage medium

Abstract

The invention provides a time sequence data storage method and device, electronic equipment and a storage medium, and relates to the technical field of data storage, the method comprises the following steps: under the condition that first time sequence data is received, caching the first time sequence data into a memory space, and storing the first time sequence data into a log file of a disk space; under the condition that the timestamp of the first time sequence data cached in the memory space meets a time condition, writing the first time sequence data into a complete file of the disk space from the memory space, and deleting the first time sequence data in the memory space; and deleting the first time sequence data in the log file under the condition that the first time sequence data is written into the complete file of the disk space. According to the method, the problem of loss of the latest time series data in the memory caused by sudden downtime or fault of the equipment can be avoided.

Description

Technical Field

[0001] The present disclosure relates to the field of data storage technology, and in particular to a method, device, electronic device and storage medium for storing time series data. Background Art

[0002] Time series data refers to a set of observation data points arranged in chronological order, each of which contains a timestamp and a corresponding measurement or observation value. In the financial field, there is a need to collect, process, store and analyze time series data from different sources to achieve real-time monitoring and management of business processes, applications, etc. With the development of the financial industry, data is written more and more frequently and in larger quantities in monitoring scenarios. How to optimize the data storage reliability of monitoring systems or monitoring equipment is an urgent problem to be solved. Summary of the invention

[0003] The present disclosure provides a method, device, electronic device and storage medium for storing time series data. The technical solution is as follows:

[0004] According to one aspect of the present disclosure, a time series data storage method is provided, the method comprising:

[0005] When receiving the first time series data, cache the first time series data in a memory space, and store the first time series data in a log file in a disk space;

[0006] When the timestamp of the first time series data cached in the memory space meets the time condition, writing the first time series data from the memory space into the complete file in the disk space, and deleting the first time series data in the memory space;

[0007] When the first time series data is written into the complete file in the disk space, the first time series data in the log file is deleted.

[0008] According to another aspect of the present disclosure, a time series data storage device is provided, the device comprising:

[0009] A first storage module is used for caching the first time series data into a memory space and storing the first time series data into a log file in a disk space when the first time series data is received;

[0010] A second storage module is configured to write the first time series data from the memory space into a complete file in the disk space and delete the first time series data in the memory space if the timestamp of the first time series data cached in the memory space meets a time condition;

[0011] A deletion module is used to delete the first time series data in the log file when the first time series data is written into the complete file in the disk space.

[0012] According to one aspect of the present disclosure, an electronic device is provided, including: a processor and a memory storing a program, wherein the program includes instructions, and when the instructions are executed by the processor, the processor executes the time series data storage method as described above.

[0013] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause the computer to execute the time series data storage method as described above.

[0014] According to another aspect of the present disclosure, a computer program product is provided, the computer program product comprising computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the above-mentioned time series data storage method.

[0015] The beneficial effects brought by the technical solution provided by the embodiments of the present disclosure include at least:

[0016] The disclosed embodiment provides a method for storing time series data: a log file is added to the disk space, and when the device receives the latest time series data, it will not only be cached to the memory space first, but also written to the log file in the disk space, so as to realize the persistence of the latest time series data, avoid the loss of the latest time series data in the memory due to sudden downtime or failure of the device, and improve the storage reliability of data; at the same time, the latest time series data is cached in the memory space, which will also improve the query efficiency of the time series data; and, after the timestamp of the time series data stored in the memory space meets the time condition, the time series data will be written to the disk space and deleted from the memory space, so as to avoid the occupation of the memory space by the historical time series data and improve the space utilization rate of the memory space; in addition, after the historical time series data is written into the complete file of the disk space, the corresponding historical time series data in the log file will also be deleted synchronously, so as to avoid the storage of duplicate time series data in the disk, and further improve the space utilization rate of the disk space. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Further details, features and advantages of the present disclosure are disclosed in the following description of exemplary embodiments in conjunction with the accompanying drawings, in which:

[0018] Figure 1 A flowchart of a method for storing time series data according to an exemplary embodiment of the present disclosure is shown;

[0019] Figure 2A flowchart of another method for storing time series data according to an exemplary embodiment of the present disclosure is shown;

[0020] Figure 3 A flowchart of another method for storing time series data according to an exemplary embodiment of the present disclosure is shown;

[0021] Figure 4 A flowchart of another method for storing time series data according to an exemplary embodiment of the present disclosure is shown;

[0022] Figure 5 It is a structural schematic diagram of a time series data storage device provided by an embodiment of the present disclosure;

[0023] Figure 6 A structural block diagram of an exemplary electronic device that can be used to implement the embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0024] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0025] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0026] The term "including" and its variations used in this document are open inclusions, that is, "including but not limited to". The term "based on" means "based at least in part". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one other embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the following description. It should be noted that the concepts of "first", "second", etc. mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units. It should be noted that the modifications of "one" and "multiple" mentioned in the present disclosure are illustrative and not restrictive. Those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more". The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are for illustrative purposes only, and are not used to limit the scope of these messages or information.

[0027] The solution of the present invention is described below with reference to the accompanying drawings, and the technical solution provided by the embodiment of the present invention is explained in detail through specific embodiments and their application scenarios.

[0028] Please refer to Figure 1 , which shows a flow chart of a method for storing time series data according to an exemplary embodiment of the present disclosure. The method is applied to an electronic device as an example for explanation. Figure 1 As shown, the method includes:

[0029] Step 101: When first time series data is received, cache the first time series data in a memory space, and store the first time series data in a log file in a disk space.

[0030] In the related art, when storing time series data, since the newly written time series data is not persisted but cached in the memory, if the device crashes or fails at this time, this part of the time series data cannot be restored after the device is restored, resulting in the loss of part of the time series data. In order to solve the problem of time series data loss, the embodiment of the present disclosure improves the data structure of the disk space and adds a log file (append-only log or log file) to store the latest time series data, so as to achieve the persistence of the latest time series data.

[0031] On the basis of adding log files to the disk space, after the electronic device (which can be a monitoring device) receives the latest first time series data, the first time series data will be cached in the memory space first, and the first time series data will also be stored in the log file of the disk space; on the one hand, the first time series data cached in the memory space can improve the query efficiency of the time series data; on the other hand, the first time series data cached in the log file can realize the persistence of the latest time series data, avoiding sudden downtime or failure of the device and the loss of the latest time series data.

[0032] Exemplarily, the first time series data is a set of observation data arranged in time, and each observation data includes a timestamp and a corresponding measurement value (or observation value).

[0033] Step 102: When the timestamp of the first time series data cached in the memory space meets the time condition, the first time series data is written from the memory space to a complete file in the disk space, and the first time series data in the memory space is deleted.

[0034] Taking into account the limited nature of memory space, and generally having higher query and processing requirements for time series data in the latest time period, in order to avoid excessive memory space being occupied, it is also necessary to write the historical time series data cached in the memory space to the disk space in a timely manner. The corresponding disk space also includes complete block files for storing historical time series data that need to be persisted. In a possible implementation, when the timestamp of the first time series data cached in the memory space meets the time condition, the first time series data is written from the memory space to the complete file of the disk space, and the first time series data in the memory space is deleted, so that the memory space continues to store the latest time series data.

[0035] Among them, the time condition can be: the time difference between the timestamp of the first time series data and the current time is greater than the time threshold. Exemplarily, the time threshold can be 2 hours, and the time threshold can be set by business personnel. For example, the timestamp of the first observation data in the first time series data is 12:20, then at 14:21, the first time series data can be cached from the memory space to the disk space. Alternatively, the time condition can also be: the time difference between the storage time of the first time series data in the memory space and the current time is greater than the time threshold.

[0036] Step 103: When the first time series data is written into a complete file in the disk space, the first time series data in the log file is deleted.

[0037] After the first time series data is written into the complete file of the disk space, it means that the first time series data is no longer the latest time series data, is not stored in the memory space, and has achieved data persistence, that is, there is no problem of time series data loss. At this time, the first time series data in the log file can be deleted to avoid invalid occupation of disk space, so that the log file can be used to store the latest time series data.

[0038] In summary, the embodiments of the present disclosure provide a method for storing time series data: a log file is added to the disk space, and when the device receives the latest time series data, it will not only be cached to the memory space first, but also written to the log file in the disk space, so as to realize the persistence of the latest time series data and avoid the loss of the latest time series data in the memory due to sudden downtime or failure of the device; at the same time, the latest time series data is cached in the memory space and the query efficiency of the time series data is improved; moreover, after the timestamp of the time series data stored in the memory space meets the time condition, the time series data will be written to the disk space and deleted from the memory space, so as to avoid the occupation of the memory space by the historical time series data and improve the space utilization rate of the memory space; in addition, after the historical time series data is written into the complete file of the disk space, the corresponding historical time series data in the log file will also be deleted synchronously, so as to avoid the storage of duplicate time series data in the disk and further improve the space utilization rate of the disk space.

[0039] Since the latest time series data is not only cached in the memory space, but also written to the log file in the disk space, the latest time series data written to the disk space will not be deleted due to a sudden device crash. While preventing the loss of time series data, the latest time series data in the log file can also be used to restore the latest time series data cached in the memory space before the device crashed.

[0040] Please refer to Figure 2 , which shows a flowchart of another method for storing time series data according to an exemplary embodiment of the present disclosure. The method is applied to an electronic device as an example for explanation. Figure 2 As shown, the method includes:

[0041] Step 201 , when first time series data is received, cache the first time series data into a memory space, and store the first time series data into a log file in a disk space.

[0042] Step 202: When the timestamp of the first time series data cached in the memory space meets the time condition, the first time series data is written from the memory space to a complete file in the disk space, and the first time series data in the memory space is deleted.

[0043] Step 203: When the first time series data is written into a complete file in the disk space, the first time series data in the log file is deleted.

[0044] The implementation of steps 201 to 203 may refer to the above embodiments, and will not be described in detail in this embodiment.

[0045] Step 204 , when the first time series data is written into a complete file in the disk space, the latest time of the first time series data written into the complete file is obtained.

[0046] Since time series data is stored in real time in the log file, when the device is down and then restored, the memory space only needs to restore the latest time series data that has not been written into the complete file. In order for subsequent devices to know which time series data to read from the log file to restore the latest time series data in the memory space during fault recovery. The disk space in the disclosed embodiment also includes a check file for storing the time series data that has been written into the complete file in the disk space. In a corresponding possible implementation, after the first time series data is written into the complete file in the disk space, the latest time of the first time series data that has been written into the complete file is also obtained.

[0047] It should be noted that the latest time is the timestamp of the last (or latest) observation data in the first time series data.

[0048] Step 205, update the latest time into the disk space inspection file.

[0049] The latest time is further updated into the check file of the disk space, so that the check file always stores the time of the latest time series data written into the complete file.

[0050] Step 206, when a device failure is detected and recovered, based on the latest time in the check file, the time series data cached in the memory space before the device failure is restored.

[0051] When recovering after detecting a device failure, the latest time can be obtained from the check file in the disk space, and the required latest time series data can be read from the log file based on the latest time to recover the time series data cached in the memory space before the device failure.

[0052] It should be noted that the purpose of adding log files is to restore the latest time series data in memory after a downtime failure by adding log files and storing the latest time series data (consistent with the latest time series data cached in memory) without changing the original time series data storage logic (the latest time series data is cached in memory, and the historical data is cached in the complete file of disk space). If the time series data is restored by the complete file, since the time series data in the complete file stored in the disk space needs to be compressed multiple times to save space (first a single time series data is compressed, and then the time series data blocks of a fixed time period are compressed as a whole), it takes a long time to search and decompress the time series data required to restore the memory. However, since the log file stores the latest time series data, the time series data storage volume is small, and the storage structure is the same as the time series data required in the memory (both are time series data compressed from a single time series data), it is convenient to find the required time series data according to time, and there is no need to decompress the time series data in the log file. The memory data can be directly restored based on the time series data in the log file, which improves the recovery efficiency of the time series data in the memory.

[0053] In an illustrative example, step 206 may include step 206A and step 206B.

[0054] Step 206A: search and read the second time series data after the latest time from the log file.

[0055] Step 206B: write the second time series data into the memory space to restore the time series data cached in the memory space before the device failure.

[0056] Since the time series data is first cached in the memory space and then written to the complete file in the disk space, when restoring the latest time series data cached in the memory space, the timestamp of the latest time series data required in the memory space must be located after the latest timestamp of the time series data written in the complete file. Correspondingly, when restoring the latest time series data cached in the memory space, it is necessary to find and read the second time series data after the latest time from the log file, and then write the second time series data to the memory space to restore the time series data cached in the memory space before the device failure. For example, if the latest time is 13:20, and the log file has cached time series data from 13:00 to 14:22, the corresponding time series data from 13:20 to 14:22 is read from the log file as the second time series data for writing into the memory space for time series data recovery.

[0057] In this embodiment, by saving the latest time of the time series data written to the complete file, when the device failure is restored, the time series data after the latest time can be read from the log file based on the latest time to restore the time series data stored in the memory space before the device failure, ensuring that data can be quickly restored after a failure occurs, thereby improving the storage reliability of the time series data.

[0058] Since the collection of time series data is relatively high-frequency and in large quantities, if the time series data is directly cached in memory or written to disk space, it will obviously greatly increase the storage cost of memory space and disk space. In order to reduce the occupation of storage space and reduce storage costs, the embodiments of the present disclosure also compress the time series data when storing it.

[0059] Please refer to Figure 3 , which shows a flowchart of another method for storing time series data according to an exemplary embodiment of the present disclosure. The method is applied to an electronic device as an example for explanation. Figure 3 As shown, the method includes:

[0060] Step 301, when first time series data is received, the first time series data is written into a writable data block in a memory space, and the first time series data is stored in a log file in a disk space.

[0061] The time series data stored in the memory space consists of a series of closed data blocks and a writable data block. Each data block is used to store the time series data of the target duration. Taking the target duration as 2 hours as an example, the writable data block is used to store the time series data within the last 2 hours, and the closed data block is used to store the time series data before 2 hours. The writable data block is a string that can only be appended. New time series data is appended to the string. Once the writable data block is full of 2 hours, it will be closed. The closed data block cannot be modified until it is deleted from the memory.

[0062] Moreover, when it is determined that the target time difference of all the first time series data written in the writable data block reaches the time difference threshold, it is necessary to copy all the first time series data written in the writable data block to the target closed data block so that the latest time series data can continue to be written to the writable data block. The target time difference is the time difference corresponding to the earliest observed data and the latest observed data written in the writable data block.

[0063] In a possible implementation, after the electronic device receives the first time series data, it writes the latest first time series data into a writable data block in the memory space, and also writes the latest first time series data into a log file in the disk space.

[0064] In order to reduce the space occupied by the time series data in the memory, the time series data is also compressed during the process of writing the time series data into the writable data block. Correspondingly, in an exemplary example, writing the first time series data into the writable data block in the memory space also includes step 301A and step 301B.

[0065] Step 301A: perform data compression processing on the first time series data to obtain compressed first time series data.

[0066] Step 301B: write the compressed first time series data into a writable data block in the memory space.

[0067] First, data compression processing is performed on the first time series data to obtain compressed first time series data, and then the compressed first time series data is written into a writable data block in the memory space.

[0068] Optionally, the process of performing data compression processing on the first time series data may also include step 301A1 and step 301A2.

[0069] Step 301A1, calculate the time difference between the observation times of adjacent observation data in the first time series data, and the numerical difference between the observation values ​​of adjacent observation data.

[0070] Among them, the first time series data is a group of observation data points sorted in chronological order, which contains multiple observation data, and each observation data includes an observation value and an observation time. Based on the characteristics of the first time series data, when compressing the first time series data, it is performed through two methods: timestamp compression and data compression. In a possible implementation, the time difference between the observation times of adjacent observation data in the first time series data and the numerical difference between the observation values ​​of adjacent observation data are calculated respectively, and then the timestamps in other observation data except the first observation data in the first time series data are converted into time differences, and the observation values ​​are converted into numerical differences, thereby significantly reducing the length of timestamps and observation values, and achieving the purpose of compressing data.

[0071] Optionally, each observation data includes a key for indicating the type of the observation data in addition to the timestamp and the observation value. For example, the observation data is <key, timestamp, observation value>.

[0072] Step 301A2, based on the time difference, the numerical difference and the first observation data in the first time series data, determine the compressed first time series data.

[0073] Furthermore, the time difference and the numerical difference corresponding to the first observation data and other observation data are determined as the compressed first time series data.

[0074] For example, if the first time series data is:<key,13:00,10%> ,<key,13:05,15%> ,<key,13:10,25%> ,<key,13:15,30%> ,<key,13:20,40%> , then the compressed first time series data can be:<key,13:00,10%> ,<key,5,5> ,<key,5,10> ,<key,5,5> ,<key,5,10> It can be seen that in the compressed first time series data, except for the first observation data<key,13:00,10%> In addition, the timestamps in other observation data all store the time difference between the timestamp corresponding to the previous observation data, and the observation values ​​all store the numerical difference between the observation values ​​corresponding to the previous observation data.

[0075] When the first time series data is subsequently restored, the timestamps and observation values ​​of other observation data can be calculated through simple mathematical operations based on the first observation data, the time difference and the value difference.

[0076] Step 302, when the target time difference of the first time series data written in the writable data block reaches the time difference threshold, the first time series data is copied to the target closed data block in the memory space, and the latest time series data is written to the writable data block.

[0077] Since the writable data block is used to store the latest target duration data, when it is determined that the target time difference of all the first time series data written in the writable data block reaches the time difference threshold, all the first time series data written in the writable data block needs to be copied to the target closed data block so that the latest time series data can continue to be written to the writable data block. Among them, the target time difference is the time difference corresponding to the earliest observation data and the latest observation data written in the writable data block.

[0078] Step 303: When the timestamp of the first time series data in the target closed data block meets the time condition, all the first time series data in the target closed data block are written into a complete file in the disk space.

[0079] When the timestamp of the first time series data in the target closed data block meets the time condition, all the first time series data in the target closed data block are written into a complete file in the disk space.

[0080] In order to save disk space, the time series data in the target closed data block can be compressed again when it is written into the disk space. In an exemplary example, writing all the first time series data in the target closed data block into a complete file in the disk space can also include steps 303A to 303C.

[0081] Step 303A, compress all first time series data in the target closed data block to obtain compressed block data.

[0082] Step 303B, obtaining the target time range corresponding to all first time series data in the target closed data block.

[0083] Step 303C, write the compressed block data and the target time range into a complete file in the disk space.

[0084] Different from compressing the first time series data, when writing all the time series data in the target closed data block to the disk space, it is necessary to perform overall compression based on the block data. In one possible implementation, all the first time series data in the target closed data block are compressed, that is, all the first time series data are compressed as a whole to obtain compressed block data; and in order to facilitate the subsequent search for the time series data of the corresponding time from the disk space, it is also necessary to obtain the target time range corresponding to all the first time series data in the target closed data block, and write the compressed block data and the target time range into the complete file of the disk space.

[0085] For example, if the target closed data block stores time series data between 13:00 and 15:00, the target time range is 13:00 to 15:00. If time series data between 13:00 and 15:00 is needed later, the corresponding compressed block data is fed back to the requesting client, and the compressed block data is decompressed in the requesting client to obtain the required time series data.

[0086] Optionally, when compressing all first time series data in the target closed data block, data compression algorithms that can be used include: LZW compression, Huffman compression, run-length coding (RLC) and other algorithms, which are not limited to this embodiment.

[0087] Step 304, deleting the first time series data in the target closed data block in the memory space.

[0088] After all the time series data in the target closed data block in the memory space are written into the disk space, all the first time series data in the target closed data block may be correspondingly deleted to store new time series data.

[0089] Step 305 , when the first time series data is written into a complete file in the disk space, the first time series data in the log file is deleted.

[0090] The implementation of step 305 may refer to the above embodiment, and this embodiment will not be described in detail here.

[0091] In this embodiment, by setting a specific compression algorithm to compress the time series data, the occupancy rate of the time series data on memory space and disk space can be reduced, further saving memory space and disk space; and the data can be restored in a specific manner subsequently, while improving space utilization and ensuring the accuracy of the time series data.

[0092] The latest time series data is cached in the memory space to improve the query efficiency of the time series data and avoid the need to search and read the latest time series data from the disk space. In addition to improving the query efficiency by setting the storage location, this embodiment further improves the query efficiency and processing efficiency of the time series data by improving the storage structure and index mechanism of the latest time series data in the memory space.

[0093] Please refer to Figure 4 , which shows a flowchart of another method for storing time series data according to an exemplary embodiment of the present disclosure. The method is applied to an electronic device as an example for explanation. Figure 4 As shown, the process of writing the compressed first time series data into the writable data block in the memory space may further include the following steps:

[0094] Step 401: Obtain the time series data name of the first time series data.

[0095] Step 402: Hash the time series data name to obtain a target shard identifier.

[0096] Step 403: write the compressed first time series data into the target data shard indicated by the target shard identifier in the writable data block.

[0097] The writable data block can be divided into multiple data slices, each of which has a corresponding slice identifier. When storing the first time series data into the writable data block, first obtain the time series data name of the first time series data, that is, the key of the first time series data, and use a hash algorithm to process the time series data name to obtain a target slice identifier, and then write the compressed first time series data into the target data slice indicated by the target slice identifier in the writable data block.

[0098] Step 404: Obtain a smart pointer pointing to the memory address of the first time series data storage.

[0099] Step 405 : construct a first relationship mapping table using the time series data name of the first time series data as a key and the smart pointer as a value.

[0100] The first relationship mapping table is a mapping of the time series data name corresponding to the time series data stored in each data shard to the smart pointer, and the smart pointer is used to indicate the memory address where the corresponding time series data is stored. After the first time series data is stored in the target data shard, a smart pointer (shared-pointers) pointing to the memory address will be created according to the memory address where the first time series data is stored, and then the first relationship mapping table is constructed with the time series data name of the first time series data as the key and the smart pointer as the value.

[0101] Step 406: construct a second relationship mapping table using the target shard identifier of the target data shard as a key and the mapping identifier of the first relationship mapping table as a value.

[0102] Since the writable data block includes multiple data shards, in order to clarify the time series data stored in each data shard, it is also necessary to use the target shard identifier of the target data shard as the key and the mapping identifier of the first relationship mapping table as the value to construct a second relationship mapping table so that it can be mapped to the corresponding first relationship mapping table according to the target shard identifier.

[0103] Then, when searching for the latest time series data from the memory space, the time series data query can be performed based on the first relationship mapping table and the second relationship mapping table. The specific query process is to first obtain the data type of the required time series data (i.e., the target time series data name) from the data query request, and calculate the target hash value based on the target time series data name. The shard identifier of the data shard where the time series data to be queried is stored can be determined according to the target hash value, and the target mapping identifier of the first relationship mapping table can be determined from the second relationship mapping table based on the shard identifier; then, the corresponding first relationship mapping table is found based on the target mapping identifier, and the target smart pointer is queried from the first relationship mapping table based on the target time series data name of the time series data to be queried; finally, based on the target smart pointer, the target memory address of the time series data to be queried is located to obtain the time series data to be queried.

[0104] In this embodiment, by sharding the optional data blocks, a single first relationship mapping table becomes small enough, thereby reducing the time required to search the single first relationship mapping table, and data search can also be performed through parallel search, thereby reducing the time required to traverse the optional data blocks, thereby improving the efficiency of time series data query.

[0105] Please refer to Figure 5 , which is a schematic diagram of the structure of a time series data storage device provided by an embodiment of the present disclosure. For example, Figure 5 As shown, the device 500 includes:

[0106] The first storage module 501 is used for caching the first time series data into a memory space and storing the first time series data into a log file in a disk space when the first time series data is received;

[0107] A second storage module 502 is configured to write the first time series data from the memory space into a complete file in the disk space and delete the first time series data in the memory space if the timestamp of the first time series data cached in the memory space meets a time condition;

[0108] The deleting module 503 is configured to delete the first time series data in the log file when the first time series data is written into the complete file in the disk space.

[0109] Optionally, the device further comprises:

[0110] an acquisition module, configured to acquire, when the first time series data is written into the complete file in the disk space, the latest time of the first time series data written into the complete file;

[0111] An updating module, used for updating the latest time into the inspection file of the disk space;

[0112] A recovery module is used to recover the time series data cached in the memory space before the device failure based on the latest time in the inspection file when a device failure is detected and recovered.

[0113] Optionally, the recovery module is further used to:

[0114] Searching and reading second time series data after the latest time from the log file;

[0115] The second time series data is written into the memory space to restore the time series data cached in the memory space before the device failure.

[0116] Optionally, the first storage module 501 is further used for:

[0117] Writing the first time series data into a writable data block in the memory space;

[0118] When the target time difference of the first time series data written in the writable data block reaches the time difference threshold, the first time series data is copied to the target closed data block in the memory space, and the latest time series data is written to the writable data block.

[0119] Optionally, the second storage module 502 is further used for:

[0120] When the timestamp of the first time series data in the target closed data block meets the time condition, writing all the first time series data in the target closed data block into the complete file in the disk space;

[0121] Delete the first time series data in the target closed data block in the memory space.

[0122] Optionally, the first storage module 501 is further used for:

[0123] Performing data compression processing on the first time series data to obtain compressed first time series data;

[0124] The compressed first time series data is written into the writable data block in the memory space.

[0125] Optionally, the second storage module 502 is further used for:

[0126] Compressing all the first time series data in the target closed data block to obtain compressed block data;

[0127] Acquire a target time range corresponding to all the first time series data in the target closed data block;

[0128] The compressed block data and the target time range are written into the complete file in the disk space.

[0129] In summary, the embodiments of the present disclosure provide a method for storing time series data: a log file is added to the disk space, and when the device receives the latest time series data, it will not only be cached to the memory space first, but also written to the log file in the disk space, so as to realize the persistence of the latest time series data and avoid the loss of the latest time series data in the memory due to sudden downtime or failure of the device; at the same time, the latest time series data is cached in the memory space and the query efficiency of the time series data is improved; moreover, after the timestamp of the time series data stored in the memory space meets the time condition, the time series data will be written to the disk space and deleted from the memory space, so as to avoid the occupation of the memory space by the historical time series data and improve the space utilization rate of the memory space; in addition, after the historical time series data is written into the complete file of the disk space, the corresponding historical time series data in the log file will also be deleted synchronously, so as to avoid the storage of duplicate time series data in the disk and further improve the space utilization rate of the disk space.

[0130] The exemplary embodiment of the present disclosure also provides an electronic device, comprising: at least one processor; and a memory connected to the at least one processor in communication. The memory stores a computer program that can be executed by the at least one processor, and the computer program is used to cause the electronic device to perform the method according to the embodiment of the present disclosure when executed by the at least one processor.

[0131] The exemplary embodiments of the present disclosure also provide a non-transitory computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor of a computer, is used to cause the computer to perform the method according to the embodiments of the present disclosure.

[0132] The exemplary embodiments of the present disclosure further provide a computer program product, including a computer program, wherein when the computer program is executed by a processor of a computer, the computer is used to enable the computer to perform the method according to the embodiments of the present disclosure.

[0133] refer to Figure 6, a block diagram of an electronic device 600 that can be used as a server or client of the present disclosure will now be described, which is an example of a hardware device that can be applied to various aspects of the present disclosure. The electronic device is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0134] like Figure 6 As shown, the electronic device 600 includes a computing unit 601, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 602 or a computer program loaded from a storage unit 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the device 600 can also be stored. The computing unit 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0135] A plurality of components in the electronic device 600 are connected to the I / O interface 605, including: an input unit 606, an output unit 607, a storage unit 608, and a communication unit 609. The input unit 606 may be any type of device capable of inputting information to the electronic device 600, and the input unit 606 may receive input digital or character information, and generate key signal inputs related to user settings and / or function control of the electronic device. The output unit 607 may be any type of device capable of presenting information, and may include but is not limited to a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. The storage unit 608 may include but is not limited to a disk, an optical disk. The communication unit 609 allows the electronic device 600 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks, and may include but is not limited to a modem, a network card, an infrared communication device, a wireless communication transceiver, and / or a chipset, such as a Bluetooth device, a WiFi device, a WiMax device, a cellular communication device, and / or the like.

[0136] The computing unit 601 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 601 performs the various methods and processes described above. For example, in some embodiments, Figure 1 , Figure 2 , Figure 3 , Figure 4 The method shown may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as the storage unit 608. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 600 via the ROM 602 and / or the communication unit 609. In some embodiments, the computing unit 601 may be configured to execute the computer program by any other suitable means (e.g., by means of firmware). Figure 1 , Figure 2 , Figure 3 , Figure 4 The method shown.

[0137] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0138] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0139] As used in this disclosure, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, apparatus, and / or device (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0140] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0141] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0142] A computer system may include clients and servers. Clients and servers are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship to each other.

Claims

1. A method for storing time series data, characterized in that: The method comprises: When receiving the first time series data, cache the first time series data in a memory space, and store the first time series data in a log file in a disk space; When the timestamp of the first time series data cached in the memory space meets the time condition, writing the first time series data from the memory space into the complete file in the disk space, and deleting the first time series data in the memory space; When the first time series data is written into the complete file in the disk space, the first time series data in the log file is deleted.

2. The method according to claim 1, characterized in that The method further comprises: When the first time series data is written into the complete file in the disk space, acquiring the latest time of the first time series data written into the complete file; Updating the latest time into the inspection file of the disk space; When a device failure is detected and recovered, the time series data cached in the memory space before the device failure is recovered based on the latest time in the inspection file.

3. The method according to claim 2, characterized in that The step of restoring the time series data cached in the memory space before the device failure based on the latest time in the inspection file includes: Searching and reading second time series data after the latest time from the log file; The second time series data is written into the memory space to restore the time series data cached in the memory space before the device failure.

4. The method according to any one of claims 1 to 3, characterized in that: The step of caching the first time series data into a memory space includes: Writing the first time series data into a writable data block in the memory space; When the target time difference of the first time series data written in the writable data block reaches the time difference threshold, the first time series data is copied to the target closed data block in the memory space, and the latest time series data is written to the writable data block.

5. The method according to claim 4, characterized in that When the timestamp of the first time series data cached in the memory space meets the time condition, writing the first time series data from the memory space into the complete file in the disk space, and deleting the first time series data in the memory space include: When the timestamp of the first time series data in the target closed data block meets the time condition, writing all the first time series data in the target closed data block into the complete file in the disk space; Delete the first time series data in the target closed data block in the memory space.

6. The method according to claim 4, characterized in that Writing the first time series data into a writable data block in the memory space includes: Performing data compression processing on the first time series data to obtain compressed first time series data; The compressed first time series data is written into the writable data block in the memory space.

7. The method according to claim 6, characterized in that Writing all the first time series data in the target closed data block into the complete file in the disk space includes: Compressing all the first time series data in the target closed data block to obtain compressed block data; Acquire a target time range corresponding to all the first time series data in the target closed data block; The compressed block data and the target time range are written into the complete file in the disk space.

8. A time series data storage device, characterized in that: The device comprises: A first storage module is used for caching the first time series data into a memory space and storing the first time series data into a log file in a disk space when the first time series data is received; A second storage module is configured to write the first time series data from the memory space into a complete file in the disk space and delete the first time series data in the memory space if the timestamp of the first time series data cached in the memory space meets a time condition; A deletion module is used to delete the first time series data in the log file when the first time series data is written into the complete file in the disk space.

9. An electronic device, comprising: processor; as well as Memory for storing programs, The program includes instructions, which, when executed by the processor, enable the processor to execute the time series data storage method according to any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to enable the computer to execute the time series data storage method according to any one of claims 1 to 7.

Images