A time series data processing method and device
By determining the target storage node based on data characteristics and improving the B+Tree structure for storing real-time data values, the problems of high resource consumption and low query efficiency caused by large amounts of time-series data are solved, achieving efficient storage and retrieval.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- LENOVO (BEIJING) LTD
- Filing Date
- 2022-06-28
- Publication Date
- 2026-07-07
AI Technical Summary
The large volume of time-series data leads to high storage resource consumption and affects data query efficiency, which is difficult to solve effectively with existing technologies.
By identifying the data characteristics based on time-series data, the target storage node is determined, and the real-time data value is stored in the target storage area. The leaf nodes of the storage tree structure are used to store the real-time data value. The B+Tree structure is improved to change the disk offset value into the real-time data value. The storage and retrieval are optimized by combining the log mechanism and the LRU mechanism.
It reduces storage resource consumption and improves the efficiency of time-series data processing and data retrieval speed.
Smart Images
Figure CN115062024B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of data processing technology, and more specifically to a time-series data processing method and apparatus. Background Technology
[0002] Time-series data is a series of data recorded in chronological order. When monitoring the status of products or equipment, time-series data needs to be stored. However, due to the large volume of time-series data, storing it consumes significant system resources and also affects the efficiency of subsequent data queries, resulting in poor processing efficiency. Summary of the Invention
[0003] In view of the above, this application provides the following technical solution:
[0004] A time-series data processing method, comprising:
[0005] Based on the data characteristics of the time-series data to be stored, the target storage node is determined;
[0006] Based on the data characteristics, determine the real-time data value that matches the time-series data to be stored;
[0007] The real-time data values are stored in the target storage area corresponding to the target storage node.
[0008] Optionally, determining the target storage node based on the data characteristics of the time-series data to be stored includes:
[0009] Obtain the data storage structure corresponding to the time-series data to be stored;
[0010] If the data storage structure is a storage tree, a target leaf node is determined in the storage tree based on the data characteristics of the time-series data to be stored. The storage tree is a multi-layer structure, each layer contains one or more nodes, and the nodes of the last layer of the storage tree are leaf nodes.
[0011] The target leaf node is determined as the target storage node.
[0012] Optionally, storing the real-time data value in the target storage area corresponding to the target node includes:
[0013] The real-time data value is stored in the data page of the target leaf node. In the storage tree, the metadata of the leaf node and the real-time data value are stored in the data page of the leaf node to form a data file. The index file of the storage tree is stored on the disk.
[0014] Optionally, determining the target storage node based on the data characteristics of the time-series data to be stored includes:
[0015] Based on the data characteristics of the time series data to be stored, determine the metadata of the time series data to be stored;
[0016] Based on the aforementioned metadata, the target storage node is determined.
[0017] Optionally, storing the real-time data value into the data page of the target leaf node includes:
[0018] Obtain the first data volume of the time-series data to be stored in the target acquisition period;
[0019] Obtain the free storage space of the data page of the target leaf node;
[0020] Based on the first data volume and the free storage space, detect whether the data page can meet the storage requirements of the real-time data value;
[0021] If so, store the real-time data value in the data page of the target leaf node;
[0022] If not, generate a target data page associated with the data page and store the real-time data value in the target data page.
[0023] Optionally, the method further includes:
[0024] In response to storing the real-time data value in the target storage area corresponding to the target storage node, a data storage log and a data storage success prompt message are generated, and the prompt message is sent to the receiving end;
[0025] If the receiving end does not receive the prompt information, it will re-execute the storage operation of storing the real-time data value to the target storage node based on the data storage log.
[0026] Optionally, the method further includes:
[0027] If real-time data values stored in the target storage area are lost, the real-time data values can be recovered based on the data storage logs.
[0028] Optionally, the method further includes:
[0029] In response to receiving a read request, the target storage node corresponding to the read request is determined;
[0030] Obtain the data value corresponding to the read request in the target storage area corresponding to the target storage node.
[0031] Optionally, the method further includes:
[0032] If the number of data values in the target storage area is greater than the target number threshold, target data that meets the target conditions and is to be deleted is identified in the target storage area and the target data is deleted.
[0033] A time-series data processing apparatus, comprising:
[0034] The first determining unit is used to determine the target storage node based on the data characteristics of the time-series data to be stored;
[0035] The second determining unit is used to determine, based on the data characteristics, a real-time data value that matches the time-series data to be stored;
[0036] A storage unit is used to store the real-time data values in the target storage area corresponding to the target storage node.
[0037] A readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the timing data processing method as described in any of the preceding claims.
[0038] An electronic device, comprising:
[0039] Memory, used to store applications and the data generated by the running of the applications;
[0040] A processor for executing the application program to implement the various steps of the timing data processing method as described in any of the preceding claims.
[0041] As can be seen from the above technical solution, this application discloses a time-series data processing method and apparatus, including: determining a target storage node based on the data characteristics of the time-series data to be stored; determining a real-time data value matching the time-series data to be stored according to the data characteristics; and storing the real-time data value in a target storage area corresponding to the target storage node. In this invention, when storing time-series data, the real-time data value corresponding to the time-series data is stored in the target area, so that only the real-time data value changes during the storage process, while the rest of the content remains unchanged, reducing the consumption of storage resources and improving the efficiency of time-series data processing. Attached Figure Description
[0042] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.
[0043] Figure 1 A flowchart illustrating a time-series data processing method provided in an embodiment of this application;
[0044] Figure 2 This is a schematic diagram of the original structure of a B+Tree;
[0045] Figure 3 This is a schematic diagram of a B+tree structure provided in an embodiment of this application;
[0046] Figure 4 This is a schematic diagram of the structure of a timing data processing device provided in an embodiment of this application. Detailed Implementation
[0047] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0048] This application provides a time-series data processing method, wherein time-series data refers to time-series data, that is, a data column of the same indicator recorded in chronological order. Since time-series data is constantly generated, there is an optimization need to reduce storage resource consumption and improve the read hit rate of stored data.
[0049] See Figure 1 This is a flowchart illustrating a time-series data processing method provided in an embodiment of this application. The method may include the following steps:
[0050] S101. Determine the target storage node based on the data characteristics of the time-series data to be stored.
[0051] The time-series data to be stored refers to the real-time data values that need to be stored based on changes in the time series data. Data characteristics can characterize the attributes of the time-series data to be stored, such as the size of the data, data type, and data structure characteristics. Typically, data storage uses corresponding data storage structures, such as storage trees, databases, or tables. A data storage structure includes multiple nodes capable of storing data. Different storage nodes may store different data characteristics, such as different storage fields (e.g., the first storage node stores temperature data, while the second stores humidity data), or different data types (e.g., the third storage node stores static data, while the fourth stores dynamic data). Therefore, based on the data characteristics of the time-series data to be stored, a matching target storage node can be determined. Furthermore, the target storage node can be determined based on the structural and data characteristics of the storage structure in which the node resides, ensuring that the target storage node can effectively store the time-series data.
[0052] In one implementation, determining the target storage node based on the data characteristics of the time-series data to be stored includes: obtaining a data storage structure corresponding to the time-series data to be stored; if the data storage structure is a storage tree, determining a target leaf node in the storage tree based on the data characteristics of the time-series data to be stored; and determining the target leaf node as the target storage node; wherein the storage tree is a multi-layer structure, each layer contains one or more nodes, and the nodes of the last layer of the storage tree are leaf nodes.
[0053] Specifically, data storage structures can include linear and non-linear storage structures. A tree structure is a non-linear storage structure that stores a collection of data elements with a one-to-many relationship. Storage tree structures are typically multi-layered, with each layer containing one or more nodes, and parent-child relationships existing between nodes in adjacent layers. For example, a storage tree with three layers of data could have the root node data in the first layer, the internal node data in the second layer, and the leaf node data in the third layer. The parent node data can store the location information of the leaf node data. In this embodiment, leaf nodes can be used to store the time-series data to be stored. Therefore, based on the data characteristics of the time-series data to be stored, a matching target leaf node is determined as the target storage node.
[0054] In another implementation, determining the target storage node based on the data characteristics of the time-series data to be stored includes: determining the metadata of the time-series data to be stored based on the data characteristics of the time-series data to be stored; and determining the target storage node based on the metadata.
[0055] Metadata describes data, primarily information about data attributes, used to support functions such as indicating storage location, historical data, resource lookup, and file records. Therefore, metadata can be used to determine the attributes of time-series data to be stored or the preset storage location, thereby identifying the target storage node.
[0056] S102. Based on the data characteristics, determine the real-time data value that matches the time-series data to be stored.
[0057] In this embodiment, the stored data is the real-time data value of the time-series data to be stored. Taking a storage tree structure as an example, the storage content of each Value in the leaf nodes is changed from disk offset value to real-time data value, that is, the actual data value corresponding to the current acquisition time. This ensures that when storing time-series data, only the data value changes, while the rest of the content remains unchanged. This achieves the use of a dynamic array to store changing values, which significantly reduces resource consumption.
[0058] S103. Store the real-time data values in the target storage area corresponding to the target storage node.
[0059] Each storage node can have at least one data page for data storage. The storage node can determine whether it can store the current real-time data value based on the amount of data already stored on the current data page. If the data page corresponding to the target storage node can store the current real-time data value, the data page is designated as the target storage area, and the real-time data value is written to the current data page.
[0060] Taking the data storage structure as a storage tree and the target storage node as the target leaf node as an example, in one implementation, storing the real-time data value to the target storage area corresponding to the target storage node includes: storing the real-time data value to the data page of the target leaf node.
[0061] In this storage tree, the metadata and real-time data values of the leaf nodes are stored in the data pages of the leaf nodes to form data files, while the index file of the storage tree is stored on disk. That is, the data files corresponding to the storage tree store the data information, and the index files store the index information. Storing the data information and index information separately makes it easier to read the data.
[0062] Taking B+Tree as an example, B+Tree is a tree data structure that is commonly used in file systems of databases and operating systems. The leaf nodes of B+Tree store disk data offsets. Through the embodiments of this application, the content stored in the leaf nodes of B+Tree can be improved, that is, the content stored in the leaf nodes is changed from disk offset values to real-time data values.
[0063] See Figure 2This is a schematic diagram of the original structure of a B+Tree. Figure 2 The B+Tree shown includes: a root node (Rootpage), internal pages (Internal pages, or intermediate index nodes), and leaf pages. The B+Tree is used to manage data on the HDD (Hard Disk Drive), such as writing and reading data. Figure 2 It also includes binary log files (bin logs) used to record information about data changes that have occurred or may occur. In a B+Tree, the root node is the top-level node, leaf nodes are the bottom-level nodes, and internal nodes are all nodes except the root and leaf nodes. Specifically, the root and internal nodes do not store data; they are only used for indexing. The leaf nodes in the original B+Tree structure store disk data offsets. Figure 2 (represented by "offset" in Chinese). In this structure, to find relevant data, you first need to find the corresponding leaf node through the index information in the root node or internal nodes, and then find the corresponding data value through the disk based on the disk data offset stored in the leaf node. Since disk operations are involved, the hit rate of data reading is reduced.
[0064] See Figure 3 This is a schematic diagram of a B+tree structure provided in an embodiment of this application. This B+tree structure is an improvement upon the original structure, including a root node, internal nodes, and leaf nodes. However, the storage content of each value in the leaf nodes is changed from disk offset values to real-time data values. Since only the data values change, the rest of the content remains the same. Therefore, using a dynamic array to store the changing values significantly reduces resource consumption. The real-time data values are accessed through the data pages corresponding to the leaf nodes (leafpages). Figure 3 Data is stored using the 'date' symbol, such as storing all data values collected within one hour. Figure 3 In Chinese, it is represented by "int hour
[1024] ", which can also store all data values collected in real time. Figure 3 In Chinese, it is represented by "int realtime
[4096] ". Time-series data typically includes metadata and real-time data values. Real-time data values are obtained through monitoring, while metadata represents the monitoring field corresponding to that real-time data value. For example, if the time-series data is the monitoring of the processor temperature of an electronic device, then the metadata is temperature, and the real-time data value represents the temperature value corresponding to each acquisition moment within the acquisition period. If the acquisition period is one month, and 100 million data points are collected in that month, then... Figure 2Using a B+tree structure for data storage would store 100 million data entries. Even if only the monitored temperature value changes while other information remains constant, it would still be stored as a single, sequential record. Figure 3 The data is stored using a B+tree structure. Disk offset values are transformed into data blocks. Invariant information (such as ID, location, and monitoring field information) is stored in the leaf nodes, while real-time changes are stored on the corresponding data pages of the leaf nodes. This way, when searching for data, the corresponding leaf node can be located using the invariant metadata, and then the corresponding data page can be retrieved to obtain all the data collected in that collection cycle. This method uses fewer storage resources than storing data record by record, and data retrieval is much faster.
[0065] Specifically, in one embodiment of this application, in response to receiving a read request, a target storage node corresponding to the read request is determined, and the data value corresponding to the read request is obtained in the target storage area corresponding to the target storage node. The read request may include the acquisition period in which the data to be read is located, the metadata fields corresponding to the data to be read, and the target storage area, such as the target leaf node in a B+Tree, is located based on this unchanging information. Then, the data page corresponding to the target leaf node is read to obtain the data value corresponding to the read request, such as the various acquired data values within the target acquisition period.
[0066] In this embodiment, since real-time data values are stored on the data page corresponding to the leaf node, typically all real-time data values collected within the acquisition period are stored on a single data page for easy retrieval. However, if the amount of collected data is large and the storage space of the currently stored data page is insufficient, multiple data pages can be allocated to meet the data storage needs. Specifically, in one implementation, storing real-time data values on the data page of the target leaf node includes: obtaining a first data volume of the target acquisition period of the time-series data to be stored; obtaining the free storage space of the data page of the target leaf node; based on the first data volume and the available storage space, monitoring whether the data page can meet the storage needs of the real-time data values; if yes, storing the real-time data values on the data page of the target leaf node; if no, generating a target data page associated with the data page and storing the real-time data values on the target data page. The target data page associated with the data page can be newly created or obtained by adjusting an existing data page. It should be noted that the information corresponding to the data page and the target data page remains unchanged, i.e., their metadata information is the same. Taking a B+tree structure as an example, when a device has 1 million data entries, if the real-time data value to be stored exceeds 4K or 16K, multiple data pages will be allocated to meet the storage requirements. In this embodiment, information with unchanged content is stored on leaf nodes, while real-time data values are scattered across data pages, with the number of data pages allocated determined based on the amount of data to be stored.
[0067] To enhance the security of data written to the storage tree and prevent data loss, this application embodiment also provides a data writing log mechanism. This mechanism generates storage logs of data writing operations to prevent data loss. Specifically, in one implementation, in response to storing real-time data values in the target storage area corresponding to the target storage node, a data storage log and a successful data storage notification are generated and sent to the receiving end. If the receiving end does not receive the notification, it re-executes the storage operation of storing the real-time data values to the target storage node based on the data storage log.
[0068] This application does not limit the specific form of the logging mechanism, as long as it can generate corresponding data storage logs. For example, it can adopt... Figure 3The journal log mechanism is illustrated. The target storage area is the data page in memory corresponding to the leaf node. When there is real-time data value to be written, it is first written to the data page in memory, then to the data storage log, and a success message is sent back to the receiving end (such as the application frontend). If the receiving end receives this message, it indicates that the data storage was successful. If a failure occurs, such as a disk power failure, the receiving end will not receive the success message, indicating that the data was not written successfully. In this case, the receiving end will perform a rewrite operation, that is, rewrite the data according to the write information recorded in the data storage log. Furthermore, it also includes: if the real-time data value stored in the target area is lost, the remaining data storage log can be used to recover the real-time data value. For example, after the disk is restarted from a power failure, if structured data is read through a snapshot, reading the data storage log can recover the written data outside the snapshot. Furthermore, data can be written periodically. When the storage tree structure is a B+Tree and the B+Tree is in memory, there are two ways to write data to disk. The first is to write a data log for each data write request received. The second is to periodically store the structured B+Tree data to disk, which is similar to a snapshot. For example, if there are 100 million records to be written, all of these records can be written to the data log. After the writing is complete, the entire structured data inside the B+Tree will be written to a file. If it is written once per minute, when another batch of data needs to be written, this data will not be written to the structured data, but to the data log. By combining these two methods, the efficiency and accuracy of data writing are improved.
[0069] In one embodiment of this application, to manage the data in a data page and further improve the data read hit rate, the data in the data page can be cleaned periodically. For example, if the number of data values in the target storage area exceeds a target threshold, target data that meets the target conditions is identified in the target storage area and deleted. Specifically, the target conditions can be determined according to a data management mechanism, for example, through... Figure 3The LRU mechanism illustrated implements data management for the target storage area. LRU stands for Least Recently Used, a commonly used page replacement algorithm that selects the least recently used page for eviction. This algorithm assigns an access field to each page, recording the time 't' elapsed since its last access. When a page needs to be evicted, the page with the largest 't' value is selected. In other words, when the number of data values in the target storage area exceeds the target data volume threshold, the LRU mechanism can be invoked to delete data that meets the criteria. Furthermore, the data is managed by the LRU mechanism; when frequently accessed, the data is cached in memory, significantly improving the data read hit rate.
[0070] In this embodiment, the storage content of each Value in the leaf nodes of the B+Tree is changed from disk offset values to actual data values. Since the performance data remains unchanged except for the data values themselves, using a dynamic array to store the changing values significantly reduces resource consumption. Write input utilizes a sequential log mechanism (journal) and periodic checks on B+Tree data. This ensures that, after the B+Tree structure improvement, single-type performance data is distributed across only one data page, rather than scattered across multiple pages. Data is managed by an LRU mechanism; when frequently accessed, the data is continuously cached in memory, thus greatly improving the data read hit rate.
[0071] This application also provides a time-series data processing apparatus, see [link to relevant documentation]. Figure 4 The device may include:
[0072] The first determining unit 401 is used to determine the target storage node based on the data characteristics of the time-series data to be stored;
[0073] The second determining unit 402 is used to determine a real-time data value that matches the time-series data to be stored based on the data characteristics.
[0074] Storage unit 403 is used to store the real-time data value to the target storage area corresponding to the target storage node.
[0075] This application discloses a time-series data processing apparatus, comprising: a first determining unit determining a target storage node based on the data characteristics of the time-series data to be stored; a second determining unit determining a real-time data value matching the time-series data to be stored based on the data characteristics; and a storage unit storing the real-time data value in a target storage area corresponding to the target storage node. In this invention, when storing time-series data, the real-time data value corresponding to the time-series data is stored in the target area, ensuring that only the real-time data value changes during the storage process, while other contents remain unchanged, thus reducing storage resource consumption and improving the efficiency of time-series data processing.
[0076] In one embodiment, the first determining unit 401 includes:
[0077] The first acquisition subunit is used to acquire the data storage structure corresponding to the time-series data to be stored;
[0078] The first determining subunit is used to determine a target leaf node in the storage tree based on the data characteristics of the time-series data to be stored if the data storage structure is a storage tree, wherein the storage tree is a multi-layer structure, each layer contains one or more nodes, and the nodes of the last layer of the storage tree are leaf nodes.
[0079] The second determining subunit is used to determine the target leaf node as the target storage node.
[0080] Furthermore, the second determining unit 402 is specifically used for:
[0081] The real-time data value is stored in the data page of the target leaf node. In the storage tree, the metadata of the leaf node and the real-time data value are stored in the data page of the leaf node to form a data file. The index file of the storage tree is stored on the disk.
[0082] In one embodiment, the first determining unit 401 includes:
[0083] The third determining subunit is used to determine the metadata of the time series data to be stored based on the data characteristics of the time series data to be stored.
[0084] The fourth determining subunit is used to determine the target storage node based on the metadata.
[0085] Optionally, the storage unit is specifically used for:
[0086] Obtain the first data volume of the time-series data to be stored in the target acquisition period;
[0087] Obtain the free storage space of the data page of the target leaf node;
[0088] Based on the first data volume and the free storage space, detect whether the data page can meet the storage requirements of the real-time data value;
[0089] If so, store the real-time data value in the data page of the target leaf node;
[0090] If not, generate a target data page associated with the data page and store the real-time data value in the target data page.
[0091] In one embodiment, the device further includes:
[0092] The generation unit is used to generate a data storage log and a data storage success prompt message in response to storing the real-time data value in the target storage area corresponding to the target storage node, and send the prompt message to the receiving end;
[0093] A re-storage unit is used to re-execute the storage operation of storing real-time data values to the target storage node based on the data storage log if the receiving end does not receive the prompt information.
[0094] Furthermore, the device also includes:
[0095] The data recovery unit is used to recover the real-time data values stored in the target storage area based on the data storage log if the real-time data values are lost.
[0096] In one embodiment, the device further includes:
[0097] The third determining unit is used to determine the target storage node corresponding to the read request in response to receiving the read request;
[0098] The reading unit is used to obtain the data value corresponding to the reading request in the target storage area corresponding to the target storage node.
[0099] In another embodiment, the device further includes:
[0100] The deletion unit is used to determine target data to be deleted in the target storage area that meets the target conditions if the number of data values in the target storage area is greater than the target number threshold, and then delete the target data.
[0101] It should be noted that the specific implementation of each unit and subunit in this embodiment can be referred to the corresponding content above, and will not be described in detail here.
[0102] In another embodiment of this application, a readable storage medium is also provided, on which a computer program is stored, which, when executed by a processor, implements the various steps of the timing data processing method as described in any of the preceding claims.
[0103] In another embodiment of this application, an electronic device is also provided, comprising:
[0104] Memory, used to store applications and the data generated by the running of the applications;
[0105] A processor for executing the application to achieve:
[0106] Based on the data characteristics of the time-series data to be stored, the target storage node is determined;
[0107] Based on the data characteristics, determine the real-time data value that matches the time-series data to be stored;
[0108] The real-time data values are stored in the target storage area corresponding to the target storage node.
[0109] Optionally, determining the target storage node based on the data characteristics of the time-series data to be stored includes:
[0110] Obtain the data storage structure corresponding to the time-series data to be stored;
[0111] If the data storage structure is a storage tree, a target leaf node is determined in the storage tree based on the data characteristics of the time-series data to be stored. The storage tree is a multi-layer structure, each layer contains one or more nodes, and the nodes of the last layer of the storage tree are leaf nodes.
[0112] The target leaf node is determined as the target storage node.
[0113] Optionally, storing the real-time data value in the target storage area corresponding to the target node includes:
[0114] The real-time data value is stored in the data page of the target leaf node. In the storage tree, the metadata of the leaf node and the real-time data value are stored in the data page of the leaf node to form a data file. The index file of the storage tree is stored on the disk.
[0115] Optionally, determining the target storage node based on the data characteristics of the time-series data to be stored includes:
[0116] Based on the data characteristics of the time series data to be stored, determine the metadata of the time series data to be stored;
[0117] Based on the aforementioned metadata, the target storage node is determined.
[0118] Optionally, storing the real-time data value into the data page of the target leaf node includes:
[0119] Obtain the first data volume of the time-series data to be stored in the target acquisition period;
[0120] Obtain the free storage space of the data page of the target leaf node;
[0121] Based on the first data volume and the free storage space, detect whether the data page can meet the storage requirements of the real-time data value;
[0122] If so, store the real-time data value in the data page of the target leaf node;
[0123] If not, generate a target data page associated with the data page and store the real-time data value in the target data page.
[0124] Optionally, the method further includes:
[0125] In response to storing the real-time data value in the target storage area corresponding to the target storage node, a data storage log and a data storage success prompt message are generated, and the prompt message is sent to the receiving end;
[0126] If the receiving end does not receive the prompt information, it will re-execute the storage operation of storing the real-time data value to the target storage node based on the data storage log.
[0127] Optionally, the method further includes:
[0128] If real-time data values stored in the target storage area are lost, the real-time data values can be recovered based on the data storage logs.
[0129] Optionally, the method further includes:
[0130] In response to receiving a read request, the target storage node corresponding to the read request is determined;
[0131] Obtain the data value corresponding to the read request in the target storage area corresponding to the target storage node.
[0132] Optionally, the method further includes:
[0133] If the number of data values in the target storage area is greater than the target number threshold, target data that meets the target conditions and is to be deleted is identified in the target storage area and the target data is deleted.
[0134] This application discloses a readable storage medium and an electronic device. The method involves a processor executing data features based on the time-series data to be stored to determine a target storage node; determining a real-time data value matching the time-series data based on the data features; and storing the real-time data value in a target storage area corresponding to the target storage node. In this invention, when storing time-series data, the real-time data value corresponding to the time-series data is stored in the target area. This ensures that only the real-time data value changes during the storage process, while other contents remain unchanged, reducing storage resource consumption and improving the efficiency of time-series data processing.
[0135] It should be noted that the specific implementation of the processor in this embodiment can be referred to the corresponding content above, and will not be described in detail here.
[0136] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.
[0137] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.
[0138] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.
[0139] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A time-series data processing method, comprising: Based on the data characteristics of the time-series data to be stored, the target storage node is determined, and the target storage node is the target leaf node in the storage tree; Based on the data characteristics, determine the real-time data value that matches the time-series data to be stored; The metadata of the time-series data to be stored is stored in the target leaf node, and the real-time data value is stored in the data page corresponding to the target leaf node. The index file of the storage tree is stored on the disk, so that the data information and the index information are stored separately. During the storage process, the only change is the real-time data value, while the rest of the content remains unchanged.
2. The method according to claim 1, wherein determining the target storage node based on the data characteristics of the time-series data to be stored includes: Obtain the data storage structure corresponding to the time-series data to be stored; If the data storage structure is a storage tree, a target leaf node is determined in the storage tree based on the data characteristics of the time-series data to be stored. The storage tree is a multi-level structure, with each level containing one or more nodes, and the nodes in the last level of the storage tree are leaf nodes. The target leaf node is determined as the target storage node.
3. The method according to claim 1, wherein determining the target storage node based on the data characteristics of the time-series data to be stored includes: Based on the data characteristics of the time series data to be stored, determine the metadata of the time series data to be stored; Based on the aforementioned metadata, the target storage node is determined.
4. The method according to claim 1, wherein storing the real-time data value on the data page corresponding to the target leaf node comprises: Obtain the first data volume of the time-series data to be stored in the target acquisition period; Obtain the free storage space of the data page of the target leaf node; Based on the first data volume and the free storage space, detect whether the data page can meet the storage requirements of the real-time data value; If so, store the real-time data value in the data page of the target leaf node; If not, generate a target data page associated with the data page and store the real-time data value in the target data page.
5. The method according to claim 1, further comprising: In response to storing the real-time data value in the target storage area corresponding to the target storage node, a data storage log and a data storage success prompt message are generated, and the prompt message is sent to the receiving end; If the receiving end does not receive the prompt information, it will re-execute the storage operation of storing the real-time data value to the target storage node based on the data storage log.
6. The method according to claim 5, further comprising: If real-time data values stored in the target storage area are lost, the real-time data values can be recovered based on the data storage logs.
7. The method according to claim 1, further comprising: In response to receiving a read request, the target storage node corresponding to the read request is determined; Obtain the data value corresponding to the read request in the target storage area corresponding to the target storage node.
8. The method according to claim 1, further comprising: If the number of data values in the target storage area exceeds the target quantity threshold, target data that meets the target conditions and is to be deleted is identified in the target storage area, and the target data is deleted.
9. A time-series data processing apparatus, comprising: The first determining unit is used to determine the target storage node based on the data characteristics of the time-series data to be stored, wherein the target storage node is the target leaf node in the storage tree; The second determining unit is used to determine, based on the data characteristics, a real-time data value that matches the time-series data to be stored; The storage unit is used to store the metadata of the time-series data to be stored in the target leaf node, and to store the real-time data value in the data page corresponding to the target leaf node. The index file of the storage tree is stored on the disk, so that the data information and the index information are stored separately. During the storage process, the only change is the real-time data value, while the rest of the content remains unchanged.