Time series data tiered storage system and method

Abstract

The present disclosure relates, among other things, to scalable data processing, storage, and / or management systems and methods. In various embodiments, a sequence number scheme associated with ingested data records may enable improved management of data stored in one or more layers (e.g., hot and / or cold data storage layers) using data merging, data deletion, and / or data compression techniques. In certain embodiments, certain data records stored in one or more segment files associated with a partition may be merged into a compressed segment file, thereby reducing storage overhead associated with storing previous records that are updated and / or deleted records. In further embodiments, data record tombstones may be used in connection with managing deleted records.

Description

[Technical field]

[0001] (Copyright Granted) Portions of the disclosure of this patent document may contain material that is subject to copyright protection. The copyright owner has no objection to the facsimile reproduction of either the patent document or the patent disclosure, as appearing in the U.S. Patent and Trademark Office patent file or records, but otherwise reserves all copyright rights whatsoever.

[0002] (CROSS REFERENCE TO RELATED APPLICATIONS) This application claims the benefit of priority under 35 U.S.C. §119(e) to U.S. Provisional Patent Application No. 63 / 329,346, filed April 8, 2022, and entitled “TIME SERIES DATA LAYERED STORAGE SYSTEMS AND METHODS,” the contents of which are incorporated herein by reference in their entirety. Summary of the Invention

[0003] FIELD OF THE DISCLOSURE The present disclosure relates generally to systems and methods for managing data, and more particularly, to systems and methods for managing time series data using tiered data storage techniques.

[0004] For example, but not by way of limitation, data applications that generate large amounts of time-series data, such as Internet-of-Things (IoT) networks, may benefit from highly scalable solutions for data ingestion, storage, and retrieval. When accessing time-series data, it may be desirable to have more recent data available for queries with relatively minimal latency, while also reducing storage costs associated with large amounts of data. However, balancing both these data storage and data access goals using conventional techniques may present certain challenges.

[0005] Embodiments of the disclosed systems and methods may use tiered data storage techniques. In certain embodiments, data may be stored in at least two storage tiers. For example, in some embodiments, a hot storage tier may be used, in which data may be stored in a record-oriented manner. Data stored in the hot storage tier may be made available for query with relatively minimal latency using a more robust storage infrastructure. Consistent with various embodiments disclosed herein, the hot storage tier may have mechanisms to expire and / or otherwise clean up (e.g., delete and / or mark and / or otherwise schedule for eventual deletion) old data based on user instructions and / or automatically after a certain period of time and / or after the data is a certain amount old (e.g., after some subsequent data entry, after the data has not been queried for a certain period of time, etc.) and / or via other automated data management processes. A cold storage tier, described in more detail below, may use similar mechanisms to expire and / or otherwise clean up old data.

[0006] In further embodiments, a cold storage tier may be used, where data may be stored in a relatively inexpensive storage infrastructure. In some embodiments, data stored in the cold storage tier may be stored in compressed and / or columnar format. Access latency for data available in the cold storage tier, which may be referred to in some instances herein as data availability latency (i.e., the time it takes for ingested data to be made available for access), may be relatively high compared to data in a hot storage tier, but storage costs for larger volumes of data may be relatively low.

[0007] By using multi-tiered data storage techniques consistent with various aspects of the disclosed embodiments, data may be stored in a manner in which certain data may be made available with relatively minimal query response latency and in which certain data may be stored with relatively low-cost and / or efficient storage solutions. Further embodiments of the disclosed systems and methods provide techniques for managing and / or otherwise updating data stored between hot and cold storage tiers using data merging and / or data compression techniques. In certain implementations, data storage and data management techniques consistent with various aspects disclosed herein may be relatively seamless from a user's perspective in terms of where the data is materialized. Indeed, in some embodiments, data may be stored in both hot and cold storage tiers, with duplicate data stored in both storage tiers being removed at query time. The working operation of the present invention will be readily understood by reference to the following detailed description taken in conjunction with the accompanying drawings. [Brief description of the drawings]

[0008] [Figure 1] 1 illustrates a non-limiting example of a data storage and / or data management services architecture consistent with certain embodiments disclosed herein. [Diagram 2] 1 illustrates a non-limiting example of a multi-dimensional data management structure using data partitions and data segments, consistent with certain embodiments disclosed herein. [Diagram 3] 1 illustrates a non-limiting example of a data compression process consistent with certain embodiments disclosed herein. [Figure 4] 1 illustrates a flowchart of a non-limiting example of a data compression process consistent with certain embodiments disclosed herein. [Diagram 5] 1 illustrates a flowchart of a non-limiting example of a data record deletion process consistent with certain embodiments disclosed herein. [Figure 6] 1 illustrates non-limiting examples of systems that can be used to implement certain embodiments of the systems and methods of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] In this specification, a description of a system and method consistent with the embodiments of the present disclosure is provided. Although several embodiments are described, it should be understood that the present disclosure is not limited to any one embodiment, but instead encompasses numerous alternatives, modifications, and equivalents. In addition, numerous specific details are set forth in the following description to provide a thorough understanding of the embodiments disclosed herein, but some embodiments can be practiced without some or all of these details. Moreover, for the sake of clarity, certain technical material that is known in the relevant art has not been described in detail to avoid unnecessarily obscuring the disclosure.

[0010] Embodiments of the present disclosure can be understood by reference to certain drawings. The components of the disclosed embodiments can be arranged and designed in a wide variety of configurations, as generally described and / or illustrated in the figures herein. Thus, the following description of embodiments of the systems and methods of the present disclosure is not intended to limit the scope of the disclosure, but is merely representative of possible embodiments of the present disclosure. In addition, the steps of any method disclosed herein do not necessarily have to be performed in any particular order, nor sequentially, nor do steps have to be performed only once, unless otherwise specified.

[0011] Hierarchical data storage and management Embodiments of the disclosed systems and methods may use tiered data storage techniques to, among other things, provide data storage and / or data management with relatively fast query response while reducing dependency on relatively expensive data storage infrastructure. Consistent with various embodiments disclosed herein, the data storage and / or data management architecture may include a hot storage tier where data may be made available for querying with relatively minimal data availability latency, and a cold storage tier where data may be stored in a compressed columnar format within a relatively inexpensive storage infrastructure.

[0012] In relation to certain Amazon Web Service ("AWS") data storage and / or other similar storage implementations, there may be a significant relative cost difference between high-speed block storage and cloud object storage. For example, AWS block storage on solid-state disk devices may be significantly more expensive than AWS cloud object storage. This cost difference may be more significant when achieving higher durability using some object cloud storage services involves data replication as part of the data management architecture. For example, cloud object storage such as AWS S3 may offer higher durability within the base service. However, achieving comparable durability using high-speed block storage may involve data replication, which may be associated with more storage space and result in extra costs.

[0013] In some implementations, data size when stored in a compressed columnar format may be smaller than when stored in a row-oriented format. Embodiments of the disclosed systems and methods may manage data storage between hot and cold storage tiers in a manner that takes into account storage cost differences between the tiers to more efficiently realize storage savings.

[0014] Consistent with various disclosed embodiments, records entering the system may be processed by a hot storage tier component. Processing the data by the hot storage tier may, in some implementations, make the incoming records available for querying with relatively low latency. Copies of the records may be added to a store, which in certain instances may be referred to as a canonical store, where the data may be stored in record-oriented compressed chunks in a cloud object store. These data chunks may be used to generate cold storage tier updates, which may be periodic in nature.

[0015] In connection with various embodiments of the disclosed systems and methods, the stored data may be partitioned. That is, in some implementations, a mechanism may be used to split data set records into data partitions such that any given data record with the same key value will be in the same data partition. In some embodiments, the object store may not allow modification of existing objects and / or files, and thus data updates may create new files that are written to the object store. For example, in certain embodiments, updates may create new updated file records, which may be columnar files, for each time bucket (e.g., a fixed size period based on the data timestamp used to split the data) and for each data partition (assuming there are records that belong to a given data partition and a given time bucket).

[0016] Data filtering, data updating, and data deduplication In various disclosed embodiments, the data search mechanism may filter out possible duplicate records. To facilitate filtering of duplicate records, input records may be numbered and / or otherwise associated with a sequence number. In certain embodiments, for partitioned data, the sequence number may be unique within a given partition (but in some implementations not necessarily globally unique) and monotonically increasing, although it will be understood that other suitable sequence number paradigms may also be used. In further embodiments, records in the system may be associated with a primary key, which may include a set of record values ​​that determine distinct records in the system, and a sequence number. In instances where there are multiple records with the same primary key, a data deduplication mechanism consistent with various aspects of the disclosed embodiments may be used such that the record of the multiple records with the highest sequence number is used (in the case of monotonically increasing sequence numbers).

[0017] An embodiment of the disclosed system and method may enable record updates by allowing the insertion of a new record with the same primary key. An embodiment of the deduplication mechanism described above may ensure that the most recent record is used. To delete a record, a concrete record (such as a record that may be referred to herein in certain instances as a data record tombstone and / or derivatives thereof) may be inserted into the storage system that has the same primary key as the record to be deleted, but does not have a data value. In various embodiments, the data record tombstone may be assigned a higher sequence number than the record that it is intended to replace and / or otherwise delete. This may allow the data record deduplication mechanism to retain the data record tombstone as the record with the highest sequence number.

[0018] Consistent with various aspects of the disclosed embodiments, column data files created by updating the cold storage tier may be partitioned. In some embodiments, the data may first be partitioned (e.g., based on some selected column values, i.e., selector values) into data partitions. The data may then be partitioned into fixed-size time periods based on data timestamps, which may be referred to as time buckets. The time buckets may include, for example, but are not limited to, daily buckets, monthly buckets, and / or yearly buckets. The selection for bucket size may be made based on expected query patterns, which may depend (at least in part) on which time periods are likely to be queried together. The data may then be partitioned into data partitions.

[0019] Merging fragmented data Certain embodiments of the disclosed data management system and method may address segmentation of data over time. To have relatively less data in the hot storage tier, the system may update the cold storage tier at some frequency. The cold storage tier updates may create new segment files for each data partition and time bucket, leading to a fragmented representation of data over time. This may increase data retrieval time, as a greater number of segment files may need to be retrieved and processed to respond to a query request. To address data fragmentation associated with data storage in the cold storage tier, embodiments of the disclosed system and method may merge multiple segment files of the same data partition and same time bucket into one or more compressed segment files.

[0020] In at least one non-limiting example, a time series data table may include temperature measurements. Each record may have three fields: sensor_id, time, and temperature. The data may be partitioned by sensor_id, i.e., each sensor data may belong to a separate data partition. In an example, a sensor may report data once per minute.

[0021] The tables may be configured to materialize data in a cold storage tier that may be updated periodically (e.g., daily). Data may be collected in time buckets (e.g., monthly time buckets). In some implementations, it may be assumed that sensors are operating accurately and therefore there are few, if any, gaps in the input data and minimal delays in data arrival. Additionally, in an illustrative example, it may be assumed that the exemplary system may perform a cold store update at 6:00 PM on February 28th.

[0022] The update may create at least one new segment file for each sensor that contains records since the last update (e.g., since 6:00 PM the previous day, February 27th). The next update, which runs at 6:00 PM on March 1st, may create two segment files for each sensor: one file for the February time bucket that contains records from the last update to midnight on March 1st, and one file for the March time bucket.

[0023] If there is delayed data included in the March 1st update (e.g., data associated with a timestamp from January), three separate files may be created belonging to the January, February, and March time buckets, respectively.

[0024] The number of segment files in any given time bucket of a data partition (e.g., exceeding a threshold) may trigger a data compression process consistent with various aspects of the disclosed embodiments. In some embodiments, the data compression process may eliminate duplicate records and / or tombstones using various aspects of the record deduplication mechanism detailed above. In further embodiments, the data compression process may be performed periodically (e.g., based on a user-specified period, etc.), and / or otherwise triggered, such as based on reaching a threshold size of segment files and / or records.

[0025] Data Storage and Duration Data storage in a storage tier consistent with various aspects of the disclosed embodiments may depend, at least in part, on one or more time periods that may be set by a user and / or otherwise adjusted as appropriate. For example, the time period for updating data stored in a cold storage tier may be T U In this case, the minimum period of time for data stored in the hot storage tier is T L +T P (where T P is the maximum period required to process the data increments available in the cold storage tier).

[0026] In at least one non-limiting illustrative embodiment, the cold storage layer may be updated daily (e.g., every 24 hours) and the cold storage layer update process may be set to not exceed one hour. U =24, T P = 1, and the time-to-live (TTL) of data in the hot storage layer is T U +T P (i.e., 25 hours). The set of potentially overlapping records is P can be entered.

[0027] In certain embodiments, instead of using both hot and cold storage tiers for data storage and data management, either the hot or cold storage tier may be used. For example, the hot storage tier may be used in applications where the most recent data needs to be stored and made available with minimal latency (and relatively low data volume). The cold storage tier may be used in applications where data availability latency requirements are relatively low, but data volume is relatively high.

[0028] Tiered Data Storage and Data Management Architecture 1 illustrates a non-limiting example of a data storage and / or data management architecture 100 consistent with certain embodiments disclosed herein. As shown, architecture 100 may include systems, services, and / or components associated with hot and cold storage tiers. Architecture 100 may further include systems, services, and / or components shared between the hot and cold storage tiers, as well as systems, services, and / or components associated with canonical storage.

[0029] Various embodiments of the disclosed systems and methods may use shared information contained in the definition metastore 102. The definition metastore 102 may provide definitions related to namespaces, which may enable different users to operate on and / or process data in particular tables while operating in different namespaces. Namespaces may be used, for example, but not limited to, to localize table names (e.g., table names may be unique within a namespace) and / or to apply access rights to namespaces. In some embodiments, information contained in the definition metastore 102 may be used to grant access rights (e.g., by an access management system and / or service) based on namespace. For example, a user may be granted privileges to access certain data tables and restricted from accessing certain other data tables.

[0030] The definition metastore 102 may further provide definitions related to data tables, which may define the logical structure of data tables stored and / or otherwise managed by the service. Definitions related to data tables may include, for example, without limitation, information about table elements and / or columns, data types, etc. In some embodiments, the definition metastore 102 may further provide information about one or more partitioning schemes (e.g., projections) supported by the data management service.

[0031] In some embodiments, the definition metastore 102 may provide definitions regarding storage tiers, such as whether and / or which data should be stored in a hot storage tier, a cold storage tier, both storage tiers, etc., retention periods for stored data that may vary depending on the tier in some implementations, updates to hot and / or cold storage tiers, criteria for data compression operations, etc. In this manner, the information contained in the definition metastore 102 may help define the logical structure of data, how it should be partitioned by services, how it should be written to storage, etc.

[0032] The hot storage layer may include streaming writers 104 and hot data stores 106. Data ingested into the data storage and data management platform may be published to one or more partitioned topics, which in some implementations may include partitioned Kafka topics. In some embodiments, each message published to a topic may have a sequence number within an associated partition. For example, each message published to a Kafka topic may have an offset within a given Kafka topic partition, which may serve as a sequence number for various data management operations consistent with embodiments disclosed herein. In some embodiments, the data storage and data management platform may expose a REST API that may enable external systems and / or services to insert data records into the data storage and / or data management service.

[0033] The data may be consumed by streaming writer 104 (e.g., consumed from each topic). In certain embodiments, streaming writer 104 may be configured to detect which data partition an incoming data record belongs to, store the record in a hot data store within the partition, and / or associate the data record with a data partition key associated with the target data partition. In some embodiments, streaming writer 104 may include a Cassandra key-value database. Streaming writer 104 may further detect new data partitions from ingested data records, potentially repartition the ingested data as needed (e.g., based on information included in definition metastore 102), add the data partitions (as needed) to a data partition index 108 that may be shared between hot and cold storage tiers, and then store the data records with the new data partition key in hot data store 106.

[0034] In certain embodiments, sequence numbers may be assigned during the data ingestion process (e.g., assigned by streaming writer 104). In various embodiments, sequence numbers may be globally unique and / or monotonically increasing. In further embodiments, sequence numbers may be monotonically increasing and / or unique within a given data partition. In some implementations, data associated with a topic ingested by a service may be associated with a unique offset number within a given topic partition (e.g., as in the case of Kafka topics), and the offset number may be used as and / or otherwise associated with the sequence number consistent with various aspects of the disclosed embodiments. It will be understood that sequence numbers may be associated with other paradigms.

[0035] In certain embodiments, data stored in the hot data store 106 may be associated with a time to live ("TTL") that specifies the time and / or duration that the data should be retained in the hot data store 106. In some embodiments, this information may be specified in the definition metastore 102. For example, in connection with a hot data store 106 implemented using the Apache Cassandra database, the associated Cassandra tables may have their TTLs set according to user-specified configuration.

[0036] The canonical storage layer may include a canonical store writer 110, a canonical store 112, and a canonical segment index 114. Data ingested into the data storage and data management service may be provided to the canonical store writer 110. The canonical store writer 110 may consume the received topic record data, process the data, and / or store the data in the canonical store 112. The canonical store 112, in some embodiments, may include a cloud-based storage service, such as, for example, but not limited to, AWS S3.

[0037] Files written to the canonical store 112 may be associated with records added to the canonical segment index 114 and may provide index information for the records stored in the canonical store 112. Data stored in the canonical store 112 may be used in connection with various cold tier storage operations, partitioning and / or repartitioning operations, data backup operations, etc., as discussed in more detail below.

[0038] In some embodiments, the cold storage tier may include a segment extraction service 116, a cold data segment store 118, a data segment indexer 120, a data segment index 122, and / or a segment compression service 126. Consistent with various disclosed embodiments, data stored in the canonical store 112 and / or index information included in the canonical segment index 114 may be used to construct data records in the cold storage tier. For example, without limitation, the segment extraction service 116 may interact with the canonical store 112 and / or the canonical segment index 114 to access data from the canonical store 112, potentially process the data (e.g., partition and / or otherwise organize the data into time buckets ordered by recording time), and store the data in the cold data segment store 118. When data is stored in the cold data segment store 118, the segment extraction service 116 may interact with the data segment indexer service 120 to generate one or more records in a data segment index 122 associated with the data stored in the cold data segment store 118.

[0039] In various embodiments, the segment extraction service 116 may store data in the cold data segment store 118 based at least in part on information included in the definition metastore 102. For example, the definition metastore 102 may include information regarding cold data storage tier data storage and / or data update schedules, including information regarding update periods, update frequencies, update data volume thresholds, etc. This information may be used by the segment extraction service 116 to schedule data recording actions and / or updates from the canonical store 112 to the cold data segment store 118. For example, but not limited to, the definition metastore 102 may include update schedule information indicating that the cold storage tier should be updated daily. Records added to the canonical store 112 for the day's period may then be retrieved by the segment extraction service 116, which may partition the records according to a partitioning scheme (which may be defined by information included in the definition metastore 102), and then write the partitioned data to the cold data segment store 118. In some embodiments, the data stored in the cold data segment store 118 may include column files.

[0040] In some embodiments, data written to the cold data segment store 118 may be divided among time periods, which may be referred to as time buckets in certain instances herein, such that data of a single data partition associated with a timestamp belonging to a given time period is stored in the same time bucket. This may facilitate, among other things, efficient data search and / or management operations. For example, in connection with a data search over a particular time range, the time bucket information may be used to quickly identify data segments for the search.

[0041] Data written to the cold data segment store 118 may be associated with one or more records included in the data segment index 122. For example, when a data record is recorded in the cold data segment store 118 by the segment extraction service 116, the segment extraction service 116 may interact with the data segment indexer 120 to add an index record to the data segment index 122 associated with the data record.

[0042] In at least one non-limiting example, the segment extraction service 116 may be implemented using Apache Spark, and the cold data segment store 118 may be implemented using Parquet and / or AWS S3 storage. A Spark job may be launched by the segment extraction service 116, potentially periodically (e.g., on a user-specified periodic basis), to populate the cold data segment store 118. The Spark job may create new data segments for storage by the cold data segment store 118 as Parquet files for the defined data partitions and time buckets. The segments may be stored in AWS S3 storage, and associated entries may be added to the data segment index 122 by the data segment indexer 120. Another Spark job (e.g., a periodic Spark job) may be run to implement segment compression for the dataset, which in some implementations may meet user-specified compression criteria.

[0043] In various embodiments, the use of a canonical storage tier in conjunction with a cold storage tier consistent with certain aspects of the disclosed systems and methods may enable certain optimized data, processing, management, search, and / or query capabilities. For example, and without limitation, the canonical store 112 may store record data in a compressed form, but the partitioning and / or division of data and use of time buckets in conjunction with the cold data segment store 118 may provide certain data processing, search, management, and / or query efficiencies that may not otherwise be achieved directly by the canonical storage tier.

[0044] In certain embodiments, the definition metastore 102 may contain information used by various systems, services, and / or components of the disclosed platform to determine which ingested topics should be recorded by the hot data storage tier and the canonical store (and, by extension, the cold data storage tier). For example, in some embodiments, the streaming writer 104 and the canonical store writer 110 may use information contained in the definition metastore 102 to determine which ingested data should be recorded to the hot data store 106 and / or the canonical store 112.

[0045] In at least one non-limiting example, based on information included in the definition metastore 102, the entire input data stream may be ingested by the canonical store writer 110 for storage in the canonical store 112 (and / or cold data storage layer), but only a subset of the data may be ingested by the streaming writer 104 for storage in the hot data store 106. In some embodiments, the subset may be associated with a particular data topic, table, and / or associated projection. For example, the definition metastore 102 may include information that directs the streaming writer 104 to process input data associated with a particular topic for storage in the hot data store 106 (e.g., if there is a hot storage materialization defined for the input topic, etc.). However, it will be appreciated that the definition metastore 102 may include information that specifies various other manners in which data included in the data stream is processed and / or otherwise ingested by the canonical store writer 110 and / or the streaming writer 104. For example, in some implementations, the definition metastore 102 may include information specifying that all input data may be ingested by both the hot storage tier and the canonical storage tier.

[0046] In various embodiments, data stored in the canonical store 112 may be used in connection with data restoration and / or backup operations. For example, if data is deleted from the hot and / or cold storage tiers but remains stored in the canonical store 112, the data may be restored from the canonical store 112 to the hot and / or cold storage tiers.

[0047] In various embodiments, data stored in the canonical store 112 may be used in connection with a data repartitioning operation. For example, but not limited to, a data storage and / or data management service and / or a user thereof may determine that it is advantageous to repartition data stored in a cold storage tier from its original materialized projection (e.g., based on how the data in the cold storage tier is being queried, etc.). Using the data stored in the canonical store 112, the data may be repartitioned and stored in the cold data storage tier consistent with the updated projection. It will be appreciated that various other events that trigger repartitioning of data in the cold data storage tier may be used in connection with various aspects of the disclosed embodiments. Additionally, it will be appreciated that data repartitioning from the canonical store 112 may similarly occur in connection with a hot storage tier.

[0048] When retrieving data from the service, the streaming read API 124 may be queried with relevant query information (e.g., identifying a data partition and / or a time period). The streaming read API 124 may query the hot and cold storage tiers based on the identified data partition and / or time period. In some embodiments, the low-level data search component may apply a filter to the fetched data. The time-ordered sequence of records belonging to the data partitions fetched from both tiers may be processed by a deduplication device, where records having the same primary key but lower sequence numbers may be discarded. Records from different data partitions may be merged into a single result, and optional post-processing such as sorting or aggregation may be performed.

[0049] Consistent with various disclosed embodiments, the segment compression service 126 may initiate segment merge and / or compression operations (e.g., by initiating associated Spark jobs) consistent with various disclosed embodiments, potentially periodically and / or according to a user-specified schedule. In some embodiments, segment compression operations may be performed and / or may otherwise implement certain user-specified compression criteria.

[0050] Multidimensional Data Structures 2 illustrates a non-limiting example of a data management architecture 200 using data partitions and data segments consistent with certain embodiments disclosed herein. As illustrated, ingested data may be organized in a multi-dimensional space having a first dimension that includes an index to a data partition 202 and a second dimension that includes an index to a data segment 204 within the data partition. In many embodiments, the data and / or entries within a data segment may be chronological.

[0051] In some embodiments, a data table may include columns from which a subset of columns may be selected to compute a data partitioning key. In some implementations, these columns of a data table may include entries that may be referred to as selectors. A selector may be associated with a given partitioning scheme (which may be referred to in certain instances herein as a data projection and / or derivative thereof). In some embodiments, a data partition key may be computed as a function (e.g., a hash function) of certain data values ​​included in the data table. As illustrated with respect to FIG. 2, a selector may be included in a data partition index 202 associated with a data partition key.

[0052] The data segment index 204 may associate a data partition key with a particular timestamp range. As shown, a data partition key may be associated with multiple segments of a particular data partition (e.g., data partition key key1 may be associated with the first data segment 206 and the second data segment 208 of the first data partition). In some cases, a data partition key may be associated with a single segment of a data partition (e.g., data partition key keyM may be associated with the first data segment 210 of the Mth data partition).

[0053] In certain circumstances, there may be multiple records associated with the same timestamp. For example, in the illustrated example, a record having timestamp 1.T may be present in both the first segment 206 and the second segment 208 of the first data partition. Consistent with various embodiments disclosed herein, ingested records and / or data may be associated with a sequence number. Multiple records associated with the same timestamp may be distinguished based on the associated sequence numbers.

[0054] In some embodiments, the sequence numbers may be globally unique and monotonically increasing. In further embodiments, the sequence numbers may be unique within a given data partition. If duplicate records exist within the system, during the data retrieval and / or query process, the duplicate records may be filtered out such that only the data and / or records having the highest sequence numbers are returned. In certain embodiments, additional table columns may be associated with data records to allow additional information to be associated with the records and used in connection with distinguishing between the records.

[0055] The sequence number associated with a data record may be used in connection with various data operations, including, for example, but not limited to, data update, data access, data deletion, and / or data compression and / or merging operations. For example, as described above, the sequence number may be used in connection with importing and retrieving updates of previously imported data records, and data records having higher sequence numbers may be retrieved as part of a query to ensure that the most recent records are retrieved. For example, as illustrated in connection with FIG. 2, a data record associated with timestamp time 1.1 in a first segment 206 of a first data partition may be associated with sequence number seg1. An update to a data record also associated with timestamp time 1.1 may be imported and stored in the same segment 206. To distinguish this updated data record from the original data record associated with timestamp time 1.1, the updated data record may be assigned sequence number seq 2. In data retrieval and / or other data processing and / or management operations, the record having the higher sequence number, i.e., sequence number seq 2, is retrieved and / or otherwise considered the most recent data record.

[0056] In at least one non-limiting embodiment, a data table has columns col1, col2, col3, and col4, where col2 and col3 are selectors for a projection p1 and a record r N is a column identifier and value (id(col1), val 1N (id(col2),val 2N (id(col3),val 3N (id(col4),val 4N ) tuples of records r N The data partitioning key in the projection p1 of N =f(val 2N ,vaI 3n), where f() is a data partition key calculation function (e.g., a hash function). In some embodiments, for processing or storing a data partition in P partitions, the associated processing / storage partition index can be calculated based on hash(key N ) mod P.

[0057] Sequence numbers consistent with various aspects of the disclosed systems and methods may enable efficient data update and / or search operations. Additionally, in some implementations, because the original data may not be deleted (and in some embodiments may not be assigned a higher sequence number) when an update is received, the use of sequence numbers consistent with various aspects of the disclosed embodiments may facilitate data auditing and / or other methods of inspecting data record history, provenance, etc. Because the original data may not be immediately deleted (and / or may be configured to be retained forever and / or for a length of time depending on how data cleanup and / or deduplication processes are configured), the use of sequence numbers consistent with aspects of the disclosed embodiments may provide data record versioning and / or backup capabilities, where a data record having a lower sequence number may be accessed to access a previous version of the data record.

[0058] Data Compression and Merging 3 illustrates a non-limiting example of a data compression process 300 consistent with certain embodiments disclosed herein. As illustrated, a first record 308 associated with a timestamp time 1.1 may be stored in a first segment 302 of a first data partition and associated with a sequence number seq 1. A second record 310 also associated with a timestamp time 1.1 may also be stored in the first segment 302 of the first data partition associated with a higher sequence number seq 2. The second record 310 may include, for example, an update to the first record 308. Consistent with various embodiments disclosed herein, the data compression process may generate a compressed segment 306 of the first data partition that includes a record having a higher sequence number, i.e., the second data record 310 associated with the timestamp time 1.1 and sequence number seq 2.

[0059] Similarly, a data record 312 associated with a timestamp time 1.T may be stored in a first segment 302 of a first data partition and associated with sequence number seq SA, and a different data record 314 associated with timestamp time 1.T may be stored in a second segment 304 of the first data partition and associated with sequence number seq S+1. A data compression process consistent with certain embodiments disclosed herein may add the record with the higher sequence number, i.e., the data record 314 associated with timestamp time 1.T and sequence number seq S+1, to the compressed data segment 306.

[0060] In the illustrated embodiment, a data record 316 included in the second segment 304 of the second data partition associated with timestamp time 1.T+1 and sequence number seq S+2 may also be added to the compressed data segment 306. In this manner, the most recent data records of the first segment 302 and the second segment 304 of the first partition may be combined into the compressed data segment 306 of the first data partition (the first segment 302 and the second segment 304 are scheduled for eventual deletion).

[0061] 4 illustrates a flow chart of a non-limiting example of a data compression process 400 consistent with certain embodiments disclosed herein. The illustrated method 400 may be implemented in a variety of ways, including using software, firmware, hardware, and / or any combination thereof. In certain embodiments, various aspects of the process 400 and / or its constituent steps may be performed by one or more systems and / or services, including systems and / or services that may implement aspects of a hot data storage tier, a cold data storage tier, a canonical data store, and / or various shared systems and / or services.

[0062] In various embodiments, the data compression process 400 and / or aspects thereof may be initiated periodically, based on a user instruction, and / or in accordance with one or more conditions and / or triggers. For example, in various embodiments, the disclosed data compression process and / or aspects thereof may be initiated based on a determination that a total number of data segments of a data partition has reached a threshold number of data segments, that a total storage size of a data partition has reached a threshold total storage size, etc.

[0063] At 402, a first time-series data record stored in a first segment of a first data partition may be identified. In various embodiments, the first data partition may be stored in a cold data store (e.g., a cold data store managed at least in part by a data management service system). In certain embodiments, the cold data store may include a cloud service data store.

[0064] The first time series data record may be associated with a first timestamp and a first sequence identifier. In various embodiments, the sequence identifier may include a sequence number, although other types of sequence identifiers may be used. In certain embodiments, the sequence identifier and / or the sequence number may be monotonically increasing and / or unique within a given partition.

[0065] At 404, it may be determined whether another record in the first data partition is associated with the same timestamp as the first timestamp (e.g., as in the case where an update, revision, and / or newer record to the first time series data record is stored in the first data partition). If so, process 400 may proceed to 406, where a second time series data record stored in the first data partition associated with the first timestamp may be identified. In various embodiments, the second time series data record may be associated with a second sequence identifier. In certain embodiments, the second time series data record may be stored in a first segment of the first data partition. In further embodiments, the second time series data record may be stored in another segment of the first data partition.

[0066] At 408, it may be determined that the second sequence identifier is greater than the first sequence identifier, indicating that the second time series data record was recorded after the first time series data record. Based on this determination, at 410, a compressed segment of the first data partition may be generated and / or stored. The compressed data segment may include the second time series data record. In certain circumstances, a record may be updated multiple times. In various embodiments, a record having the largest sequence identifier (e.g., indicating that it is the most recently updated record) that shares a timestamp with other records may be identified and included in the compressed data segment.

[0067] Following generation and / or storage of the compressed data segment, the first segment of the first data partition may be marked for deletion at 412. In certain embodiments, when a new segment is stored in the data partition, a cold storage tier data segment index may be updated.

[0068] In some embodiments, a third time series data record may be identified in the first data segment of the first data partition associated with the second timestamp and the third sequence identifier. A fourth time series data record stored in the second segment of the first data partition may be further identified, also associated with the second timestamp and further associated with a fourth sequence identifier. The fourth sequence identifier may be determined to be greater than the third sequence identifier, and the generated compressed segment of the first data partition may further include the fourth time series data record. In this manner, the compressed segment may include records resulting from multiple data segments of the data partition. Similar to the first data segment, the second data segment may be marked for deletion as part of the compression process.

[0069] In certain embodiments, marking one or more data segments for deletion may include scheduling the data segments for deletion. For example, without limitation, deletion of the data segments may be scheduled to occur at a particular deletion time. The deletion time may include, for example, a next scheduled deletion time, which may be scheduled periodically and / or by a user, a deletion time determined at least in part based on determining that a total number of data segments of the first data partition has reached a threshold number of data segments, determining that a total storage size of the first data partition has reached a threshold total storage size, etc.

[0070] Data Record Deletion and Tombstones 5 illustrates a flow chart of a non-limiting example of a data record deletion process 500 using data record tombstones consistent with certain embodiments disclosed herein. The illustrated method 500 may be implemented in a variety of ways, including using software, firmware, hardware, and / or any combination thereof. Certain embodiments may include various aspects of the process 500 and / or its constituent steps may be performed by one or more systems and / or services, including systems and / or services that may implement aspects of a hot data storage tier, a cold data storage tier, a canonical data store, and / or various shared systems and / or services.

[0071] At 502, a time series data record may be received for storage in a data partition (e.g., a data partition of a cold data store). Consistent with various disclosed embodiments, the time series data record may be associated with a timestamp and a first sequence identifier. At 504, the time series data record may be stored in the data partition.

[0072] At 506, a request to delete the time series data record may be received. In response to the delete request, at 508, a time series data record tombstone may be generated and stored (stored in the data partition in the same and / or different segments within the data partition). Consistent with embodiments disclosed herein, the data record tombstone may be associated with a timestamp and a second sequence identifier, which may be higher and / or greater than the first sequence identifier, indicating that the tombstone was recorded after the time series data record. In some embodiments, the time series data record tombstone may not include any data values ​​(i.e., may be an empty data record) and / or may include information indicating and / or otherwise identifying the record as a tombstone record.

[0073] Consistent with certain embodiments disclosed herein, tombstone records may be identified and used in connection with a data management process as part of a data cleanup, compression, and / or merging process consistent with various disclosed embodiments. For example, without limitation, in some embodiments, a record associated with a tombstone record may not be included in a compressed data segment (which may or may not include a tombstone record) generated as part of a data compression and / or merging process.

[0074] Exemplary System Architecture 6 illustrates an example of a system 600 that may be used to implement certain embodiments of the systems and methods of the present disclosure. Various systems, services and / or devices used in connection with aspects of the disclosed embodiments may be communicatively coupled using a variety of networks and / or network connections (e.g., network 608). In certain embodiments, network 608 may include a variety of network communication devices and / or channels and may utilize any suitable communication protocols and / or standards that facilitate communication between the systems and / or devices.

[0075] Network 608 may include the Internet, a local area network, a virtual private network, and / or any other communications network utilizing one or more electronic communications technologies and / or standards (e.g., Ethernet, etc.). In some embodiments, network 608 may include a wireless carrier system, such as a personal communications system ("PCS"), and / or any other suitable communications system incorporating any suitable communications standards and / or protocols. In further embodiments, the network 608 may include analog and / or digital mobile communications networks, for example, utilizing code division multiple access (CDMA), Global System for Mobile Communications or Groupe Special Mobile (GSM), frequency division multiple access (FDMA), time division multiple access (TDMA) standards, 4G and / or 5G communications standards (e.g., Long-Term Evolution (LTE), 5G New Radio (NR), orthogonal frequency division multiple access (OFDMA), etc.). In certain embodiments, the network 608 may incorporate one or more satellite communications links. In still further embodiments, the network may utilize the IEEE 802.11 standard, Bluetooth®, ultra-wide band (“UWB”), Zigbee®, and / or any other suitable standard or specifications.

[0076] The various systems and / or devices used in connection with aspects of the disclosed embodiments may comprise a variety of computing devices and / or systems, including any computing system or systems suitable for implementing the systems and methods disclosed herein. For example, the connected devices and / or systems may include a variety of computing devices and systems, including laptop computer systems, desktop computer systems, server computer systems, distributed computer systems, smartphones, tablet computers, etc.

[0077] In certain embodiments, the systems and / or devices may comprise at least one processor system configured to execute instructions stored on an associated non-transitory computer-readable storage medium. As discussed in more detail below, systems used in connection with implementing various aspects of the disclosed embodiments may further comprise a secure processing unit ("secure processing unit, SPU") configured to perform sensitive operations such as trusted credential and / or key management, cryptographic operations, secure policy management, and / or other aspects of the systems and methods disclosed herein. The systems and / or devices may further comprise software and / or hardware configured to enable electronic communication of information between the devices and / or systems over a network using any suitable communication technology and / or standard.

[0078] As illustrated in FIG. 6 , an exemplary system 600 may include a processing unit 602; system memory 604, which may include high-speed random access memory (“random access memory” (“RAM”)), nonvolatile memory (“nonvolatile memory” (“ROM”)), and / or one or more bulk non-volatile non-transitory computer-readable storage media (e.g., hard disk, flash memory, etc.) for storing programs and other data for use and execution by the processing unit 602; a port 614 for interfacing with removable memory 616, which may include one or more diskettes, optical storage media (e.g., flash memory, thumb drives, USB dongles, compact discs, DVDs, etc.) and / or other non-transitory computer-readable storage media; a network interface 606 for communicating with other systems over one or more network connections and / or networks 608 using one or more communications technologies; a user interface 612, which may include a display and / or one or more input / output devices, such as, for example, a touch screen, keyboard, mouse, track pad, etc.; and one or more buses 618 for communicatively coupling the elements of the system.

[0079] In some embodiments, system 600 may alternatively or additionally include SPU 610 that is protected from tampering by users of system 600 or other entities by utilizing secure physical and / or virtual security techniques. SPU 610 may help enforce security of sensitive operations such as personal information management, trusted credentials and / or key management, privacy and policy management, and other aspects of the systems and methods disclosed herein. In certain embodiments, SPU 610 may be configured to operate in a logically secure processing domain and to protect and manipulate sensitive information as described herein. In some embodiments, SPU 610 may include internal memory that stores executable instructions or programs configured to enable SPU 610 to perform secure operations as described herein.

[0080] Operation of the system 600 may generally be controlled by the processing unit 602 and / or SPU 610, which operate by executing software instructions and programs stored in the system memory 604 (and / or other computer-readable media, such as removable memory 616). The system memory 604 may store various executable programs or modules for controlling the operation of the system 600. For example, the system memory may include an operating system ("operating system, OS") 620, which may at least in part manage and coordinate system hardware resources and provide common services for the execution of various applications, and a trust and privacy management system 622 for implementing trust and privacy management functions, including protection and / or management of personal data through management and / or enforcement of associated policies. The system memory 604 may further include, without limitation, communications software 624 configured to enable, in part, communication with and by the system 600, one or more applications, a data management service 626 configured to implement various aspects of the disclosed systems and / or methods, and / or any other information and / or applications configured to implement embodiments and / or aspects of the systems and methods disclosed herein.

[0081] The systems and methods disclosed herein are not inherently related to any particular computer, electronic control unit, or other apparatus, but may be implemented by any suitable combination of hardware, software, and / or firmware. A software implementation may include one or more computer programs that include executable code / instructions that, when executed by a processor, cause the processor to perform a method defined at least in part by the executable instructions. The computer programs may be written in any form of programming language, including compiled or interpreted languages, and may be deployed in any form, such as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. Furthermore, the computer programs may be deployed to be executed on one computer or on multiple computers at one site or distributed multiple sites, interconnected by a communication network.

[0082] A software embodiment may be implemented as a computer program product including a computer program and a non-transitory storage medium configured to store instructions that, when executed by a processor, are configured to cause the processor to perform a method in accordance with the instructions. In certain embodiments, the non-transitory storage medium may take any form capable of storing processor-readable instructions on the non-transitory storage medium. The non-transitory storage medium may be embodied by a compact disc, a digital video disk, a magnetic disk, a flash memory, an integrated circuit, or any other non-transitory digital processing apparatus memory device.

[0083] Although the foregoing has been described in some detail for clarity, it will be apparent that certain changes and modifications may be made without departing from the principles of the present invention. For example, it will be understood that several modifications may be made to the various embodiments, systems, services, and / or components presented in connection with the drawings and / or associated description within the scope of the body of work of the present invention, and that the examples presented in the drawings and described herein are provided for purposes of illustration and description, not limitation. It is further noted that there are many alternative ways of implementing both the systems and methods described herein. Thus, the present embodiment should be considered as illustrative and not restrictive, and the embodiments of the present invention should not be limited to the details given herein, but may be modified within the scope of the appended claims and their equivalents.

Claims

1. 1. A method for managing time series data performed by a data management service system, the data management service system comprising at least one processor and a non-transitory computer readable medium storing instructions that, when executed by the at least one processor, cause the data management service to perform the method, the method comprising: identifying a first time series data record stored in a first segment of a first data partition, the first time series data record being associated with a first timestamp and a first sequence identifier; identifying a second time series data record stored in the first data partition, the second record associated with the first timestamp and a second sequence identifier; determining that the second sequence identifier is greater than the first sequence identifier; generating a compressed segment of the first data partition based at least in part on determining that the second sequence identifier is greater than the first sequence identifier, the compressed segment including the second time series data record associated with the first timestamp and the second sequence identifier; storing the compressed segment in the first data partition; marking the first segment of the first data partition for deletion by the data management services system.

2. The method of claim 1 , wherein the second time-series data record is stored in the first segment of the first data partition.

3. The method of claim 1 , wherein the second time-series data record is stored in a second segment of the first data partition.

4. The method of claim 1 , wherein the first data partition is stored in a cold data store that is at least partially managed by the data management service system.

5. The method of claim 4 , wherein the cold data store comprises a cloud services data store.

6. The method of claim 1 , wherein the first sequence identifier and the second sequence identifier are unique to the first data partition.

7. The method of claim 6 , wherein the first sequence identifier is greater than a sequence identifier assigned to a previous data record.

8. The method of claim 1 , wherein the second time-series data record includes an update to the first time-series data record.

9. The method comprises: identifying a third time series data record stored in the first segment of the first data partition, the third time series data record being associated with a second timestamp and a third sequence identifier; 2. The method of claim 1, further comprising identifying a fourth time series data record stored in a second segment of the first data partition, the fourth time series data record being associated with the second timestamp and a fourth sequence identifier.

10. The method comprises: determining that the fourth sequence identifier is greater than the third sequence identifier; 10. The method of claim 9, wherein generating the compressed segment of the first data partition is further based at least in part on determining that a fourth sequence number is greater than a third sequence number, and wherein the compressed segment includes the fourth time series data record associated with the second timestamp and the fourth sequence identifier.

11. The method of claim 10 , further comprising marking the second segment of the first data partition for deletion by the data management service system.

12. 2. The method of claim 1, wherein the first data partition is stored in a cold data store, the method further comprising updating a cold storage tier data segment index in response to storing the compressed segment in the first data partition.

13. The method of claim 1 , wherein the first sequence identifier comprises a first sequence number and the second sequence identifier comprises a second sequence number.

14. 2. The method of claim 1 , wherein marking the first segment of the first data partition for deletion by the data management service system comprises scheduling the first data segment of the first data partition for deletion.

15. 15. The method of claim 14, wherein scheduling the first data segment of the first data partition for deletion comprises scheduling the first data segment of the first data partition for deletion at a next deletion time by the data management service system.

16. The method of claim 15 , wherein the next purge time comprises a periodic purge time.

17. The method of claim 16 , wherein the periodic deletion times include periodic deletion times defined by a user of the data management service system.

18. 16. The method of claim 15, wherein the next deletion time is determined based at least in part on determining that a total number of data segments of the first data partition has reached a threshold number of data segments.

19. 16. The method of claim 15, wherein the next deletion time is determined based at least in part on determining that a total storage size of the first data partition has reached a threshold total storage size.

20. The method of claim 1 , further comprising deleting the first segment of the first data partition.

Images