Managing real-time data stream processing

Abstract

A method (300) of managing processing of data (12) includes receiving a data query (170) from a user (10) of a data query system (150), data of the data query being stored in a data store (142) in communication with the data query system. The method also includes receiving a staleness parameter (174) indicating an upper time bound of the data query. The upper time bound limits a query response (172) to data within the data store that is earlier in time than the upper time bound. The method also includes determining whether data stored within the data store satisfies the staleness parameter. When a portion of the data within the data store does not satisfy the staleness parameter, the method includes generating the query response excluding the portion of the data that does not satisfy the staleness parameter.

Description

[0001] This application is a divisional application of the following application,

[0002] International application number of the original application: PCT / US2021 / 026929

[0003] The original international filing date was April 12, 2021.

[0004] The original application's national application number was 202180042569.2.

[0005] Original application title: Managing Real-Time Data Stream Processing Technical Field

[0006] This disclosure relates to the management of real-time data stream processing. Background Technology

[0007] As today's applications generate massive amounts of data, analytics tools have become increasingly powerful. In other words, these tools can extract depth from rich data to support their functionality. With massive data volumes, analytics tools such as dashboards, real-time alerts, and other data visualizations are being deployed more frequently by users to understand their data. With this increased use, some analytics tools require minimal latency while wanting to analyze real-time or near-real-time data. Since the data powering these tools is often stored in distributed systems (e.g., cloud storage), these distributed storage systems are addressing these needs in ways that meet these requirements, especially when the data they store arrives as a continuous data stream. Summary of the Invention

[0008] One aspect of the present invention provides a method for managing data processing. The method includes receiving, at data processing hardware, a data query from a user of a data query system, the data of which is stored in a data memory communicating with the data query system. The method further includes, at the data processing hardware, receiving a stagnation parameter indicating an upper time boundary of the data query, the upper time boundary limiting a query response to data in the data memory earlier than the upper time boundary. The method also includes determining, by the data processing hardware, whether the data stored in the data memory satisfies the stagnation parameter. When a portion of the data in the data memory does not satisfy the stagnation parameter, the method includes the data processing hardware generating a query response that excludes the portion of the data written to the data memory more recently than the upper time boundary of the stagnation parameter.

[0009] Implementations of this disclosure may include one or more of the following optional features. In some implementations, receiving the stagnation parameter includes receiving the stagnation parameter from the user of the data query system. The method may include: determining an upper time boundary of the stagnation parameter by data processing hardware receiving user data from the user; ingesting the received user data into the data storage to form one or more log files at a first time; converting the one or more log files into a column data format at a second time, the column data format being optimized for a corresponding query; determining a time difference between the first time and the second time; and assigning the time difference to the upper time boundary of the stagnation parameter.

[0010] In some examples, the method includes: identifying a log file for the data stored in the data storage device by data processing hardware; and determining, by the data processing hardware, a corresponding timestamp within the log file that most closely matches the upper time boundary of the stagnation parameter. In this example, each log file includes multiple rows of data, each row including a timestamp. Also in this example, for a query response, the method includes reading data from the log file by the data processing hardware that is earlier than the timestamp of the upper time boundary that most closely matches the stagnation parameter. Here, the timestamp may indicate when the data storage device generated the corresponding data row in the corresponding log file.

[0011] In some configurations, the method includes receiving a data set from the user at data processing hardware, having the data processing hardware ingest the data set into the data storage, and having the data processing hardware generate one or more log files for the data set. In this configuration, each log file includes data rows corresponding to data from the data set, and the generation of the one or more log files generates a timestamp for each row of data within the respective log file.

[0012] Another aspect of this disclosure provides a method for managing data processing. The method includes receiving a query at data processing hardware, the query requesting data stored in a storage system for a corresponding time interval. The method also includes determining, by the data processing hardware, whether any portion of the corresponding time interval corresponds to static data cached from a previous query. Static data corresponds to stored data that remains unchanged during storage operations in the storage system. When a portion of the corresponding time interval corresponds to static data cached from the previous query, the method includes generating a corresponding response by the data processing hardware to the query, the query including the cached static data corresponding to the portion of the corresponding time interval, without performing a read operation on the stored data within the portion of the corresponding time interval in the storage system.

[0013] This aspect may include one or more of the following optional features. In some implementations, the method includes: receiving an initial query at data processing hardware, the initial query requesting data stored in the storage system at an initial time interval; determining by the data processing hardware that a portion of the data stored in the storage system at the initial time interval includes static data; and generating a response by the data processing hardware to the initial query, the initial query including the requested data and including the portion of data corresponding to the corresponding static data at the initial time interval. The data processing hardware caches the portion of data corresponding to the static data at the initial time interval. Here, determining that the portion of the data stored in the storage system at the initial time interval includes static data includes identifying a timestamp associated with a column format of the data stored in the storage system, the timestamp indicating the last modification date of the stored data in a given column. Optionally, determining that the portion of the data stored in the storage system at the initial time interval includes static data includes determining that a timestamp associated with a given column of the stored data in the storage system satisfies a static data threshold, the timestamp indicating the last modification date of the stored data in the given column, the static data threshold indicating the probability that the stored data in the given column will remain unchanged during future storage operations of the storage system.

[0014] In some configurations, generating the corresponding response to the query includes performing a read operation on the stored data in the storage system within the corresponding time interval, but not within the portion of the corresponding time interval corresponding to the static data cached from the previous query. When any portion of the corresponding time interval fails to correspond to static data cached from the previous query, the method may include the data processing hardware determining that a portion of the data stored in the storage system during the corresponding time interval includes static data. Here, the method may further include: generating the corresponding response to the corresponding query by the data processing hardware, the corresponding query including the requested data and including the portion of data corresponding to the corresponding static data of the corresponding time interval; and caching the portion of data corresponding to the static data of the corresponding time interval by the data processing hardware.

[0015] In some examples, the storage operation includes storing a continuous data stream of real-time data from users of the storage system. The method involves the data processing hardware transmitting a response to the query to a data analysis system configured to generate a graphical representation of the response.

[0016] Another aspect of this disclosure provides a system for managing data processing. The system includes data processing hardware and memory hardware in communication with the data processing hardware. The memory hardware stores instructions that, when executed on the data processing hardware, cause the data processing hardware to perform operations. The operations include receiving a data query from a user of a data query system, the data of which is stored in a data memory in communication with the data query system. The operations also include receiving a stagnation parameter indicating an upper time boundary of the data query. The upper time boundary limits the query response to data in the data memory that is earlier than the upper time boundary. The operations further include determining whether the data stored in the data memory satisfies the stagnation parameter. When a portion of the data in the data memory does not satisfy the stagnation parameter, the operations include generating the query response that excludes the portion of the data written to the data memory more recently than the upper time boundary of the stagnation parameter.

[0017] Implementations of this aspect of the disclosure may include one or more of the following optional features. In some embodiments, receiving the stagnation parameter includes receiving the stagnation parameter from the user of the data query system. The system may include: receiving user data from the user, ingesting the received user data into the data storage to form one or more log files at a first time, converting the one or more log files into a column data format at a second time, optimizing the column data format for a corresponding query, determining the time difference between the first time and the second time to determine the upper time boundary of the stagnation parameter, and allocating the time difference to the upper time boundary of the stagnation parameter.

[0018] In some examples, the system includes identifying log files for the data stored in the data storage device, determining a corresponding timestamp within the log file that best matches the upper time boundary of the stagnation parameter, and, in response to a query, reading data from the log file that is earlier than the timestamp of the upper time boundary that best matches the stagnation parameter. In this example, each log file includes multiple lines of data, each line including a timestamp. Here, the timestamp can indicate when the data storage device generated the corresponding data line in the corresponding log file.

[0019] In some configurations, the system includes receiving a data set from the user, ingesting the data set into the data storage, and generating one or more log files for the data set. In this configuration, each log file includes data rows corresponding to data from the data set, and the generation of the one or more log files involves generating a timestamp for each row of data within the respective log file.

[0020] Another aspect of this disclosure provides a system for managing data processing. The system includes data processing hardware and memory hardware in communication with the data processing hardware. The memory hardware stores instructions that, when executed on the data processing hardware, cause the data processing hardware to perform operations. The operations include receiving a query that requests data stored in a storage system for a corresponding time interval. The operations also include determining whether any portion of the corresponding time interval corresponds to static data cached from a previous query. Static data corresponds to stored data that remains unchanged during storage operations in the storage system. When a portion of the corresponding time interval corresponds to static data cached from the previous query, the operations include generating a corresponding response to the query, the query including the cached static data corresponding to the portion of the corresponding time interval, without performing a read operation on the stored data within the portion of the corresponding time interval in the storage system.

[0021] Implementations of this aspect of the present disclosure may include one or more of the following optional features. In some examples, the system includes: receiving an initial query that requests data stored in the storage system at an initial time interval; determining that a portion of the data stored in the storage system at the initial time interval includes static data; generating a response to the initial query, the initial query including the requested data and including the portion of data corresponding to the corresponding static data at the initial time interval; and caching the portion of data corresponding to the static data at the initial time interval. Here, determining that the portion of the data stored in the storage system at the initial time interval includes static data includes identifying a timestamp associated with a column format of the data stored in the storage system, the timestamp indicating the last modification date of the stored data in a given column. Optionally, determining that the portion of the data stored in the storage system at the initial time interval includes static data includes determining that a timestamp associated with a given column of the stored data in the storage system satisfies a static data threshold, the timestamp indicating the last modification date of the stored data in the given column, the static data threshold indicating the likelihood that the stored data in the given column will remain unchanged during future storage operations of the storage system.

[0022] In some configurations, generating a response to a query includes performing a read operation on stored data in the storage system within the corresponding time interval, but not within a portion of the corresponding time interval corresponding to static data cached from the previous query. When any portion of the corresponding time interval fails to correspond to static data cached from the previous query, the system may include: determining that a portion of the data stored in the storage system during the corresponding time interval includes static data; generating a response to a corresponding query, the corresponding query including the requested data and including a portion of the data corresponding to the corresponding static data of the corresponding time interval; and caching the portion of the data corresponding to the static data of the corresponding time interval. The storage operation may include storing a continuous data stream of real-time data from a user of the storage system. The system may include transmitting a response to the query to a data analysis system configured to generate a graphical representation for the response.

[0023] Details of one or more implementations of this disclosure are set forth in the accompanying drawings and the following description. Other aspects, features, and advantages will be apparent from the specification, the drawings, and the claims. Attached Figure Description

[0024] Figure 1 This is a schematic diagram of an example data management environment.

[0025] Figure 2A yes Figure 1 A schematic diagram of an example data manager for a data management environment.

[0026] Figure 2B yes Figure 2A A schematic diagram of an example ingestor for the data manager.

[0027] Figure 2C and 2D It is used for Figure 2A A schematic diagram of an example converter for the data manager.

[0028] Figure 2E yes Figure 2A A schematic diagram of an example operator for a data manager.

[0029] Figure 2F It is used for Figure 2A A schematic diagram of an example cache for a data manager.

[0030] Figure 3 and 4 This is a flowchart illustrating an example of the arrangement of operations in a data processing method.

[0031] Figure 5This is a schematic diagram of an exemplary computing device that can be used to implement the systems and methods described herein.

[0032] In the various figures, the same reference numerals denote the same elements. Detailed Implementation

[0033] Real-time applications generate massive amounts of data in real time and push this data to data warehouses for storage and future data analysis. For example, some applications can generate dashboards (i.e., visual analytics), real-time alerts, or other analyses based on data stored in a data warehouse. When queries related to the data stored in the data warehouse occur, these queries typically require minimal latency to ensure that dashboards or other data-driven analyses convey accurate information. It is not uncommon for such query processes to execute within milliseconds. Therefore, the goal of query-enabled data storage systems is to have data in the data warehouse in a format that allows for querying as quickly as possible.

[0034] Typically, applications running fast queries often employ a technique called column data management. Data typically arrives at the storage system in a row-oriented ingestion. Column data management requires converting the row-oriented data to a column-oriented format. This conversion is a processor-intensive process (i.e., demanding high processing resources such as CPU). Furthermore, the data ingestion process itself is processor-intensive. Since both types of processing are resource-intensive, the ingestion system does not want to tax resources to the point of potentially causing ingestion problems (i.e., problems while the storage system is receiving data). Thus, column data management transposes the data after it has been ingested in a row-oriented format. Because column data management occurs after ingestion, the storage system aims to ingest data as quickly as possible so that column data management can occur shortly thereafter. Based on this approach, the storage system receives the most recent user data in a row-oriented format and writes the received data to the permanent storage portion of the storage system. Then, a background process converts the stored data into a query-friendly format, allowing queries on the stored data to be optimized for rapid execution.

[0035] Ideally, when a storage system receives a query, it prefers that the stored data involved in the query is already in a query-friendly format (e.g., also known as a query-optimized format). Otherwise, if the stored data involved in the query has not been converted to a query-optimized format, the query may be delayed. For example, when the stored data involved in the query is not in a query-optimized format, the query read time will increase due to the non-query-optimized form of the stored data. In other words, data in a query-optimized format has been optimized to reduce the read time of the non-query-optimized format. Therefore, by requesting a query on the most recently ingested data by the storage system, the query process inherently experiences additional read time due to reading data in a non-query-optimized format (e.g., row-oriented format).

[0036] In one approach to overcoming potential latency issues and / or optimizing data queries, query-based storage systems can optimize queries, if possible, to avoid querying data stored in a non-query-optimized format. In other words, due to the characteristics of the storage system and the time required to transform the ingested data into a query-optimized format, queries can be restricted to reading only the data stored in the query-optimized format or maximizing the amount of data read in the query-optimized format. For example, a storage system processes real-time streaming data from start to finish. Since streaming data can be continuous, the data stream itself is often referred to as a portion of the data stream. Here, the head refers to the beginning of the data stream or a specified starting reference point within the data stream, while the tail refers to the data currently being generated in the data stream. Therefore, the head of data is essentially data that is earlier than, for example, the tail. For simplicity, for query-based storage systems, the head of data refers to the portion of the data that has both been ingested by the storage system and transformed into a query-optimized format, while the tail refers to the latest or most recent data that is being ingested or has been ingested but has not yet been transformed into a query-optimized format. Here, if the query system restricts the query to data from the head (e.g., data that has been fully processed and is in a query-optimized form) and excludes data from the tail, the query system will minimize latency.

[0037] Excluding data from the tail is a staleness approach to data query optimization. The staleness approach allows users to allow (e.g., specify or agree to) an acceptable amount of staleness with respect to the results of their data queries. In other words, a user can instruct the query system that a query for their data does not need to look at data from the most recent five minutes (e.g., data from the tail of the data stream). Here, this five minutes indicates that the user is content to have their query results stale for five minutes (i.e., missing results from the most recent five minutes). In this example, the user's query focuses on data from five minutes ago. This approach is a trade-off between query latency and staleness. In other words, if the user allows a certain prescribed level of staleness (e.g., five minutes), his or her query will occur with less latency because the query system does not need to respond to information about more recent data (e.g., within five minutes). Therefore, staleness indicates the user's tolerance for query responses to slightly earlier data (i.e., not recent data) so that the query system can provide a faster response to queries from the user.

[0038] While query-based storage systems ideally attempt to avoid increasing read times by retrieving data in non-query-optimized formats, this isn't always possible. For example, a user or user analytics might request queries for recently stored data that is still in a non-query-optimized format. Therefore, another approach to query system optimization is query caching. Query caching can be used in conjunction with stagnation methods or as an alternative. The core idea of ​​query caching is that even if a query might require reading some data stored in a non-query-optimized format, caching can further reduce the query time associated with query-optimized data.

[0039] Query caching typically operates on the principle that most of the data requested by a query remains constant. This can be advantageous when queries on a query-based storage system occur frequently on a routine basis. For example, data queries supporting user analytics might be configured to occur periodically to maintain relatively up-to-date analytics tools. As an example, a user dashboard might be configured to represent the most recent 90 days of data relevant to the user in the data store. Therefore, the user would program queries at a specific frequency to retrieve the most recent 90 days of data to populate the user's dashboard with the actual data from those 90 days. However, within those 90 days, most of the stored data for that 90-day period remains constant (e.g., 88 out of 90 days remain unchanged). This means that typically only a small fraction of the 90 days of data changes. For example, only the data from the last day or two changes. In other words, when new data is ingested by the storage system, that new data (e.g., the last two days) will be new to the user's analytics because it was not previously represented. Conversely, previously represented data (e.g., the previous 88 days) remains largely unchanged. Here, the query system can leverage a typical 90-day query frequency and the simple principle that storing new, real-time streaming data has minimal impact on most of the queried data. When a query generates results from the most recent 90 days, it can cache those results. By caching results, subsequent queries can avoid reading stored data already present in the cache of previous queries. Using this method, the current query might have to read the most recent data in a non-query-optimized form (e.g., the last two days), but completely avoids reading large chunks of stored data (e.g., even if it were query-optimized). By employing a stagnation method, a query caching method, or some combination of both, the overall query time of the query system can be reduced.

[0040] Figure 1An example of a data management environment 100 is shown. User device 110 associated with user 10 generates user data 12 while executing its computing resources 112 (e.g., data processing hardware 114 and / or memory hardware 116). For example, user 10 generates user data 12 using a real-time application running on the data processing hardware 114 of user device 110. A real-time application typically refers to an application or program that user 10 interprets or generally considers to be immediate or current. Because real-time applications strive to be current, they often require low latency in the execution time of the set of tasks that perform the tasks or use computing resources (e.g., computing resource 112). Some examples of real-time applications include video conferencing applications, community storage applications, e-commerce transactions, and other applications with real-time communication capabilities. Because real-time applications have the ability to generate large amounts of user data 12, user 10 typically utilizes other systems (e.g., remote system 130, storage system 140, query system 150, or analytics system 160) for user data storage and / or user data management.

[0041] In some examples, user equipment 110 is a local device (e.g., associated with the location of user 10) that uses its own computing resources 112 capable of communicating with one or more remote systems 130 (e.g., via network 120). Additionally or optionally, user equipment 110 utilizes its access to remote resources (e.g., remote computing resource 132) to operate applications for user 10. User data 12 generated using user equipment 110 can be initially stored locally (e.g., in data memory 118 of memory hardware 116) and then transferred to remote system 130, or sent to remote system 130 at creation time via network 120. For example, user equipment 110 uses remote system 130 to transfer user data to storage system 140.

[0042] In some examples, user 10 utilizes computing resources 132 of remote system 130 (e.g., a cloud computing environment) to store user data 12. In these examples, remote system 130 may receive streaming user data 12 as it is being generated by various user applications. Here, a data stream (e.g., user data 12) refers to a continuous or typically continuous feed of data arriving at remote system 130 for storage and / or further processing. In some configurations, user 10 and / or remote system 130 configure user data 12 to be sent at frequent intervals, rather than continuously feeding user data stream 12 to remote system 130, so that remote system 130 has a continuous supply of user data 12 to process. Much like user equipment 110, remote system 130 includes computing resources 132, such as remote data processing hardware 134 (e.g., servers and / or CPUs) and memory hardware 136 (e.g., disks, databases, or other forms of data storage).

[0043] In some configurations, remote computing resource 132 is a resource used by various systems associated with and / or communicating with remote system 130. For example... Figure 1 As shown, these systems may include storage system 140, query system 150, and / or analysis system 160. In some examples, the functionality of these systems 140, 150, and 160 may be integrated in different combinations (e.g., built into each other) or as separate systems capable of communicating with each other. For example, storage system 140 and query system 150 may be combined into a single system (e.g., as shown in the diagram). Figure 1 (Dash lines are shown around these systems). Remote system 130 and its computing resources 132 can be configured to host one or more functions of these systems 140, 150, 160. In some embodiments, remote system 130 is a distributed system, with its computing resources 132 distributed across one or more locations accessible via network 120.

[0044] In some examples, storage system 140 is configured to operate data warehouse 142 (e.g., data storage and / or multiple databases) as a means of data storage for user 10 (or multiple users). Generally, data warehouse 142 stores data from one or more sources and can be designed to analyze, report on, and / or integrate data from its sources. Data warehouse 142 enables users (e.g., organizational users) to have a central storage repository and storage data access points. By including user data 12 in a central repository such as data warehouse 142, data warehouse 142 can streamline data retrieval functions such as data analysis and / or data reporting (e.g., via analysis system 160). Furthermore, data warehouse 142 can be configured to store large amounts of data, allowing user 10 (e.g., organizational users) to store large amounts of historical data to understand data trends. Since data warehouse 142 can be the primary or sole data storage repository for user data 12, storage system 140 can typically receive large amounts of data (e.g., gigabytes per second, megabytes per second, or more) from user devices 110 associated with user 10. Additionally or optionally, as storage system 140, storage system 140 and / or storage warehouse 142 can be configured for data security (e.g., data redundancy), for multiple users from a single data source (e.g., multiple employees within an organization), and / or for simultaneous multi-user access. In some configurations, data warehouse 142 is persistent and / or non-volatile, such that by default, data is not overwritten or erased by newly arriving data.

[0045] Query system 150 is configured to request information or data from storage system 140 in the form of query 170. In some examples, query 170 is initiated by user 10 as a request for user data 12 within storage system 140. For example, user 10 operates through query system 150 (e.g., an interface associated with query system 150) to retrieve user data 12 stored in data warehouse 142 of storage system 140. Here, query 170 can be user-initiated (i.e., directly requested by user 10) or system-initiated (i.e., configured by query system 150 itself). In some examples, query system 150 configures routine or recurring queries 170 (e.g., at a specified frequency) to allow user 10 to perform analysis or monitoring of user data 12 stored in storage system 140.

[0046] The format of query 170 can vary, but it may include references to user data 12 stored in storage system 150 and / or specific user data 12 for a specific period of time. For example, query 170 requests user data 12 from the previous seven days. In some configurations, user 10 sends user data 12 to storage system 140 in a specific format, allowing query system 150 to generate query 170 based on information about the specific format (e.g., using attributes of the format). For example, data storage system 140 receives user data 12 in a tabular format, where user data 12 populates the rows and columns of a table. Using a tabular format, user data 12 within the table can have rows and columns corresponding to a schema or heading associated with user data 12. For example, user data 12 may refer to a business transaction conducted by user 10. In this example, user data 12 may include columns for seller, buyer, transaction price, transaction quantity, and other transaction data collected by user 10 regarding their transaction. Here, each row may have a heading or schema, such as a transaction number or identifier and / or a time entry associated with the transaction. Because storage system 140 can receive user data 12 in a specific format (e.g., a transaction table format), storage system 140 is configured to store user data 12 such that query system 150 can access elements of a format (e.g., relation, header, or other schema) associated with user data 12 (e.g., providing further context or definitions to user data 12). In other words, query system 150 generates a query 170 requesting transaction prices for the previous seven days.

[0047] In response to query 170, query system 140 generates a query response 172 that satisfies or attempts to satisfy the request of query 170 (e.g., a request for specific user data 12). Generally, query response 172 includes the user data 12 requested by query system 150 in query 170. Query system 150 may return the query response 172 to the entity that initiated query 170 (e.g., user 10) or another entity or system communicating with query system 150. For example, query 170 itself or query system 150 may specify that query system 150 deliver one or more query responses 172 to a system associated with user 10, such as analytics system 160. For example, user 10 uses analytics system 160 to perform analysis on user data 12. Analytics system 160 may be configured to generate reports, warnings, graphical representations of the data (e.g., dashboard 162), or other data-based visualizations. To generate these analyses on user data 12, the analysis system 160 can work in conjunction with the query system 150 to retrieve user data 12 on the basis of the analysis (e.g., initiating or configuring a query 170 on behalf of user 10). Typically, the query system 150 is set up to generate routine queries 170 on user data 12 within the storage system 140, thereby enabling the analysis system 160 to perform its analyses (e.g., at a specific frequency). For example, the query system 150 executes a daily query 170 to pull transaction data from the most recent seven days for the analysis system 160 to analyze and / or represent. Here, in Figure 1 In this context, dashboard 162 represents an example of how analytics system 160 expresses user data 12 in query response 172 received from query system 150.

[0048] Further reference Figure 1 The data management environment 100 also includes a manager 200. The manager 200 is typically configured to optimize query-based data storage. The manager 200 performs this optimization by executing and / or coordinating operations (e.g., storage operations, query operations, and / or data analysis operations) related to systems 140, 150, and 160 for user 10. The manager 200's functionality can be centralized (e.g., residing in one of systems 140, 150, and 160) or, depending on its design, distributed among systems 140, 150, and 160. In some examples, for instance... Figure 1Manager 200 is configured to receive user data 12 from user 10 and facilitate storage operations at storage system 140. For example, manager 200 ingests user data 12 upon receipt and can convert user data 12 into a query-optimized format. Here, ingestion refers to obtaining user data 12 and / or importing user data 12 into storage system 140 (e.g., importing into data warehouse 142) to allow systems to use the ingested user data (e.g., via query system 150 and / or analysis system 160). Generally, data can be ingested in real time, where manager 200 imports data as it is emitted from a source (e.g., user 10 or user device 110 of user 10), or ingested in batches, where manager 200 imports discrete blocks of data at periodic time intervals. During ingestion, manager 200 can validate the format of user data 12 (e.g., to conform to an acceptable format of storage system 140) and route user data 12 to data warehouse 142 (e.g., a specific data storage location in data warehouse 142 specified for user data 12 of user 10). Typically, data ingestion can be challenging for streaming data, especially when data storage system 140 attempts to optimize the data received for other operations (e.g., queries). Here, the manager 200's functionality supports streaming data and, more specifically, helps optimize aspects of data management for streaming data during query-based operations. For example, manager 200 integrates stagnation methods, query caching methods, or some combination of both to reduce the overall query time of the stored user data 12.

[0049] refer to Figures 2A-2F The manager 200 includes an ingestor 210, a converter 220, and an operator 230. In a configuration where the manager 200 uses a caching method, the manager 200 also includes a cacher 240. For example, in... Figure 2A The cache 240 is shown in dashed outline to indicate that the manager 200 does not always use the caching method 240.

[0050] Intake unit 210 is configured to perform an intake process, causing it to import user data 12 into data warehouse 142. For example... Figure 2A As shown, ingestor 210 can receive streaming user data 12 and process the user data 12 from start to finish to generate ingested user data referred to as log file 212. In some examples, when ingestor 210 ingests user data 12, ingestor 210 associates ingestion timestamp 214 with user data 12. For example, ingestion timestamp 214 indicates the time when ingestor 210 ingests (e.g., imports) each specific portion of user data 12 into log file 212. In some implementations, for example... Figure 2BIngestor 210 receives user data 12 in a tabular format with rows and columns. When ingestor 210 receives the tabular user data 12, it is configured to generate log file 212 in a row-oriented format. In this row-oriented format, ingestor 210 stores a given row R of data along with all its column values ​​(e.g., in a contiguous manner, such that multiple column values ​​of that row are adjacent). This row-oriented ingestion helps preserve the schema or headings corresponding to the columns of a row in a specific relationship. For example, returning to the previously discussed transaction sales example, by ingesting a given row R of data along with all its column values, the data storage format keeps the seller, buyer, transaction price, transaction quantity, and other transaction data together for a given business transaction. By ingesting user data 12 in a row-oriented format, ingestor 210 can minimize the additional processing required to modify the ingested format during ingestion.

[0051] Reference Figure 2B The ingestor 210 ingests user data 12 into multiple log files 212, 212an. When the ingestor 210 ingests user data 12, it creates log file 212 and transmits user data 12 to line R of log file 212. 1-n For example, user data 12 corresponds to the last fifteen minutes of user data 12 from a real-time application, where the header of user data 12 corresponds to fifteen minutes ago, and the tail of user data corresponds to the current time. When ingestor 210 receives the header of user data 12, ingestor 210 creates first log files 212, 212a, and generates the first line R1 of the first log file 212a from the first line of user data 12. Here, each column C of log file 212... 1-n It can be configured to match or correspond to the header or pattern of user data 12. When ingestor 210 generates lines R of log file 212, ingestor 210 can also associate the timestamp 214,T of the current time (e.g., the ingestion time of line R) with the lines R of user data 12 generated in log file 212. The size of each log file 212 can be determined based on the preferences of storage system 140 or data warehouse 142. Once ingestor 210 generates log file 212, the number R of lines of user data 12 is equal to the size capacity configured for log file 212 (e.g., displayed as five lines R). 1-5 If the user data 12 is not ingested, the ingestor 210 creates a new log file (e.g., a second log file 212, 212b) and continues ingesting user data into the new log file 212. Utilizing the user data stream, the ingestor 210 continues the process of generating and transmitting user data 12 to log file 212 as long as the user data stream continues. In batch processing, the ingestor 210 repeats this process until the batch is complete and no more user data 21 is ingested.

[0052] exist Figure 2B In this process, the ingestor 210 generates three log files 212 and 212ac. For simplicity, if we assume user 10 is generating user data 12 at a constant rate, then the ingestor 210 generates a first log file 212 and 212a containing the last 11 to 15 minutes of user data 12, a second log file 212b containing the last 6 to 10 minutes of user data 12, and a third log file 212c containing the last 5 minutes of user data 12. Once the log file 212 generated by the ingestor 210 is full, the ingestor 210 transmits the complete log file 212 to the storage system 140 for storage in the data warehouse 142.

[0053] refer to Figure 2A After being ingested by ingestor 210, the converter 200 of manager 200 transforms the user data 12 stored in one or more log files 212 into one or more query files 222. Here, the format of query file 222 is considered a query-optimized format because read operations related to query 170 take less time to execute on query file 222 than on log file 212. Furthermore, real-time applications that generate user data streams 12 often find it convenient to generate data in row-oriented databases or tables. However, for querying and / or data analysis, column-oriented formats are generally more compatible with querying and / or data analysis than row-oriented formats. For example, column-oriented formats are more advantageous for read operations (i.e., taking less time) than row-oriented formats because rows may have longer data streams to read. Additionally, the columnar form of data inherently allows related data to be grouped together (e.g., all buyers in transaction data), which prevents read operations from reading data that is not needed to be retrieved by query 170. For example, buyer query 170 does not need to read seller data. For at least these reasons, in some examples, query file 222 has a column data format.

[0054] refer to Figure 2C and 2D This shows that converter 220 generates query files 222,222 from log file 212. 1-n In some implementations, all rows R of a single column C in log file 212 1-n The data storage block (e.g., denoted as C) is transmitted to the data storage block of the definition query file 222. 1-N (Data storage blocks). In such as Figure 2C In some configurations, converter 200 includes or transmits the ingestion timestamp 214 associated with each row R to the column storage block (e.g., denoted as T1-T). n Additionally or optionally, Figure 2DThe query file 222 (e.g., a column block) may include additional time values ​​T, which can be used during the period from query 170 to storage system 140. For example, time T 1-N This can refer to local extrema (e.g., minimum and / or maximum values) of timestamp values ​​within the query file 222. In another example, time T... 1-N This indicates the last modification time of the data entry in file 222. Here, when time T... 1-N When the column representing user data 12 is last modified, cache 230 can read that time T from query file 222 to understand whether query file 222 includes static data that has not changed over a period of time (e.g., by ingestion or transformation).

[0055] Reference Figure 2E The operator 230 of the manager 200 is configured to receive query 170 from the query system 150 and respond to query 170 with query response 172. In some examples, the operator 230 is also configured to implement a stagnation approach for query optimization. For example, the operator 230 receives a stagnation parameter 174 from user 10 or the query system 150. Here, the stagnation parameter 174 refers to the upper time boundary of query 170, where the upper time boundary restricts query response 172 to user data 12 within the data warehouse 142 that is earlier than the upper time boundary. The stagnation parameter 174 can be configured such that it can be associated with a single query 170, all queries 170 for a specific user 10, and / or a specific set of queries 170 (e.g., repeated or scheduled queries 170).

[0056] Figure 2EStagnation parameter 174 is also outlined with a dashed line to indicate that stagnation parameter 174 does not necessarily originate from user 10 or query system 150. In some configurations, operator 230 generates stagnation parameter 174. While stagnation parameter 174 can potentially indicate how much user 10 is willing to trade off between latency and stagnation, it ideally attempts to prevent query 170 from reading stored user data 12 in a non-query-optimized format (e.g., row-oriented format of log file 212). Therefore, operator 230 can determine its own stagnation parameter 174. For example, operator 230 and / or manager 200 recommend stagnation parameter 174 to user 10. Operator 230 can determine its own stagnation parameter 174 by determining the time taken from the ingestion point of user data 12 to the transformation point at transformer 220 where log file 212 is transformed into query file 222. In some examples, operator 230 utilizes timestamp 214 generated during the ingestion process to determine when log file 212 is ingested by ingestor 210. For example, the timestamps 214 of the first line R (e.g., first line R1) and the last line R (e.g., fifth line R5) of log file 212 typically estimate how long the ingestion process at ingestor 210 takes for a given log file 212. Using this time, operator 230 can determine the time taken for the transformation process of converting log file 212 into query file 222 using the time value T associated with the transformation process of converter 220. For example, the last modified time T is configured to indicate when converter 220 converts log file 212 into query file 222. Based on the times of the ingestion and transformation processes, operator 230 generates a stagnation parameter 174, which is greater than or equal to the time taken to perform the ingestion and transformation processes. For example, operator 230 assigns the total time taken to perform the ingestion and transformation to the upper time boundary of stagnation parameter 174. In this approach, stagnation parameter 174 typically ensures that user data 12 is in a query-optimized format for query 170. Additionally or optionally, operator 230 can be configured to independently track or monitor the time of ingestion and transformation, without relying on timestamps or time values ​​associated with log file 212 and / or query file 222.

[0057] When operator 230 receives or generates stagnation parameter 174, operator 230 is configured to determine whether user data 12 stored in storage system 140 (e.g., data warehouse 142) satisfies stagnation parameter 174. In some examples, stored user data 12 satisfies stagnation parameter 174 when timestamp 214 indicates that line R of user data 12 in log file 212 is earlier than the upper time boundary of stagnation parameter 174. In some implementations, such as Figure 2EAs shown, the operator 230 identifies the log file 212 storing user data 12 in the data warehouse 142 and determines the timestamp 214 within the log file 212 that matches or is closest to the stagnation parameter 174. Figure 2E In the process, operator 230 has determined that the timestamp 214,T3 associated with the third row R3 of user data 12 in log file 212n most closely matches the stagnation parameter 174. Based on this determination, operator 230 is configured to exclude rows R with timestamp 214 earlier than the timestamp 214,T3 associated with the third row R3 from the read operation in response to query 170. For example, Figure 2E This shows that operator 230 did not perform a read operation on the fourth line R4 and the fifth line R5 of log file 212. This is in contrast to the third line R3 (e.g., line R...). 1-3 For the associated timestamp 214, T3, and an earlier timestamp 214, operator 230 will perform read operations on the user data 12 within the row R associated with these earlier timestamps 214. In some configurations, operator 230 only evaluates whether the timestamp 214 of log file 212 satisfies the stagnation parameter 174, because log file 212 is in a non-query-optimized format, which will inevitably cause some delay for query 170 when querying data in a non-query-optimized format. Therefore, this approach attempts to minimize the number of read operations required on user data 12 that has not yet been converted to query file 222.

[0058] Reference Figure 2F The manager 200 can utilize caching methods for query optimization. For example, storage system 140 and systems preparing to store data in storage system 10 (e.g., ingestor 210 and / or converter 220) typically partition the data based on time units. For instance, column C associated with query file 222 includes a column timestamp T, which indicates the time unit in which column C is partitioned. These time units can be minutes, hours, days, weeks, months, etc. Figure 2F The query file 222 shows a week, with its time partitioned by day (e.g., column timestamp T). 1-7Caching 240 is configured to determine whether data stored within storage system 140 is static data. For example, caching 222 determines whether user data 12 stored in query file 222 is static data. Static data refers to data that remains unchanged (i.e., unaffected) during storage operations to storage system 140. The presence of static data is due to storage system 140, with data warehouse 142, being configured as a central data storage location for large amounts of user data 12. While storing a substantial amount of user data 12 allows user 10 to understand data trends or perform data analysis on user data 12, it also means that most of the data stored within data warehouse 142 may be rarely changed (e.g., only undergoing read operations). By identifying static or cold user data within storage system 140, caching 240 can cache query responses 172 for identified static user data over time to reduce query processing for future read operations on static user data.

[0059] To illustrate, Figure 2F The cache 230 has identified the first to fifth columns C of user data 12 in the query file 222. 1-5 This corresponds to static data. In other words, of the user data 12 stored in storage system 140, the earliest five days have remained unchanged, while the most recent two days have still undergone some storage operations. In some examples, cache 230 determines whether user data 12 in query file 222 is static data by using the last modified time associated with query file 222. Here, last modified data refers to the timestamp T associated with column C of the data (e.g., query file 222), which indicates the last time any data within column C has undergone a change or modification. In some implementations, cache 240 determines that the last modified time associated with query file 222 meets a static data threshold 246. Static data threshold 246 is a value (e.g., configured by cache 230) indicating the probability that stored data in query file 222 will remain unchanged during future storage operations of storage system 140. Figure 2F In this context, cache 230 determines that the fifth column timestamp T5 associated with the fifth column C5 of user data 12 satisfies the static data threshold 246. In some examples, when cache 230 determines that a particular timestamp T satisfies the static data threshold 246, cache 230 is configured to assume that query files 222 partitioned by time before the particular timestamp T also satisfy the static data threshold 246 (i.e., may contain static data). For example, Figure 2F The first to fifth columns C of the data determined by cache 242 are shown. 1-5 This corresponds to static data because the timestamp T5 in column C5 satisfies the static data threshold of 246.

[0060] In some configurations, cache 240 automatically caches user data 12, determining it as static user data by converting it into cached user data. In other configurations, once cache 240 determines that column C of user data 12 (e.g., query file 222) includes static user data, it caches that static user data during the next time it is provided as query response 172. By waiting until manager 200 provides static user data as query response 172, cache 240 can conserve processing resources by not caching large amounts of data that are irrelevant to the required data (i.e., the data requested by query 170).

[0061] Using a caching method, when operator 230 receives query 170, operator 230 is configured to retrieve user data 12 from storage system 140 requested by query 170. Utilizing the caching method, cache 240 is configured to receive or identify the stored user data 12 associated with query 170 (e.g., displayed as query file 222) and determine whether any portion of the stored user data 12 corresponds to user data 244 cached at cache 242 associated with cache 240. In some examples, query 170 requests stored user data for a specific time interval 176 (e.g., buyers from the past seven days). Using the specific time interval 176, cache 240 determines whether any portion of time interval 176 corresponds to cached user data 244. Here, cache 240 has previously cached five days of the past seven days (e.g., timestamp T as in cache 242). 1-5 (As shown). Because cache 240 determines that a portion of the requested data has already been cached, cache 240 provides cached user data 222 to partially (or completely) complete query 170 without requiring a read operation on storage system 140 for a portion of the time interval 176 associated with cached user data 244. For any remaining portion of the time interval 176 that is not satisfied by cached user data 244, operator 230 facilitates a read operation to retrieve the corresponding stored user data 12.

[0062] In some implementations, a portion of time interval 176 corresponds to static user data stored in storage system 140, but cache 240 has not yet converted the static user data into cached user data 244. Here, cache 240 cannot yet provide cached user data 244 to complete a portion of query 170. However, when operator 230 provides static user data in its query response 172, cache 240 recognizes that the static user data is likely the subject of future query 170 and caches the static user data. In other words, if, during or before query 170 for buyer history over the past seven days, cache 240 identifies the earliest five days of buyer history as static user data (e.g., column C),... 1-5 However, the identified static user data has not yet been cached. When the manager 200 returns query response 172 or shortly thereafter, the cache 240 will cache the earliest five days of buyer history.

[0063] Figure 3 This is a flowchart illustrating an example arrangement of operations for method 300 of data management. In operation 302, method 300 receives a data query 170 from user 10 of query system 150, containing data 12 stored in data storage 142 communicating with data query system 150. In operation 304, method 300 receives a stagnation parameter 174 indicating an upper time boundary for data query 170. The upper time boundary restricts query response 172 to data 12 within data storage 142 that is earlier than the upper time boundary. In operation 306, method 300 determines whether the data stored in data storage 142 satisfies stagnation parameter 174. When a portion of the data 12 within data storage 142 does not satisfy stagnation parameter 174, in operation 308, method 300 generates a query response 172 that excludes the portion of data 12 that does not satisfy stagnation parameter 174. For example, method 300 generates a query response 172 that excludes the portion of data 12 that was most recently written to data storage, which is later than the upper time boundary of the stagnation parameter.

[0064] Figure 4This is a flowchart illustrating another example arrangement of the operation of method 400 for data management. In operation 402, method 400 receives a query 170 requesting data 12 for a corresponding time interval 176 to be stored in storage system 140. In operation 404, method 400 determines whether any portion of the corresponding time interval 176 corresponds to static data cached from the previous query 170. Static data corresponds to stored data 12 that remains unchanged during storage operations in storage system 140. When a portion of the corresponding time interval 176 corresponds to static data cached from the previous query 170, in operation 406, method 400 generates a corresponding response 172 to query 170. Response 172 includes cached static data 244 corresponding to the portion of the corresponding time interval 176, without performing a read operation on storage system 140 for the stored data of the portion of the corresponding time interval 176.

[0065] Figure 5 This is a schematic diagram of an example computing device 500 that can be used to implement the systems (e.g., manager 200, remote system 130, storage system 140, query system 150, analysis system 160) and methods (e.g., methods 300, 400) described in this document. The computing device 500 is intended to represent various forms of digital computers, such as laptops, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. The components shown herein, their connections and relationships, and their functions are merely exemplary and are not intended to limit the implementation of the invention described and / or claimed in this document.

[0066] Computing device 500 includes a processor 510 (e.g., data processing hardware), a memory 520 (e.g., memory hardware), a storage device 530, a high-speed interface / controller 540 connected to the memory 520 and a high-speed expansion port 550, and a low-speed interface / controller 560 connected to a low-speed bus 570 and the storage device 530. Each of components 510, 520, 530, 540, 550, and 560 is interconnected using various buses and may be suitably mounted on a common motherboard or otherwise mounted. Processor 510 can process instructions executed within computing device 500, including instructions stored in memory 520 or on storage device 530, to display graphical information of a graphical user interface (GUI) on an external input / output device (e.g., a display 580 coupled to high-speed interface 540). In other embodiments, multiple processors and / or multiple buses, as well as multiple memories and memory types, may be suitably used. In addition, multiple computing devices 500 can be connected, each of which provides a portion of the necessary operation (e.g., as a server group, a set of blade servers, or a multiprocessor system).

[0067] Memory 520 stores information non-transitorily within computing device 500. Memory 520 may be a computer-readable medium, a volatile memory cell, or a non-volatile memory cell. Non-volatile memory 520 may be a physical device for temporarily or permanently storing programs (e.g., instruction sequences) or data (e.g., program state information) used by computing device 500. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electrically erasable programmable read-only memory (EEPROM) (e.g., typically used for firmware, such as boot programs). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase-change memory (PCM), and magnetic disks or magnetic tapes.

[0068] Storage device 530 provides mass storage for computing device 500. In some embodiments, storage device 530 is a computer-readable medium. In various embodiments, storage device 530 may be a floppy disk device, hard disk device, optical disk device, magnetic tape device, flash memory or other similar solid-state storage device, or device array, including devices in a storage area network or other configurations. In other embodiments, a computer program product is tangibly contained in an information carrier. The computer program product contains instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer or machine-readable medium, such as memory 520, storage device 530, or memory on processor 510.

[0069] High-speed controller 540 manages bandwidth-intensive operations of computing device 500, while low-speed controller 560 manages less bandwidth-intensive operations. This allocation of responsibilities is merely exemplary. In some embodiments, high-speed controller 540 is coupled to memory 520, display 580 (e.g., via a graphics processor or accelerator), and high-speed expansion port 550, which can accept various expansion cards (not shown). In some embodiments, low-speed controller 560 is coupled to storage device 530 and low-speed expansion port 590. Low-speed expansion port 590, which may include various communication ports (e.g., USB, Bluetooth, Ethernet, Wireless Ethernet), may be coupled to one or more input / output devices, such as keyboards, pointing devices, scanners, or networking devices, such as switches or routers, for example, via a network adapter.

[0070] As shown in the figure, the computing device 500 can be implemented in a variety of different forms. For example, it can be implemented as a standard server 500a or multiple times in a group of such servers 500a, as a laptop 500b, or as part of a rack server system 500c.

[0071] Various implementations of the systems and techniques described herein can be implemented in digital electronic and / or optical circuits, integrated circuits, specially designed ASICs (Application-Specific Integrated Circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementations in one or more computer programs executable and / or interpretable on a programmable system, the programmable system including at least one programmable processor, which may be dedicated or general-purpose, coupled to receive data and instructions from a storage system, at least one input device, and at least one output device, and to send data and instructions to the storage system, at least one input device, and at least one output device.

[0072] These computer programs (also referred to as programs, software, software applications, or code) include machine instructions for a programmable processor and can be implemented using high-level procedural and / or object-oriented programming languages ​​and / or assembly / machine languages. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, non-transitory computer-readable medium, device, and / or apparatus (e.g., disk, optical disk, memory, programmable logic device (PLD)) used to provide machine instructions as machine-readable signals. The term "machine-readable signal" refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0073] The processes and logical flows described in this specification can be executed by one or more programmable processors that execute one or more computer programs to perform functions by manipulating input data and generating output. The processes and logical flows can also be executed by special-purpose logic circuitry, such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits). For example, processors suitable for executing computer programs include general-purpose and special-purpose microprocessors, as well as any one or more processors of any type of digital computer. Typically, the processor receives instructions and data from read-only memory or random access memory, or both. The basic elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include, or be operatively coupled to, one or more mass storage devices for receiving or transferring data to one or more mass storage devices, such as magnetic disks, magneto-optical disks, or optical disks. However, a computer need not have such devices. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; CD-ROMs and DVD-ROMs. Processors and memory can be supplemented by or integrated into dedicated logic circuits.

[0074] To provide interaction with the user, one or more aspects of this disclosure can be implemented on a computer with a display device, such as a CRT (cathode ray tube), LCD (liquid crystal display), or a touchscreen for displaying information to the user, and optionally also includes a keyboard and a pointing device, such as a mouse or trackball, through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including sound, speech, or tactile input. Furthermore, the computer can interact with the user by sending documents to and receiving documents from the device used by the user; for example, by sending a webpage to a web browser on the user's client device in response to a request received from a web browser.

[0075] Various embodiments have been described. However, it should be understood that various modifications can be made without departing from the spirit and scope of this disclosure. Therefore, other embodiments are also within the scope of the appended claims.

Claims

1. A method for managing data processing, characterized in that, The method includes: Receive queries for data stored in the data storage of the storage system at corresponding time intervals; Receive a stagnation parameter, the stagnation parameter indicating the upper time boundary of the query, the upper time boundary being used to limit the query response to data in the data memory earlier than the upper time boundary; Determine whether the data stored in the data memory satisfies the stagnation parameter; When a portion of the data in the data storage does not meet the stagnation parameter, a query response is generated that excludes the portion of data that does not meet the stagnation parameter. The method further includes: Determine whether any portion of the corresponding time interval corresponds to static data cached from a previous query, the static data corresponding to stored data that remains unchanged during storage operations of the storage system; When a portion of the corresponding time interval corresponds to static data cached from the previous query, a corresponding response to the query is generated, the corresponding response including the cached static data corresponding to the portion of the corresponding time interval, without performing a read operation on the stored data of the storage system for the portion of the corresponding time interval.

2. The method as described in claim 1, characterized in that, The operation also includes: Receive initial query request data, which is stored in the storage system at initial time intervals; It is determined that a portion of the data stored in the storage system during the initial time interval includes static data; Generate a response to the initial query, the initial query including the requested data and including said portion of data corresponding to the corresponding static data of the initial time interval; and The cache is a portion of the data corresponding to the static data of the initial time interval.

3. The method as described in claim 2, characterized in that, The determination of the portion of the data stored in the storage system during the initial time interval includes static data including a timestamp identifying a column format associated with the data stored in the storage system, the timestamp indicating the last modification date of the stored data in a given column.

4. The method as described in claim 2, characterized in that, Also includes: Determining that a portion of the data stored in the storage system during the initial time interval includes static data includes determining that a timestamp associated with a given column of the stored data in the storage system satisfies a static data threshold, the timestamp indicating the last modification date of the stored data in the given column, the static data threshold indicating the likelihood that the stored data in the given column will remain unchanged during future storage operations of the storage system.

5. The method as described in claim 1, characterized in that, Generating the corresponding response to the query includes performing a read operation on the stored data in the storage system within the corresponding time interval, but not within the portion of the corresponding time interval corresponding to the static data cached from the previous query.

6. The method as described in claim 1, characterized in that, The operation also includes: When any portion of the corresponding time interval fails to correspond to static data cached from a previous query, it is determined that a portion of the data stored in the storage system during that corresponding time interval includes static data; Generate a response to the corresponding query, the corresponding query including the requested data and including a portion of the data corresponding to the corresponding static data of the corresponding time interval; and The cache is a portion of the data corresponding to the static data of the corresponding time interval.

7. The method as described in claim 1, characterized in that, The storage operation includes storing a continuous data stream of real-time data from users of the storage system.

8. The method as described in claim 1, characterized in that, The operation also includes transmitting the corresponding response to the query to a data analysis system configured to generate a graphical representation of the corresponding response.

9. A system for managing data processing, characterized in that, include: Data processing hardware; and Memory hardware that communicates with the data processing hardware, the memory hardware storing instructions that, when executed on the data processing hardware, cause the data processing hardware to perform the following operations: Receive queries for data stored in the data storage of the storage system at corresponding time intervals; Receive a stagnation parameter, the stagnation parameter indicating the upper time boundary of the query, the upper time boundary being used to limit the query response to data in the data memory earlier than the upper time boundary; Determine whether the data stored in the data memory satisfies the stagnation parameter; When a portion of the data in the data storage does not meet the stagnation parameter, a query response is generated that excludes the portion of data that does not meet the stagnation parameter. The operation also includes: Determine whether any portion of the corresponding time interval corresponds to static data cached from a previous query, the static data corresponding to stored data that remains unchanged during storage operations of the storage system; When a portion of the corresponding time interval corresponds to static data cached from the previous query, a corresponding response to the query is generated, the query including the cached static data corresponding to the portion of the corresponding time interval, without performing a read operation on the stored data of the storage system for the portion of the corresponding time interval.

10. The system as described in claim 9, characterized in that, The operation also includes: Receive initial query request data, which is stored in the storage system at initial time intervals; It is determined that a portion of the data stored in the storage system during the initial time interval includes static data; Generate a response to the initial query, the initial query including the requested data and including said portion of data corresponding to the corresponding static data of the initial time interval; and The cache is a portion of the data corresponding to the static data of the initial time interval.

11. The system as claimed in claim 10, characterized in that, The determination of the portion of the data stored in the storage system during the initial time interval includes static data including a timestamp identifying a column format associated with the data stored in the storage system, the timestamp indicating the last modification date of the stored data in a given column.

12. The system as described in claim 10, characterized in that, Determining that a portion of the data stored in the storage system during the initial time interval includes static data includes determining that a timestamp associated with a given column of the stored data in the storage system satisfies a static data threshold, the timestamp indicating the last modification date of the stored data in the given column, the static data threshold indicating the likelihood that the stored data in the given column will remain unchanged during future storage operations of the storage system.

13. The system as described in claim 9, characterized in that, Generating the corresponding response to the query includes performing a read operation on the stored data in the storage system within the corresponding time interval, but not within the portion of the corresponding time interval corresponding to the static data cached from the previous query.

14. The system as described in claim 9, characterized in that, The operation also includes: When any portion of the corresponding time interval fails to correspond to static data cached from a previous query, it is determined that a portion of the data stored in the storage system during that corresponding time interval includes static data; Generate a response to the corresponding query, the corresponding query including the requested data and including a portion of the data corresponding to the corresponding static data of the corresponding time interval; and The cache is a portion of the data corresponding to the static data of the corresponding time interval.

15. The system as described in claim 9, characterized in that, The storage operation includes storing a continuous data stream of real-time data from users of the storage system.

16. The system as described in claim 9, characterized in that, It also includes transmitting the corresponding response to the query to a data analysis system configured to generate a graphical representation of the corresponding response.

Images