Query restartability

Abstract

The present disclosure relates to query restartability. Methods, systems, and apparatus for restarting a query using a token, including a computer program encoded on a computer storage medium. One method includes receiving a query from a requesting device by a computer; determining a current result responsive to the query using a data storage system; generating a restart token using the current result, which represents an operation performed to determine a plurality of results responsive to the query including the current result, and which can be used to determine a new result responsive to the query that is not included in the plurality of results responsive to the query; and providing a message to the requesting device, the message including a) first data for a restart token, the restart token representing an operation performed to determine the plurality of results responsive to the query, and b) second data for the current result responsive to the query.

Description

[0001] Description of the case

[0002] This application is a divisional application of Chinese invention patent application No. 201710827669.1, filed on September 14, 2017. Technical Field

[0003] The present disclosure relates to query restartability. Background Art

[0004] The system can receive queries from devices, such as client devices. The system can delegate the task of performing the query, or a portion of the query, to a specific computer, such as a server included in the system. When the system allows for parallel processing, the system can provide messages to multiple computers, each message representing a task to perform a query, and receive responses from the computers. The system can combine data from the responses to determine the data to send to the device.

[0005] For example, each computer can be assigned to manage a specific portion of the database. These various portions of the database can be referred to as "tables" or "shards" and can include a portion of the data from one or more tables in the distributed database. When a device requests an operation on specific data in the distributed database (e.g., read specific data or write specific data), the shard (or in some cases multiple shards) containing the specific data is identified, and the computer to which the shard is assigned is instructed to perform the requested operation on the data in the shard.

[0006] Sometimes the system can generate multiple responses, each including a partial result in response to a query. The system can stream multiple responses to the device from which the system received the query. Summary of the Invention

[0007] In some embodiments, the system processing a query can generate a restart token for the results responsive to the query, e.g., when the results do not include all of the data in the system responsive to the query. The results can include a batch of results or a single result item, e.g., for data from individual rows in a database. The system can create a message that includes both the restart token and the results. The message can include the restart token attached to the results or can include the restart token attached to the last item in the batch of results. The system can provide the message to the client device from which the query was received.

[0008] If the connection between the system and the client device is lost, for example due to a network failure, a computer restart, a computer crash, or data being moved from a first computer in the system to a second computer, the client device provides the query and the restart token to the system so that the computer in the system can continue to process the query. Similarly, when the connection between two computers in the system is lost, the primary computer can provide the query and the restart token to the secondary computer so that the secondary computer can continue to process the query. The secondary computer can be the same computer that previously processed the query, a different computer, or both.

[0009] The restart token allows the system or an auxiliary computer to restart processing of a query and determine the data of the next result to be sent in response to the query without having to determine any intermediate results previously provided to a requesting device, such as a client device. For example, when the system receives the restart token and the query, the system can determine which results, including the most recent result sent to the requesting device and thereby creating the restart token, have been sent to the requesting device and the next result to be sent to the requesting device. When the response data is represented by a key, the restart token can include the key of the last data sent to the requesting device or the key of the next data that should be sent to the requesting device.

[0010] In some examples, the restart token can be a query parameter that changes the results received by the requesting device by ignoring the results previously received by the requesting device. For example, the requesting device can request the first five results that respond to a particular query from a computer in the system. After the requesting device receives the first five results, the requesting device can process the five results, for example, inputting data from the results into a user interface for presentation. The requesting device can then create a new request for the particular query that includes the restart token representing the first five results, so that the computer does not include the first five results as any additional results received by the requesting device in response to the new request. The parameter identifying the restart token can be included in the header of the new request, the body of the new request, or other appropriate part of the new request.

[0011] Generally, one innovative aspect of the subject matter described herein can be embodied as a method comprising the following acts: receiving, by a computer, a specific query from a requesting device; determining, using a data storage system, a current result responsive to the specific query; generating, using the current result, a restart token representing an operation performed to determine a plurality of results responsive to the specific query, including the current result, and which can be used to determine a new result responsive to the specific query that is not included in the plurality of results; and providing, to the requesting device, a message comprising a) first data representing the restart token performed to determine a plurality of results responsive to the specific query and b) second data for the current result responsive to the specific query. Other embodiments of this aspect include corresponding computer systems, devices, and computer programs recorded on one or more computer storage devices, each configured to perform the acts of the method. The one or more computer systems can be configured to perform specific operations or actions by installing software, firmware, hardware, or a combination thereof on the system, which, in operation, causes the system to perform the actions. The one or more computer programs can be configured to perform specific operations or actions by including instructions that, when executed by a data processing device, cause the device to perform the actions.

[0012] Generally, one innovative aspect of the subject matter described in this specification can be embodied as a method comprising the following acts: sending a specific query to a computer; receiving a message from the computer, the message comprising a) first data for a current result responsive to the specific query and b) second data for representing an operation performed to determine a plurality of results currently received by the system and capable of being used to determine a new result responsive to the specific query that is not included in the plurality of results responsive to the specific query. Other embodiments of this aspect include corresponding computer systems, apparatuses, and computer programs recorded on one or more computer storage devices, each configured to perform the acts of the method. The one or more computer systems can be configured to perform specific operations or actions by installing software, firmware, hardware, or a combination thereof on the system, which, in operation, causes the system to perform the actions. The one or more computer programs can be configured to perform specific operations or actions by including instructions that, when executed by a data processing apparatus, cause the apparatus to perform the actions.

[0013] Each of the foregoing and other embodiments can optionally include one or more of the following features, alone or in combination. Generating a restart token using a current result responsive to a particular query, which represents an operation performed to determine a plurality of results responsive to the particular query including the current result responsive to the particular query, can include generating a restart token using the current result responsive to the particular query, the restart token representing an operation performed to determine a plurality of results responsive to the particular query, the plurality of results including i) the current result responsive to the particular query and ii) one or more prior results responsive to the particular query previously provided to the requesting device. Determining the current result responsive to the particular query using a data storage system can include determining one or more current rows responsive to the particular query using the data storage system. Generating the restart token can include: determining that the one or more current rows responsive to the query stored in the data storage system are sorted by a key; and determining an index of the one or more current rows responsive to the particular query in response to determining that the one or more current rows responsive to the query stored in the data storage system are sorted by a key. Providing a message to a requesting device, the message including a) first data representing a restart token for an operation performed to determine a plurality of results responsive to a particular query and b) second data for a current result responsive to the particular query, may include providing a message to the requesting device, the message including a) first data for an index of the one or more current rows responsive to the particular query and b) second data identifying content of the one or more current rows responsive to the particular query. Determining the index of the one or more current rows responsive to the particular query may include determining an index of a last row of the one or more current rows responsive to the particular query.

[0014] In some embodiments, determining current results responsive to a particular query using a data storage system may include determining one or more current rows responsive to the particular query using the data storage system. Generating a restart token may include determining that one or more rows responsive to the query stored in the data storage system are not sorted by a particular key; and identifying a plurality of deterministic decisions made to determine the one or more current rows responsive to the particular query in response to determining that the one or more rows responsive to the query stored in the data storage system are not sorted by the particular key. The plurality of deterministic decisions may allow the system to determine the plurality of results responsive to the query including the current result responsive to the particular query. Providing a message to a requesting device, the message including a) first data representing the restart token for an operation performed to determine the plurality of results responsive to the particular query and b) second data for the current result responsive to the particular query, may include providing a message to the requesting device including a) first data identifying each of a plurality of deterministic decisions made to determine the one or more current rows responsive to the particular query and b) second data identifying the content of the one or more current rows responsive to the particular query. The method may include: determining, by the computer, that two or more specific operations for generating a result for the specific query can be performed in parallel; generating corresponding subqueries for at least two of the two or more specific operations; and assigning a corresponding subquery to an iterator for each subquery, the iterator to perform the corresponding specific operation so that two or more of the iterators process the corresponding subqueries in parallel. Determining that the one or more rows responsive to the query stored in the data storage system are not sorted by a specific key may include determining that the two or more iterators are processing the corresponding subqueries in parallel.

[0015] In some implementations, generating the restart token may include generating the restart token without storing the restart token in persistent storage. Generating the restart token may include storing the restart token in volatile memory without storing the restart token in non-volatile memory. The method may include: receiving the specific query and the restart token from the requesting device after providing the message to the requesting device, the restart token representing the operation performed to determine multiple results including the current result that are responsive to the specific query, and which can be used to determine a new result responsive to the specific query that is not included in the multiple results responsive to the specific query; using a second data storage system and the restart token to determine a second result responsive to the specific query that is not included in the multiple results; determining a second restart token, which represents the operation performed to determine the second result responsive to the specific query and the multiple results responsive to the specific query; and providing a second message to the requesting device, the second message including a) third data of the second restart token for representing the operation performed to determine the second result responsive to the specific query and the multiple results responsive to the specific query and b) fourth data for the second result responsive to the specific query.

[0016] In some embodiments, receiving a message from a computer that includes a) first data for a current result responsive to a particular query and b) second data representing a restart token for an operation performed to determine a plurality of results currently received by a system, including the current result, may include receiving a message from the computer that includes a) first data for a current result responsive to a particular query and b) second data identifying a particular state of the computer to determine the plurality of results currently received by the system. The second data may include an index identifying the plurality of results currently received by the system. The second data may include second data identifying one or more deterministic steps performed by the computer to determine the plurality of results currently received by the system.

[0017] In some embodiments, sending a specific query to a computer may include sending the specific query to the computer across a connection. The method may include, after receiving a message for each of the plurality of results responsive to the specific query, determining a most recent restart token received from the computer, the restart token being usable to determine a new result responsive to the specific query that is not included in the plurality of results responsive to the specific query; providing the specific query and the most recent restart token to a second computer; and receiving a message from the second computer, the message including third data for a second result responsive to the specific query that is not included in the plurality of results responsive to the specific query and fourth data for a second restart token representing an operation performed to determine the second result responsive to the specific query and the plurality of results. The method may include, after receiving a message for each of the plurality of results responsive to the specific query, determining that a connection to the computer no longer exists. Determining the most recent restart token received from the computer may be responsive to determining that a connection to the computer no longer exists. The method may include, after receiving a message for each of the plurality of results responsive to the specific query, determining to request additional results for the specific query from the computer. Determining the most recent restart token received from the computer may be responsive to determining to request additional results for the particular query.

[0018] In some embodiments, the method may include: receiving the specific query from a requesting device before sending the specific query to the computer. Sending the specific query to the computer may include: in response to receiving the specific query from the requesting device, sending a first sub-query representing a first portion of an operation for the specific query to the computer; sending a second sub-query representing a second portion of the operation for the specific query to a second computer; for each second current result in a second plurality of results responsive to the specific query: receiving a second message from the second computer, the second message including third data for the second current result responsive to the specific query and fourth data for representing a second restart token for an operation performed using the second sub-query to determine the second plurality of results currently received by the system including the second current result; determining a most recent current result and a most recent second current result; and sending a response to the requesting device, the response identifying i) the most recent current result and the corresponding restart token and ii) the most recent second current result and the corresponding second restart token. The method may include: determining to assign the second sub-query representing the second portion of the operation for the particular query to the second computer; and before sending the second sub-query representing the second portion of the operation for the particular query to the second computer: determining that a second current result has not been received from the second computer; determining that a current result and a corresponding restart token have been received from the computer; and sending a first response identifying the current result and the corresponding restart token to the requesting device. Sending the second sub-query representing the second portion of the operation for the particular query to the second computer may occur after sending the first response identifying the current result and the corresponding restart token to the requesting device. In some embodiments, the system may include a computer. The system may include a second computer.

[0019] The subject matter described herein can be implemented in certain embodiments and can result in one or more of the following advantages. In some embodiments, a system can include state information for a query in a restart token, which can be used to reduce or eliminate the storage of state information in the log, to keep the restart state smaller, or both. In some embodiments, a system using restart tokens can be scalable, for example, not requiring storage space to store restart information. For example, a system can use restart tokens without storing restart information in permanent storage. In some embodiments, a system using restart tokens can minimize the amount of redo work after a connection is lost, reduce or eliminate the delivery of duplicate information to a requesting system, or both. In some embodiments, the systems and methods described below reduce query restart latency, enable query restarts for systems with low delivery latency, allow queries to run over longer periods of time, allow a requesting system to restart a query without paging the system and receiving duplicate results, reduce the number of transactions aborted due to transient errors, or a combination of two or more of the above. For example, query restart tokens, such as low-latency query restart tokens, can help minimize tail delays caused, for example, by the need to return failed queries, can simplify the programming model by removing the need for retry loops in code, such as client code, or both. The simplified programming model can be used for read-only transactions that do not perform locking, such as transactions using snapshot isolation. In some examples, query restart tokens can enable processing of queries on the system when the probability of failure without restarting is high, such as when the probability of failure exceeds a threshold probability, approaches one hundred percent, or both. In some embodiments, a system or method using query restart tokens can be able to process a single long-running query, for example, without requiring multiple queries that each use a large number of results to respond to the single query, without analyzing the consistency of data in response to each of the multiple queries, or both.

[0020] The details of one or more implementations of the subject matter described in this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 is an example of an environment where a computer includes a restart token with a batch of query results.

[0022] Figure 2 is a flow diagram of a process for generating a message including a result batch in response to a query and a restart token.

[0023] Figure 3is a block diagram of a computing system that can be used in conjunction with the computer-implemented methods described herein.

[0024] Like reference numbers and designations in the various drawings indicate similar elements. DETAILED DESCRIPTION

[0025] 1.0 Overview

[0026] In some embodiments, the systems and methods described below store data representing the progress of work on a query in a restart token and stream a query restart token with each result for the query. The result is one of a plurality of results, each of which responds to the query and is included in a separate message to the requesting device. The result can be included in a batch of partial results, each of which responds to the query, such as in a batch of two or more results, or as a single result that is part of a result stream provided to the requesting device. The examples described below generally refer to a batch of results. The same system and method can be used for a single result as a partial result for a query. The system can include a restart token for each message that includes at least one result, such as a single result or a batch of results.

[0027] The system can generate a restart state for each batch of results and provide the restart state as a restart token in messages with data representing the batch of results. When the system experiences a failure and loses state information for a query, the system uses the restart state to continue streaming results after the failure, such as the most recent batch of results received from the requesting device.

[0028] For example, the system can add a restart token to each batch of results sent to a requesting device. A batch of results can be used for a partial query when the search operation for a query is divided into parts, and each part of the search operation operates as an independent sub-query. The system can divide the search operation into parts based on the shards that include the data responsive to the query, such that each sub-query is used for a specific shard. The system provides a corresponding restart token for each batch of results, the corresponding restart token having data specific to the corresponding batch of results to the requesting device from which the system received the query, such as whether the query was the original query or a sub-query.

[0029] When the system is no longer receiving data in response to a query and requires additional response data, the system uses a query restart token from a batch of results to continue working on the query. The system may no longer be receiving data in response to a query because the system is requesting only a specific number of results or because the system experiences a connection interruption, such as between the system and a requesting device or between multiple computers included in the system. For example, when a connection between a first computer in the system and a requesting device is interrupted, a second computer in the system receives a request for a query from the requesting device, the request including a restart token from a batch of results that the system previously provided to the requesting device (e.g., a most recent batch of results) to enable the system to continue working on the query. When a connection is interrupted between two computers included in the system, such as when the first of the two computers is a primary computer that delegates work to a secondary computer, which retrieves partial results for the query, the primary computer may use the restart token and the batch of results received from the secondary computer to enable the secondary computer or another computer in the system to continue generating results for the query. The interruption in connection may be caused by a network failure, a computer restart, a computer crash, the movement of data from one computer to another, or a combination of two or more of these.

[0030] In some examples, a computer processing a query may stop processing the query because the data responsive to the query has been moved to another computer, the data has become contaminated, or the computer is overloaded. A computer may be able to stop processing a query when the data responsive to the query is replicated across multiple computers. For example, a computer may initially host a shard copy of the data responsive to the query. When the computer determines that the shard copy has been moved to a different computer and that the computer can no longer access the shard copy, the computer stops processing the query. The computer may provide a message to the requesting device indicating that the computer is no longer processing the query. In some embodiments, the computer does not provide a message to the requesting device indicating that the computer is no longer processing the query.

[0031] When a system, e.g., a single computer or multiple computers, determines a batch of results in response to a query, the system can generate a restart token by capturing the state of runtime structures, e.g., stateful runtime structures, that executed the query after the system generated the last row in the batch of results. The system's runtime structures, e.g., runtime state, can include states of multiple iterators organized as an iterator tree, e.g., of arbitrary size or depth, or both.

[0032] The iterator tree can represent a runtime query plan for processing a query. Each node in the iterator tree can represent a single iterator that processes a portion of an operation for the query (e.g., a subquery of the query). The computer can execute the portion of the operation for the query represented by the iterator in the iterator tree. The iterator can have zero, one, or multiple input values. Each input value can be a tuple. The iterator can have an output, such as an output tuple. The output tuple can represent the result of the query, for example, as part of a batch of results or a single result.

[0033] In some examples, the system can convert a query request into an iterator tree during query compilation and use the state of the iterator tree to generate restart tokens. The shape of the iterator tree and the specific data being processed can control the order in which the system provides control to the iterators in the iterator tree.

[0034] The system can use the iterator tree to generate restart tokens. For example, the system can use the state of all iterators that participated in producing the last row in a particular result batch to generate a restart token for a particular result batch. The system can use a depth-first traversal of the iterator tree to determine the data for the restart token. For example, the system can determine the first iterator node in each branch that includes state information and use the determined state information to generate the restart token.

[0035] In some embodiments, the system can use the order in which the iterators perform operations to generate restart tokens. For example, for a particular batch of results, the system can cause the iterators to generate state information in the same order that the iterators generated data for that particular batch of results. When generating restart tokens, the system combines the state information in the order in which the iterators generated the state information, e.g., so that the restart token is later used to reproduce the steps performed to generate the particular batch of results in the same order.

[0036] When the system uses a restart token to restart work on a query, for example, the same computer that originally generated the results for the query or a different computer in the system, the system generates an iterator tree and provides the iterators in the iterator tree with their restart state from the restart token. The system can provide the iterators with their restart state only when the iterators can control the generation of data. For example, the system provides a batch of results in response to a particular query and a particular restart token to the requesting device. The system stops processing the query, for example, the system experiences a driver failure and crashes or otherwise stops processing the query. For example, a different computer or the same computer system receives the particular query and the particular restart token from the requesting device and generates an iterator tree for the particular query. The system uses the data from the particular restart token to update the state information of the iterators in the iterator tree.

[0037] When the current iterator uses data from its restart state, the system can cause the iterator to set the same values as before the restart, for example, when creating rows used to generate additional result batches. For example, the restart token can cause the current iterator to see the same input values that the current iterator saw before the restart.

[0038] In some examples, the system can receive from each current iterator for the last row in a particular result batch a variable describing the state of the corresponding iterator.The variable may not include data from the result batch.

[0039] The state of each current iterator can be less than a few bytes. For example, the state of all current iterators can be tens of bytes, so as to allow the system to create a restart token of tens of bytes, for example less than one hundred bytes.

[0040] In some embodiments, the data accessed by the system is deterministic data. As used in this description, deterministic data is data determined by the system in a specific order given a specific query, so that the system will determine the same deterministic data in the specific order at any time the system receives the specific query. For example, given a specific query, the system determines that the deterministic data that responds to the specific query is sorted by a specific key. In some examples, the data can be sorted by row number, data identifier, or another appropriate value. In these embodiments, the restart token can include a specific key for the last data included in the result batch. The restart token can include a specific key for the next data included in the new result batch.

[0041] In some embodiments, the system can determine deterministic data by always performing the same sequence of operations to identify data responsive to a particular query, e.g., when no other operations are available to determine the deterministic data. In some embodiments, the system can determine deterministic data by performing the sequence of operations in an order that determines the deterministic data.

[0042] In some examples, the system can determine deterministic data responsive to a particular query, e.g., based on new data added to the database, that includes more data responsive to the particular query than previously identified. The previously identified subset of deterministic data will be in the same order as previously identified and will be identified by the system before a different subset of previously unidentified deterministic data, e.g., the new data will have an identifier that is sequentially subsequent to the identifier for the old data.

[0043] In some embodiments, the data accessed by the system is non-deterministic data. As used in this description, non-deterministic data is data that a system can determine in one of a plurality of different orders for a given particular query, the plurality of different orders depending on the order of operations (e.g., non-deterministic steps) performed by the system when determining the non-deterministic data. In some examples, the system can determine the non-deterministic data when one or more operations are performed in parallel to determine the non-deterministic data. For example, the system can generate non-deterministic data when the system divides the search operation for a particular query into several parts (e.g., subqueries), assigns each subquery to a particular computer in a computer group, and causes multiple computers from the computer group to perform operations for their corresponding subqueries.

[0044] When the system performs non-deterministic steps to determine the result in response to a query, such as when the results are not sorted by a particular key, the system can include historical data in the restart token. The historical data can include data indicating the deterministic steps taken to identify the result in response to the query. The system can use the historical data upon restart to determine the steps taken to identify data previously provided to the requesting device, for example, to ensure that only data that was not previously sent to the requesting device is provided to the requesting device upon restart.

[0045] 2.0 Sample Operating Environment

[0046] Figure 1 is an example of an environment 100 where the query system 103 includes a restart token with a query result batch. For example, a client device 102 in the environment 100, such as a requesting device, may A During the period T, the client device 102 sends a query to the computer 104 in the query system 103 to request data. In response, the client device 102 E During this time, a response including partial results responsive to the query and a restart token is received from computer 104. Client device 102 can later use the restart token to request additional results responsive to the query, for example, if client device 102 becomes disconnected from computer 104 or otherwise does not receive additional results from computer 104.

[0047] For example, the computer 104 may A When the computer determines a batch of results responsive to the query without receiving data from other computers, the computer determines a first batch of results and a corresponding first restart token for the first batch of results. The first restart token can represent a state of the computer 104 used to determine the first batch of results, such as a state of one or more iterators on the computer 104.

[0048] In time period T E During this time, computer 104 provides the first batch of results and the first restart token to client device 102. Computer 104 may include the first batch of results along with the first restart token in a message. Computer 104 may send multiple data packets to client device 102, the multiple data packets representing the message with the first batch of results and the first restart token. Computer 104 may send a single data packet to client device 102, the single data packet representing the message with the first batch of results and the first restart token.

[0049] Computer 104 may include a database, such as a data storage system or a portion of a database such as a shard, with multiple results responsive to a query. Computer 104 may implement the database using any suitable technology. Computer 104 queries the database to determine a batch of results responsive to a query received from client device 102. For example, computer 104 may use a first query to determine a first batch of results, a second query to determine a second batch of results, and so on. Computer 104 may generate the first and second queries as subqueries of the query received from client device 102.

[0050] Computer 104 can provide multiple batches of results to client device 102, allowing client device 102 to begin processing the data included in each batch of results as the results are determined. For example, computer 104 can be part of a search system that provides search results in response to a query. Computer 104 can provide batches of results specific to a particular query to client device 102, allowing client device 102 to present some of the search results in a search results user interface, for example, after potentially enriching the search results with data obtained from another system. As client device 102 receives batches of results from computer 104, client device 102 can construct a first search results page. When client device 102 receives another batch of results, for example, without sending multiple requests to computer 104, client device 102 can add content to the first search results page, cache results from the other batch for use in generating a second search results page, or both. For example, because the results are responsive to the same query, client device 102 can cache results for a second search results page without having to determine how to make the data in the second search results page consistent with the first search results page.

[0051] When computer 104 is a master computer that delegates subqueries to multiple auxiliary computers 106a-b, computer 104, for example, as a requesting device, may BDuring this process, a subquery is assigned to each of the auxiliary computers 106a-b. For example, computer 104 may determine a plurality of data shards that include data responsive to the query. Computer 104 may determine that each of the auxiliary computers 106a-b stores one of the determined shards. Computer 104 determines a subquery for each of the corresponding shards and sends the subquery, for example, in a message, to the corresponding auxiliary computers 106a-b.

[0052] Each auxiliary computer 106a-b performs one or more operations while processing a corresponding subquery to determine a batch of data responsive to the query. Each auxiliary computer 106a-b determines a corresponding state of the corresponding auxiliary computer used to generate the data in the batch of data and includes the data for the state in a corresponding restart token. For example, the first auxiliary computer 106a may use an iterator to determine a first batch of data responsive to the query. The first auxiliary computer 106a may use the state of the iterator as a restart token or as part of a restart token. The first auxiliary computer 106a may use the state information of any suitable number of iterators to determine the restart token.

[0053] In time period T C During this time, computer 104 receives the results of the batch and the corresponding restart tokens from auxiliary computers 106a-b. For example, computer 104 may receive a message sent in one or more packets from each of auxiliary computers 106a-b, the message including data for the results of the batch and data for the corresponding restart token for a particular auxiliary computer 106a-b.

[0054] Computer 104, during time period T D During the process, the results and restart tokens of the batch are combined. For example, the computer 104 can concatenate the data representing the results of each batch from the auxiliary computers 106a-b. The computer 104 can store a global iterator tree in memory that represents the restart tokens for the auxiliary computers 106a-b. Each local iterator node in the global tree can store the restart token for one of the auxiliary computers 106a-b. The computer 104 can use the global tree to generate a global restart token for the results of the batch currently stored on the computer 104 and not yet sent to the client device 102 (e.g., the combined corresponding batch results received from the auxiliary computers 106a-b).

[0055] Computer 104 in time period T EDuring this time, the results of the combined batches and the combined restart token are provided to the client device 102. For example, the computer 104 sends a message to the client device 102 that includes data representing the results of the connected batches and data for the global restart token.

[0056] In some examples, computer 104 may generate a global restart token that includes state information for one or more batches of results previously sent to client device 102 and to do so for a period of time T E During this time, data will not be sent to client device 102. For example, computer 104 can assign a subquery to three auxiliary computers (not shown). Computer 104 can receive a corresponding first result batch and a first restart token from each of the three auxiliary computers. During, for example, time period T0, computer 104 provides client device 102 with a first global result batch representing the first result batch from the three auxiliary computers and a first global restart token representing the first restart token. During, for example, time period T1 after time period T0, computer 104 receives a second result batch and corresponding restart tokens from the first and second auxiliary computers. The computer does not receive the second result batch from the third auxiliary computer, for example, because the third auxiliary computer is restarted and no longer works on the subquery previously assigned to the third auxiliary computer.

[0057] The computer 104 determines that the third auxiliary computer will not provide a second batch of results and generates a second global batch of results using the two second batches of results from the first and second auxiliary computers. The computer 104 generates a second global restart token using the second restart tokens received from the first and second auxiliary computers and the first restart token received from the third auxiliary computer. The computer 104 generates a second global restart token, for example, at a time period T after time period T1. E During this time, the second global result batch and the second global restart token are provided, for example, in a message, to the client device 102. The computer 104 may determine that the connection with the third auxiliary computer is lost, for example, because the third auxiliary computer crashed or was rebooted. The computer 104 may provide the third subquery previously assigned to the third auxiliary computer and the first restart token received from the third auxiliary computer to the third auxiliary computer or another computer.

[0058] The computer 104 continues to provide result batches and corresponding restart tokens to the client device 102 until it is determined that no additional data is responsive to the query. The computer 104 may include data in the final result batch sent to the client device 102 indicating that the final result batch is not included with the restart token, e.g., data indicating that no additional data will be received by the client device 102 for the query.

[0059] In some examples, computer 104 may provide result batches and corresponding restart tokens to client device 102 until computer 104 receives a stop request from client device 102 indicating that computer 104 should no longer provide result batches to client device 102. In some implementations, computer 104 may provide result batches and corresponding restart tokens to client device 102 until a predetermined period of time after receiving the query, until a predetermined number of result batches have been provided to client device 102, or until another suitable stop condition is met.

[0060] When the client device 102 is no longer receiving result batches from the computer 104 , e.g., based on a stop condition or when the connection between the client device 102 and the computer 104 becomes disconnected, the client device 102 may use the most recently received restart token and the original query to request additional result batches from the computer 104 , e.g., so that the client device 102 does not receive from the computer 104 any result batches that the client device 102 previously received.

[0061] Time period T A ,T B ,T C ,T D and T E Can be any suitable length. In some examples, the time period T A ,T B ,T C ,T D and T E One or more of may overlap. For example, computer 104 may D and T E During this period, the restart tokens are combined and a global result batch and a corresponding global restart token are provided. At the same time, for example, in the time period T C During this time, result batches and corresponding restart tokens continue to be received from the auxiliary computers 106a-b.

[0062] In some embodiments, a particular global results batch may include multiple results batches from a single secondary computer 106a-b. For example, computer 104 may create a global results batch having two results batches from a first secondary computer 106a and one results batch from a second secondary computer 106b.

[0063] Client device 102 can be any suitable device. For example, client device 102 can include a personal computer, a mobile communication device, or another device that can send and receive data over network 108. Network 108, such as a local area network (LAN), a wide area network (WAN), the Internet, or a combination thereof, connects client device 102 to computers in query system 103. Network 108, or another network, can connect computer 104 to auxiliary computers 106a-b in query system 103. Although a single computer 104 is described, some embodiments of environment 100 may use a single computer 104 or multiple computers 104 operating in conjunction with each other, including, for example, a collection of remote computers deployed as a cloud computing service.

[0064] Each of computer 104 and auxiliary computers 106a-b can be a physical computer or a virtual computer executing on a physical computer. For example, computer 104 can be a first virtual machine executing on a physical computer, first auxiliary computer 106a can be a second virtual machine executing on a physical computer, and second auxiliary computer 106b can be a third virtual machine executing on a physical computer. One or more virtual machines can execute on the same physical computer. Computers 104 and 106a-b can be any suitable combination of physical computers and software executing on the physical computers.

[0065] In some implementations, one or more of computers 104 and 106a-b may be containers executing on an operating system. For example, first auxiliary computer 106a may be a first container accessing a first shard of data stored in memory on underlying physical hardware, and second auxiliary computer 106b may be a second container accessing a second shard of data stored in memory on underlying physical hardware. The first container and the second container may execute on the same operating system instance, on the same physical computer, or both. In some examples, the first container and the second container may execute on different operating system instances, for example, of the same operating system type or different operating system types, on different physical computers, or both.

[0066] 3.0 Example Processing Flow

[0067] Figure 2 is a flow diagram of a process 200 for generating a message including a batch of results responsive to a query and a restart token. For example, process 200 may be used by computer 104 from environment 100 or one of auxiliary computers 106a-b.

[0068] The computer receives a specific query (202). For example, the computer may receive the specific query from a requesting device. When the computer is a secondary computer, a client device, or another suitable device, the requesting device may be a primary computer. The specific query may be, for example, an initial query received from a client device, or a subquery received from another computer.

[0069] The computer determines, using a data storage system, a current batch of results responsive to a particular query (204). The computer may use data from a shard included on the computer (e.g., a shard owned by the computer) to determine the results responsive to the particular query. For example, the computer may include a database to store data, including data responsive to the particular query. The computer may access the database to determine the current batch of results.

[0070] The computer determines whether the current batch of results responsive to the query includes deterministic data (206). The computer can determine whether the particular key is used to sort the data, whether multiple computers are simultaneously processing subqueries identifying search operators for the query, such as when each subquery is used for sharding, whether multiple non-deterministic steps are to be performed to sort the data, or a combination of two or more of these. When the computer determines that the particular key is used to sort the data, the computer can determine that the current batch of results includes deterministic data. When the computer determines that the particular key is not used to sort the data, the computer can determine that the current batch of results does not include deterministic data.

[0071] In response to determining that the current batch of results responsive to the query does not include deterministic data, the computer generates a restart token representing an operation performed to determine multiple batches of results responsive to the specific query including the current batch of results (208). For example, the computer determines state information representing steps performed by the computer to generate the current batch of results. The state information allows the computer or another computer to repeat a process, such as a sequence of operations performed, to generate the current batch of results, such as so that no results from the current batch of results are provided to the requesting device in another batch of results. For example, if the computer does not receive the state information, the computer may provide results from the current batch of results to the requesting device, such as the requesting device may receive some results multiple times. Assuming that the data is non-deterministic and can be determined in different orders depending on the sequence of operations performed to determine the data, the computer can use the state information to ensure that the sequence of operations is performed in the same order. In some examples, the computer can also use the current batch of results to generate the restart token, for example, in addition to the state information.

[0072] The computer may not include the result in the state information, e.g., to ensure that the size of the restart token is less than a maximum threshold size. The computer may not store the state information, e.g., in persistent storage, to reduce memory requirements, e.g., when the computer would have to store state information for multiple queries, so that the requesting device can, e.g., use the restart token to restart the query with another computer, or both.

[0073] In response to determining that the current batch of results responsive to the query includes deterministic data, the computer generates a restart token (210) representing a state of the current batch of results. The state can be the state of the computer after determining the current batch of results. The computer or another computer can use the state to determine which results have been provided to the requesting device, which results should be sent next to the requesting device, or both.

[0074] The computer provides a message (212) including first data for a restart token and second data for results of a current batch. The computer may generate a message including first data for a restart token and second data for results of a current batch. The computer uses a network to provide the message to a requesting device from which the computer received the query. The computer may provide the message using any suitable method, such as as one or more packets.

[0075] The computer receives a specific query and a restart token (214) representing an operation performed to determine the plurality of batches of results. For example, when the computer is disconnected from the requesting device, the requesting device may determine that the connection is lost and that no additional batches of results have been received from the computer. In response, the computer receives the specific query and the restart token from the requesting device.

[0076] The computer uses the special query and the restart token to determine the results of the plurality of batches (216). For example, whether the result batches include deterministic data or non-deterministic data, the computer uses the special query and the restart token to recreate the state of the computer when the computer provided the last batch of results to the requesting device, such as to cause the computer to not provide any results that the requesting device has already received to the requesting device.

[0077] The computer uses the data storage system to determine a new batch of results responsive to the specific query that does not include any results from the multiple batches of results (218). For example, the computer accesses the database to determine the new batch of results. The results in the new batch of results do not include any results previously received by the requesting device for the specific query.

[0078] In some examples, the database may contain different data than previously stored on the computer. For example, when the computer includes a data shard and the shard is later split, such as during a period when the computer is disconnected from the requesting device, the computer determines the data included in the remaining portion of the shard that responded to the particular query. The computer sends the determined data as a batch of results, along with a corresponding restart token, to the requesting device.

[0079] In some embodiments, when a shard split occurs, the computer can use a specific query and a restart token to determine that a response to the specific query has been made and that all data stored on the computer has been sent to the requesting device. The computer can generate a message indicating that all response data has been provided to the requesting device. The message can include an indication of the data stored on the computer. The message can also include an indication that a shard split has occurred.

[0080] The computer determines a new restart token that represents the operations performed to determine the new batch of results and the multiple batches of results (220). For example, the computer can determine whether the data is deterministically determined, such as by performing step 206. In response, the computer includes appropriate data in the new restart token. The computer can use the new batch of results to determine the new restart token.

[0081] The computer provides a second message including third data for the new restart token and fourth data for the new batch of results (222). The computer may generate the second message using the new restart token and the new batch of results. The computer provides the second message to the requesting device.

[0082] The order of the steps in process 200 described above is illustrative only, and generating a message including a batch of results responding to a query and a restart token can be performed in a different order. For example, the computer can determine whether the result is determined deterministically or non-deterministically, and then determine the result for the current batch. In some examples, the computer can determine whether the result is determined deterministically or non-deterministically and determine the result for the current batch simultaneously.

[0083] In some implementations, process 200 may include additional steps, fewer steps, or some steps may be divided into multiple steps. For example, a computer may perform steps 202 through 212 without performing one or more of steps 214 through 222. In some examples, a computer may perform steps 202 through 204 and steps 210 through 212.

[0084] 4.0 Optional Implementation Details

[0085] In some embodiments, the system can generate a restart token that does not have a significant impact on network costs, processor costs, or both. The system can generate a restart token of any appropriate size. The system can generate a restart token of a size of several thousand bytes (e.g., less than ten thousand bytes), preferably a few hundred bytes (e.g., less than a thousand bytes), and more preferably a few tens of bytes (e.g., less than a hundred bytes). For example, when the system receives less than a few bytes of state information from each current iterator, the total size of the restart token can be less than three hundred bytes, preferably less than one megabyte.

[0086] The system can control the size of the restart token. For example, the size of the restart token can be based on the query complexity, the degree of parallelism used to process the query (for example, each subquery running in parallel at this time may require an independent restart state), or both. The system can adjust the query complexity, the degree of parallelism, or both to adjust the size of the restart token, for example, to keep the size of the restart token below a threshold size. For example, the system can use the size of the iterator tree representing the runtime query plan, for example, the shape of the iterator tree representing the degree of parallelism used to process the query, or both to determine the size of the restart token. The system can adjust the size of the iterator tree, the shape of the iterator tree, or both to adjust the size of the corresponding restart token. In some examples, the system may include data for the runtime query plan represented by the iterator tree as part of the restart token, for example, when the response data is determined non-deterministically. In some examples, the system does not include data for the runtime query plan represented by the iterator tree as part of the restart token, for example, when the response data is determined deterministically.

[0087] In some embodiments, the system can allow for parallel processing when the order in which results are generated is unimportant. For example, when a requesting device providing a query to the system indicates that the order in which results are provided to the requesting device is unimportant, or the system otherwise determines that the order is unimportant, the system can determine to generate subqueries for different operations used to generate query results. The system can assign subqueries for different operations to different iterators. Each iterator generates results for the corresponding subquery in parallel.

[0088] When multiple computers perform operations to determine a batch of results in response to a query, a master computer, such as computer 104, can generate a global iterator tree that includes state information from each of the multiple computers, such as a runtime query plan. For example, the master computer can assign a portion of a distributed query to each of the multiple computers that are nodes of the global iterator tree, and the computer will perform work for that portion of the distributed query, such as the node processing data from one shard of the input data. The master computer can use a specific local iterator to remotely call a subquery of the query on a computer, such as one of the nodes. The master computer can receive the subquery result including the corresponding restart token from one of the computers and provide the result to the local iterator. The local iterator can be part of a global iterator tree stored on the master computer, in which each local iterator corresponds to one of the computers that is assigned a portion of the distributed query. The master computer uses the local iterator to update the global tree using the results and the included restart token.

[0089] The host computer can use the global iterator tree to generate a global restart token. For example, the host computer receives multiple sub-query result batches, each sub-query result batch comes from one of the computers that is processing the sub-query and represents one of the iterator nodes in the global iterator tree. Each sub-query result batch includes a corresponding restart token. The host computer can provide the restart token to the corresponding local iterator in the global iterator tree so that the local iterator updates the global iterator tree. The host computer uses the sub-query results that have not yet been provided to the requesting device to generate the current result batch, and the host computer receives the original query from the requesting device. The host computer uses the local iterator for the sub-query results included in the current result batch to generate the current restart token. The host computer sends the current result batch and the current restart token to the requesting device.

[0090] In some examples, the master computer can use the number of shards of the input data to dynamically adjust the number of local iterators included in the global iterator tree, and the corresponding number of auxiliary computers performing operations for the query. For example, the master computer can use the number of computers that are performing work for a distributed query to adjust the number of local iterators, e.g., where each computer includes one shard of the input data. When a shard storing data for the query changes, e.g., is merged or split, the master computer updates the local iterator to represent the change in shard. The master computer can generate restart tokens that include state information for each shard, e.g., in a restart token for a particular result batch, the relevant portion of the set of shards can be round-tripped so that upon restart, the iterator receives exactly the same data that the iterator saw before the restart.

[0091] During query processing, when the system dynamically changes how data partitions are grouped into shards across multiple computers, the system can use multiple stages to generate restart tokens. For example, the system can fix the set of shards when a computer takes control of processing a query. When the system determines that a shard has been split after the system assigned a subquery to a particular computer that previously hosted the shard, the system ensures that the particular computer processes data in the remaining portion of the shard so that the particular computer generates a restart state specific only to the remaining portion of the shard, rather than data previously included in the shard before the split.

[0092] For example, the primary computer can begin by ignoring dynamic sharding and a fixed set of shards when the primary computer gains control of the query. The primary computer can set up the sharding by creating a global iterator tree for the sharding, e.g., where an iterator node in the global iterator tree corresponds to each shard in the sharding. For each iterator node in the global iterator tree, the primary computer assigns the corresponding operation to the secondary computer. As secondary computers in the system, such as the primary computer's system, process corresponding subqueries, the shard assignments to different secondary computers in the system can be dynamically changed, e.g., because the shard set may become stale as query processing progresses.

[0093] When a logical shard included in a shard set is split after a primary computer assigns a subquery to a secondary computer hosting the logical shard, the system can constrain the secondary computer to iterate sequentially through the logical partitions of the underlying input data, for example, when executing an operation for the subquery, so that reads of physical data on the secondary computer never jump back and forth between data records, which can be split between different shards, computers such as nodes, or both.

[0094] When a host computer detects a loss of connection to a computer, the host computer can use the current resume token for the computer to cause the computer, or another computer hosting the same shard, to continue working on a subquery previously assigned to the computer. When a requesting device detects a loss of connection to a host computer, the requesting device can use the current global resume token to cause the host computer, or a different host computer, to continue working on a query.

[0095] In some embodiments, the host computer may group the shards into an appropriate number of shard groups and allow subquery processing between the shard groups to run in parallel. For example, the host computer may receive corresponding state information generated for the shards in the corresponding shard group from each shard group. The host computer combines the state information from the shard groups to create a global restart token. The host computer may select the number of shard groups based on the maximum restart token size. In some examples, the host computer may create three to twenty shard groups. In some examples, the host computer may create five to twelve shard groups. The host computer may use any appropriate number as the total number of shard groups. The host computer may use the maximum restart token size, the maximum message size, or both to determine the number of shard groups.

[0096] The system can cause computers hosting shards in a shard group to process subquery results sequentially. For example, a master computer can assign shards only to shard groups that together form a contiguous data set. The master computer can select a first computer hosting the first shard in a shard group and send a first subquery, e.g., for the first shard, to the first computer. While the first computer processes the first subquery, the master computer receives batches of results from the first computer, along with a corresponding restart token for each batch of results. The master computer also receives data for other shard groups from other computers.

[0097] When the host computer determines that the first computer has finished processing the first subquery, the host computer selects a second computer to host the second shard in the shard group. The second shard is sequentially located after the first shard and has no intervening shards. The host computer sends the second subquery to the second computer and, in response, receives a batch of results and a corresponding restart token from the second computer.

[0098] The master computer uses the restart token of the current shard in the shard group as the restart token for the shard group. For example, when a first computer processes a first subquery, the master computer uses the first restart token from the first computer as the restart token for the shard group to which the first computer belongs. When a second computer processes a second subquery, after the first computer finishes processing the first subquery, the master computer uses the second restart token from the second computer as the restart token for the shard group to which the first and second computers belong.

[0099] In some embodiments, when the system generates restart state in the order in which the iterators perform operations, the system may use a minimal set of iterators to generate the restart state. In some embodiments, when the system causes the iterators to generate state information in the same order that the iterators generate data for a particular batch of results, the system may not propagate calls to generate restart state to subtrees of the iterator tree that did not participate in the computation of the value in the last row of the particular batch of results. In some embodiments, the main computer may not request iterator state information from the computer processing the subquery, but may instead use the current restart token and the last batch of results received from that computer.

[0100] In some embodiments, when the system causes a secondary computer to iterate sequentially through the logical partitions of data following a split of a shard previously hosted on the secondary computer, a primary computer included in the system may be able to associate a logical partition key with each row received by the primary computer from the secondary computer. The primary computer may store the logical partition key in a global restart token for later use during a restart. The system may use the logical partition key to allow a restart after the shard boundaries have changed since the restart token was generated.

[0101] In some embodiments, the computer may receive a specific query that includes a stop condition. For example, the stop condition may specify that the computer should send a specific number of results, a specific number of batches of results, or batches of results for a specific time period to the requesting device. The computer sends batches of results to the requesting device until the stop condition is met.

[0102] The computer may later receive the specific query and a restart token for the last batch of results sent to the requesting device. In response, the computer provides additional batches of results to the requesting device, e.g., until a second stop condition is met or there are no additional results to send to the requesting device.

[0103] In some embodiments, the system can use multiple different types of restart tokens. For example, when the system has not started processing input, the system can, for example, keep the restart token empty and allocate space in memory for the restart token. The system can allocate space for the restart token in volatile memory, for example, not allocate or use space in non-volatile memory for the restart token. When the system is processing a batch of results and still has more results to send to the requesting device, the system can use data representing the state of the processing of the query as the restart token. The system stores the state in the allocated space in volatile memory as the data for the restart token. When the system has finished processing the input and determined the final batch of results in response to the query, the system can use a variable, for example, which represents infinity or a compact representation of infinity, as the restart token.

[0104] The requesting device can use the restart token to determine the status of the query. For example, the requesting device can analyze the restart token to determine whether there are additional results or batches of results that the requesting device will receive. When the requesting device uses the restart token to determine that there are no additional results or batches of results, the requesting device can use the batch of results received from the computer, stop waiting for additional batches of results, or both.

[0105] When the requesting device uses the restart token to determine that there are additional batches of results, the requesting device can use the results of the currently received batch, determine whether the results of the batch have not been received within a predetermined time period, or both. For example, the requesting device can cause the data included in some batches of results to be presented.

[0106] If the requesting device determines that the batch of results has not been received from the computer within a predetermined time period, the requesting device may determine whether the connection to the computer has been lost. In response to determining that the connection to the computer has been lost, the requesting device may establish a second connection to the computer or a second connection to another computer. The requesting device uses the second connection to provide a query and a most recent restart token as part of a request for results responsive to the query. The requesting device may provide the query and most recent restart token to the computer or another computer.

[0107] In some examples, for example, in addition to the state of the system processing the corresponding query, a restart token can indicate the number of rows of the current input to skip in determining the current batch of results. In some examples, the restart token can indicate an input number that identifies an iterator from an iterator tree that generated the last entry included in the most recent batch of results as the state of the system processing the corresponding query. When a computer receives a restart token, it can update its state based on the restart token. The computer can use the restart token to request data from the current input and skip a given number of rows identified by the restart token.

[0108] In some embodiments, the system can use a depth-first traversal, such as a conditional depth-first traversal, to generate restart tokens for an iterator tree. For example, the system can navigate nodes in an iterator tree, such as each node representing an iterator for processing a query, and also determine a top-most node in the iterator tree that includes state information. The system uses the state information for the top-most node to generate the restart token. The system can determine a top-most node for each branch of the iterator tree, such as when different branches include different levels of state information.

[0109] In some examples, the system can determine whether to proceed deeper into the iterator tree based on whether a child node has a non-empty restart state. The system can use the non-empty restart state to create a restart token for the corresponding branch of the iterator tree. The system can create a restart token for each branch of the iterator tree and combine the restart tokens of each branch to create a restart token for the entire iterator tree.

[0110] The system can traverse the iterator tree in the same manner, wherein control is passed through the iterators represented by the iterator tree during query execution. For example, the system can use the iterator tree to determine which portion of the operation for a particular query to send to a corresponding iterator, such as a computer. As the system traverses the iterator tree to determine the operation, the system can generate corresponding restart tokens for the iterators, for example, to generate restart tokens for the entire iterator tree.

[0111] In some implementations, the system may include historical data with a restart token. For example, the historical data may represent the deterministic steps taken to generate a batch of results. The historical data may describe how a system with non-deterministic outputs can perform specific steps, such as deterministic steps, to order the data used to determine the batch of results. For example, the historical data may indicate how the system can reproduce a certain number of rows in a given order so that upon restart, the system or another system will only skip those rows previously included in the batch of results sent to the requesting device, without skipping additional rows.

[0112] When the system receives the restart token and historical data, it can use replay mode to replay all steps identified in the historical data. Replay mode can be, for example, sequential mode or parallel mode, where the system extracts input data identified by the historical data in the order in which the input data was identified in the historical data. Once the system completes replaying the historical data, it switches to normal mode, where it retrieves data and creates result batches. In some examples, while the system is executing steps sequentially in replay mode, the system can also execute steps in parallel in normal mode, for example, when the system begins generating results to provide to a requesting device.

[0113] When iterator behavior depends on both input values or constants and variable values produced by descendant iterators outside the iterator or in the iterator tree, the system can generate history data. For example, a system, such as a relational query processor, can include iterators such as "cross apply" or "external apply" that produce dependent variable values. When the system generates a restart token for an iterator that produces dependent variable values, the system can, for example, generate both the restart token and history data that allows replay of values produced by the system by the iterator before the restart occurs, e.g., instead of indicating whether the iterator was stopped in processing the query before the restart occurred.

[0114] When the historical data includes data identifying the order in which iterators were called, the iterators called each other, or both to produce the last result item, the system can determine whether the restart token, the historical data, or both include data for a particular iterator in the iterator tree. For example, when a query is restarted using a restart token and historical data, the system provides historical data to iterators that will continue the work previously performed for that query. The system may not provide historical data to any iterator that will not continue the work previously performed for that query. For example, for a particular iterator that does not have any historical data, does not need to deliver results in a repeatable order based on historical data, or both, the system will not provide the restart token or historical data to the particular iterator. The system may provide to a particular iterator only a subquery that identifies the search operation for that particular iterator.

[0115] In some examples, when a particular iterator is starting its child iterators from the beginning, e.g., without history data or a restart token, the particular iterator may not request restart state information from its child iterators when generating a restart token. A restart token may indicate that no work has yet been performed. In some embodiments, when a particular iterator does not depend on a repeatable order of inputs, the particular iterator may not request a restart token from its child iterators. The particular iterator may use the results that the particular iterator has already received from its child iterators to generate a restart token or a restart token and history data.

[0116] In some embodiments, the restart token may include an identifier indicating whether the restart token includes historical data. For example, the restart token may include a history identifier. The history identifier may be a Boolean value indicating whether the restart token includes historical data.

[0117] In some embodiments, a system may include a history token that includes history data separate from a restart token. In general, any system that includes both history data and restart state information with a batch of results may be described as including a restart token with history data, regardless of whether the history data and restart state information are in sequential parts of a message or separated by other data, such as headers, response data, or other data.

[0118] In some implementations, the restart token may include an identifier for data that corresponds to a batch of results. For example, the system may include an identifier for a database, a shard, a portion of a table, a portion of memory, an iterator, or a combination of two or more of these in the restart token. The system may use the identifier to determine which steps correspond to which data, which batches of results, or which subqueries.

[0119] In some implementations, the system can generate history data that includes identifiers of data to which the history data was applied. For example, the system can include multiple keys in the history data, such that each key indicates the data to which the corresponding history data, such as a step, was applied. When the system includes keys in the history data, the system can use the keys to replay the history after a split. For example, when a split reduces the boundaries of a shard, the system can use the keys to determine which steps in the history apply to a specific new shard and execute only those steps during replay.

[0120] In some embodiments, the system may include multiple levels of auxiliary computers. For example, a primary computer may assign subqueries to a first level of auxiliary computers. Each of the first level of auxiliary computers may assign further subqueries to a second level of auxiliary computers, such that each first level auxiliary computer is associated with a specific second level auxiliary computer.

[0121] The iterator tree can represent the number of computers that process the operation for the query. For example, each iterator in the iterator tree can represent a computer that processes the operation for the query. In some examples, a single computer can perform operations for multiple iterators in the iterator tree, such as when a single computer executes a virtual machine for each of two iterators from the iterator tree. The leaf nodes in the iterator tree can represent specific computers that generate results and provide results to the computers represented by the non-leaf nodes in the iterator tree. The computers represented by the non-leaf nodes in the iterator tree can receive results from multiple other computers (for example, computers represented by leaf nodes or non-leaf nodes) and combine the received results. The computers represented by the non-leaf nodes in the iterator tree can communicate with their child iterators to generate restart tokens or both restart tokens and historical data.

[0122] In some embodiments, the system can use historical data when the system includes two or more hierarchical levels of auxiliary computers, and, for example, rather than processing a query on a single computer or when a primary computer and one hierarchical level of auxiliary computers process operations for a query. In some embodiments, the system can use historical data when the system includes parallel processing of two or more auxiliary computers, for example, regardless of the hierarchy of the auxiliary computers. In some embodiments, the system can use historical data when the system includes two or more hierarchical levels of auxiliary computers and uses parallel processing of at least some of the auxiliary computers.

[0123] In some embodiments, when the system generates result batches in parallel and creates historical data for these result batches, the system can determine whether the size of the historical data meets a threshold size. For example, the threshold size can be a maximum size and the system can determine whether the size of the historical data is greater than or almost greater than the maximum size.

[0124] When the size of the historical data meets a threshold size, the system can stop generating results in parallel and begin generating results sequentially. For example, the system can determine that the historical data is too large and does not meet the requirement to keep the size of the restart token below the threshold size. The system can continue to include the historical data as part of the restart token while updating the deterministic value within the restart token, and, for example, without increasing the size of the historical data. The system can include a processing type identifier, such as a flag, in the restart token that instructs the system to switch from parallel processing to sequential processing. The system can include data that indicates when the system switches from parallel processing to sequential processing.

[0125] In some embodiments, the system can generate historical data identifying the steps taken to generate the batch of results sent to the requesting device, including the results of the current batch, such as current historical data. In some examples, the system can generate historical data identifying the steps taken or to be taken to generate the batch of results that should be sent to the requesting device, such as next historical data.

[0126] In some embodiments, the system can generate different types of historical data. For example, the system can determine whether to use current historical data, next historical data, or both. The system can create a restart token using the selected type of historical data. When the system receives the restart token, the system can determine the type of historical data included in the restart token. The system can replay the steps identified in the historical data and then determine the next steps to perform based on the type of historical data. When the historical data identifies current historical data for a batch of results sent to the requesting device, the system can determine a new batch of results to send to the requesting device. For example, the system can perform additional steps to determine the new batch of results. When the historical data identifies next historical data for a batch of results that has not yet been sent to the requesting device, the system generates a new batch of results, for example, by selecting the results represented by performing the operation indicated in the next historical data.

[0127] When the system receives a restart token generated for a query using both parallel and sequential processing, the system can initialize a state of the system starting with parallel processing. During initialization, when the system recognizes a value of a processing type identifier indicating a switch from parallel processing to sequential processing, the system switches from parallel processing of the query to sequential processing of the query.

[0128] When sequentially processing queries that can be processed in parallel, the system can determine whether to continue processing the query using sequential processing or whether the size of the historical data will be small enough to begin processing the data in parallel. For example, the system can have a current restart token that includes historical data. As the system determines the results of additional batches, the system can update the historical data by removing some of the historical data that is no longer needed to determine the current state of the system and to determine whether the historical data is small enough to allow parallel processing again.

[0129] The system may include an iterator interface. The iterator interface may include flow control methods. The flow control methods may include methods for transitioning an iterator through various stages to indicate that data responsive to a query is ready, to indicate that no additional data responsive to a query is available, or a combination of two or more of these. For example, the iterator interface may include a signaling mechanism that allows an iterator to indicate that it has produced all of its output tuples and that the iterator has no more to output.

[0130] The iterator interface may include data exchange methods. The data exchange methods may include methods for identifying the location of input values, output values, or both. The system may use the internal state of the iterators in the iterator tree after determining the final tuple or intermediate tuples and the values, such as output values, accessible through the data exchange methods to determine the status of query execution.

[0131] In some embodiments, the system may use an iterator tree that indicates separate restart tokens, each for a different subquery of an operation generated for a query. The system may assign each of the subqueries to an auxiliary computer that performs the operation to determine a batch of results for the corresponding subquery. For some nodes in the iterator tree, the system may use a "start flag" that indicates whether the corresponding auxiliary computer has begun processing the corresponding subquery, an "end flag" that indicates whether the corresponding auxiliary computer has completed processing the corresponding subquery, or both. When the system processes subqueries in parallel, the system may use one or two flags. When the system processes subqueries sequentially, the system may use one or two flags.

[0132] When the system determines that a shard split has occurred, the system can generate separate subqueries for each split. For example, when a split widens the boundaries of a shard, the system can create a first subquery for the data originally included in the shard and a second subquery for the data newly added to the shard. The system provides the first subquery and the second subquery to a secondary computer hosting the shard so that the computer determines a result batch responsive to each of the subqueries and receives the result batch from the secondary computer. For example, when the system has shards {user(user1) / thread(1,10); user(user1) / thread(10,20); user(user1) / thread(20,30)} and determines that one of the shards becomes user(user1) / thread(15,25); the system can create a first subquery for user(user1) / thread(15,20) and a second subquery for user(user1) / thread(20,25) and provide the first subquery and the second subquery in different requests to the computer hosting the shard user(user1) / thread(15,25).

[0133] When a split reduces a shard boundary, the system can create new subqueries for the new shard boundary and include the same restart token for each subquery. The system sends the subqueries to the corresponding secondary computers hosting the corresponding shards. Each secondary computer determines whether the restart token indicates whether all data responsive to the subquery has been previously provided to the system. In response to determining that all data responsive to the subquery has been previously provided to the system, the secondary computer provides a message to the system indicating that all data has been previously provided to the system. In response to determining that all data responsive to the subquery has not been provided to the system, the secondary computer determines whether the restart token applies to the shards included on the secondary computer. If the restart token applies to the shards included on the secondary computer, the secondary computer uses the restart token to determine data that has not been previously provided to the system and generates a result batch using the determined data that has not been previously provided to the system. If the restart token does not apply to the shards included on the secondary computer, the secondary computer ignores the restart token and processes the data in the shard as if the secondary computer had not received the restart token.

[0134] 5.0 Additional Implementation Details

[0135] Embodiments of the subject matter and functional operations described in this specification may be implemented using digital electronic circuits, tangibly embodied computer software or firmware, virtual machines, applications executed on virtual machines, computer hardware, including the structures disclosed in this specification and their structural equivalents, or a combination of one or more of the above. Embodiments of the subject matter described in this specification may be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible, non-transitory program carrier for execution by a data processing device or for controlling the operation of the data processing device. Alternatively or in addition, the program instructions may be encoded on an artificially generated propagation signal, such as a machine-generated electrical, optical, or electromagnetic signal, which is generated to encode information for transmission to a suitable receiver device for execution by the data processing device. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory array or device, or a combination of one or more of the above.

[0136] The term "data processing apparatus" refers to data processing hardware and includes all kinds of devices, equipment and machines for processing data, including, for example, a programmable processor, a computer, a virtual machine, multiple processors or computers. The apparatus may also be or further include special-purpose logic circuitry, such as an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). In addition to hardware, the apparatus may optionally include code that creates an execution environment for the computer program, such as code constituting processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of the above.

[0137] A computer program, which may also be referred to or described as a program, software, container, software application, module, software module, script or code, may be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages, and may be deployed in any form, including as a separate program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program may, but need not, correspond to a file in a file system. A computer program may be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program, or in multiple coordinated files (e.g., files that store one or more modules, subroutines, or portions of code). A computer program may be deployed to execute on one computer or on multiple computers located at one site or distributed across multiple sites and interconnected by a communication network.

[0138] The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by, and apparatus can be implemented as, special purpose logic circuitry, such as an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit).

[0139] The computer that is suitable for the execution of computer program comprises for example general or special purpose microprocessor or both, or any other type of central processing unit.Usually, central processing unit will receive instruction and data from read-only memory or random access memory or both.The essential element of computer is the central processing unit for executing instruction and one or more memory devices for storing instruction and data.Usually, computer will also comprise one or more large capacity storage devices for storing data, such as magnetic, magneto-optical disk or optical disk, or operation coupling to receive data therefrom or transmit data thereto.But, computer need not have such device.Moreover, computer can be embodied as other equipment, such as mobile phone, personal digital assistant (PDA), mobile audio or video player, game console, global positioning system (GPS) receiver or portable storage device, such as universal serial bus (USB) flash drive, this only lists a few.

[0140] Computer-readable media suitable for storing computer program instructions and data include all forms of nonvolatile memory, media, and storage devices, including, for example, semiconductor memory devices, such as EPROM, EEPROM, and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The processor and memory can be supplemented by, or incorporated in, special purpose logic circuitry.

[0141] To provide for interaction with a user, embodiments of the subject matter described in this specification may be implemented on a computer having a display device, such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user, and a keyboard and pointing device, such as a mouse or trackball, that the user can use to provide input to the computer. Other types of devices may also be used to provide for interaction with a user; for example, feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user may be received in any form, including acoustic, voice, or tactile input. In addition, a computer may interact with a user by sending documents to and receiving documents from a device used by the user; for example, by sending a web page to a web browser on the user's device in response to a request received from the web browser.

[0142] The embodiments of the subject matter described herein can be implemented by a computing system comprising a back-end component such as a data server, or comprising a middleware component such as an application server, or comprising a front-end component such as a client computer having a graphical user interface or a web browser with which a user can interact with the embodiments of the subject matter described in this specification, or a combination of one or more such back-end, middleware, or front-end components. The components of the system can be interconnected by digital data communication of any form or medium, such as a communication network. Examples of communication networks include local area networks (LANs) and wide area networks (WANs), such as the Internet.

[0143] A computing system may include a client and a server. The client and server are typically remote from each other and typically interact via a communication network. The client-server relationship arises due to computer programs running on the respective computers in a client-server relationship with each other. In some embodiments, the server transmits data, such as an HTML page, to a user device, for example, to display data to a user interacting with the user device as a client and to receive user input from the user. Data generated on the user device, such as the results of the user interaction, may be received from the user device at the server.

[0144] An example of this type of computer is Figure 3, which shows a schematic diagram of a general computer system 300. According to one embodiment, system 300 can be used for the operations described in association with any of the computer-implemented methods described above. System 300 includes a processor 310, a memory 320, a storage device 330, and an input / output device 340. Each of components 310, 320, 330, and 340 are interconnected using a system bus 340. Processor 310 is capable of processing instructions for execution within system 300. In one embodiment, processor 310 is a single-threaded processor. In another embodiment, processor 310 is a multi-threaded processor. Processor 310 is capable of processing instructions stored in memory 320 or on storage device 330 to display graphical information for a user interface on input / output device 340.

[0145] Memory 320 stores information within system 300. In one embodiment, memory 320 is a computer-readable medium. In one embodiment, memory 320 is a volatile memory unit. In another embodiment, memory 320 is a non-volatile memory unit.

[0146] The storage device 330 can provide mass storage for the system 300. In one embodiment, the storage device 330 is a computer-readable medium. In various embodiments, the storage device 330 can be a floppy disk device, a hard disk device, an optical disk device, or a magnetic tape device.

[0147] The input / output device 340 provides input / output operations for the system 300. In one embodiment, the input / output device 340 includes a keyboard and / or a pointing device. In another embodiment, the input / output device 340 includes a display unit for displaying a graphical user interface.

[0148] Although this specification contains many specific implementation details, these should not be interpreted as limitations on the scope of what may be claimed, but rather as descriptions of features that may be specific to a particular embodiment. Certain features described in this specification in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented separately in multiple embodiments or in any suitable subcombination. Moreover, although features may be described above as acting in certain embodiments, or even initially claimed as such, one or more features from a claimed combination may in some cases be omitted from the combination, and the claimed combination may be directed to a subcombination or variation of a subcombination.

[0149] Similarly, although operations are described in a particular order in the accompanying drawings, this should not be understood as requiring that such operations be performed in the particular order shown or in a sequential order, or that all illustrated operations be performed, to achieve the desired results. In certain circumstances, multitasking and parallel processing may be advantageous. Moreover, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments. It should be understood that the program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0150] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve the desired results. As an example, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve the desired results. In some cases, multitasking and parallel processing may be advantageous.

Claims

1. A computer-implemented method, which, when executed by data processing hardware, causes the data processing hardware to perform operations comprising: receiving, at a master computing node, an initial query from a requesting device at a first instance of time, the initial query requesting a query response based on data stored in a distributed database, the master computing node corresponding to one of a plurality of computing nodes of a query system that coordinates queries to the plurality of computing nodes; Parsing the initial query into a query tree comprising a plurality of sub-queries by the master computing node, each sub-query generating a corresponding result when executed in parallel with one or more other sub-queries to form a portion of the query response; Assigning, by the master computing node, each corresponding subquery of the plurality of subqueries to a corresponding computing node of the plurality of computing nodes coordinated by the master computing node; generating, by the master computing node, a runtime query plan representing the query tree, the query tree including the assigned plurality of sub-queries; and During runtime execution of the initial query, a current restart token of the runtime query plan at the primary compute node is updated by the primary compute node based on the corresponding results of the multiple sub-queries, the current restart token identifying the executed sub-query among the multiple sub-queries to represent a current point in the progress of execution of the initial query.

2. The method according to claim 1, wherein The operations also include storing the runtime query plan at the primary compute node.

3. The method according to claim 1, wherein The operations further include: receiving, at the primary computing node, a second query from the requesting device at a second instance of time, the second instance of time being subsequent to the first instance of time; determining, by the master computing node, a second response to the second query including results from the initial query; and Based on the determination that some portion of the second response to the second query includes results from the initial query, the query system is prevented, by the master computing node, from re-executing a corresponding portion of the second query that generated the results from the initial query.

4. The method according to claim 3, wherein: The operations further include: generating, by one or more of the plurality of computing nodes, query results for the second query that do not include results from the initial query; and The query result of the second query excluding the result from the initial query is transmitted by the master computing node as a second query response to the second query.

5. The method according to claim 1, wherein: The runtime query plan includes a stateful runtime structure; and The operations further include: receiving, at the master computing node, a second query from the requesting device at a second instance of time, the second instance of time being after the first instance of time; and A second runtime query plan for the second query is generated by the master compute node based on the current restart token of the runtime query plan of the initial query.

6. The method according to claim 5, wherein: Generating the second runtime query plan includes modifying the runtime query plan of the initial query by adjusting a number of a plurality of subqueries of the runtime query plan of the initial query.

7. The method according to claim 1, wherein: The runtime query plan includes a stateful runtime structure; and The operations further include, during runtime of executing the initial query: receiving a first result from a first compute node executing a first subquery assigned to the first compute node; receiving a second result from the second compute node executing the second subquery assigned to the second compute node; and The current restart token of the runtime query plan at the master compute node is updated by the master compute node based on the first result from the first compute node for the first subquery and the second result from the second compute node for the second subquery.

8. The method according to claim 7, wherein: The operations also include transmitting, from the master compute node to the requesting device, the current restart token of the runtime query plan and a query response to the initial query, the query response including the first result from the first compute node and the second result from the second compute node.

9. The method according to claim 1, wherein The operations further include, during runtime of executing the initial query: receiving, at the master compute node, a first result from a first compute node assigned a first subquery of the query tree and a second result from a second compute node assigned a second subquery of the query tree; and concatenating, by the master computing node, the first result from the first computing node and the second result from the second computing node to form the query response to the initial query; and The query response including the concatenated first result and second result is transmitted from the master computing node to the requesting device.

10. The method according to claim 1, wherein: Assigning each respective subquery of the plurality of subqueries to the respective compute node of the plurality of compute nodes coordinated by the master compute node includes: assigning a first subquery to a first compute node of the plurality of compute nodes and assigning a second subquery to a second compute node of the plurality of compute nodes, the first compute node having a first data shard including data responsive to the initial query and the second compute node having a second data shard including data responsive to the initial query; and During the execution of said initial query at runtime: Determining, by the master computing node, that the data of the first data shard in response to the initial query has changed storage location to a third computing node among the plurality of computing nodes; and The second subquery is reassigned by the master computing node to the third computing node among the plurality of computing nodes.

11. A system, comprising: Data processing hardware; as well as Memory hardware in communication with the data processing hardware, the memory hardware storing instructions that, when executed on the data processing hardware, cause the data processing hardware to perform operations comprising: receiving, at a master computing node, an initial query from a requesting device at a first instance of time, the initial query requesting a query response based on data stored in a distributed database, the master computing node corresponding to one of a plurality of computing nodes of a query system that coordinates queries to the plurality of computing nodes; Parsing the initial query into a query tree comprising a plurality of sub-queries by the master computing node, each sub-query generating a corresponding result when executed in parallel with one or more other sub-queries to form a portion of the query response; Assigning, by the master computing node, each corresponding subquery of the plurality of subqueries to a corresponding computing node of the plurality of computing nodes coordinated by the master computing node; generating, by the master compute node, a runtime query plan representing the query tree, the query tree including the assigned plurality of sub-queries; and During runtime execution of the initial query, a current restart token of the runtime query plan at the primary compute node is updated by the primary compute node based on the corresponding results of the multiple sub-queries, the current restart token identifying the executed sub-query among the multiple sub-queries to represent a current point in the progress of execution of the initial query.

12. The system according to claim 11, wherein The operations also include storing the runtime query plan at the primary compute node.

13. The system according to claim 11, wherein: The operations further include: receiving, at the primary computing node, a second query from the requesting device at a second instance of time, the second instance of time being subsequent to the first instance of time; determining, by the master computing node, a second response to the second query including results from the initial query; and Based on the determination that some portion of the second response to the second query includes results from the initial query, the query system is prevented, by the master computing node, from re-executing a corresponding portion of the second query that generated the results from the initial query.

14. The system according to claim 13, wherein: The operations further include: generating, by one or more of the plurality of computing nodes, query results for the second query that do not include results from the initial query; and The query result of the second query excluding the result from the initial query is transmitted by the master computing node as a second query response to the second query.

15. The system of claim 11, wherein: The runtime query plan includes a stateful runtime structure; and The operations further include: receiving, at the master computing node, a second query from the requesting device at a second instance of time, the second instance of time being after the first instance of time; and A second runtime query plan for the second query is generated by the master compute node based on the current restart token of the runtime query plan of the initial query.

16. The system according to claim 15, wherein: Generating the second runtime query plan includes modifying the runtime query plan of the initial query by adjusting a number of a plurality of subqueries of the runtime query plan of the initial query.

17. The system of claim 11, wherein: The runtime query plan includes a stateful runtime structure; and The operations further include, during runtime of executing the initial query: receiving a first result from a first compute node executing a first subquery assigned to the first compute node; receiving a second result from the second compute node executing the second subquery assigned to the second compute node; and The current restart token of the runtime query plan at the master compute node is updated by the master compute node based on the first result from the first compute node for the first subquery and the second result from the second compute node for the second subquery.

18. The system according to claim 17, wherein: The operations also include transmitting, from the master compute node to the requesting device, the current restart token of the runtime query plan and a query response to the initial query, the query response including the first result from the first compute node and the second result from the second compute node.

19. The system according to claim 11, wherein: The operations further include, during runtime of executing the initial query: receiving, at the master compute node, a first result from a first compute node assigned a first subquery of the query tree and a second result from a second compute node assigned a second subquery of the query tree; and concatenating, by the master computing node, the first result from the first computing node and the second result from the second computing node to form the query response to the initial query; and The query response including the concatenated first result and second result is transmitted from the master computing node to the requesting device.

20. The system of claim 11, wherein: Assigning each respective subquery of the plurality of subqueries to the respective compute node of the plurality of compute nodes coordinated by the master compute node includes: assigning a first subquery to a first compute node of the plurality of compute nodes and assigning a second subquery to a second compute node of the plurality of compute nodes, the first compute node having a first data shard including data responsive to the initial query and the second compute node having a second data shard including data responsive to the initial query; and During the execution of said initial query at runtime: Determining, by the master computing node, that the data of the first data shard in response to the initial query has changed storage location to a third computing node among the plurality of computing nodes; and The second subquery is reassigned by the master computing node to the third computing node among the plurality of computing nodes.

Images