Low-latency demultiplexer for propagating ordered data to multiple sinks
Patent Information
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- GOOGLE LLC
- Filing Date
- 2023-06-19
- Publication Date
- 2026-07-31
Smart Images
Figure 0007898549000001 
Figure 0007898549000002 
Figure 0007898549000003
Abstract
Description
Technical Field
[0001] The present disclosure relates to a low-latency demultiplexer for propagating ordered data.
Background Art
[0002] As the popularity of large-scale distributed databases (i.e., "cloud" databases) continues to grow, the need to propagate data from a database to data sinks (e.g., applications) also continues to increase. However, when each sink consumes data at a different rate, it can be problematic to continuously propagate ordered data from a database to multiple sinks. A simple solution to this problem involves dealing with data propagation to each sink individually. However, this requires separate and independent database queries for each stream of data to a given sink. Alternatively, each sink may be limited to the rate of the slowest sink, in which case the latency of all sinks except the slowest increases.
Summary of the Invention
[0003] One aspect of this disclosure provides a method for a low-latency demultiplexer that propagates data to multiple sinks. The method, when performed by a computer, causes the data processing hardware to perform an operation. The operation includes receiving a series of writes. Each write in the series of writes requests that its respective data be stored in a database communicating with the data processing hardware. For each write in the series of writes, the operation includes storing the respective data in a first buffer associated with a first data streaming application, storing the respective data in a second buffer associated with a second data streaming application, and sending the respective data to the database. The operation also includes receiving confirmation from the database that the respective data for each write has been committed to the database. The confirmation includes an order indicator indicating the order in which the respective data for each write was committed to the database relative to the other writes in the series of writes. In response to receiving confirmation that the data for each write has been committed to the database, the operation includes sending the data for each write from a first buffer to a first data streaming application based on an order indicator indicating that the data for each write has been committed to the database for the other writes in the sequence of writes, and sending the data for each write from a second buffer to a second data streaming application based on an order indicator indicating that the data for each write has been committed to the database for the other writes in the sequence of writes.
[0004] Embodiments of the present disclosure may include one or more of the following optional features. In some embodiments, sending each data from a first buffer to a first data streaming application based on the order in which each data is committed to the database for each other in a series of writes includes using an order indicator to determine that an acknowledgment of a preceding write has been received. Each data of a preceding write was committed to the database immediately before each data of the respective write. In some of these embodiments, determining that an acknowledgment of a preceding write has been received includes determining the watermark level of the first buffer. Optionally, sending each data from a second buffer to a first data streaming application based on the order in which each data is committed to the database for each other in a series of writes includes using an order indicator to determine that no acknowledgment of a preceding write has been received, that each data of a preceding write was committed to the database immediately before each data of the respective write, and after determining that no acknowledgment of a preceding write has been received, receiving an acknowledgment that each data of the preceding write has been committed to the database. The operation may also include sending each of the preceding writes from the second buffer to the second data streaming application in response to receiving confirmation that the data for each preceding write has been committed to the database, and sending each of the data for each subsequent write from the second buffer to the second data streaming application after sending the data for each preceding write.
[0005] In some examples, the sequence indicator includes an increment identifier. In some embodiments, the sequence indicator includes a timestamp. In some of these embodiments, the operation further includes using the timestamp of each received confirmation to generate a linked list that orders a series of writes in the order in which the data for each of each write in the series of writes was committed to the database. Generating the linked list may include generating a hash map.
[0006] Optionally, receiving confirmation from the database that each data for each write has been committed to the database includes determining that a threshold period has elapsed without receiving confirmation, and in response to determining that a threshold period has elapsed without receiving confirmation, retrieving a change log from the database. The operation may further include determining from the change log that each data for each write has been committed to the database. The database may include a Structured Query Language (SQL) database.
[0007] Other aspects of this disclosure provide a system for a low-latency demultiplexer that propagates data to multiple sinks. The system includes data processing hardware and memory hardware that communicates with the data processing hardware. The memory hardware stores instructions that cause the data processing hardware to perform an operation when executed by the data processing hardware. An operation includes receiving a series of writes. Each write in the series of writes requests that its respective data be stored in a database that communicates with the data processing hardware. For each write in the series of writes, the operation includes storing its respective data in a first buffer associated with a first data streaming application, storing its respective data in a second buffer associated with a second data streaming application, and sending its respective data to the database. The operation also includes receiving confirmation from the database that each data for each write has been committed to the database. The confirmation includes an order indicator that indicates the order in which each data for each write was committed to the database relative to the other writes in the series of writes. In response to receiving confirmation that the data for each write has been committed to the database, the operation includes sending the data for each write from a first buffer to a first data streaming application based on an order indicator indicating that the data for each write has been committed to the database for the other writes in the sequence of writes, and sending the data for each write from a second buffer to a second data streaming application based on an order indicator indicating that the data for each write has been committed to the database for the other writes in the sequence of writes.
[0008] This embodiment may include one or more of the following optional features. In some embodiments, sending each data from a first buffer to a first data streaming application based on the order in which the respective data of each write was committed to the database relative to other writes in a series of writes includes using an order indicator to determine that an acknowledgment of a preceding write has been received. Each data of a preceding write was committed to the database immediately before each data of the respective write. In some of these embodiments, determining that an acknowledgment of a preceding write has been received includes determining the watermark level of the first buffer. Optionally, sending each data from a second buffer to a first data streaming application based on the order in which the respective data of each write was committed to the database relative to other writes in a series of writes includes using an order indicator to determine that an acknowledgment of a preceding write has not been received, that each data of a preceding write was committed to the database immediately before each data of the respective write, and after determining that an acknowledgment of a preceding write has not been received, receiving an acknowledgment that each data of the preceding write has been committed to the database. The operation may also include sending each of the preceding writes from the second buffer to the second data streaming application in response to receiving confirmation that the data for each preceding write has been committed to the database, and sending each of the data for each subsequent write from the second buffer to the second data streaming application after sending the data for each preceding write.
[0009] In some examples, the sequence indicator includes an increment identifier. In some embodiments, the sequence indicator includes a timestamp. In some of these embodiments, the operation further includes using the timestamp of each received confirmation to generate a linked list that orders a series of writes in the order in which the data for each of each write in the series of writes was committed to the database. Generating the linked list may include generating a hash map.
[0010] Optionally, receiving confirmation from the database that each data for each write has been committed to the database includes determining that a threshold period has elapsed without receiving confirmation, and in response to determining that a threshold period has elapsed without receiving confirmation, retrieving a change log from the database. The operation may further include determining from the change log that each data for each write has been committed to the database. The database may include a Structured Query Language (SQL) database.
[0011] Details of one or more embodiments of this disclosure are described in the accompanying drawings and the following description. Other embodiments, features, and advantages will become apparent from the description and drawings, as well as from the claims. [Brief explanation of the drawing]
[0012] [Figure 1] This is a schematic diagram of an exemplary system for a low-latency demultiplexer for propagating data to multiple sinks. [Figure 2] Figure 1 is a schematic diagram of an exemplary buffer data structure for the database client of the system shown. [Figure 3A] Figure 2 is a schematic diagram of the database client receiving an out-of-order write configuration. [Figure 3B] Figure 2 is a schematic diagram of the database client receiving an out-of-order write configuration. [Figure 3C] Figure 2 is a schematic diagram of the database client receiving an out-of-order write configuration. [Figure 4] This is an exemplary operational configuration flowchart for a method of providing a low-latency demultiplexer for propagating data to multiple sinks. [Figure 5] This is a schematic diagram of an exemplary computing device that may be used to implement the systems and methods described herein. [Modes for carrying out the invention]
[0013] Similar reference symbols in various drawings refer to the same elements. Large-scale distributed databases (i.e., "cloud" databases) are in ever-increasing demand. With this growing demand, it's common for a given database to have hundreds or thousands of data sinks that require any updates to the database in real-time or near real-time. For example, these data sinks require that any data written to the database be streamed to the data sinks instantly with minimal latency. Generally speaking, these data sinks need to receive data in the same order in which it was written to or committed to the database. However, if each sink consumes data at a different rate, sequentially propagating ordered data from the database to multiple sinks can become problematic.
[0014] A simple solution to this problem involves dealing with data propagation to each sink individually. However, this requires separate and independent database queries for each stream of data to a given sink. Alternatively, each sink may be limited by the rate of the slowest sink, in which case the latency of all sinks except the slowest one increases. Another alternative is for the system to take a database log (e.g., write-a-log (WAL)) directly from the database to decide whether to update the database, but this technique has several drawbacks. Firstly, the number of data streams is limited by the number of connections the database can maintain, which is usually a few hundred or less. This lacks the scalability required to support modern distributed computing systems. Furthermore, this technique also requires database polling, consuming considerable database resources if polling is frequent, and data latency if polling is infrequent.
[0015] Typically, a database includes a database client that provides an interface between the database and the user. The database client receives data (for example, from one or more users) to write to the database. The database client provides data to the database. Once the data is successfully written to the database (i.e., committed), the database provides a commitment confirmation to the database client, which may then notify the user(s).
[0016] Embodiments of this specification include a database client that runs in the application layer and mediates between the user and the database. The database client enables the sequential propagation of ordered data in an efficient manner with minimal latency, without requiring an additional storage system. By running along the data write path to the database rather than along the database log read path, the database client can retrieve written data without having to perform database reads. This allows the database client to avoid serializing data objects to and from the binary data format required by many databases (e.g., Structured Query Language (SQL) databases). The database client includes one or more buffer data structures that buffer data to be written to the database in transient or volatile memory. When the database client receives confirmation that the data has been committed to the database, the database client propagates the data from the buffer to each pending data sink. Each data sink can receive data from its buffer at any rate that the data sink can handle. The database client may maintain separate “watermarks” or pointers to properly order the data individually to each data sink.
[0017] Referring here to Figure 1, in some embodiments, the exemplary data propagation system 100 includes a remote system 140 that communicates with one or more user devices 10 via a network 112. The remote system 140 may be a single computer, multiple computers, or a distributed system (e.g., a cloud environment) having scalable / elastic resources 142 including computing resources 144 (e.g., data processing hardware) and / or storage resources 146 (e.g., memory hardware). A data store 150 (i.e., a remote storage device) may be overlaid on the storage resources 146 to enable scalable use of the storage resources 146 by one or more of the clients (e.g., user devices 10) or computing resources 144. The data store 150 is configured to store sets of data blocks 154, 154a-n (also referred to herein simply as data 154) in one or more tables or databases 156 (i.e., cloud databases), each containing, for example, multiple rows and columns.
[0018] The remote system 140 is configured to receive a series of writes 152, 152a-n from a user device 10 associated with each user 12, for example, via a network 112. The user device 10 may be any computing device such as a desktop workstation, laptop workstation, or mobile device (i.e., smartphone). The user device 10 includes computing resources 18 (e.g., data processing hardware) and / or storage resources 16 (e.g., memory hardware). The user 12 may construct a query 20 using an SQL interface. Each write 152 in the series of writes 152 includes the respective data 154 to be written to the database 156. The database 156 (e.g., an SQL database) may be stored in a data store 150.
[0019] The database client 160 mediates communication between user 12 and database 156 (for example, at the application layer). In addition to interface with database 156, the database client 160 propagates ordered data 154 to one or more applications 170, 170a-n (or any other data sink), such as one or more data streaming applications 170. Each application 170 receives the ordered data 154 with minimal latency as soon as the data 154 is committed to database 156. That is, applications 170 receive the data 154 in the same order in which it is written to database 156.
[0020] For each write 152 in a series of writes 152 received by the database client 160, the database client 160 stores the data 154 of the write 152 in buffers 210, 210a-n associated with each application 170 to which the database client 160 streams the data 154. For example, if the database client 160 streams the data 154 to three applications 170, the database client 160 stores the data 154 from each write 152 in a first buffer 210 associated with the first application 170, a second buffer 210 associated with the second application 170, and a third buffer 210 associated with the third application. In another example, the database client 160 stores the data 154 from each write 152 in a single buffer 210 associated with all three applications 170. Each buffer 210 represents a data structure that temporarily stores the data 154 using transient storage (e.g., volatile memory).
[0021] Database client 160 sends each piece of data 154 to database 156 for each write 152. For each write 152, database 156 stores or commits each piece of data 154 to database 156 (i.e., persistent or non-volatile storage), and then sends acknowledgments 220, 220a~n to database client 160 confirming that data 154 has been committed to database 156. Each acknowledgment 220 includes an order indicator 222 that indicates the order in which the data 154 of each write 152 was committed to database 156 relative to the data 154 of other writes 152 within a series of writes 152. As discussed in more detail below, order indicator 222 may include an incrementing identifier or a timestamp.
[0022] In response to receiving an acknowledgment 220 that the data 154 of a write 154 has been committed to database 156, database client 160 sends each piece of data 154 from each buffer 210 to the corresponding application 170. For example, when it is confirmed that the first buffer 210 is committed to database 156, the data 154 is sent to the first application. In this example, database client 160 uses the second buffer 210 to send data 154 to the second application 170. In this way, database client 160 propagates data 154 to application 170 with minimal latency by not having to fetch or read data 154 from database 156 or from the database log. Instead, database client 160 "intercepts" data 154 on its way to database 156 and stores data 154 locally in buffer 210 until data 154 is committed to database 156. When database client 160 determines that data 154 has been committed to database 156, database client 160 may propagate data 154 to application 170 using buffer 210.
[0023] Referring here to Figure 2, in some examples, application 170 (or any other data sink) consumes data 154 at different rates. For example, one application 170 consumes data at an unlimited rate (i.e., the same rate at which database client 160 sends data 154), while another application 170 consumes data at a rate that is at least temporarily slower than the rate at which database client 160 can send data 154 (i.e., slower than the rate at which database 156 receives data 154). In some embodiments, by implementing watermarks 212, 212a~n for each buffer 210, database client 160 is able to track which data 154 has been sent to each application 170 and which data 154 has not yet been sent. When the database client 160 sends new data 154 to the application 170, the database client 160 updates the corresponding watermark 212 to reflect the new position of the watermark 212 (i.e., to reflect that new data 154 has been sent).
[0024] In some examples, the database client 160 expires (e.g., removes or deletes) the data 154 from the buffer 210 based on the watermark 212. That is, if the watermark 212 indicates that the data 154 has been sent to the corresponding application 170, the database client 160 may recover space in the buffer 210 by deleting or overwriting the sent data 154. In some embodiments, multiple applications 170 share a single buffer 210. In these embodiments, each application 170 sharing the buffer 210 has an independent watermark 212 to track which data 154 has been sent to which application 170. In this case, the database client 160 may ensure that the data 154 is expired only when the data 154 has been sent to all applications 170 that use the buffer 210 (i.e., based on each of the independent watermarks 212).
[0025] In the example shown in the schematic diagram 200, the database client 160 receives four writes 152a - d with data 154a - d to be written to the database 156. The database client 160 also streams the data 154 to two applications 170a - b. Each application 170 is associated with a corresponding buffer 210a - b. When the database client 160 receives the writes 154a - d, it queues the data 154a - d to their respective buffers 210a - b. The database client 160 also sends the data 154a - d to the database 156. In this example, the database 156 responds with four acknowledgments 220a - d (each including a sequence indicator 222) that confirm that the data 154a - d has been committed. Here, the sequence indicator 222 is an incrementing identifier (i.e., an incrementing integer). That is, the database 156 increments the sequence indicator 222 each time a commit to the database 156 is successful.
[0026] However, the first application 170a can only consume data 154a~b (and cannot yet consume data 154c~d), and the second application 170b can only consume data 154a~c (and cannot yet consume data 154d). Therefore, the database client 160 updates the first watermark 212a to reflect that it has sent data 154a~b to the first application 170a, and similarly updates the second watermark 212b to reflect that it has sent data 154a~c to the second application 170b. Unsent data 154 remains in the buffer 210 until it indicates that the application 170 can consume or receive data 154 (i.e., data 154c~d remains in the first buffer 210a and data 154d remains in the second buffer 210b). Therefore, the database client 160 propagates data 154 to a different sink (i.e., application 170) without relying on a connection to database 156, even though the sinks have different data consumption rates.
[0027] Referring here to Figure 3A, in some embodiments, when the database client 160 sends each data 154 of each write 152 to the database 156 from the buffer 210 to the application 170, based on the order in which the respective data 154 of each write 152 sent to the database 156 by the database client 160, the database client 160 uses the order indicator 222 to determine that an acknowledgment 220 of a preceding write 152 has been received, indicating that the respective data 154 of the preceding write 152 has been committed to the database 156, immediately before each data 154 of the respective write 152. For example, the database client 160 determines that an acknowledgment 220 of a preceding write has been received by determining the level of the watermark 212 in the buffer 210.
[0028] For example, as shown in schematic diagram 300a, database client 160 receives four writes 152a-d and queues the corresponding data 154a-d in two buffers 210a-b of two applications 170a-b. Database client 160 sends data 154a-d to database 156. In this example, database client 160 receives confirmations 220a, b, and d that data 154a, b, and d have been committed, but for unknown reasons (e.g., network congestion or malfunction), does not receive confirmation that data 154c has been committed. In this scenario, database client 160 sends data 154a-b from the first buffer 210a to the first application 170a, and data 154a-b from the second buffer 210b to the second application 170b. This is to reflect the order in which data 154 was committed to database 156. However, despite receiving confirmation 220d that data 154d will be committed to database 156, database client 160 does not send data 154d to applications 170a~b. This is because database client 160 determines (for example, based on sequence indicator 222) that there may be data 154 (i.e., data 154c) that was committed before data 154d for which database client 160 has not received confirmation.
[0029] Referring here to Figure 3B, in some embodiments, the database client 160 waits for confirmation of data 154c so that the data 154 is propagated to the application 170 in the correct order (i.e., the order in which the database 156 committed the data 154). For example, the database client 160 uses the order indicator 222 to determine that no confirmation 220 for a preceding write 154 has been received, and after determining that no confirmation 220 for a preceding write 154 has been received, it receives confirmation 220 that each of the data 154 for the preceding write 152 has been committed to the database 156. In response to receiving confirmation 220 that each of the data 154 for the preceding write 152 has been committed to the database 156, the database client 160 may send each of the data 154 for the preceding write 152 from the buffer 210 to the corresponding application 170. After sending each of the preceding writes 152 data 154, the database client 160 may send each of the subsequent writes 152 data 154 from the buffer 210 to the application 170.
[0030] For example, continuing with the example in Figure 3A, after sending data 154a~b to applications 170a~b, database client 160 receives confirmation 220c that data 154c has been committed to database 156. Thus, in this example, data 154a~d are committed to database 156 in a different order than the order in which confirmations 220a~d are received by database client 160. After receiving confirmation 220c, database client 160 sends the remaining buffered data 154c~d from each buffer 210a~b to the corresponding applications 170a~b. In this example, applications 170a~b consume data at the same rate, as discussed with respect to Figure 2, although application 170 may consume data at different rates, and database client 160 relies on buffer 210 to buffer data 154 until all of data 154 has been consumed by application 170.
[0031] Referring now to Figure 3C and continuing the example from Figure 3A, in some embodiments, the database client 160 is unable to receive a missing data confirmation 220c within the threshold period. In these embodiments, the database client may include a fallback controller 310, which retrieves a change log 330 from the database 156 after determining that the threshold period has been met (e.g., at least 5ms have elapsed). For example, in response to the threshold period being met, the fallback controller 310 sends a change log request 320 requesting the change log 330. From the change log 330, the fallback controller 310 may determine that each data 154 has been committed to the database 156. In this example, the change log 330 reveals that data 154c has been committed to the database 156, and the database client 160 sends the remaining buffered data 154c-d from each buffer 210a-b to the corresponding applications 170a-b.
[0032] In the example shown, the sequence indicator 222 is an increment identifier, such as a monotonically increasing integer. Some databases 156 (e.g., SQL databases) use such a scheme with commitment confirmations. Such a scheme allows the database client 160 to easily determine which data 154 was committed to database 156 immediately before each write 152, so that the data 154 can be properly ordered to application 170. For example, if database client 160 receives a confirmation 220 for a first write 152 containing sequence identifier "4", database client 160 may determine that the data 154 associated with the confirmation 220 for a second write 152 containing sequence identifier "3" was committed to database 156 immediately before the first write 152, and therefore the data 154 for the second write 152 needs to be propagated to application 170 before the data 154 for the first write 152. Database 156 does not need to commit the data in the order in which it is received from database client 160. The database 156 may provide an indication (in addition to the sequence indicator 222) of which data 154 is committed by the confirmation 220 (using, for example, a description, identifier, hash, etc.) to enable the database client 160 to match the confirmation 220 with specific data 154.
[0033] Some databases 156 may not include an increment identifier and instead include a timestamp as an order indicator 222. In this scenario, the database client 160 needs to determine the order in which to propagate the data 154 based on the timestamp received in confirmation 220. In some embodiments, the database client 160 uses the timestamps to generate a linked list that orders a series of writes 152 in the order in which each data 154 of each write 152 was committed to the database 156. For example, the database client 160 generates a hash map of the data 154, which may be a link to the ordered linked list. The database client 160 may insert the data 154 and / or writes 152 into the hash map using a pointer to the immediately preceding write timestamp. By maintaining a pointer to the most recent data 154 propagated to the application 170, the database client 160 buffers the data 154 in the hash map until it is ready to propagate. In some examples, the database client 160 modifies the schema of the database 156 to help determine the order relative to the timestamps. For example, a modified schema may cause each write to update a field in a different row of the database using the exact same timestamp as the timestamp of confirmation 220. In this example, the database client 160 may perform a "stale read" of the modified row (for example, at the point immediately preceding the timestamp) to determine the row's previous state. Such stale reads tend to be computationally very inexpensive (i.e., much less expensive than directly pulling data from the change log table).
[0034] The examples herein show a buffer 210 that stores data 154 until a commitment confirmation 220 is received, but other buffers or storage locations may be used similarly. For example, data 154 may initially be stored in a first storage location (e.g., a first buffer) until a confirmation 220 is received. Once a confirmation 220 is received that data 154 is committed, the data 154 can be moved to buffer 210, which may mitigate the difficulty of purging uncommitted data 154. In this scenario, data 154 is provided from buffer 210 to application 170 with the same speed at which application 170 can consume data 154, since all of the data 154 in buffer 210 has already been committed.
[0035] Figure 4 is a flowchart of an exemplary operational configuration for Method 400, which provides a low-latency demultiplexer for propagating data to multiple sinks. When the computer implementation Method 400 is executed by the data processing hardware 144, it causes the data processing hardware 144 to perform an operation. In operation 402, Method 400 includes receiving a series of writes 152, each requesting that the respective data 154 be stored in a database 156 that communicates with the data processing hardware 144. For each write 152 in the series of writes 152, in operation 404, Method 400 includes storing the respective data 154 in a first buffer 210 associated with a first data streaming application 170 and storing the respective data 154 in a second buffer 210 associated with a second data streaming application 170. In operation 406, Method 400 includes sending the respective data 154 to the database 156. Method 400 includes, in operation 408, receiving a confirmation 220 from database 156 that each data 154 of each write 152 has been committed to database 156. The confirmation 220 includes an order indicator 222 indicating the order in which each data 154 of each write 152 was committed to database 156 for other writes 152 in a series of writes 152.In response to receiving confirmation 220 that each data 154 of each write 152 has been committed to the database 156, the method 400 includes, in operation 410, sending each data 154 of each write 152 from a first buffer 210 to a first data streaming application 170 based on an order indicator 222 indicating that each data 154 has been committed to the database 156 for the other writes 152 in a series of writes 152, and sending each data 154 of each write 152 from a second buffer 210 to a second data streaming application 170 based on an order indicator 222 indicating that each data 154 has been committed to the database 156 for the other writes 152 in a series of writes 152.
[0036] Figure 5 is a schematic diagram of an exemplary computing device 500 that can be used to implement the systems and methods described in this document. The computing device 500 is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The components shown herein, their connections and relationships, and their functions are for illustrative purposes only and are not intended to limit the embodiments of the invention described and / or claimed in this document.
[0037] The computing device 500 includes a processor 510, memory 520, a storage device 530, a high-speed interface / controller 540 connected to memory 520 and a high-speed expansion port 550, and a low-speed bus 570 and a low-speed interface / controller 560 connected to storage device 530. Each component 510, 520, 530, 540, 550, and 560 is interconnected using various buses and may be mounted on a common motherboard or otherwise present as needed. The processor 510 processes instructions for execution within the computing device 500, including instructions stored in memory 520 or storage device 530, to display graphical information of a graphical user interface (GUI) on an external input / output device such as a display 580 connected to the high-speed interface 540. In other embodiments, multiple processors and / or multiple buses may be used as needed, along with multiple memories and memory types. Also, multiple computing devices 500 may be connected, with each device providing a required portion of the operation (e.g., as a server bank, a group of blade servers, or a multiprocessor system).
[0038] Memory 520 stores information non-temporarily within the computing device 500. Memory 520 may be a computer-readable medium, a volatile memory unit, or a non-volatile memory unit. Non-temporarily stored memory 520 may be a physical device used to temporarily or permanently store programs (e.g., instruction sequences) or data (e.g., program state information) for use by the computing device 500. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electronically erasable programmable read-only memory (EEPROM) (e.g., typically used for firmware such as boot programs). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase-change memory (PCM), and disk or tape.
[0039] The storage device 530 can provide high-capacity storage to the computing device 500. In some embodiments, the storage device 530 is a computer-readable medium. In various different embodiments, the storage device 530 may be a device array including floppy disk devices, hard disk devices, optical disk devices, or tape devices, flash memory or other similar solid-state memory devices, or devices in a storage area network or other configuration. In additional embodiments, the computer program product is tangibly embodied in an information carrier. The computer program product includes instructions that perform one or more of the methods described above at runtime. The information carrier is a computer-readable medium or machine-readable medium such as memory 520, the storage device 530, or memory on the processor 510.
[0040] The high-speed controller 540 manages the bandwidth-intensive operation of the computing device 500, and the low-speed controller 560 manages the low-bandwidth-intensive operation. Such role assignments are merely examples. In some embodiments, the high-speed controller 540 is coupled to memory 520, a display 580 (e.g., via a graphics processor or accelerator), and a high-speed expansion port 550 that can accept various expansion cards (not shown). In some embodiments, the low-speed controller 560 is coupled to a storage device 530 and a low-speed expansion port 590. The low-speed expansion port 590, which may include various communication ports (e.g., USB, Bluetooth®, Ethernet®, Wireless Ethernet), may be connected to one or more input / output devices such as a keyboard, pointing device, scanner, or network devices such as switches or routers, for example, via a network adapter.
[0041] The computing device 500 can be implemented in many different forms, as shown in the figure. For example, it may be implemented as a standard server 500a, or multiple times within a group of such servers 500a, as a laptop computer 500b, or as part of a rack server system 500c.
[0042] Various embodiments of the systems and technologies described herein can be realized in digital electronic and / or optical circuits, integrated circuits, specially designed ASICs (application-specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs executable and / or interpretable on a programmable system comprising at least one programmable processor, at least one input device, and at least one output device, which may be specialized or general-purpose, coupled to receive data and instructions from and transmit data and instructions to a storage system.
[0043] A software application (i.e., a software resource) can refer to computer software that causes a computing device to perform a task. In some examples, a software application may be called an “application,” “app,” or “program.” Exemplary applications include, but are not limited to, system diagnostic applications, system administration applications, system maintenance applications, word processing applications, spreadsheet applications, messaging applications, media streaming applications, social networking applications, and game applications.
[0044] These computer programs (also known as programs, software, software applications, or code) include machine instructions for a programmable processor and may be implemented in high-level procedural and / or object-oriented programming languages and / or assembly / machine languages. As used herein, the terms “machine-readable medium” and “computer-readable medium” refer to any computer program product, non-transient computer-readable medium, apparatus and / or device (e.g., magnetic disks, optical disks, memory, programmable logic devices (PLDs)) used to provide machine instructions and / or data to a programmable processor that includes a machine-readable medium that receives machine instructions as machine-readable signals. The term “machine-readable signal” refers to any signal used to provide machine instructions and / or data to a programmable processor.
[0045] The processing and logic flows described herein can be performed by one or more programmable processors, also called data processing hardware, executing one or more computer programs to act on input data and produce outputs. Processing and logic flows can also be performed by special-purpose logic circuits, such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits). Processors suitable for executing computer programs include, for example, one or more processors in both general-purpose and special-purpose microprocessors, and in either type of digital computer. Generally, processors receive instructions and data from read-only memory, random-access memory, or both. The basic elements of a computer are a processor for executing instructions, and one or more memory devices for storing instructions and data. Generally, a computer also includes one or more mass storage devices for storing data, such as magnetic disks, magneto-optical disks, or optical disks, or is operablely connected to them to receive data from them, or to transmit data to them, or both. However, a computer is not required to have such devices. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, such as semiconductor memory devices including EPROM, EEPROM, and flash memory devices; magnetic disks such as internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. Processors and memory may be complemented by or incorporated into dedicated logic circuits.
[0046] To provide user interaction, one or more embodiments of this specification may be implemented on a computer having, for example, a display device such as a CRT (cathoderay tube), LCD (liquid crystal display) monitor, or touchscreen for displaying information to the user, and optionally a keyboard and a pointing device such as a mouse or trackball, by which the user can provide input to the computer. Other types of devices may also be used to provide user interaction, for example, the feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or haptic feedback, and input from the user may be received in any form, including acoustic, speech, or haptic input. Furthermore, the computer may interact with the user by sending and receiving documents to and from the user's device, for example, by sending a web page to a web browser on the user's client device in response to a request received from a web browser.
[0047] Several embodiments have been described. Nevertheless, it is understood that various modifications can be made without departing from the spirit and scope of this disclosure. Accordingly, other embodiments are within the scope of the following claims.
Claims
1. A computer execution method (400) performed by the data processing hardware (144) to enable the data processing hardware (144) to operate, The aforementioned operation is, The operation includes receiving a series of writes (152), each write (152) in the series of writes (152) requests that its respective data (154) be stored in a database (156) that communicates with the data processing hardware (144), and the operation is as follows: For each of the writes (152) in the series of writes (152) mentioned above, Each of the aforementioned data (154) is stored in a first buffer (210) associated with a first data streaming application (170), The respective data (154) are stored in a second buffer (210) associated with a second data streaming application (170), Each of the aforementioned data (154) is transmitted to the database (156), The operation includes receiving a confirmation (220) from the database (156) that the respective data (154) of each of the writes (152) has been committed to the database (156), the confirmation (220) including an order indicator (222) indicating the order in which the respective data (154) of each of the writes (152) was committed to the database (156) for other writes (152) in the series of writes (152), and further the operation includes, For each of the aforementioned writes (152) within the series of writes (152), In response to receiving the confirmation (220) that the respective data (154) of each of the aforementioned writes (152) has been committed to the database (156), Based on the sequence indicator (222) which indicates the order in which each of the aforementioned data (154) is committed to the database (156) relative to other writes (152) in the series of writes (152), the respective data (154) of each of the respective writes (152) is transmitted from the first buffer (210) to the first data streaming application (170), Based on the sequence indicator (222) which indicates the order in which each of the aforementioned data (154) is committed to the database (156) relative to other writes (152) in the series of writes (152), the respective data (154) of each of the respective writes (152) is transmitted from the second buffer (210) to the second data streaming application (170), Methods that include...
2. Sending the respective data (154) of each write (152) to the first data streaming application (170) from the first buffer (210) based on the order in which the respective data (154) of each write (152) in the series of writes (152) was committed to the database (156) includes determining, using the order indicator (222), that the confirmation (220) of the write (152) preceding each write (152) has been received, and that the respective data (154) of each write (152) is committed to the database (156) The method according to claim 1 (400), wherein immediately before being committed to (156), the respective data (154) of the write (152) preceding each of the writes (152) is committed to the database (156), and after transmitting the respective data (154) of the writes (152) preceding each of the writes (152) from the first buffer (210) to the first data streaming application (170), the respective data (154) of each of the writes (152) is transmitted from the first buffer (210) to the first data streaming application (170).
3. The method of claim 2 (400), wherein determining that the confirmation (220) of a write (152) preceding each of the aforementioned writes has been received includes determining the level of the watermark (212) of the first buffer (210).
4. Sending the respective data (154) of each write (152) to the second data streaming application (170) from the second buffer (210) based on the order in which the respective data (154) of each write (152) was committed to the database (156) for other writes (152) within the series of writes (152) is: Using the sequence indicator (222), it is determined that the confirmation (220) for the write (152) preceding each of the writes (152) has not been received. After determining that the confirmation (220) for each of the preceding writes (152) has not been received, the confirmation (220) is received that the respective data (154) for each of the preceding writes (152) has been committed to the database (156), In response to receiving the confirmation (220) that the respective data (154) of the write (152) preceding each of the aforementioned writes (152) has been committed to the database (156), The data (154) of the write (152) preceding each of the writes (152) is transmitted from the second buffer (210) to the second data streaming application (170), After transmitting the respective data (154) of the write (152) preceding each of the aforementioned writes (152), the respective data (154) of each of the aforementioned writes (152) is transmitted from the second buffer (210) to the second data streaming application (170), The method according to claim 2 (400), including the method according to claim 2.
5. The method according to claim 1 (400), wherein the sequence indicator (222) includes an increment identifier.
6. The method according to claim 1 (400), wherein the sequence indicator (222) includes a timestamp.
7. The method of claim 6 (400), further comprising using the timestamp of each received confirmation (220) to generate a linked list that orders the series of writes (152) in the order in which the respective data (154) of each write (152) of the series of writes (152) was committed to the database (156).
8. The method according to claim 7 (400), wherein generating the link list includes generating a hash map.
9. Receiving the confirmation (220) from the database (156) that the respective data (154) of each of the writes (152) has been committed to the database (156) means that It is determined that the threshold period has elapsed without receiving the aforementioned confirmation (220), In response to determining that the threshold period has elapsed without receiving the aforementioned confirmation (220), the change log (330) is obtained from the database (156), From the change log (330), it is determined that the respective data (154) of each of the writes (152) has been committed to the database (156), The method according to claim 1 (400), including the method according to claim 1.
10. The method according to claim 1 (400), wherein the database (156) includes a structured query language (SQL) database.
11. Data processing hardware (144), and A system (100) including memory hardware (146) that communicates with the data processing hardware (144), A system wherein the memory hardware (146) stores a program, and when the program is executed by the data processing hardware (144), it causes the data processing hardware (144) to execute the method described in any one of claims 1 to 10.
12. A program that causes the data processing hardware (144) to execute the method described in any one of claims 1 to 10.