Real-time transaction-consistent change notification

Abstract

A method includes executing an initial instance of a change log process (500) for a distributed system (200), each instance of the change log process configured to store a transaction history (132) of transactions (204, 214) executed on the distributed system. The method also includes receiving a transaction request (138) for execution of a corresponding transaction and determining a change log load based on the received transaction request (322). The method includes executing at least one subsequent instance of the change log process when the change log load satisfies a threshold load. When multiple instances of the change log process are executing, the method includes stopping execution of at least one subsequent instance of the change log process and merging a transaction history of the initial instance of the change log process and a transaction history of the at least one subsequent instance of the change log process.

Description

[0001] Case Analysis

[0002] This application is a divisional application of Chinese invention patent application 201780011478.6, filed on April 27, 2017. Technical Field

[0003] This disclosure relates to providing consistent change notifications of data for a monitor subscription to one or more monitors within a distributed system. Background Technology

[0004] Distributed storage systems store data on memory hardware across one or more memory locations. Monitors can issue standby queries to subscribe to specific pieces of data stored on the distributed system and receive change notifications whenever the subscribed data undergoes a mutation. For example, a writer can execute write transactions to modify / mutate specific pieces of data stored on the distributed system, and the monitor subscribed to the data is notified when the data is modified. However, when multiple changes / writes occur simultaneously, the increased demand on the distributed system causes it to overload and fail to stream change notifications in real time. It is known to completely abandon subscriptions during periods of high demand to alleviate the load on the distributed system. For this reason, some systems are designed not to guarantee in-order delivery, and the ordering is often left to the client. However, this results in the subscribed monitor receiving inconsistent snapshots of the distributed data. Furthermore, it is known to use change logs to record each transaction to the data stored on the distributed system. These change logs record every change made to the data store, and when a read transaction occurs on the data store, the change log acts as an on / off checkpoint for all data for a given snapshot. Summary of the Invention

[0005] Using persistent change logs on distributed systems generally requires truncating the change log onto a single machine at regular intervals, leading to bottlenecks while transactions await execution. Therefore, persistent change logs negate the distributed functionality provided by distributed systems due to their inherent scalability limitations during periods of high demand. One aspect of this disclosure provides a method for creating transactionally consistent change notifications. The method includes: executing an initial instance of a change logging process for a distributed system by data processing hardware; receiving a transaction request at the data processing hardware for executing a corresponding transaction on the distributed system; and determining a change log load by the data processing hardware based on the received transaction request. Each instance of the change logging process is configured to store a transaction history of transactions executed on the distributed system on memory hardware communicating with the data processing hardware. When the change log load meets a threshold load, the method includes executing at least one subsequent instance of the change logging process by the data processing hardware. The method also includes determining by the data processing hardware whether multiple instances of the change logging process are currently executing. When multiple instances of a change logging process are executing, the method includes stopping the execution of at least one subsequent instance of the change logging process by data processing hardware and merging the transaction history of the initial instance of the change logging process and the transaction history of at least one subsequent instance of the change logging process by data processing hardware.

[0006] Implementations of this disclosure may include one or more of the following optional features. In some implementations, the method includes storing each transaction history in a change log cache of the memory hardware. Storing each transaction history may include: segmenting the transaction history into data stripes by data processing hardware; copying each data stripe; and storing the data stripes and the copied data stripes at multiple storage locations in the memory hardware. When the change log load does not meet a threshold load after executing at least one subsequent instance of the change log process, the method may include stopping the execution of at least one subsequent instance of the change log process by the data processing hardware, and merging the transaction history of the initial instance of the change log process and the transaction history of at least one subsequent instance of the change log process by the data processing hardware.

[0007] In response to receiving each transaction request, the method may include: obtaining the real time from the data processing hardware; determining a maximum commit time based on the real time from the data processing hardware; and passing the maximum commit time from the data processing hardware to at least one change log cache in the storage hardware. The maximum commit time can specify the maximum time required to complete the execution of the corresponding transaction on the distributed system.

[0008] In some examples, at least one changelog cache sets a notification lock from real time up to the maximum commit time. The notification lock prevents the corresponding changelog cache from notifying one or more subscription monitors for transactions with a corresponding commit time earlier than the maximum commit time. At least one changelog cache can remove the notification lock at the end of the maximum commit time. The removed notification lock allows the corresponding changelog cache to notify one or more subscription monitors for each transaction with a corresponding commit time earlier than the maximum commit time. After passing the maximum commit time, the method may include having the data processing hardware commit the execution of the corresponding transaction on the distributed system. Here, the transaction includes the maximum commit time. The method may then also include: having the data processing hardware obtain a transaction result indicating that the execution of the corresponding transaction on the distributed system is accepted when the corresponding transaction includes a corresponding commit time at or before the maximum commit time; and passing the transaction result from the data processing hardware to at least one changelog cache.

[0009] In some examples, the received transaction result causes at least one changelog cache to record at least one mutation associated with the corresponding transaction. At least one changelog cache may propagate the recorded mutation to one or more monitor agents executing on the data processing hardware. Each monitor agent may be configured to provide notification to the corresponding subscribed monitor for the corresponding transaction. The notification may include the recorded mutation and optionally a consistent snapshot of the transaction history. Here, the consistent snapshot in the notification includes the corresponding transaction as well as any completed transactions with a corresponding commit time earlier than the commit time of the corresponding transaction.

[0010] In some implementations, after transmitting the maximum commit time, the method includes having the data processing hardware commit the execution of the corresponding transaction on the distributed system. Here, the transaction includes the maximum commit time. The method may then include: having the data processing hardware obtain a transaction result indicating that the execution of the corresponding transaction on the distributed system was rejected when the corresponding transaction failed to commit at or before the maximum commit time; and transmitting the transaction result from the data processing hardware to at least one changelog cache. In some examples, the received transaction result causes at least one changelog cache to create a gap in the transaction history between the actual time when the transaction process transmitted the maximum commit time to at least one changelog cache and the maximum commit time.

[0011] In some examples, at least one changelog cache passes gaps in the transaction history to one or more monitor agents executing on the data processing hardware. Each monitor agent can be configured to provide a notification to the corresponding subscribed monitor for the transaction at the end of the maximum commit time, including a cache miss during the gap in the transaction history.

[0012] The method may also include having the data processing hardware determine to restart the change logging process after an offline event, and obtaining the real time at the change logging process executing on the data processing hardware. The method may also include: having the data processing hardware determine the current sequence number based on the real time when the change logging process restarts; and having the data processing hardware wait from the current sequence number for the global maximum commit time delay before updating the transaction history of any transactions executed on the distributed system. Transactions executed on the distributed system before the restart of the change logging process can complete within the global maximum commit time delay. The method may also include sending the merged transaction history to one or more subscription monitors. Each subscription monitor may have a constant query for changes to data stored on the memory hardware.

[0013] Another aspect of this disclosure provides a system for creating transactionally consistent change notifications. The system includes data processing hardware for a distributed system and memory hardware communicating with the data processing hardware. The memory hardware stores instructions that, when executed on the data processing hardware, cause the data processing hardware to perform operations. The operations include: executing an initial instance of a change logging process for the distributed system; receiving a transaction request for executing a corresponding transaction on the distributed system; and determining a change logging load based on the received transaction request. Each instance of the change logging process is configured to store a transaction history of transactions executed on the distributed system on the memory hardware. When the change logging load meets a threshold load, the system includes executing at least one subsequent instance of the change logging process and determining whether multiple instances of the change logging process are executing. When multiple instances of the change logging process are executing, the system includes stopping the execution of at least one subsequent instance of the change logging process and merging the transaction history of the initial instance of the change logging process and the transaction history of at least one subsequent instance of the change logging process.

[0014] This aspect may include one or more of the following optional features. In some implementations, the operation further includes storing each transaction history in a change log cache of the memory hardware. Storing each transaction history may include: splitting the transaction history into data stripes; copying each data stripe; and storing the data stripes and the copied data stripes at multiple storage locations in the memory hardware. The operation may also include: stopping the execution of at least one subsequent instance of the change log process when the change log load does not meet a threshold load after executing at least one subsequent instance of the change log process; and merging the transaction history of the initial instance of the change log process and the transaction history of at least one subsequent instance of the change log process. In response to receiving each transaction request, the operation may include: obtaining the real time; determining the maximum commit time based on the real time; and passing the maximum commit time to at least one change log cache in the memory hardware. The maximum commit time may specify the maximum time required to complete the execution of the corresponding transaction on the distributed system.

[0015] At least one changelog cache can set a notification lock from real time up to the maximum commit time. The notification lock prevents the corresponding changelog cache from notifying one or more subscribed monitors for transactions with a corresponding commit time earlier than the maximum commit time. At least one changelog cache can remove the notification lock at the end of the maximum commit time. The removed notification lock allows the corresponding changelog cache to notify one or more subscribed monitors for each transaction with a corresponding commit time earlier than the maximum commit time.

[0016] In some examples, the operation includes committing the execution of the corresponding transaction on the distributed system after transmitting the maximum commit time. The transaction includes the corresponding commit time. The operation then includes: obtaining a transaction result indicating that the execution of the corresponding transaction on the distributed system was accepted when the corresponding transaction included a corresponding commit time at or before the maximum commit time; and passing the transaction result to at least one changelog cache. The received transaction result may cause at least one changelog cache to record at least one mutation associated with the corresponding transaction. At least one changelog cache may pass the recorded mutation to one or more monitor agents executing on the data processing hardware. Each monitor agent may be configured to provide a notification to the corresponding subscribed monitor of the corresponding transaction, the notification including the recorded mutation. The notification may also include a consistent snapshot of the transaction history. The consistent snapshot may include the corresponding transaction and any completed transactions with a corresponding commit time earlier than the commit time of the corresponding transaction.

[0017] In some implementations, the operation includes committing the execution of the corresponding transaction on the distributed system after the maximum commit time has been passed. Here, the transaction includes the maximum commit time. The operation may then further include: obtaining a transaction result indicating that the execution of the corresponding transaction on the distributed system was rejected when the corresponding transaction failed to commit at or before the maximum commit time; and passing the transaction result to at least one changelog cache. The received transaction result may cause at least one changelog cache to create a gap in the transaction history between the actual time when the transaction process passed the maximum commit time to at least one changelog cache and the maximum commit time. At least one changelog cache may pass the gap in the transaction history to one or more monitor agents executing on the data processing hardware. Each monitor agent may be configured to provide a notification to the corresponding subscribed monitor of the corresponding transaction at the end of the maximum commit time, the notification including a cache miss during the gap in the transaction history.

[0018] In some examples, the operations also include: determining to restart the changelogging process after an offline event; obtaining the real time; determining the current sequence number based on the real time when the changelogging process restarts; and waiting from the current sequence number for the global maximum commit time delay before updating the transaction history of any transactions executed on the distributed system. Transactions executed on the distributed system before the restart of the changelogging process can complete within the global maximum commit time delay. The operations may also include sending a merged transaction history to one or more subscription monitors, each with a standby query for changes to data stored on the memory hardware.

[0019] Details of one or more embodiments of this disclosure are set forth in the following drawings and description. Other aspects, features, and advantages will be apparent from the specification, drawings, and claims. Attached Figure Description

[0020] Figure 1 This is a schematic diagram of an example system used to provide user devices with access to a non-transitory data storage in a distributed system.

[0021] Figure 2 This is a schematic diagram of an example system used to provide one or more user devices with access to non-transitory data storage in a distributed storage system.

[0022] Figure 3 This is a schematic diagram of an example component for a notification system that provides near real-time notifications for frequently requested queries.

[0023] Figure 4A and Figure 4B Showing through Figure 3 A schematic diagram illustrating example operations performed by the backend of the notification system and at least one change log cache.

[0024] Figure 5 This is a diagram illustrating an example change log process.

[0025] Figure 6 This is a line graph showing the commit time window set by a change log process executed on a distributed system.

[0026] Figure 7 This is a schematic diagram of an example computing device that communicates with memory hardware.

[0027] In the various figures, similar reference numerals indicate similar elements. Detailed Implementation

[0028] Distributed systems with in-memory change logs or change log caches store the transaction history of transactions executed on the distributed system to process monitor subscriptions more quickly and thus provide near-real-time updates as transactions execute. Therefore, the in-memory cache provides increased scalability because a single change log cache stores the transaction history of transactions executed across many (e.g., a thousand) machines in the distributed system. Generally, a given data store on a distributed system can handle more transactions per second than the number of tasks that a change log cache can execute per second. To mitigate the throughput reduction of the change log cache at the fan-in point in processing all transaction requests issued to the data store, implementations include dynamically partitioning the change log cache into multiple change log caches (e.g., multiple instances of a change log process) executing on different computing resources when demand is high. Multiple change log caches and their associated transaction histories can then be opportunistically merged together to maintain a guarantee or consistency of the transaction history for one or more subscribed monitors at the fan-out point.

[0029] refer to Figure 1 and Figure 2In some embodiments, system 100 includes one or more user devices 120a-n associated with user 102. User device 120 communicates via network 132 with distributed storage system 200 having scalable / resilient non-transitory data storage 150. In some embodiments, distributed storage system 200 executes computing device 112 to manage access to data storage 150. User device 120 writes data 202 to data storage 150 when computing device 112 executes write transaction 204, and monitors or reads data 202 from data storage 150 when computing device 112 executes read transaction 214. In some examples, computing device 112 executes write transaction 204 when it receives write access request 128 from user device 120 via network 132. Similarly, computing device 112 executes read transaction 214 when it receives read access request 130 from user device 120 via network 132. In some examples, read access request 130 corresponds to a standby query previously issued by distributed system 200, which allows user device 120 to consistently monitor or read data 202 from data storage 150 without having to issue new read access requests 130 repeatedly.

[0030] In some implementations, the distributed storage system 200 executes a change log process 500 for the transaction history 312 of the transaction 204 to be executed on the distributed system 200. Figure 3 The change log data 202 is stored in a non-persistent change log cache (CLC) 310. In some scenarios, the distributed storage system 200 executes multiple instances of the change log process 500 during periods of high demand, such as when the number of write access requests 128 received by the distributed system 200 increases, thereby increasing the change log load on the distributed system 200. In some implementations, a backend 304 executing on the distributed system 200 can fan in or distribute multiple shards of the mutation data 202 from the corresponding transaction 204 in a non-persistent state across multiple non-persistent CLCs 310, and then a shard manager 318 executing on the distributed system 200 can fan out or merge shards of the mutation data 202 from the multiple CLCs 310 to provide a consistent snapshot of the data storage 150 to one or more user devices 120 interested in these corresponding transactions.

[0031] User device 120 can be any computing device capable of communicating with computing device 112 via network 132. User device 120 includes, but is not limited to, desktop computing devices and mobile computing devices, such as laptops, tablets, smartphones, and wearable computing devices (e.g., headsets and / or watches). User device 120 can also execute a graphical user interface (GUI) 222 on display 122 to write data 202 to data storage 150 and / or read data 202 from data storage 150 of distributed storage system 200. User device 120 may include a writer 320 that sends write access requests 128 to update / change / mutate data 202 stored in data storage 150. Figure 3 ), and also includes sending read access request 130 (e.g., standby query) to obtain the reader 322 of data 202 at any time that data 202 changes or mutates. Figure 3 ).

[0032] Network 132 may include various types of networks, such as Local Area Network (LAN), Wide Area Network (WAN), and / or the Internet. While network 132 can represent a long-distance network (e.g., the Internet or a WAN), in some embodiments, network 132 includes shorter-distance networks, such as a Local Area Network (LAN). In some embodiments, network 132 uses standard communication technologies and / or protocols. Therefore, network 132 may include links using technologies such as Ethernet, Wi-Fi (e.g., 802.11), Global System for Microwave Access Interoperability (WiMAX), 3G, Long Term Evolution (LTE), Digital Subscriber Line (DSL), Asynchronous Transfer Mode (ATM), InfiniBand, PCI Express Advanced Switching, Bluetooth, Bluetooth Low Energy (BLE), etc. Similarly, networking protocols used on network 132 may include Multiprotocol Label Switching (MPLS), Transmission Control Protocol / Internet Protocol (TCP / IP), User Datagram Protocol (UDP), Hypertext Transfer Protocol (HTTP), Simple Mail Transfer Protocol (SMTP), File Transfer Protocol (FTP), etc. Data 132 exchanged over the network can be represented using technologies and / or formats including Hypertext Markup Language (HTML), Extensible Markup Language (XML), etc. Furthermore, conventional encryption technologies such as Secure Sockets Layer (SSL), Transport Layer Security (TLS), Virtual Private Network (VPN), and Internet Protocol Security (IPsec) can be used to encrypt all or some of these links. In other examples, instead of or in addition to the data communication technologies described above, network 132 may also use custom and / or dedicated data communication technologies.

[0033] refer to Figure 2In some implementations, the distributed storage system 200 includes loosely coupled memory hosts 110, 110a-n (e.g., computers or servers), each memory host having computing resources 112 (e.g., one or more processors or central processing units (CPUs)) that can be used to cache data, such as storage resources 114 (e.g., memory hardware, flash memory, dynamic random access memory (DRAM), phase-change memory (PCM), and / or disks). Data storage 150 (e.g., storage abstraction) overlaid on storage resources 114 allows scalable use of storage resources 114 by one or more user devices 120, 120a-n. User devices 120 can communicate with memory hosts 110 via network 132 (e.g., via remote procedure calls (RPC)).

[0034] In some implementations, the distributed storage system 200 is "one-sided," thereby eliminating the need for any server job on its corresponding memory host 110 to respond to RPCs from user device 120 that write data 202 during write transaction 204 or read data 202 during read transaction 214. Instead, dedicated hardware can be relied upon to handle remote write access request 128 and read access request 130 separately. "One-sided" refers to a method that can be used to perform most request processing on memory host 110 in hardware rather than through software executing on CPU 112 of memory host 110. Instead of having the processor 112 of memory host 110 (e.g., a server) execute a server process 118 that routes access to the corresponding storage resource 114 (e.g., non-transitory memory) to user process 126 executing on user device 120, user device 120 can directly access storage resource 114 through network interface controller (NIC) 116 of memory host 110. Additional ideas and features relating to a single-sided distributed caching system can be found in U.S. Patent 9,164,702, which is hereby incorporated in its entirety by reference.

[0035] Distributed storage system 200 can place data 202 in the memory hardware 114 of a remote memory host 110 (e.g., data storage 150) and retrieve data 202 from the remote memory host 110 via RPC or via a network interface controller 116 supporting Remote Direct Memory Access (RDMA). The network interface controller 116 (also referred to as a network interface card, network adapter, or LAN adapter) can be a computer hardware component that connects computing devices / resources 112 to a network 132. Memory hosts 110a-n and user devices 120 can each have a network interface controller 116 for network communication. A host process 118 executing on the computing processor 112 of the memory host 110 can allocate memory locations 250a-n when performing a write transaction 204 to write updated data 202.

[0036] In some implementations, one or more user devices 120 (e.g., monitors 322) issue a read access request 130 associated with a standby query once via distributed system 200 to subscribe to a specific piece of data 202. The standby query (e.g., read access request 130) allows the corresponding user device 120 (e.g., subscribed monitor 322) to consistently monitor or read data 202 from data storage 150 without having to repeatedly issue new read access requests 130. Therefore, whenever distributed system 200 receives a write access request 128 to perform a write transaction 204 on distributed system 200 to modify data 202 in data storage 150, distributed system 200 streams the modified / mutated data 202 to one or more monitors 322 that have subscribed to data 202.

[0037] In some implementations, distributed system 200 streams updated data 202 to one or more subscription monitors 322 in real time and can scale based on the number of transaction requests that distributed system 200 is currently receiving at a given point in time. For example, distributed system 200 can provide a consistent snapshot (e.g., transaction history 312) at a given moment when multiple mutations in data 202 in data storage 150 cause it to crash under high load (i.e., distributed system 200 is receiving a large number of write access requests 128). This scalability allows distributed system 200 to stream updated data 202 to subscription monitors 322 as close to real time as possible without having to abandon streaming data 202 to monitors 322 due to overload of distributed system 200. Thus, when the execution of write transaction 204 completes successfully, read transaction 214 through monitor 322 reflects the current consistent state of data storage 150, such that monitor 322 receives the most recent copy of data 202 modified by write transaction 204. Monitor 322 receives a single, consistent snapshot of data at a given moment that may not reflect every change that has occurred since the last snapshot.

[0038] Figure 3 This is a block diagram of an example implementation of components of a notification system 300 for sending notifications 350 to one or more monitors 322 that subscribe to data 202 stored in data storage 150. As used herein, a monitor 322 may subscribe to specified data 202 by issuing a standby query (e.g., a read access request 130), such that changes to the specified data 202 in data storage 150 are streamed to the monitor 322 as notifications 350 in near real-time. The notification system 300 includes a distributed system 200 communicating via a network 132 with remote user devices 120 associated with one or more remote writers 320, 320a-n and one or more remote monitors 322, 322a-n. The distributed system 200 includes a data storage backend 304 and a shard manager 318 executed on a computing device 112, and data storage 150 of memory hardware 114.

[0039] In some implementations, computing device 112 executes instructions for a change log process 500, which provides a non-persistent change log cache (CLC) 310 within memory 114 for storing a portion of the transaction history 312 of transactions 204 executed on distributed system 200 in a non-persistent state. The CLC 310 can be partitioned within memory 114 by comparing it with a persistent change log that records each change made to data 202 within data storage 150, allowing computing device 112 to scale the CLC 310 up or down by executing additional instances of the change log process to provide two or more CLCs 310a-n. In this way, change log process 500 can also stop the execution of one or more instances based on the current load on distributed system 200 at a given time. Without the scalability provided by the non-persistent CLC 310, a higher number of write access requests 128 would cause a bottleneck effect, with each write transaction 204 waiting to execute at its corresponding location within data storage 150. Generally, a single data store 150 is assigned a corresponding CLC 310 for recording changes to that data store 150. However, multiple data stores 150 can be assigned to the same CLC 310. For example, a notification system 300 can include one million data stores 150 and one hundred CLCs 310, such that each CLC 310 will record changes to one thousand data stores 150. In some examples, it is necessary to scale up the number of CLCs 310 when additional data stores 150 are added to handle increased load on the distributed system 200.

[0040] In some examples, computing device 112 (e.g., data processing hardware) executes one or more monitor agents 314, 314a-n, each agent comprising a stateful machine with a separate subscription to monitor 322. For example, each monitor 322 has a one-to-one connection to a corresponding monitor agent 314. Therefore, each monitor agent 314 multiplexes between the corresponding monitor 322 and one or more CLCs 310, providing the corresponding monitor agent 314 with a mutation or cache miss. In a scenario where a cache miss occurs, monitor agent 314 can query backend 700 352 to retrieve / obtain data 202 from data store 202.

[0041] In some implementations, whenever a new write transaction 204 commits to update data 202 within data storage 150, backend 304 performs transaction processing to update transaction history 312 stored in CLC 310. The transaction process may use CLC 310 to maintain a cache of ordered change logs (e.g., transaction history 312) during fan-in phase 301. In some examples, the transaction process segments the cache of ordered change logs maintained by CLC 310 into stripes of data 202, copies each stripe of data 202, and stores the data stripes and the copied data stripes at multiple storage locations 250 on memory hardware 114. When backend 304 receives a write access request 128 from remote writer 320, backend 304 can obtain the real time (TT) from real time clock 316 and determine the maximum commit time TD based on that TT. Max TD Max Specifies the maximum time required to commit the corresponding write transaction 204 to execution on distributed system 200. For example, TD Max Force the upper limit of the time limit for distributed system 200 to commit the corresponding transaction 204. Therefore, if distributed system 200 reaches TD Max If transaction 204 cannot be committed, the distributed system will abort the corresponding transaction 204.

[0042] In some examples, during the fan-in phase 301, the backend 304 sends a prepare lock message 330 to the CLC 310 before executing a write transaction 204 (e.g., commit write) to modify / mutate / alter data 202 within the data store 150. The prepare lock message 330 includes a TD for the corresponding pending transaction 204. Max And make CLC 310 submit within the 610-minute time window ( Figure 6 During this period, a notification lock is set. Here, the commit time window 610 corresponds to the period from TT to TD when CLC 310 receives the ready lock message 330. Max The time window. Notification locking prevents CLC 310 from processing submissions earlier than TD during the submission time window 610. Max And any completed transaction 204 that is later than the start time of the pending transaction will notify one or more monitors 322. As used herein, the term "commit time" refers to the time when the execution of the corresponding write transaction 204 is completed. After sending the Ready Lock message 330 to CLC 310, the backend 304 commits the execution of the write transaction 204 to update the data 202 in the data store 150.

[0043] In some implementations, backend 304 obtains transaction result 332 after the corresponding transaction 204 is committed and provides transaction result 332 to CLC 310. In some examples, transaction result 332 indicates when the corresponding transaction 204 is included in TD. Max The execution of the corresponding transaction 204 was accepted at a previous commit time. In these examples, transaction result 332 may also include at least one mutation of the data 202 produced by the corresponding transaction 204. In other examples, transaction result 332 indicates whether the corresponding transaction 204 failed or data storage 150 was not completed before TD. Max When the corresponding transaction cannot be completed / committed, the execution of the corresponding transaction 204 is rejected. Therefore, as used herein, transaction result 332, indicating that transaction 204 is "rejected," can refer to the transaction in TD. Max Transaction 204, which was previously formally aborted and whose consequences are known, or which can refer to the situation where transaction 204 leads to TD. Max The result of transaction 204 is unknown as it was not committed at the time.

[0044] Backend 304 may provide CLC 310 with a lock release message 334 after committing write transaction 204. Lock release message 334 includes the transaction result 332 of the corresponding transaction 204 and causes CLC 310 to release / remove the notification lock. For example, CLC 310 may remove the notification lock immediately upon receiving lock release message 334. In an implementation where transaction result 332 indicates that the corresponding transaction has been accepted, lock release message 334 causes CLC 310 to record at least one mutation in data 202 associated with the corresponding transaction 204. Conversely, lock release message 334 causes CLC 310 to create a gap in transaction history 312 during the commit time window 610 of the corresponding transaction 204. The gap created by CLC 310 indicates that transaction 204 is pending during that gap, but the outcome of whether transaction 204 has been fully committed is unknown.

[0045] In some implementations, one or more monitor agents 314 execute during the fan-out phase 302 of a transaction process to receive transaction history 312 from CLC 310. Transaction history 312 includes any transaction 204 executed on distributed system 200 with a corresponding commit time earlier than the oldest prepare lock message used for the corresponding transaction 204. In some examples, CLC 310 transmits at least one mutation of data 202 when the corresponding transaction 204 is accepted by recording a mutation of data 202 in transaction history 312 and providing transaction history 312 to monitor agent 314. In these examples, each monitor agent 314 includes the mutation of data 202 in a notification 350 and provides notification 350 to the corresponding subscribed monitor 322. In other examples, when the corresponding transaction 204 is rejected, CLC 310 provides monitor agents 314 with a gapped transaction history 312, causing each monitor agent 314 to push a cache miss in notification 350 and provide notification 350 to the corresponding subscribed monitor 322. Here, a cache miss reflects the unknown result of an pending transaction 204 during the gap in transaction history 312. In some examples, monitor broker 314 queries backend 304 352 to obtain a snapshot of data store 150 during the gap in transaction history 312 to determine the result of pending transaction 204 and provides that result to the corresponding subscribed monitor 322. Conversely, when the corresponding transaction 204 is due to a cache miss, the cache miss reflects the unknown result of pending transaction 204 during the gap in transaction history 312. Max If a formal abort is previously rejected, the corresponding transaction 204 is simply omitted from the transaction history 312. Furthermore, when a new monitor 322 connects to the distributed system 200, the corresponding monitor agent 314 can integrate all changes in the data store 150 that affect the standby queries associated with the monitor 322, enabling the monitor 322 to receive a consistent snapshot of the data store 150.

[0046] In cases where a single CLC 310 lacks sufficient throughput to record all mutations as the number of transactions 204 submitted to the data storage 150 increases, the mutations of data 204 can be fragmented across multiple CLCs 310. In some implementations, a fragment manager 318 merges transaction histories 312 from multiple CLCs 310, each of which includes one or more corresponding fragments of the mutated data 204 to satisfy a given standby query (e.g., read access request 130) from one or more monitors 322.

[0047] Figure 4A and Figure 4B Figures 400a and 400b provide example operations performed via the backend 304 of the distributed system 200 and at least one change log cache 310. See also... Figure 3The notification system 300 is used to describe Figures 400a and 400b. The vertical y-axis indicates time increasing from top to bottom. At time 1, the backend 304 receives a transaction request (e.g., write access request 128) from the remote writer 320. The remote writer 320 may send a transaction request to perform a write transaction 204 to mutate data 202 in the data storage 150 stored on the memory hardware 114. At time 2, the backend 304 assigns the corresponding transaction 204 as pending. Here, the backend 304 obtains TT from the real time clock 316 and determines TD for the corresponding transaction 204. Max A backend 304 can create a TD (Time Deposit) field. Max The preparation lock message 330.

[0048] At time 3, backend 304 sends a prepare lock message 330 to CLC 310, and at time 4, CLC 310 retrieves TT from the real time clock 316 in response to receiving the prepare lock message 330. At time 5, CLC 310 compares the TT at the time the prepare lock message 330 was invoked (e.g., at time 3) with the TD used for the corresponding transaction 204. Max A notification lock is set during the commit time window 610. The notification lock prevents CLC 310 from notifying one or more monitors 322 of any completed transaction 204 that falls within the commit time window 610. Subsequently, at time 6, backend 304 commits the execution of the corresponding transaction 204 to modify (e.g., mutate) the data 202 within data store 150.

[0049] At time 7, data storage 150 returns the commit time to backend 304, and backend 304 obtains transaction result 332 based on the returned commit time. Figure 4A Shown in TD Max Alternatively, the commit time returned at time 7 can be used to provide transaction result 332 indicating that the corresponding transaction 204 has been accepted. In this scenario, at time 8, the backend 304 notifies the writer 320 that transaction 204 has been accepted, and at time 9, at least one mutation of data 202 produced by the corresponding transaction 204 is provided to the CLC 310. At time 10, the CLC releases / removes the notification lock and records at least one mutation of data 202. By notifying the writer 320 that transaction 204 has been accepted at time 8, the writer 320 is notified of a successful write transaction 204 before it can consider the result of transaction 204 as a subscription monitor 322. However, in other scenarios, the backend 304 may notify the writer 320 that the transaction has been accepted at time 10 while the CLC records at least one mutation of data 202. In these scenarios, it is guaranteed that at least one mutation of data 202 is recorded before the call to the writer 320 returns.

[0050] on the other hand, Figure 4B This shows that transaction 204 corresponds to TD. Max Failed to complete / commit or transaction 204 in TD Max This refers to a scenario where a transaction was previously formally aborted but was rejected. For example, at time 7, data storage 150 could return a 304 error to backend 304 indicating that transaction 204 failed due to TD. Max The transaction was rejected or indicated as 204 due to failure to submit in TD. Max Previously, the transaction result 332 was formally aborted and rejected. At time 8, backend 304 notifies remote writer 320 that transaction 204 has been rejected, and at time 9, backend 304 sends a release lock message 334 to CLC 314. Here, release lock message 334 includes transaction result 332 indicating that the corresponding transaction 204 has been rejected and notifies CLC 310 to release the notification lock. At time 10, CLC 310 releases the notification lock previously set during commit time window 610 and creates a gap associated with the rejected corresponding transaction 204. More specifically, CLC 310 can use TT and TD for the corresponding transaction 204 during commit time window 610 (i.e., when the prepare lock message 330 is invoked (e.g., at time 3)). Max A gap is created in transaction history 312 between transactions. Therefore, the gap created by CLC 310 indicates that transaction 204 is pending during the gap, but the outcome of whether transaction 204 is completed is unknown.

[0051] Figure 5 A sample changelog process 500 is provided, executed on distributed system 200 to provide scalability based on the current changelog load (e.g., demand) on distributed system 200. See also... Figure 3 A notification system 300 describes the change log process 500. Data processing hardware (e.g., computing device 112) executes an initial instance of the change log process 500 to provide an initial non-persistent CLC 310a of a transaction history 312 of transactions (e.g., write transaction 204) executed on the distributed system 200, configured to be stored on memory hardware 114 communicating with the data processing hardware. Therefore, each CLC 310 (e.g., each instance of the change log process 500) is responsible for maintaining a cache of ordered change logs.

[0052] During the fan-in phase 301, the backend 304 sends a ready lock message 330 to CLC 310a in response to receiving a transaction request 128 (e.g., a write access request) from the remote writer 320 for executing the corresponding transaction 204. Here, CLC 310a is guaranteed to TD via a record. MaxEach prepare lock message 330 is processed by entries of the corresponding transaction 204 that were committed at the time but not yet committed in the transaction history 312 (e.g., change log entries). When the backend 304 receives multiple transaction requests 128, the change log load on CLC 310a increases because CLC 310a must process multiple prepare lock messages 330. In some examples, data store 150 is able to process more commits per second than the number of prepare lock messages 330 that each CLC 310 can concurrently process before experiencing throughput loss. For example, CLC 310a may be able to process 10,000 incoming prepare lock messages 330 per second before cache performance degradation, while data store 150 is able to process 300,000 commits per second. Therefore, when the change log load meets (e.g., exceeds) a threshold load, CLC 310a may lack the throughput to process incoming prepare log messages 330. As a result, incoming prepare lock messages 330 may become a bottleneck while waiting for processing at the corresponding CLC 310. Here, the threshold load corresponds to the maximum number of ready lock messages 330 per second that the CLC 310a can process without experiencing cache performance degradation. The data processing hardware can determine the change log load based on the number of transaction requests (e.g., write access requests 128) received at a given time.

[0053] At operation 502, when the corresponding transaction 204 is successfully executed, the backend 304 obtains the transaction result 332 from the data store 150, which includes the mutation data 202. As explained above, when the change log load meets the threshold load, CLC 310a may not be able to record all changes that occur at the data store 150. To provide scalability for caching the transaction history 312 in the memory hardware 114 when the change log load meets the threshold load, the data processing hardware at operation 504 segments the mutation data 202 from the transaction result 332 into multiple shards 202a-d and executes at least one subsequent instance of the change log process 500, which provides at least one subsequent non-persistent CLC 310b-d for storing the shards 202a-d of the mutation data 202. As with the initial CLC 310a, each subsequent CLC 310a-d is configured to store the corresponding transaction history 312 of the transaction executed on the distributed system 200. For example, Figure 5 Three subsequent instances of changelog process 500 executed at operation 504 are shown to provide multiple CLC 310a, 310b, 310c, 310d data processing hardware. This data processing hardware can implement a "sharder" component to monitor changelog load and, if overloaded, decide to split mutation data 202 into multiple shards 202a-d.

[0054] During the fan-out phase 302, each CLC 310a-d (e.g., each instance of the change log process 500 executing on distributed system 200) can provide the corresponding transaction history 312a, 312b, 312c, 312d to the shard manager 318 executing on distributed system 200. Here, each transaction history 312a-d indicates one or more shards 202a-d of the mutation data 202 of the corresponding transaction 204 that have been accepted.

[0055] At operation 506, while multiple instances of the change log process (e.g., multiple CLCs 310a-d) are executing, the shard manager 318 stops the execution of at least one subsequent instance of the change log process 500 and merges shards 202a-d containing the mutation data 202 in the transaction history 312 of the initial instance (e.g., CLC 310a) and the transaction history 312 of at least one subsequent instance (e.g., CLC 310b-d) of the change log process 500. As described above, the data processing hardware can implement a "sharder" component to decide to merge shards 202a-d of the mutation data 202 due to underloading of the change log load. For example, Figure 5 The diagram illustrates transaction histories 312a-d fan-out from multiple CLCs 310a-d into a shard manager 318, which merges data shards 202a-d from the transaction histories 312 of the multiple CLCs 310a-d to provide a consistent snapshot of the transaction histories 312 at a given moment to a subscribed monitor 322. Here, the consistent snapshot of the transaction histories 312 can provide one or more batches of mutation data 202 of the corresponding accepted transaction 204 and / or one or more batches of gaps in the transaction histories 312 of the corresponding rejected transaction 204. At operation 508, one or more monitor agents 314 send a notification 350 to each subscribed monitor 322 including a consistent snapshot of the transaction histories 312 at a given moment. In the case of gaps in the transaction history, the monitor agent 314 provides a cache miss to the corresponding remote monitor 322. The monitor agent 314 can determine whether multiple instances of the change log process 500 are executing during the fan-out phase 302.

[0056] In some implementations, one or more monitor agents 314 provide a recovery token to monitor 322 along with each batch of mutation data 202 included in a consistent snapshot of the transaction history 312 at a given moment. The recovery token is globally valid and long-lived across all tasks, allowing its future use. The recovery token may contain a range representing the current state of changes sent to the client and an associated timestamp. Therefore, the use of the recovery token can provide the ability for monitor 322 to be temporarily disconnected from the distributed system, and upon reconnection, to efficiently receive notifications 350 from monitor agents 314 even if the distributed system 200 has lost all states of the temporarily disconnected monitor 322.

[0057] Figure 6 A line graph 600 illustrates a change log process 500 executed by data processing hardware (e.g., computing device 112) on distributed system 200 to update the transaction history 312 stored in at least one CLC 310 whenever a new write transaction 204 commits to update data 202 within data store 150. The horizontal x-axis represents the real time of distributed system 200, where the global sequence number of the real time increases from left to right. The dashed vertical lines 601-607 indicate the corresponding global sequence number during the real time period.

[0058] The first commit time window 610a is set by the corresponding CLC 310 between the virtual vertical lines 601 and 603 for executing the corresponding first transaction 204a on the distributed system 200. Here, the first commit time window 610a is related to the first maximum commit time TC. Max_1 Relatedly, the TC Max_1 Guarantee that the first transaction 204a will not be in TC Max_1 Submit afterwards. Figure 6 This shows TC at time 13 when the actual time equals the value at the imaginary vertical line 603. Max_1 Define the first commit window 610a and when the real time equals 4 o'clock at the virtual vertical line 601, the data processing hardware commits the first transaction 204a.

[0059] Similarly, the second commit time window 610b is set by the corresponding CLC 310 between the virtual vertical lines 602 and 604 for executing the corresponding second transaction 204b on the distributed system 200. Here, the second commit time window 610b is related to the second maximum commit time TC. Max_2 Relatedly, the TC Max_2 Guarantee that the second transaction 204b will not be in TC Max_2 Submit afterwards. Figure 6 This shows TC at 21 o'clock when the actual time equals the position of the imaginary vertical line 604. Max_2The second commit time window 610b is defined, and the data processing hardware commits the second transaction 204b when the real time equals 12 at the virtual vertical line 602. The CLC 310 associated with the first commit time window 610a and the second commit time window 610b can be the same or different.

[0060] The execution of the first transaction 204a precedes the first TC. Max_1 The submission time is complete. For example, Figure 6 The commit time for the first transaction 204a is shown to be equal to 9, and the first TC is also shown. Max_1 The value is 13. Because the first transaction 204a is not locked by another pending transaction 204 with an earlier prepare lock time, the data processing hardware can immediately obtain the first consistent snapshot of transaction history 312 when it receives a commit time when the real time equals 9. When the second transaction 204b commits at the virtual vertical line 602 before the first consistent snapshot of transaction history 312, the second transaction 204b is not fully completed until after the first consistent snapshot. For example, the second transaction 204b includes a commit time equal to 18. Therefore, the first consistent snapshot includes any mutations associated with the first transaction 204a but not the second transaction 204b.

[0061] when Figure 6 It also shows that it includes earlier than the second TC. Max_2 When the second transaction 204b with the corresponding commit time is reached, the data processing hardware takes a second consistent snapshot of the transaction history 312 at a corresponding commit time equal to 18, because the second transaction 204b is not locked by another pending transaction 204 with an earlier prepare lock time. Here, the second consistent snapshot merges the transaction history associated with the two commit time windows 610a, 610b to include the mutations associated with the first transaction 204a and the second transaction 204b. The data processing hardware may execute at least one monitor agent 314 to provide the merging of the transaction history within the second consistent snapshot.

[0062] In some implementations, the data processing hardware determines to restart the change log process 500 after offline event 650. For example, Figure 6 This illustrates that an offline event 650 occurs at virtual vertical line 605 until the changelog process 500 subsequently restarts at virtual vertical line 606 when the real time equals 30. Because the changelog cache does not have persistent state, any state previously held by the changelog cache before offline event 650 is lost when the changelog process 500 restarts. In some examples, the data processing hardware relies on the Global Maximum Commit Time Delay (TCD). MaxThis ensures that all transactions 204 executed on distributed system 200 are completed, so that all subsequent snapshots of the transaction history 312 are consistent after the restart of change log process 500. In other words, TCD Max This ensures that any transaction committed before the change log process restarts (500 error) will return to TCD (Transaction Control Center). Max The time period at which the change log processing ends. The data processing hardware determines the current sequence number based on the real time when change log processing restarts. For example, when change log process 500 restarts, the current sequence number is equal to 30 at the virtual vertical line 606. Thereafter, the data processing hardware waits for the TCD from the current sequence number (e.g., virtual vertical line 606) before updating the transaction history 312 of transaction 204 executed on the distributed system. Max As used in this paper, real time (TT) effectively discretizes time so that a measurement of real time on a given system corresponds to actual time. Therefore, TT is defined as the interval within the range between the lower bound (TTnow_lowerbound) and the upper bound (TTnow_upperbound) of the current time probability. Consequently, the value of TT cannot be directly prepared, and the condition for an event to occur at a second real time (TT2) after a first real time (TT1) is that TT2_lowerbound is greater than TT1_upperbound. Figure 6 TCD is shown Max It equals 20 and occurs between the imaginary vertical lines 606 and 607. Therefore, when TCD Max When the time equals 20, the data processing hardware does not update the transaction history 312 until the virtual vertical line 607 when the real time equals 50.

[0063] In some examples, when the real time is 25 and an offline event 650 occurs during the third commit time window 610c, the third commit time window 610c is set by the corresponding CLC 310. This is because the third commit time window 610c is a third TC known before the occurrence of the offline event 650. Max_3 Upon expiration, as a result of offline event 650 between the virtual vertical lines 605 and 606, the corresponding transaction 204 associated with the third commit time window 610c is lost in CLC 310. Therefore, during the change log process 500, the monitor agent 314 waits for TCD. Max This provides a consistent snapshot of the transaction history 312, including the corresponding transaction 204 associated with the third commit time window 610c. In the event of gaps in the transaction history 312, the monitor agent 314 retrieves a consistent state from the data store during the time frame of those gaps.

[0064] In some implementations, the fourth commit time window 610d is set by the corresponding CLC 310 after the restart event and included in the TCD. Max The fourth TC that occurred before the end Max_4 Assume the corresponding transaction associated with the fourth commit time window 610d arrives at the fourth TC. Max_4 If the submission is successful, then in TCD Max The consistent snapshot of the transaction history 312 at the end merges the transaction history associated with the two commit time windows 610c and 610d to include the mutations associated with their corresponding transactions 204. In contrast, with the TCD... Max The period began, but in TCD Max The corresponding transaction 204 associated with the fifth commit window 610e that expires thereafter will not be included in the TCD. Max In the consistent snapshot at the end, because of the fifth TC associated with the fifth commit time window 610d. Max_5 Occurred in TCD Max after.

[0065] Figure 7 This is a schematic diagram of an example computing device 700 that can be used to implement the systems and methods described in this document, such as computing resource 112 and data storage 150. The computing device 700 is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. The components shown herein, their connections and relationships, and their functions are intended to be exemplary only and are not intended to limit the implementation of the inventions described and / or claimed in this document.

[0066] Computing device 700 includes a processor 710 (i.e., data processing hardware), a memory 720, a storage device 730, a high-speed interface / controller 740 connected to the memory 720 and a high-speed expansion port 750, and a low-speed interface / controller 660 connected to a low-speed bus 770 and the storage device 730. Each of components 710, 720, 730, 740, 750, and 760 is interconnected using various buses and may be mounted on a common motherboard or otherwise, as appropriate. The processor 710 can process instructions that execute within the computing device 700, including instructions stored in the memory 720 or on the storage device 730 to display graphical information for a GUI on an external input / output device, such as a display 780 coupled to the high-speed interface 740. In other embodiments, multiple processors and / or multiple buses, as well as multiple memories and multiple types of memory, may be used as appropriate. Additionally, multiple computing devices 700 may be connected, with each device providing a portion of the necessary operation (e.g., as a server group, a set of blade servers, or a multiprocessor system).

[0067] Memory 720 non-transitory stores information within computing device 700. Memory 720 may be a computer-readable medium, a volatile memory cell, or a non-volatile memory cell. Non-transitory memory 720 may be a physical means for storing programs (e.g., sequences of instructions) or data (e.g., program state information) on a temporary or permanent basis for use by computing device 700. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electronically erasable programmable read-only memory (EEPROM) (e.g., commonly used for firmware, such as boot programs), as well as magnetic disks or magnetic tapes. Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), and phase-change memory (PCM).

[0068] Storage device 730 provides mass storage for computing device 700. In some embodiments, storage device 730 is a computer-readable medium. In various embodiments, storage device 730 may be a floppy disk device, hard disk device, optical disk device, magnetic tape device, flash memory, or other similar solid-state storage device, or an array of such devices, including devices in a storage area network or other configuration. In additional embodiments, a computer program product is tangibly embodied in an information carrier. The computer program product contains instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer or machine-readable medium, such as memory 720, storage device 730, or memory on processor 710.

[0069] High-speed controller 740 manages bandwidth-intensive operations of computing device 700, while low-speed controller 760 manages less bandwidth-intensive operations. This allocation of responsibilities is merely exemplary. In some embodiments, high-speed controller 740 is coupled to memory 720, display 780 (e.g., via a graphics processor or accelerator), and high-speed expansion port 750, which can accept various expansion cards (not shown). In some embodiments, low-speed controller 760 is coupled to storage device 730 and low-speed expansion port 770. Low-speed expansion port 770, which may include various communication ports (e.g., USB, Bluetooth, Ethernet, Wireless Ethernet), may be coupled to one or more input / output devices, such as keyboards, pointing devices, scanners, or networking devices, such as switches or routers, for example via a network adapter.

[0070] As shown in the figure, the computing device 700 can be implemented in many different forms. For example, it can be implemented as a standard server 700a or multiple times in a group of such servers 700a, as a laptop computer 700b, or as part of a rack server system 700c.

[0071] In some implementations, the computing device 700 implementing computing resource 112 communicates with data storage 150 (e.g., in memory 720). Computing resource 112 (executing on data processing hardware 710) executes an initial instance 310 of a change logging process 500 for distributed system 200, each instance 310 of change logging process 500 being configured to store a transaction history 312 of transactions 204 (e.g., write transactions) executed on distributed system 200 on memory hardware 720 communicating with computing resource 112. Computing resource 112 may receive transaction requests and determine change logging load based on the received transaction requests. In some examples, computing resource 112 executes at least one subsequent instance 310 of change logging process 500 when the change logging load meets a threshold load and determines whether multiple instances of change logging process 500 are executing. While multiple instances of the change log process 500 are executing, compute resource 112 can stop the execution of at least one subsequent instance of the change log process 500 and merge the transaction history of the initial instance 310 of the change log process 500 and the transaction history of at least one subsequent instance 310 of the change log process 500. In some examples, compute resource 112 provides a consistent snapshot reflecting the merged transaction history to all subscription monitors 322 of the mutation data.

[0072] A software application (i.e., software resource 110s) can refer to computer software that enables a computing device to perform tasks. In some examples, a software application may be referred to as an "application," "app," or "program." Example applications include, but are not limited to, system diagnostic applications, system management applications, system maintenance applications, word processing applications, spreadsheet applications, messaging applications, media streaming applications, social networking applications, and game applications.

[0073] The non-transitory memory 110hm can be a physical device for storing programs (e.g., sequences of instructions) or data (e.g., program state information) on a temporary or permanent basis for use by the computing device 110hc. The non-transitory memory 110hm can be volatile and / or non-volatile addressable semiconductor memory. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electronically erasable programmable read-only memory (EEPROM) (e.g., commonly used in firmware, such as bootloaders). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase-change memory (PCM), and magnetic disks or magnetic tapes.

[0074] Various implementations of the systems and techniques described herein may be implemented using digital electronic and / or optical circuits, integrated circuits, specially designed ASICs (Application-Specific Integrated Circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementations in one or more computer programs executable and / or interpretable on a programmable system, said programmable system including at least one programmable processor, which may be dedicated or general-purpose, coupled to receive data and instructions from a storage system, at least one input device, and at least one output device, and to transmit data and instructions to the storage system, at least one input device, and at least one output device.

[0075] These computer programs (also referred to as programs, software, software applications, or code) include machine instructions for a programmable processor and can be implemented using high-level procedural and / or object-oriented programming languages ​​and / or assembly / machine languages. As used herein, the terms “machine-readable medium” and “computer-readable medium” refer to any computer program product, non-transitory computer-readable medium, device, and / or apparatus (e.g., disk, optical disk, memory, programmable logic device (PLD)) used to provide machine instructions and / or data to a programmable processor, including machine-readable media that receive machine instructions as machine-readable signals. The term “machine-readable signal” refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0076] The implementations of the subject matter and functional operations described in this specification can be carried out using digital electronic circuits or computer software, firmware, or hardware (including the structures disclosed in this specification and their equivalents), or a combination thereof. Furthermore, the subject matter described in this specification can be implemented as one or more computer program products (i.e., one or more modules of computer program instructions encoded on a computer-readable medium for execution by or control of the operation of a data processing device). The computer-readable medium can be a machine-readable storage device, a machine-readable storage substrate, a memory device, a composition of substances that influence machine-readable propagation signals, or a combination thereof. The terms "data processing device," "computing device," and "computing processor" encompass all devices, apparatuses, and machines for processing data, including, for example, programmable processors, computers, or multiple processors or computers. In addition to hardware, a device may include code that creates an execution environment for the computer program, such as code constituting processor firmware, a protocol stack, a database management system, an operating system, or a combination thereof. Propagation signals are artificially generated signals, such as machine-generated electrical, optical, or electromagnetic signals, which are generated to encode information for transmission to a suitable receiver device.

[0077] A computer program (also referred to as an application, program, software, software application, script, or code) can be written in any programming language, including compiled or interpreted languages, and can be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program does not necessarily correspond to a file in a file system. It may be stored as a part of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to said program, or in multiple coordinating files (e.g., a file storing portions of one or more modules, subroutines, or code). A computer program can be deployed to execute on a single computer or on multiple computers located at a single site or distributed across multiple sites and interconnected via a communication network.

[0078] The processes and logic flows described in this specification can be executed by one or more programmable processors executing one or more computer programs to perform functions by manipulating input data and generating output. The processes and logic flows can also be executed by dedicated logic circuitry, and the device can also be implemented as dedicated logic circuitry, such as an FPGA (Field-Programmable Gate Array) or an ASIC (Application-Specific Integrated Circuit).

[0079] As an example, processors suitable for executing computer programs include both general-purpose microprocessors and special-purpose microprocessors, as well as any one or more processors in any kind of digital computer. Generally, a processor receives instructions and data from read-only memory or random access memory, or both. Essential components of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Generally, a computer will also include, or be operatively coupled to, receiving data from, transferring data to, or both of one or more mass storage devices (e.g., magnetic disks, magneto-optical disks, or optical disks) for storing data. However, a computer may not necessarily have such devices. Furthermore, a computer may be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio player, a global positioning system (GPS) receiver, etc. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, as examples, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and CD-ROMs and DVD-ROMs. The processor and memory can be supplemented by dedicated logic circuitry, or incorporated into dedicated logic circuitry.

[0080] To provide interaction with a user, one or more aspects of this disclosure may be implemented on a computer having a display device (e.g., a CRT (cathode ray tube), LCD (liquid crystal display) monitor, or touchscreen) for displaying information to the user and optionally a keyboard and pointing device, such as a mouse or trackball, for the user to provide input to the computer. Other types of devices may also be used to provide interaction with the user; for example, feedback provided to the user may be any form of sensory feedback, such as visual, auditory, or tactile feedback; and input from the user may be received in any form, including sound, speech, or tactile input. Furthermore, the computer may interact with the user by sending documents to and receiving documents from a device used by the user; for example, by sending a web page to a web browser on the user's client device in response to a request received from a web browser.

[0081] One or more aspects of this disclosure may be implemented in a computing system comprising back-end components (e.g., as a data server), or middleware components (e.g., an application server), or front-end components (e.g., a client computer having a graphical user interface or web browser that a user can use to interact with embodiments of the subject matter described herein), or any combination of one or more such back-end, middleware, or front-end components. The components of the system may be interconnected via digital data communication (e.g., a communication network) of any form or medium. Examples of communication networks include local area networks (“LANs”) and wide area networks (“WANs”), interconnected networks (e.g., the Internet) and peer-to-peer networks (e.g., self-organizing peer-to-peer networks).

[0082] A computing system may include clients and servers. Clients and servers are generally geographically distant and typically interact via a communication network. The client-server relationship is established by means of computer programs running on respective computers and having a client-server relationship with each other. In some implementations, the server sends data (e.g., HTML pages) to the client device (e.g., for the purpose of displaying data to a user interacting with the client device and receiving user input from the user interacting with the client device). Data generated at the client device (e.g., the result of user interaction) may be received at the server from the client device.

[0083] While this specification contains numerous details, these should not be construed as limiting the scope of this disclosure or potentially claimed protection, but rather as descriptions of features specific to particular embodiments of this disclosure. Certain features described in this specification in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments. Furthermore, although features may have been described above as functioning in certain combinations and even initially claimed in this way, one or more features from a claimed combination may be removed from that combination in some cases, and the claimed combination may be for sub-combinations or variations thereof.

[0084] Similarly, although operations are depicted in a specific order in the accompanying drawings, this should not be construed as requiring such operations to be performed in the specific order shown or in sequential order, or requiring all illustrated operations to be performed to achieve the desired result. In some cases, multitasking and parallel processing can be advantageous. Furthermore, the separation of various system components in the above embodiments should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0085] Many implementations have been described. However, it should be understood that various modifications can be made without departing from the spirit and scope of this disclosure. Therefore, other embodiments are within the scope of the following claims. For example, the actions described in the claims can be performed in a different order and still achieve the desired result.

Claims

1. A method for consistent change notification, comprising: The data processing hardware receives requests from users to monitor data events within the distributed storage system. The data processing hardware identifies the request seeking to receive notification of a corresponding data event occurring at the distributed storage system; The timing of the corresponding data event is determined by the data processing hardware, and the timing is consistent with the time when the transaction load on the distributed storage system meets the degradation threshold, wherein the degradation threshold indicates the state in which the transaction load degrades the cache performance; Based on the timing of the corresponding data events, the data processing hardware generates a new event log file to store the records of the corresponding data events, instead of storing the records of the corresponding data events in a pre-existing event log file; The data processing hardware records the corresponding data events in the new event log file; The data processing hardware integrates the new data log file and the pre-existing event log file to form an integrated event history; as well as The data processing hardware delivers notifications to the user based on the integrated event history.

2. The method according to claim 1, wherein, The corresponding data events include miss events.

3. The method according to claim 2, further comprising: The data processing hardware generates a snapshot of the user data, the snapshot capturing the state of the user data; as well as The data processing hardware determines the result of the miss event based on the snapshot.

4. The method according to claim 1, wherein, The transaction load exceeds the degradation threshold.

5. The method according to claim 1, wherein, Delivering the notification includes passing the notification to multiple users of the distributed storage system, the multiple users including the user and corresponding to subscribers who request to receive a corresponding notification regarding changes to data stored in the distributed storage system.

6. The method according to claim 1, wherein, The notification includes the integrated event history.

7. The method according to claim 1, wherein, Each event log file includes a non-persistent file format.

8. The method according to claim 1, wherein, The corresponding data events include write transactions for the data.

9. The method according to claim 1, wherein, The notification is transmitted in real time as the corresponding data event occurs.

10. A system for consistent change notification, comprising: Data processing hardware for distributed systems; as well as Memory hardware communicating with the data processing hardware, the memory hardware storing instructions that, when executed on the data processing hardware, cause the data processing hardware to perform operations, the operations including: Receive requests from users to monitor data events within the distributed storage system; The request is identified as seeking to receive notification of a corresponding data event occurring at the distributed storage system; The timing of the corresponding data event is determined to be consistent with the time when the transaction load on the distributed storage system meets the degradation threshold, wherein the degradation threshold indicates the state in which the transaction load degrades the cache performance; Based on the timing of the corresponding data events, a new event log file is generated to store the records of the corresponding data events, instead of storing the records of the corresponding data events in a pre-existing event log file; The corresponding data events are recorded in the new event log file; Integrate the new data log file and the pre-existing event log file to form an integrated event history; and The notification will be delivered to the user based on the integrated event history.

11. The system according to claim 10, wherein, The corresponding data events include miss events.

12. The system according to claim 11, wherein, The operation further includes: The data processing hardware generates a snapshot of the user data, the snapshot capturing the state of the user data; and The data processing hardware determines the result of the miss event based on the snapshot.

13. The system according to claim 10, wherein, The transaction load exceeds the degradation threshold.

14. The system according to claim 10, wherein, Delivering the notification includes passing the notification to multiple users of the distributed storage system, the multiple users including the user and corresponding to subscribers who request to receive a corresponding notification regarding changes to data stored in the distributed storage system.

15. The system according to claim 10, wherein, The notification includes the integrated event history.

16. The system according to claim 10, wherein, Each event log file includes a non-persistent file format.

17. The system according to claim 10, wherein, The corresponding data events include write transactions for the data.

18. The system according to claim 10, wherein, The notification is transmitted in real time as the corresponding data event occurs.

Images