Delivery of event notifications from a distributed file system

Abstract

In a fault-tolerant method for event information delivery within a file system cluster, one or more processors determine event information associated with file system activity performed by nodes of the cluster. The one or more processors add the event information to an event log buffer in memory. The one or more processors receive a first log sequence number (LSN) associated with flushing recovery information from a recovery log buffer. The one or more processors determine event information in the event log buffer that has a log sequence number less than or equal to the first log sequence number, and determine that the event information includes a log sequence number less than or equal to the first log sequence number, and the one or more processors flush the corresponding event information from the event log buffer to disk storage.

Description

Technical Field

[0001] This invention relates generally to the field of distributed file system operations, and more specifically to the reliable persistence of event notifications for file system operations. Background Technology

[0002] Enterprises operating across multiple geographic environments and utilizing clustered file systems often hosted in the cloud are increasingly focused on data security and monitoring of data access and other file activity events. Protecting business-critical data from internal threats and external security vulnerabilities has led many companies to include file system auditing capabilities.

[0003] Financial institutions and healthcare companies have operational and legal needs to determine access to and editing of ledgers, accounts and transactions, medical records, and personnel documents. Some consulting services rely on secure file system access and operation to support client needs and remain competitive in the market. In distributed computing environments that can employ clustered file systems, the recovery of file system activity before node restarts may involve crashes or suspensions that could include critical information or even be required by compliance rules.

[0004] A distributed file system typically comprises multiple nodes that perform operations on files, objects, or records on the system. Nodes may include message queues distributed across multiple nodes, which support the collection of file activities or events performed on files by the respective nodes. The publication of event activities to the message queues is performed by a "producer" component running within a daemon on the node. This "producer" component applies policies for configuring which files and event activities to view. Summary of the Invention

[0005] This invention provides a fault-tolerant method, computer program product, and system for event information delivery to nodes within a computer node cluster. The method provides one or more processors to determine event information associated with file system activity performed by nodes in the computer node cluster. The one or more processors add the event information to an event log buffer in memory. The one or more processors receive a first log sequence number matching a recovery log sequence number corresponding to information flushed from the recovery log buffer. The one or more processors determine that the event information included in the event log buffer has a log sequence number less than or equal to the first log sequence number, and in response to determining that the event information included in the event log buffer corresponds to a log sequence number less than or equal to the first log sequence number, the one or more processors flush the event information having the corresponding log sequence number from the event log buffer to disk storage. Attached Figure Description

[0006] Figure 1 This is a functional block diagram illustrating a distributed data processing environment according to an embodiment of the present invention.

[0007] Figure 2 This is a functional block diagram illustrating an example of flushing an event log buffer to a persistent disk storage device according to an embodiment of the present invention.

[0008] Figure 3 It is a depiction of an embodiment according to the present invention. Figure 1 A flowchart illustrating the operational steps of an event log program operating in a distributed data processing environment.

[0009] Figure 4 A block diagram depicting components of a computing system according to an embodiment of the present invention, the computing system including components configured to perform... Figure 3 The computing device for the event log program. Detailed Implementation

[0010] Embodiments of the present invention recognize that certain applications running on distributed computing environments, including clustered file systems, are configured to generate and distribute event notifications associated with event activity related to files. Embodiments of the present invention recognize that applications monitoring file systems require consistent and reliable state and activity associated with the file system and its component data. File system monitoring applications rely on event notifications delivered to the application as input to the state, access, and changes to the file system's data content, and interruptions to event notifications, which may occur due to node crashes, restarts, or other delivery issues, can lead to loss of event information and require significant intervention to correct. The resilience of event notifications from the monitored file system is a critical issue for the reliability of such applications. Storage systems can be specified for storing files and documents, or can be configured to store items as objects.

[0011] Embodiments of the present invention provide reliable delivery of event notifications for storage management operations and activities, regardless of the type of content stored. For the sake of brevity, the discussion of embodiments will refer to “files,” “file systems,” and “file system clusters,” and is not limited to “objects” or other units of content.

[0012] Embodiments of the present invention provide methods, computer program products, and computer systems for reliably delivering event notifications for a clustered file system. In embodiments of the invention, an event log buffer is included in each operating node of the clustered file system and receives event metadata of file events executed by the respective node. In some embodiments of the invention, the event log buffer has a corresponding event log that receives (i.e., flushes) event information of file accesses and activities processed on the respective node from the event log buffer and provides persistent storage (i.e., disk storage) of the event information.

[0013] In embodiments of the invention, the event log buffer is synchronized with the recovery log buffer such that the event log buffer flushes the event information stored in memory to disk before flushing the corresponding data from the recovery log buffer to the recovery log on disk. In some embodiments, in response to an interruption in the delivery of event notifications to the sink, event information is replayed from the event log stored on disk, thereby ensuring complete delivery of file system event information.

[0014] In some embodiments of the invention, nodes in a computer cluster operating a clustered file system include running daemons that have event generators that produce event notifications corresponding to file system activities performed on that node. Events in the clustered file system may include, but are not limited to, activities such as access (and access ID), creation, opening, closing, removal, edit changes, ownership changes, security level changes, and relocation. The node producer sends event information as metadata to an event log buffer and a destination. In one embodiment, the node producer sends event metadata to a message queue. Embodiments of the invention recognize that on-disk message queues and sequential logs are embodiments of the destination. Message queues are stream processing software platforms that provide real-time, high-throughput data feeds. Message queues act as a repository for collecting event information notifications from the corresponding distributed nodes of the file system cluster, so that the information can be consumed by applications of interest. In one embodiment, the node producer writes event metadata to a sequential log file on persistent storage. The destination provides an access point for file system monitoring applications to receive event notifications of file system activity and file metadata or object status, change, and access information.

[0015] Event information includes additional metadata associated with the corresponding file system event, beyond the data sent to the recovery log buffer and stored on disk in the recovery log. Embodiments of the invention recognize that, in the event of a node failure, the event information required to generate an event notification is not included in the recovery log of the system component designed to perform the recovery operation. The event log includes metadata such as, but not limited to, the inode number of the event, partial name, storage pool, user ID, owner ID, file attributes, file event activity, and timestamp.

[0016] Embodiments of the present invention ensure reliable maintenance and delivery of event notifications even in the event of crashes, reboots, or other system outages by providing an event log buffer that flushes event information to persistent disk storage before flushing recovery information included in the recovery log buffer. The event log provides event information under fault conditions where normal event activity information is not delivered to the destination.

[0017] The invention will now be described in detail with reference to the accompanying drawings. Figure 1 This is a functional block diagram illustrating a distributed data processing environment, generally represented by 100, according to an embodiment of the present invention. Figure 1 This illustration provides only one implementation and does not imply any limitation regarding the environments in which different embodiments may be implemented. Those skilled in the art can make many modifications to the described environments without departing from the scope of the invention as set forth in the claims.

[0018] The distributed data processing environment 100 includes multiple server nodes, represented as a first server node 110, a second server node 120, and intermediate server nodes (not shown) up to and including an Nth server node 130, forming a distributed file system cluster. The distributed data processing environment 100 also includes a destination 140, a storage system application 160, and a storage system monitoring application 170, all interconnected via a network 150. The network 150 can be, for example, a local area network (LAN), a wide area network (WAN) (such as the Internet), a virtual local area network (VLAN), or any combination that may include wired, wireless, or optical connections. Generally, the network 150 can be any combination of connections and protocols that support communication and data transfer between the first server node 110, the second server node 120, and the intermediate server nodes up to the Nth server node 130, the destination 140, the storage system application 160, and the storage system monitoring application 170.

[0019] First server node 110, second server node 120, and intermediate nodes (not shown) up to Nth server node 130 are nodes of a distributed file system cluster. First server node 110 includes copies of event logger 300, daemon 115, generator 117, event log 113, and recovery log 119. Second server node 120 includes copies of event logger 300, daemon 125, generator 127, event log 123, and recovery log 129. Nth server node 130 includes copies of event logger 300, daemon 135, generator 137, event log 133, and recovery log 139. In some embodiments of the invention, event logger 300 may be remotely hosted from first server node 110, second server node 120, and Nth server node 130, and provide programmable instructions to the respective server nodes.

[0020] In some embodiments, the first server node 110, the second server node 120, and the Nth server node 130 are computing devices configured to operate in a distributed environment and perform file system operations. In some embodiments, the first server node 110, the second server node 120, and the Nth server node 130 may be blade servers, web servers, laptop computers, desktop computers, standalone mobile computing devices, smartphones, tablet computers, or other electronic devices or computing systems capable of receiving, sending, and processing data. In other embodiments, the first server node 110, the second server node 120, and the Nth server node 130 may be computing devices that interact with applications and services hosted and operated in a cloud computing environment. In another embodiment, the first server node 110, the second server node 120, and the Nth server node 130 may be netbook computers, personal digital assistants (PDAs), or other programmable electronic devices capable of generating input and transmitting input to and receiving programming instructions from the event logger 300. Alternatively, in some embodiments, the first server node 110, the second server node 120, and the Nth server node 130 may be communicatively connected to a remotely operated event logger 300. The first server node 110, the second server node 120, and the Nth server node 130 may include internal and external hardware components. Figure 4 A more detailed description is provided below.

[0021] Daemon processes 115, 125, and 135 are computer programs that run as background processes. Daemon processes 115, 125, and 135 enable generators 117, 127, and 137 to determine file system event activities occurring on the corresponding nodes and generate event information associated with specific file or object activities.

[0022] Generators 117, 127, and 137 are entities operating within daemons 115, 125, and 135, respectively. Generators 117, 127, and 137 determine file system event activity on the corresponding nodes, collect event metadata, and publish the events and metadata to message queues. In embodiments of the invention, generators 117, 127, and 137 send event metadata to the event log buffer of the corresponding node, which... Figure 2 The discussion will describe this in more detail.

[0023] Recovery logs 119, 129, and 139 are collections of log files within the first server node 110, the second server 120, and the Nth server 130, respectively. These logs include activity and status information that enables the corresponding nodes to recover from crashes, restarts, or other processing interruptions. Recovery logs 119, 129, and 139 reside on persistent storage devices within the first server node 110, the second server node 120, and the Nth server node 130, respectively. When the corresponding recovery log buffer for the corresponding server node reaches or approaches its capacity limit, recovery logs 119, 129, and 139 receive recovery information and flush the information from the corresponding recovery log buffer in memory to the corresponding recovery log on disk.

[0024] Event logs 113, 123, and 133 are log files of event activity and associated metadata on the first server node 110, the second server node 120, and the Nth server node 130. Event logs 113, 123, and 133 reside in persistent storage and provide consistent event information in the event of interruptions or failures in the delivery of event information to the designated destination via message queues. Event logs 113, 123, and 133 receive event information when the corresponding event log buffer on the respective server node reaches or approaches its capacity limit and flush the event information from the corresponding event log buffer in memory to the corresponding event log on disk.

[0025] Event logger 300 is described as operating on each of the first server node 110, the second server node 120, and the Nth server node 130. In some embodiments of the invention, event logger 300 is a component of a distributed application that manages data in a clustered file system and can be scaled to operate with multiple nodes operating in that file system cluster. File system monitoring applications use file system event notifications to determine the status and condition of file system component files or objects, directories, and operational activities involving the creation and access of files / objects. Applications monitoring the file system may experience resilience issues, where there may be interruptions in the delivery of event notifications from the application to its access event notification destination. Applications monitoring the file system may not be able to tolerate delivery failures, and embodiments of the invention provide a reliable source of event notifications to replay and deliver to the event notification destination in the event of node failure conditions in the file system cluster.

[0026] In some embodiments, the event logger 300 determines event activity and instructs the corresponding generator of the node to send event information containing event metadata to the node's event log buffer. In some embodiments, the event logger 300 appends event metadata for a specific event to a JSON file stored in the event log buffer. Each instance of the event information receives a Log Sequence Number (LSN) corresponding to the same Log Sequence Number of the recovery information associated with the event, stored in the node's recovery log buffer. For a given event of file system activity (such as a "write" activity to a file), the LSN is assigned to the recovery information associated with the write event and sent to the recovery log buffer, and the same LSN is assigned to the event information of the write event sent to the event log buffer.

[0027] In some embodiments of the invention, the event logger 300 receives LSNs corresponding to a set of recovery information in the recovery log buffer. This set of recovery information includes all recovery information whose LSNs are equal to or less than the LSN received by the event logger 300. For example, the recovery log buffer may be nearing its capacity limit for recovery information, and recovery information needs to be flushed from the buffer to the recovery log in persistent disk storage. Flushing refers to the process of writing the contents of volatile memory (i.e., the buffer) to persistent storage on disk. The recovery log buffer identifies LSNs corresponding to events previously executed on the node, such as LSN = 1000. The recovery log buffer also includes multiple LSNs executed on the node prior to the event corresponding to LSN = 1000. The recovery log buffer identifies the highest sequence number scheduled to be flushed to disk storage, along with all other recovery information having LSNs less than the identified LSN = 1000 (i.e., LSN = 999, LSN = 998, etc.), and thus clears the recovery log buffer to be able to accept subsequent recovery information from file system activities executed on the node.

[0028] Before flushing recovery information to disk storage, the LSN identified by the recovery log buffer is received by the event log program 300, and the event log program 300 determines event information in the event log buffer that has an LSN equal to or less than the LSN identified by the recovery log buffer. For example, the event log program 300 receives LSN = 1000 corresponding to recovery information in the recovery log buffer, which is intended to be flushed to persistent disk storage in the recovery log. The event log program 300 determines event information associated with events having an LSN equal to or less than LSN = 1000. The event log program 300 flushes the event information of events with LSN = 1000 or less in the event log buffer to the event log in persistent storage on disk. The event log program 300 communicates the completion of flushing the event information to disk to the recovery log buffer, after which the recovery log buffer performs flushing of the recovery information from the buffer to the recovery log in persistent storage on disk.

[0029] In some embodiments, the event logger 300 determines that the event log buffer is at or near capacity and initiates a flush of event information to the event log on disk without initiating an LSN recognized by the recovery log buffer. In some embodiments, the event logger 300 determines that the event log buffer does not contain event information with an LSN equal to or less than the LSN recognized by the recovery log buffer. In this case, the event logger 300 does not perform a flush of event information but instead communicates completion to the recovery log buffer to begin flushing the recovery log buffer.

[0030] Host 140 is a software application that receives event notification deliveries via message queues from the corresponding generators 117, 127, and 137 of the first server node 110, the second server node 120, and the Nth server node 130. Host 140 provides an interface for storage system monitoring applications to access event notifications, determine the status and condition of the file system cluster components, and perform monitoring activities for the file system cluster.

[0031] Storage system application 160 includes one or more applications that provide file system storage management and coordination services for a file system cluster, which includes a first server node 110, a second server node 120, an intermediate server node (not shown), and an Nth server node 130. In some embodiments, event log program 300 is executed as a module component of storage system application 160.

[0032] Storage system monitoring application 170 includes a collection of client-side applications that access event notifications from destination 140 and monitor the corresponding status, condition, file, and directory activities of the file system. Storage system monitoring application 170 utilizes uninterrupted delivery of event notifications to maintain consistent monitoring of the corresponding file system components and content, and uses event metadata to determine the status, condition, and activities associated with cataloging operations applied to the clustered file system. Storage system monitoring application 170 uses the delivered event notifications (including activities such as system status, file count, directory count, and archive detection) to determine user file access, determine content attributes (confidential files), identify heavy access, and provide services across the distributed storage environment.

[0033] Figure 2 This is a functional block diagram illustrating an example of flushing an event log buffer to persistent disk storage according to an embodiment of the present invention. Figure 2 This includes daemon 215, recovery log buffer 220, recovery log 225, event log buffer 230, event log 235, flush LSN 240, and completion response 250. Daemon 215 includes generator 217 and policy 219. Daemon 215 runs as a background application on the nodes of the file system cluster, as described above with respect to daemons 115, 125, and 135. Generator 217 determines file system event activity on the node, collects event metadata, and publishes the events and metadata to a message queue. Generator 217 sends event information, including metadata associated with events executed by the node, to event log buffer 230. In some embodiments, generator 217 sends recovery information to recovery log buffer 220.

[0034] Policy 219 includes rules and conditions associated with accessing and logging event information related to the file system. In some embodiments, policy 219 determines which files and events are included in notifications published by generator 217.

[0035] Recovery log buffer 220 collects recovery information associated with event activity of the file system executed on the corresponding node. Recovery log buffer 220 resides in volatile memory and writes the file system recovery information held in recovery log buffer 220 to recovery log 225, which resides in persistent memory on disk. Recovery log buffer 220 receives information for recovery from interruptions in the process, such as crashes, node or component failures, or reboots.

[0036] Recovery log buffer 220 determines when its capacity is full or nearly full and sends a flush LSN 240 to event log buffer 230. Event log program 300 uses the flush LSN 240 to determine event information included in event log buffer 230 that corresponds to the recovery information in recovery log buffer 220, to flush to event log 235 residing in persistent storage on disk. The flush LSN 240 enables event log program 300 to keep event log buffer 230 synchronized with recovery log buffer 220. After flushing event information corresponding to or smaller than the flush LSN 240 from event log buffer 230 to event log 235 residing in persistent storage on disk, recovery log buffer 220 receives a completion response 250 from event log program 300.

[0037] Event log buffer 230 receives event information associated with corresponding event activities performed on the node from generator 217. Event log buffer 230 appends the event information of the corresponding event activity to a JSON file. Flush LSN 240 is received by event logger 300 and reflects a set of recovery information to be flushed from recovery log buffer 220 when the recovery log buffer becomes full. This set of recovery information includes recovery information with corresponding LSNs equal to or less than flush LSN 240. Event logger 300 uses the received flush LSN 240 to determine whether event log buffer 230 includes event information corresponding to the set of recovery information identified by flush LSN 240. If event log buffer 230 includes event information corresponding to LSNs less than or equal to flush LSN 240, then event logger 300 begins flushing the identified event information from event log buffer 230 to event log 235 on persistent storage on disk. Flushing the event log buffer before flushing the recovery log buffer 220 ensures that the event log information on persistent storage contains event information consistent with the file system state used for event notification delivery in the event of a failure interruption via message queues and the destination file system event notification delivery.

[0038] The flush LSN 240 includes a log sequence number corresponding to recovery information specifying file system activity flushed from recovery log buffer 220 to recovery log 225. The flushing of recovery log buffer 220 also includes recovery information corresponding to LSNs smaller than the current flush LSN 240 in recovery log buffer 220. Event logger 300 may use the flush LSN 240 to determine event information in event log buffer 230 for flushing to event log 235. The flushing of event information includes event information having a corresponding LSN smaller than the flush LSN 240 included in event log buffer 230.

[0039] Completion response 250 is a communication from event logger 300 that acknowledges the completion of flushing event log buffer 230 and indicates that flushing recovery log buffer 220 may begin. In embodiments of the invention, event log buffer 230 flushes event information to event log 235 based on event information having an LSN equal to or less than flushing LSN 240, before flushing recovery information from recovery log buffer 220 to recovery log 225. In some embodiments, event log buffer 230 may flush event information to event log 235 because event log buffer 230 is near or at capacity. In embodiments where event logger 300 determines that event log buffer 230 does not contain event information with a corresponding LSN equal to or less than flushing LSN 240, event logger 300 transmits a completion response to recovery log buffer 220 to begin flushing recovery information to recovery log 225.

[0040] Figure 3 This illustrates an embodiment of the invention. Figure 1 The flowchart illustrates the operational steps of an event logger operating in a distributed data processing environment. Event logger 300 receives log sequence numbers reflecting a set of recovery information designated as recovery logs flushed from the node's recovery log buffer to persistent storage. The designated flush includes recovery information with an LSN equal to or less than the received log sequence number. Event logger 300 determines whether the event log buffer includes event information with a corresponding LSN equal to or less than the LSN received from the recovery log buffer.

[0041] Event logger 300 determines file system event activities of the nodes in the file system cluster (step 310). Event logger 300 determines the initiation of file access activities performed by the nodes. Event logger 300 determines the type of activity and the metadata associated with the event activity, which are included as event information. For example, first server node 110 receives an instruction to perform a write operation on a file in the file system. Event logger 300 of first server node 110 detects file event activities and prepares to receive metadata corresponding to the event activities performed on that node.

[0042] Event logger 300 adds event information associated with event activities of a node to the event log buffer (step 320). Event logger 300 receives event information for event activities, which includes metadata associated with the activity and the file that performed the event activity. Event logger 300 receives event information uniquely associated with a specific event activity to form a JSON message and appends the message to the event log buffer residing in volatile memory. The event log buffer collects event information from multiple event activities on the node and coordinates with the addition of recovery information associated with a specific event activity provided to the recovery log buffer, which also resides in volatile memory.

[0043] For example, having received event information including specific metadata associated with a write operation to a file on the file system performed on the first server node 110, the event logger 300 appends the event information to a JSON file in the event log buffer. The event information includes metadata specifically associated with the write operation performed on the file system targeted by the write operation. The event information receives a log sequence number (LSN) that uniquely identifies the metadata associated with a specific write event activity. The write event activity also generates a corresponding set of recovery information uniquely associated with the specific write activity and sent to a recovery log buffer residing in memory. The event information includes significantly more metadata details associated with the write event activity to a specific file, while the recovery information only includes information supporting fault recovery on the node. The recovery log buffer receives the same log sequence number (LSN) for the received recovery information as the event log buffer received for the same write event activity.

[0044] Event logger 300 receives a first log sequence number (LSN) associated with flushing the recovery log buffer (step 330). The term "first" log sequence number is used to distinguish a particular LSN from other LSNs and does not necessarily correspond to an initial LSN, such as 0001. The recovery log buffer continues to receive recovery information associated with event activity on the node. When the recovery log buffer reaches or approaches a capacity level for adding recovery information to the recovery log buffer, event logger 300 receives an LSN. The LSN received by event logger 300 corresponds to the higher or highest sequence number LSN of the recovery information currently held in the recovery log buffer for flushing to disk, and flushing of recovery information includes all LSNs equal to or less than the first LSN. Event logger 300 receives the same first LSN as the first LSN designated for flushing the recovery log buffer to establish synchronization between the recovery log buffer and the event log buffer.

[0045] For example, recovery log buffer 220 ( Figure 2The recovery log buffer 220 is determined to be nearing capacity and recovery information held in the recovery log buffer 220 needs to be flushed to the recovery log 225 located on disk. The recovery log buffer 220 sends an LSN (flush LSN 240) to the event log program 300 for flushing the event log buffer 230. The sent flush LSN 240 corresponds to the same LSN with a higher or higher sequence number of the recovery information currently held in the recovery log buffer 220 intended to be flushed. The flushing of recovery information includes all LSNs equal to or less than the flush LSN 240, and the event log program 300 uses the same flush LSN 240 to determine the event information to be flushed to the event log 235 on disk.

[0046] Event logger 300 determines the LSN of the event information included in the event log buffer (step 340). Event logger 300 uses the LSN of the event information included in the event log buffer to establish the range of LSNs currently held in the event log buffer. The LSN of the unique instance of the event information added to the event log buffer is used to determine the set of event information corresponding to the individual event activities to be flushed to the event log residing on disk.

[0047] Event logger 300 determines whether the event log buffer contains an LSN less than or equal to the first LSN (determination step 350). If event logger 300 determines that there is currently no LSN in the event log buffer equal to or less than the received first LSN (step 350, "No" branch), event logger 300 will send a completion confirmation message, which initiates flushing the recovery information in the recovery log buffer to the recovery log residing on the disk (step 370).

[0048] For example, event logger 300 receives a first LSN = 1000, indicating a flush of event information having an LSN corresponding to LSN less than or equal to LSN = 1000. Event logger 300 determines from the flushing action associated with event information approaching the capacity of event log buffer 230 that all LSNs in the event log buffer exceed LSN = 1000, which occurs independently of coordination with recovery log buffer 220. Event logger 300 sends a completion response 250. Figure 2 This initiates a flush of the recovery log buffer 220.

[0049] After sending a completion acknowledgment to initiate the flushing of the recovery log buffer, the event logger 300 terminates, but remains active for detecting additional event activity performed on the node. In some embodiments of the invention, the event logger 300 detects a failure condition in the delivery of interrupted event notifications from event logs residing in persistent storage and initiates a replay of the event notifications starting from the last acknowledged event information LSN. Providing secure storage for the replay of event notifications ensures the reliability of delivering event notifications to internal and / or external destinations.

[0050] If event logger 300 determines that the event log buffer includes event information with an LSN less than or equal to the received first LSN (step 350, "Yes" branch), event logger 300 flushes the event information with an LSN less than or equal to the first LSN to the event log on disk (step 360). Event logger 300 initiates flushing of a set of event information to the event log residing in persistent storage on disk, where each event information corresponds to a different event activity performed on the node.

[0051] After flushing the event information specified by an LSN less than or equal to the received first LSN, the event log procedure 300 sends a completion acknowledgment to initiate a flush of the recovery log buffer (step 370) and continues as described above. Embodiments of the invention recognize that the recovery information and event information flushed to disk are coordinated by a specified LSN range defined by an event LSN less than or equal to the first LSN. Embodiments further recognize that by flushing the event log buffer before flushing the recovery log buffer, a reliable, protected source of event notifications and corresponding metadata is protected and can be used for replay in the event of a failure condition or interruption in the delivery of event notifications from the node.

[0052] After sending a completion acknowledgment to initiate the flushing of the recovery log buffer, the event logger 300 terminates, but remains active for detecting additional event activity performed on the node. In some embodiments of the invention, the event logger 300 detects a failure condition that interrupts the delivery of event notifications from the event log residing in persistent storage and initiates a replay of the event notifications starting from the last acknowledged event information LSN. Providing secure storage for the replay of event notifications ensures the reliability of delivering event notifications to internal and / or external destinations.

[0053] Figure 4 A block diagram depicts components of a computing system according to an embodiment of the present invention, the computing system including a computing device 405 configured to include or be operatively connected to Figure 1 The components described herein, and have operable execution capabilities. Figure 3 The ability of the event logger 300.

[0054] According to an exemplary embodiment of the present invention, computing device 405 includes a first server node 110, a second server node 120, and an Nth server node 130. Figure 1 Similar components and functionalities to those of [other components]. It should be understood that... Figure 4 This illustration provides only one possible implementation and does not imply any limitation regarding the environments in which different embodiments may be implemented. Many modifications can be made to the depicted environment.

[0055] The computing device 405 includes a communication structure 402 that provides communication between a computer processor 404, a memory 406, persistent memory 408, a communication unit 410, and an input / output (I / O) interface 412. The communication structure 402 can be implemented using any architecture designed to transfer data and / or control information between processors (such as microprocessors, communication and network processors, etc.), system memory, peripheral devices, and any other hardware components within the system. For example, the communication structure 402 can be implemented using one or more buses.

[0056] Memory 406, cache memory 416, and persistent memory 408 are computer-readable storage media. In this embodiment, memory 406 includes random access memory (RAM) 414. Generally, memory 406 may include any suitable volatile or non-volatile computer-readable storage medium.

[0057] In one embodiment, the event log program 300 is stored in persistent memory 408 for execution by one or more of the respective computer processors 404 via one or more of the memories 406. In this embodiment, persistent memory 408 includes a magnetic hard disk drive. Alternatively, or in addition to a magnetic hard disk drive, persistent memory 408 may include a solid-state hard disk drive, a semiconductor storage device, a read-only memory (ROM), an erasable programmable read-only memory (EPROM), flash memory, or any other computer-readable storage medium capable of storing program instructions or digital information. In one embodiment of the invention, recovery log 225 and event log 235 are included in persistent memory 408.

[0058] The media used by persistent memory 408 can also be removable. For example, a removable hard disk drive can be used for persistent memory 408. Other examples include optical discs and disks, thumb drives and smart cards, which are inserted into the drive to be transferred to another computer-readable storage medium that is also part of persistent memory 408.

[0059] In these examples, communication unit 410 provides communication with other data processing systems or devices, including resources of the distributed data processing environment 100. In these examples, communication unit 410 includes one or more network interface cards. Communication unit 410 can provide communication using either or both physical and wireless communication links. Event log program 300 can be downloaded to persistent storage 408 via communication unit 410.

[0060] I / O interface 412 allows for data input and output to other devices that can be connected to computing system 400. For example, I / O interface 412 can provide connectivity to external device 418 (such as a keyboard, keypad, touchscreen, and / or other suitable input devices). External device 418 may also include portable computer-readable storage media, such as thumb drives, portable optical discs or disks, and memory cards. Software and data used to implement embodiments of the invention (e.g., event log program 300) may be stored on such portable computer-readable storage media and may be loaded onto persistent memory 408 via I / O interface 412. I / O interface 412 is also connected to display 420.

[0061] The display 420 provides a mechanism for displaying data to the user and can be, for example, a computer monitor.

[0062] The procedures described herein are identified based on their implementation in specific embodiments of the invention. However, it should be understood that any particular procedural terminology used herein is for convenience only, and therefore the invention should not be limited to use only in any particular application identified and / or implied by such terminology.

[0063] This invention can be a system, method, and / or computer program product with any possible level of technical detail integration. The computer program product may include a computer-readable storage medium having computer-readable program instructions thereon for causing a processor to execute aspects of the invention.

[0064] Computer-readable storage media can be tangible means for retaining and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example, but not limited to, electronic storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of computer-readable storage media includes: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable optical disc read-only memory (CD-ROM), digital universal disc (DVD), memory sticks, floppy disks, mechanical encoding devices (e.g., punched cards or protrusions in slots with instructions recorded thereon), and any suitable combination of the foregoing. As used herein, computer-readable storage media should not be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses passing through fiber optic cables), or electrical signals transmitted through wires.

[0065] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to a suitable computing / processing device via a network (e.g., the Internet, a local area network, a wide area network, and / or a wireless network), or to an external computer or external storage device. The network may include copper cables, optical fibers, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to a computer-readable storage medium within the suitable computing / processing device.

[0066] Computer-readable program instructions used to perform the operations of this invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, integrated circuit configuration data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​(such as Smalltalk, C++, etc.) and procedural programming languages ​​(such as the "C" programming language or similar programming languages). The computer-readable program instructions may be executed entirely on a user's computer, partially on a user's computer, as a standalone software package, partially on a user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter case, the remote computer may be connected to the user's computer via any type of network (including a local area network (LAN) or a wide area network (WAN)) or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry including, for example, programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs) may execute computer-readable program instructions by utilizing state information from the computer-readable program instructions to personalize the electronic circuitry in order to perform aspects of this invention.

[0067] The present invention will now be described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0068] These computer-readable program instructions may be provided to a processor of a computer or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / actions specified in one or more blocks of a flowchart and / or block diagram. These computer-readable program instructions may also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner, such that the computer-readable storage medium storing the instructions includes an article of manufacture containing instructions that implement aspects of the functions / actions specified in one or more blocks of a flowchart and / or block diagram.

[0069] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus, or other device to produce a computer-implemented process, such that the instructions executed on the computer, other programmable apparatus, or other device perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0070] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. Each block in a flowchart or block diagram may represent a module, segment, or portion of instructions, including one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the figures. For example, two blocks shown consecutively may actually be completed as a single step, executed simultaneously, substantially simultaneously, or with partial or complete temporal overlap, or the blocks may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action or executes a combination of dedicated hardware and computer instructions.

Claims

1. A fault-tolerant method for delivering event information within a computer node cluster, the method comprising: One or more processors determine the event information associated with file system event activities executed by nodes in a cluster of computer nodes; The one or more processors add the event information of the file system event activity to the event log buffer of the node, the event information including a log sequence number that uniquely corresponds to the event activity and matches the recovery log sequence number of the event activity, and wherein the event information includes metadata associated with the event activity, the metadata not being in the recovery log information associated with the event activity; The one or more processors receive a first log sequence number that matches a recovery log buffer sequence number corresponding to recovery information flushed from the recovery log buffer of the node, based on the capacity limit of the recovery log buffer; The one or more processors determine whether at least some of the event information included in the event log buffer has a log sequence number that is less than or equal to a first log sequence number; as well as In response to determining that at least some of the event information included in the event log buffer corresponds to a log sequence number less than or equal to a first log sequence number, the one or more processors flush the event information with the corresponding log sequence number less than or equal to the first log sequence number from the event log buffer to disk storage, and then flush the recovery log buffer information with the log sequence number less than or equal to the first log sequence number.

2. The method according to claim 1, further comprising: In response to determining that there is no correspondence between the event information included in the event log buffer and a log sequence number less than or equal to the first log sequence number, the one or more processors send an acknowledgment response to the recovery log buffer, initiating a flush of the recovery information included in the recovery log buffer, without flushing the event information from the event log buffer.

3. The method according to claim 1 or 2, further comprising: In response to detecting a fault state of a node in the computer node cluster, the one or more processors send the event information of at least one file system activity executed on the node that has been flushed to disk storage to the destination.

4. The method according to claim 1, wherein the corresponding node of the computer node cluster includes a generator and a message queue as a destination, and wherein the destination is an event notification access repository outside the computer node cluster.

5. The method of claim 1, wherein the respective nodes of the computer node cluster include a generator and an event log file on persistent storage as a destination, and wherein the destination is an event notification access repository outside the computer node cluster.

6. The method of claim 1, further comprising replaying undelivered event information caused by a fault state of a node within a computer node cluster, the replay comprising: The fault state of the first node in the computer node cluster is determined by the one or more processors. The one or more processors access event information associated with file system event activities executed on the first node of the computer node cluster, stored in an event log residing in disk storage, wherein the event notification includes the delivery of event information of the event activities; The one or more processors determine the log sequence number (LSN) of the last event notification delivered to the destination of the file system, wherein the destination is a receiving repository for multiple event notifications of the file system; One or more event notifications are identified by the one or more processors as having an LSN larger than the LSN of the last event notification delivered to the destination; as well as The one or more processors replay the one or more event notifications from the event log to the destination of the file system, where the LSN of the last event notification is larger than that delivered to the destination.

7. The method according to claim 6, wherein, The host is an event notification access repository located outside the computer node cluster.

8. A fault-tolerant computer program product for delivering event information to nodes within a computer node cluster, the computer program product comprising program instructions, the program instructions including: Program instructions used to determine event information associated with file system event activities executed by nodes in a cluster of computer nodes; Program instructions for adding event information of the file system event activity to the event log buffer of the node, the event information including a log sequence number that uniquely corresponds to the event activity and matches the recovery log sequence number of the event activity, and wherein the event information includes metadata associated with the event activity, the metadata not being in the recovery log information associated with the event activity; Program instructions for receiving a first log sequence number that matches a recovery log buffer sequence number corresponding to a recovery information flushed from the recovery log buffer of the node, based on the capacity limit of the recovery log buffer; Program instructions for determining whether at least some of the event information included in the event log buffer has a log sequence number less than or equal to a first log sequence number; as well as Program instructions for responding to determining that at least some of the event information included in the event log buffer corresponds to a log sequence number less than or equal to a first log sequence number, flushing the event information with the corresponding log sequence number less than or equal to the first log sequence number from the event log buffer to disk storage, and then flushing the recovery log buffer information with the log sequence number less than or equal to the first log sequence number.

9. The computer program product according to claim 8, further comprising program instructions for: In response to determining that there is no correspondence between the event information included in the event log buffer and a log sequence number less than or equal to the first log sequence number, an acknowledgment response is sent to the recovery log buffer, and a flushing of the recovery information included in the recovery log buffer is initiated, without flushing the event information from the event log buffer.

10. The computer program product of claim 8, further comprising program instructions for: In response to the detection of a fault state of a node in the computer node cluster, the event information to be flushed to disk storage is sent to the destination.

11. The computer program product according to claim 8, wherein, The corresponding nodes in the computer node cluster include generators and message queues that serve as destinations.

12. The computer program product according to claim 8, wherein, The corresponding nodes in the computer node cluster include event log files on persistent storage as their destination.

13. A fault-tolerant computer system for delivering event information to nodes within a computer node cluster, the computer system comprising: One or more computer processors; One or more computer-readable storage media, and program instructions stored on the one or more computer-readable storage media, the program instructions comprising: Program instructions used to determine event information associated with file system event activities executed by nodes in a cluster of computer nodes; Program instructions for adding event information of the file system event activity to the event log buffer of the node, the event information including a log sequence number that uniquely corresponds to the event activity and matches the recovery log sequence number of the event activity, and wherein the event information includes metadata associated with the event activity, the metadata not being in the recovery log information associated with the event activity; Program instructions for receiving a first log sequence number that matches a recovery log buffer sequence number corresponding to a recovery information flushed from the recovery log buffer of the node, based on the capacity limit of the recovery log buffer; Program instructions for determining whether at least some of the event information included in the event log buffer has a log sequence number less than or equal to a first log sequence number; and Program instructions for responding to determining that at least some of the event information included in the event log buffer corresponds to a log sequence number less than or equal to a first log sequence number, flushing the event information with the corresponding log sequence number less than or equal to the first log sequence number from the event log buffer to disk storage, and then flushing the recovery log buffer information with the log sequence number less than or equal to the first log sequence number.

14. The computer system of claim 13, further comprising program instructions stored on the computer-readable storage medium and executable by at least one of the one or more processors, for: In response to determining that there is no correspondence between the event information included in the event log buffer and a log sequence number less than or equal to the first log sequence number, an acknowledgment response is sent to the recovery log buffer, and a flushing of the recovery information included in the recovery log buffer is initiated, without flushing the event information from the event log buffer.

15. The computer system of claim 13, further comprising program instructions stored on the computer-readable storage medium and executed by at least one of the one or more processors, for replaying undelivered event information caused by a failure state of a node within a cluster of computer nodes, the program instructions comprising: Program instructions used to determine the fault status of the first node of the computer node cluster; Program instructions for accessing event information associated with file system event activities executed on the first node of the computer node cluster, stored in an event log residing in disk storage, wherein the event notification includes the delivery of event information of the event activities; Program instructions for determining the log sequence number (LSN) of the last event notification delivered to the destination of the file system, wherein the destination is a receiving repository of multiple event notifications of the file system; Program instructions for identifying one or more event notifications having an LSN that is larger than the LSN of the last event notification delivered to the destination; as well as Program instructions for replaying one or more event notifications, having an LSN larger than the LSN of the last event notification delivered to the destination, from the event log to the destination in the file system.

16. The computer system according to claim 15, wherein, The host is an event notification access repository located outside the computer node cluster.

Images