System and method for maintaining a comprehensive catalog of hierarchical file data

Abstract

Systems and methods for maintaining a comprehensive catalog of hierarchical file data. Systems and methods for maintaining a comprehensive catalog of metadata records that describe hierarchical file data stored in one or more file systems capture event notifications related to the contents of directories affected within the file system, integrate the transformed event notifications, and transmit to the catalog an instruction to display as "unsynchronized" the metadata corresponding to the affected directories, including a modular change processor (MCP). An enumerator computer (EC) receives from the catalog the IDs of directories for which the metadata is displayed as "unsynchronized"; receives the updated contents of the "unsynchronized" directories from the file system; generates replacement metadata for the updated contents; optionally replaces the metadata displayed as "unsynchronized" with the corresponding replacement metadata, and transmits to the catalog an instruction to display the replacement metadata as "in synchronization".

Description

Technical Field

[0001] Cross - reference to Related Applications This application claims the benefit of U.S. Provisional Patent Application No. 63 / 353,686, filed on June 20, 2022, entitled "System and Method for Maintaining a Comprehensive Catalog of Hierarchical File Data", the content of which is hereby incorporated by reference in its entirety for all purposes.

[0002] The present invention relates to a mechanism for maintaining a file storage system and a catalog (metadata) of information that describes and classifies files stored in such a system. More specifically, the present invention includes a mechanism for keeping the catalog up - to - date without having to crawl the entire file system to enumerate its contents.

Background Art

[0003] An information catalog can be kept up - to - date by iteratively and thoroughly enumerating the contents of the catalog using a file system crawler. This process may be suitable for relatively small catalogs, but it is overly slow and may generate an excessive amount of processing overhead for use in applications that include relatively large catalogs.

[0004] Attempts have been made to improve the large-scale file system crawling speed, and there is a method as described in SmartScan: Non-Patent Document 1 discusses efficient crawling techniques when dealing with large-capacity file systems. This approach uses the recent modification history of directories in two dimensions, namely spatio-temporal analysis and statistical analysis, to target directories that can be scanned more frequently while reducing scans elsewhere. This approach may be effective in some applications, but generally, it will miss new project areas that are not temporally or spatially arranged in the previous areas (because those not known cannot be recognized).

[0005] Other approaches focus on parallelizing scans, which can work well to a certain extent but are prone to failure as the scale increases and can have a serious adverse impact on the scan targets. The more parallelization of threads and nodes, the more the target storage device can be affected by information requests. When the act of updating the catalog has an adverse impact on other users who attempt to read and write files or entities (the main use case), the existing storage device deactivates the update to save the main utility.

[0006] It may be possible to implement target crawling or directly collect events to improve the efficiency of catalog maintenance, but such methods have difficulties in dealing with a large number of failures and missed events. See, for example, Non-Patent Document 2.

Prior Art Documents

Non-Patent Documents

[0007]

Non-Patent Document 1

Non-Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0008] Therefore, there is a need for an improved mechanism for information catalog (metadata) maintenance to solve the above-described problems.

Means for Solving the Problems

[0009] The appended claims can serve to summarize the present invention.

[0010] The elements and advantages described herein do not include all of them, and many additional elements and advantages will be apparent to those skilled in the art, especially when considering the drawings, the specification, and the claims. Also, note that the language used herein was mainly selected for readability and explanatory purposes and is not intended to limit the scope of the present invention.

Brief Description of the Drawings

[0011]

Figure 1

Figure 2

Figure 3

Best Mode for Carrying Out the Invention

[0012] The present invention will be described by way of example with reference to the accompanying drawings, in which like reference numerals indicate like elements.

[0013] In the following detailed description, reference is made to the accompanying drawings which form a part hereof, and specific embodiments in which the invention may be practiced are illustrated by way of example. It should be understood that such embodiments are described in sufficient and detailed manner to enable those skilled in the art to practice the invention, and that other embodiments may be utilized and structural, procedural, and systematical changes may be made without departing from the spirit and scope of the present invention. Also, well-known structures, circuits, and techniques not to impede the understanding of this specification are not shown in detail. Therefore, the following detailed description should not be taken in a limiting sense, and the scope of the present invention is defined by the appended claims and their equivalents.

[0014] As used in this specification and the appended claims, the singular forms include plural referents unless the context clearly dictates otherwise. For example, the reference to "computer" includes a plurality of computers. In yet another example, the reference to "enumerator" includes a plurality of enumerators.

[0015] Although specific terms are used in this specification, they are used only in a general and descriptive sense, not for restrictive purposes. All terms, including technical and scientific terms used in this specification, shall have the same meaning as commonly understood by those of ordinary skill in the art to which the present invention pertains, unless otherwise specifically defined. It will be further understood that terms such as those defined in commonly used dictionaries shall be interpreted to have the meaning commonly understood by those of ordinary skill in the art to which the present invention pertains. It will be further understood that terms such as those defined in commonly used dictionaries shall be interpreted to have a meaning consistent with their meaning in the context of the related art and this disclosure. Such commonly used terms shall not be interpreted in an idealized or overly formal sense unless specifically defined otherwise in this disclosure.

[0016] As used in this disclosure, the term "file system" or "filesystem" refers to a database (which may also be referred to interchangeably with indexes and / or volumes) that includes the physical location of data on a device. Data is generally organized into folders called directories, which may contain other folders and files. Thus, a file system is used to control how data is stored and retrieved. Attributes (such as "read-only") and access rights are assigned to directories and files. Also, the term "file system" or "filesystem" as used in this specification encompasses all types of file storage methods or file storage systems, including object storage, AWS / S3 buckets, etc., and "file" is used in the sense of encompassing files of data stored in a hierarchical structure, objects, storage blobs, and other names similar to files.

[0017] As used herein, the term "computer" is used in the sense of encompassing workstations, personal computers, tablets, wireless phones or other suitable computing devices including processors, computer-readable media on which computer-readable program code (including instructions and / or data) may be disposed, and user interfaces. Terms such as "server", "application", "engine", "component", "module", "agent", etc. are for referring to computer-related entities including hardware or combinations of hardware and software. For example, an engine can be, but is not limited to, a processor, a computer, etc. including a process, an object, an executable file, an execution thread, and / or a program executed by the processor. Also, various computer-related entities can be localized to one computer or distributed among two or more computers. The terms "real-time" and "on-demand" mean to sense and respond to the occurrence of external events substantially simultaneously (e.g., within milliseconds or microseconds) or without intentional delay, taking into account the processing limits of the system and the time required to accurately respond to inputs.

[0018] The file data catalog is maintained to enable performing reports, data analysis, metadata storage and retrieval, work on files stored in the file system, and the like. The catalog is most useful when maintained in an up-to-date state, and due to the continuously growing size of the file system under management, it often experiences considerable difficulty even in just ensuring an accurate inventory. When there are tens of billions of files, it is preferable to have an efficient process for collecting catalog change items.

[0019] One aspect of the present invention is that, even when using state-of-the-art computing resources, it may take days to complete the process of enumerating an entire file system containing billions of entities. As a result, real-time insights cannot be obtained, and it has been found that resources that could be used more productively elsewhere may be tied up. Such problems are exacerbated when the process of enumerating the entire file system is frequently repeated to maintain a catalog, potentially rendering maintenance impossible.

[0020] The inventors have also recognized that alternative mechanisms for maintaining a catalog, such as directly using an event stream (e.g., basic event notification collection), what is offered in file systems and file repositories such as Lustre, CEL, iNotify on Linux, USN Journal on Windows®, StorNext, BeeGFS, and SNS for AWS; or using a general event stream change notification or differentiation engine that provides snapshot or policy-based differences for file systems such as Isilon OneFS, GPFS, Qumulo, and ZFS, are often problematic in applications.

[0021] The inventors recognized that in the case where events or event metadata are missed or events are not captured in sequence, it can be difficult, if not impossible, to maintain an accurate catalog with existing approaches. For example, when a large number of changes occur, such as an HPC user creating and quickly deleting one billion temporary files, the "direct event processing" approach processes twice as many events. Since event streams generally generate many more events than the number of events that are actually of interest, this scenario is not uncommon. There may be read, write, and close events that have no value for catalog maintenance, but all other events may overwhelm the number of events such that a relatively high-processing system with CPU processing overhead has to be used. Therefore, using basic event notifications can lead to catalog items being missed or incorrect, and a major bottleneck can occur when trying to process events in sequence. When a large-scale event storm occurs, when trying to process such a large number of events, the file system may try to send more events than it can handle, and all activity may come to a halt, which is a serious failure scenario. The event processing system must not harm the system being monitored.

[0022] Referring to FIG. 1, embodiments of the present invention solve such problems by integrating (effectively aggregating) events occurring within a directory (or the prefix of an object-based collection) and displaying such a directory or prefix as "unsynchronized" or stale. Such events indicate changes to the contents of file systems 1, 2, and 3. For example, a notification for some changed metadata is received by MCP20 for / dir1 / dir2 / file1 (which can be all metadata such as file size, owner, etc.), and shortly thereafter a notification is also received by MCP20 for / dir1 / dir2 / file2. Both of these notifications constitute the "events" used in this specification. Since all such events target the same directory ( / dir1 / dir2), such a directory is once displayed as "unsynchronized" at 11, i.e., such events are effectively combined (e.g., aggregated) in a single marking activity. Such integration enables embodiments of the present invention to simply display the directory with an accurate status indicator (e.g., "unsynchronized") regardless of how many changes there have been to a given directory during a particular observation period.

[0023] Embodiments of the present invention additionally include a timestamp with an unsynchronized mark, which represents a significant improvement in many applications. The inventors recognized that simply displaying a directory as unsynchronized (e.g., a boolean true or false) may be insufficient in some applications where items become stale (e.g., when a directory is repeatedly displayed as synchronized and unsynchronized due to many events).

[0024] Instead, embodiments of the present invention use an "unsynchronized time" attribute set by a timestamp. If the "unsynchronized time" > "synchronized time", the catalog item is stale; otherwise, the item is up-to-date.

[0025] An exemplary use of a timestamp is shown with reference to FIG. 1, in which the enumerator 9 queries, at (a) 8, a directory (and / or any synchronization time mark, a directory with an unsynchronized time mark) where the unsynchronized time is greater than the synchronization time for the "unsynchronized" directories of file systems 1, 2, and 3 in catalog 7; enumerates, at (b) 12, the contents of each of the identified "unsynchronized" directories; and updates, at (c) 10, the catalog 7 (e.g., with modified metadata) to display the directories that were previously shown at "unsynchronized time" as "synchronized time" using the current timestamp (e.g., "synchronization time"). Thus, the process of the present invention includes replacing the outdated metadata of catalog 7 with the updated metadata generated by the enumerator 9 after the modular change processor 20 of the present invention displays the outdated metadata of catalog 7 as unsynchronized at 11.

[0026] Note in a particular embodiment that the merging process serves to display directories (not simple files) as "unsynchronized" in one process 11. Thereafter, an agent (enumerator 9) scans the directories in a process separate from the merging process. The marks displayed in the metadata of catalog 7 effectively serve as an interface between these two processes.

[0027] This approach has the following advantages

[0028] (a) As described above, since the system displays the directories affected by the initial merging process as "unsynchronized", problems that occur when events are out of order (regardless of whether / a / b / c or / a / b is updated first) can be prevented. (b) Scanning the entire affected directory to prevent problems related to event leakage. (c) Concentrate on the areas known to have changed so as not to adversely affect the target storage device. (d) Prevent the problem of important detailed information leaking due to insufficient classification (Boolean vs. timestamp), although the directory is displayed as not synchronized many times. (e) Provide substantially real-time data for insight and analysis.

[0029] Note that the concept of the present invention does not simply update Catalog 7, but involves using the directory in a way that allows a significant amount of "event compression" (i.e., merging) to avoid the aforementioned problems associated with the conventional approach of collecting events. For example, when 1,000,000 files are added to a single directory, instead of identifying each of the 1,000,000 changes to the directory, the system of the present invention effectively captures them all with a single "directory not synchronized" display, thus generating only a single event that is reduced by a factor of 1,000,000 compared to the existing approach. Also, embodiments of the present invention can include a number of enumerators and catalogs that are not closely synchronized with time. To ensure that there are no problems maintaining synchronized time between Catalog 7 and the modular change processor 20, such embodiments consider the directory to be out-of-date and add a tolerance to the out-of-date time only when out-of-date time - time + tolerance > synchronized time.

[0030] Also, it must be recognized that an additional advantage of such embodiments is that the necessary processing overhead may be less compared to existing approaches, since there is no need to process and collect events at the rate at which they are generated. That is, events can be processed asynchronously rather than in real time. For example, consider a system in which events are generated in relation to updates to metadata: / dir1 / dir2 / file1 is added, / dir1 / dir2 / file2 is added, and the / dir1 / dir2 / dir3 directory is added (where all new files and new directories are added). In a conventional approach, the second and third events may be lost during transmission and these two events (including other events that may occur in dir3 potentially) may be missed in the catalog of metadata. However, embodiments of the present invention effectively prevent such problems by presenting the three events as a single event, i.e., by presenting / dir1 / dir2 as "unsynchronized". Subsequently, the enumerator 9 requests the catalog 7 to identify the "unsynchronized" directory, and the catalog responds with "dir1 / dir2" at 8. Then, the enumerator 9 fetches a list containing dir3, file1, and file2 at 12 in the corresponding file system containing dir1 / dir2 to ensure that no items are missed. As another example, assume that / dir1 / foo is a file. Someone deletes this file and replaces it in the exact same location with a directory, so that / dir1 / foo is now a directory. It can be said that the event stream has a / dir1 / foo removal event occurring first, followed by a dir1 / foo directory creation event. Or the event stream may not provide this information in order, which can mean something completely different. In the described embodiments, after the upper directory (dir1) of the event is presented as "unsynchronized" and then scanned to categorically identify the net changes, it is possible to effectively disengage from such out-of-order events.

[0031] Referring to FIGS. 1 to 3, embodiments of the present invention will be described in more detail. As shown in FIG. 1, the catalog 7 is filled with hierarchical metadata for the items of the file repository. The initial population can be accomplished by any number of conventional approaches. Once the population of this catalog is complete, embodiments of the present invention are used to keep the catalog up to date.

[0032] The modular change processor 20 is configured to receive change notifications in various formats. For example, the change may result from a file system 1 that generates events with a notifier (real-time notification), such as Lustre (lustre.org). File system 2 that generates difference snapshots (events that occur during a predefined interval), such as Isilon OneFS (Dell, Inc., located in Round Rock, Texas), GPFS (IBM, located in Armonk, New York); a policy-based event processor that generates a change list through a change waiting queue, such as a file system 3 with a substantially different signal mechanism that can indicate that a change has occurred in a storage device. Agents 4, 5, 6 within the processor 20 process and effectively convert such different event streams and their specific formats into a normalized format associated therewith.

[0033] The modular change processor 20 identifies the directory in which an incoming event has occurred and displays such directory as "unsynchronized" in the catalog 7 at 11. Thus, the modular change processor 20 integrates events that affect the same directory so that the directory is displayed only once. Previously displayed directories are maintained in the displayed state.

[0034] In a particular embodiment, when a major change is detected (e.g., a directory replacement or a change in directory name at a higher level), the entire tree becomes marked as "unsynchronized" with a special "tree not synchronized" timestamp. In this way, all directories of the corresponding tree are displayed as "unsynchronized". Thereafter, when additional "unsynchronized" events occur for the displayed directories, no additional work needs to be done other than updating the "unsynchronized" timestamp. For example, assume that tree / a with subdirectories / a / micro, / a / macro, / a / intro is replaced by tree / b with subdirectories / b / micro, / b / supera, / b / ultra. It doesn't matter how many (or how few) directories overlap between / a and / b; the entire / a and / b trees are displayed as unsynchronized, and they are scanned again recursively down to display each subdirectory as "unsynchronized".

[0035] Enumerator 9 receives, at 8, the IDs of all directories displayed as "unsynchronized" from catalog 7. Thereafter, enumerator 9 enumerates, at 12, the contents of the corresponding directories in file systems 1, 2, and 3, either non-recursively or recursively as necessary (e.g., through targeted crawling of "unsynchronized" or "tree-unsynchronized" directories), updates, at 10, the catalog 7 metadata associated with the corresponding directories, and displays each directory as "synchronizing" along with a time (e.g., a timestamp). If a new directory that was previously unknown is discovered at this stage, it is added to the "unsynchronized" directory queue and enumerated. For example, if directory / a / b is added, the example displays / a as unsynchronized. If directory / a / c is added but this event is missed or delayed, it becomes "previously unknown", but is automatically added when / a, which was displayed as unsynchronized, is scanned / enumerated. Note that the enumeration can be processed by one or more of the plurality of independent operating enumerators 9 as illustrated and described below in connection with FIG. 2. This process is repeated indefinitely as necessary: 1. Search for directories displayed as "unsynchronized" (including "tree-unsynchronized"); 2. Enumerate such directories for their contents at 12; 3. Display new subdirectories that are unknown (exceeding synchronization time) and add them to the queue, repeating (1) and (2). 4. Update this directory as "within synchronization time" at 10.

[0036] Unsynchronized Merge and Collection -- Modular Change Processor Algorithm The following algorithm shows simplified pseudo-code for an example that integrates events using different change monitors and displays directories as unsynchronized.

[0037] File System 1: Change Log / Notifier Collection (e.g., Lustre) 1. Repeat until aborted: a. Collect and repeat 1,000,000 events: i. Analyze and filter events (file / directory add / remove / change), discarding unwanted events (e.g., reads, opens, closes). ii. If the event path is a directory 1. If the event is a rename or move of this directory name a. Generate a memory marker for the new target directory name as unsynchronized in the tree synchronization. b. Generate a memory marker for the parent directory of the source directory as unsynchronized. c. Generate a memory marker for the parent directory of the target directory as unsynchronized. 2. Otherwise (new, modified, etc.) a. Generate a memory marker for the parent directory as unsynchronized. iii. Otherwise (not a directory) 1. If the event is a file name change (including source and target) a. Fetch the parent directories for the source and target file names of the move or rename. b. Generate memory markers for the two parent directories as unsynchronized. 2. Otherwise (not a name change) a. Fetch the parent directory for the object of 1.a.iii. b. Generate a memory marker for the parent directory of 1.a.iii.2 as unsynchronized. b. For all directory markers in 1: i. Check if the list is deduplicated. ii. Display directories that are not properly catalog-synchronized or tree-synchronized with timestamps.

[0038] File System 2: Snapshot-based Event Processor (e.g., OneFS) 1. Repeat until aborted a. Process snapshot differences: i. Receive snapshot ii. If a previous snapshot exists, receive the difference between the two snapshots "alpha" and "beta". iii. If a swap of two directories is detected, this is treated as a special case and each candidate directory is presented as an unsynchronized tree. iv. Collect the parent directories of the changed items identified in 1.a.ii. v. Generate a deduplication memory marker for the parent directories b. For the unique and deduplicated dir markers of 1.a. i. Present directories or unsynchronized trees that are not properly synchronized with timestamps File System 3: Policy-based Event Processor (e.g., GPFS) 1. Repeat until aborted a. Differentiated collection and processing i. Run a GPFS policy scan to find items with a new "change time" or "modification time" since the previous policy scan. ii. Collect and deduplicate the parent directories of the items in 1.a.i. iii. Generate a memory marker for the parent items of 1.a.ii b. For the unique dir markers of 1.a.iii. i. Present directories that are not synchronized with timestamps

[0039] A similar approach applies to other types of differentiation engines.

[0040] 1. Event / Action Processing (Processed as a stream or the difference between two points in time) 2. The upper directories are extracted and duplicate removed. 3. Directory display not synchronized for scanning 4. Any special cases (e.g., directory swap or name change) are processed as required and displayed without tree synchronization.

[0041] Enumerator algorithm One or more enumerators can be executed simultaneously as shown in Figure 2 to keep the catalog up to date. The simplified algorithm they use is as follows: Repeat until aborted 1. Request directories in Catalog 7 that are not synchronized at time 8 and where the non - synchronized time > synchronization time. 2. Enumerator 9 at 12 receives the directory list identified in step 1 from the storage device. 3. Update Catalog 7 with the new directory list. a. If any new (previously unknown) items of the directory are themselves directories, add these sub - directories to Catalog 7 as new directories and display them as non - synchronized with timestamps. 1. Update Catalog 7 when the synchronization time of the directory obtained at 1.2 = current time.

[0042] Referring now to Figure 2, various embodiments include multiple enumerators 9 operating on one or more machines to coordinate and enumerate a number of storage devices 1, 2, 3, etc. and update Catalog 7. It must also be recognized that Catalog 7 can exist collectively on multiple machines. Similarly, enumerator 9 can exist on one or more machines having one or more different operating regimes. Other aspects of the system, including MCP20 and Catalog 7, may likewise be in one or more machines.

[0043] FIG. 3 shows a diagrammatic representation of a machine in an exemplary form of a computer system 300, within which a series of instructions can be executed to cause the machine to perform any of the methodologies described above. In other embodiments, the machine can include a network router, a network switch, a network bridge, a web appliance, or any machine capable of executing a series of instructions that specify actions to be taken by that machine. The precise nature of the machine is not important for the examples, and can be present in any hardware capable of executing a compatible operating regime that can execute instructions.

[0044] The exemplary computer system 300 includes a processor 302, a main memory 304, and a static memory 306, which communicate with each other via a bus 308. The computer system 300 can further include a video display unit 310 (e.g., a liquid crystal display (LCD), a plasma display, a cathode ray tube (CRT), etc.). The computer system 300 can also include an alphanumeric input device 312 (e.g., a keyboard or a touch screen), a cursor control device 314 (e.g., a mouse), a drive unit 316 (e.g., a disk, a flash memory, etc.), a signal generation device 320 (e.g., a speaker), and a network interface device 322.

[0045] The drive unit 316 includes a computer-readable medium 324 on which is stored a series of instructions (i.e., software) 326 embodying any one or all of the methodologies described above. The software 326 is also illustrated as being resident completely, or at least partially, within the main memory 304 and / or within the processor 302. The software 326 can further be transmitted or received via the network interface device 322. For the purposes of this specification, the term "computer-readable medium" is considered to include any medium that can store or encode a series of instructions for execution by a computer, such that the computer can perform any of the methodologies of the present invention, as further described below.

[0046] Some parts of the above description present the features of the present invention in terms of symbolic representations of operations on algorithms and information. Such descriptions and representations of algorithms are means that those skilled in the art use to most effectively convey the essence of the operations to those skilled in the art. Such operations are understood to be functionally or logically described but implemented by a computer program. Also, it has been proven that it is sometimes convenient to refer to such an operation array by a module or functional name without losing generality. The process steps and instructions of the present invention can be implemented in software, firmware, or hardware. When implemented in software, it should be noted that it can be downloaded, resident, and operated on various platforms used in real-time network operation systems. Also, the specific naming of components, capitalization of terms, attributes, data structures, or other programming or structural aspects are not essential or important, and the present invention or the mechanism implementing its features can have other names, forms, or protocols.

[0047] The present invention is very suitable for a variety of computer network systems across many topologies. In this field, the configuration and management of large-scale networks are composed of storage devices and computers communicatively connected to each other through a network such as the Internet to different computers and storage devices.

[0048] Without departing from the scope of the present disclosure, modifications, additions, or omissions may be made to the systems, apparatuses, and methods described herein. For example, the components of the systems and apparatuses may be integrated or separated. Also, the operations of the systems and apparatuses disclosed herein may be performed by more, fewer, or other components, and the methods described may include more, fewer, or other steps. Also, the steps may be performed in any suitable order. It should be understood that any of the features described in connection with one of the embodiments described herein may be similarly applied to the other embodiments described herein without departing from the scope of the invention. As used in this document, "each" refers to each member of a set or each member of a subset of a set.

[0049] The present invention has been described in particular detail in connection with various possible embodiments, and those skilled in the art will recognize that the invention may be practiced in other embodiments. First, the specific naming of components, capitalization of terms, attributes, data structures, or other programming or structural aspects are not essential or critical, and the invention or the mechanisms embodying its elements may have different names, forms, or protocols. Also, the system may be embodied through a combination of hardware and software as described or entirely by hardware elements. Also, the specific functional divisions between the various system components described herein are merely exemplary and not essential, and the functions performed by a single system component may be performed by many components, and the functions performed by many components may be alternatively performed by a single component.

[0050] Also, unless otherwise explicitly specified in the above discussion, throughout the description, discussions using terms such as "processing" or "computing" or "calculating" or "determining" or "displaying" are to be understood as referring to the operations and processes of a computer system or similar electronic computing device that manipulates and transforms data represented as physical (electronic) quantities within a computer system memory or register or other such information storage locations.

[0051] Embodiments of the present invention also relate to an apparatus for performing operations herein. This apparatus can include a computer specially configured for the required purpose or a computer selectively activated or reconfigured by a computer program stored on a computer-readable medium accessible to the computer. Such a computer program can be stored on a tangible non-transitory computer-readable storage medium such as (but not limited to) all types of disks like floppy disk (registered trademark), optical disk, CD-ROM, magneto-optical disk, read-only memory (ROM), random access memory (RAM), EPROM, EEPROM, magnetic or optical card, application specific integrated circuit (ASIC), other suitable static, dynamic or volatile memory or data storage device or other types of media adapted to store electronic instructions, and all types of disks each coupled to a computer system bus. Also, the computers referred to herein can be of an architecture that includes a single processor or uses a multi-processor design for improved computing performance.

[0052] Also, the present invention is not described with reference to any particular programming language. A variety of programming languages can be used as described herein to embody the teachings of the present invention, and all references to a particular language can be understood to be provided for disclosing the embodiments and best mode of the present invention.

[0053] A variety of systems can be used with a program that follows the teachings of this specification, and it may be convenient to configure more specialized apparatus to perform the required method steps. The structures required for such a variety of systems will be apparent to those skilled in the art with equivalent variations.

[0054] Note that, to assist the Patent Office and all readers of the patents issued on this application in interpreting the claims appended hereto, the applicant does not intend any of the appended claims or elements of the claims to be subject to the application of 35 U.S.C. 112(f) unless the words "means for" or "step for" are expressly used in a particular claim.

[0055] Finally, note that the language used herein has been principally selected for readability and instructional purposes, and not for the purpose of describing or limiting the subject matter of the invention. Accordingly, the disclosure of the invention is illustrative, not limiting, of the scope of the invention as set forth in the following claims. Further, it should be understood that any of the elements described in connection with one of the embodiments described herein can be equally applied to other embodiments described herein without departing from the scope of the invention.

Claims

1. A system for maintaining a comprehensive catalog of metadata records that describe hierarchical file data stored in one or more file systems, a modular change processor (MCP) communicatively coupled to one or more special file systems containing hierarchical file data and a catalog containing metadata corresponding to the hierarchical file data, an enumerator computer (EC) communicatively coupled to the catalog and the one or more special file systems, comprising: the MCP having a first memory, a first processor, and a first storage program in the first memory executable by the first processor, the first storage program comprising: (a) causing the MCP to capture event notifications in one or more basic forms generated by the one or more special file systems, the event notifications being related to the contents of directories affected within the hierarchical file data, (b) causing the MCP to convert the event notifications to a common form, (c) causing the MCP to merge the converted event notifications into one or more combined notifications for a specific directory among the directories affected, (d) configured to cause the MCP to generate an instruction to mark the metadata corresponding to the affected directory as "out-of-sync" using the combined notifications and transmit it to the catalog, the EC comprising a second memory, a second processor, and a second storage program in the second memory executable by the second processor, the second storage program comprising: (e) causing the EC to receive the ID of a directory (an "out-of-sync" directory) in which metadata is marked as "out-of-sync" from the catalog, (f) causing the EC to receive updated contents of the "out-of-sync" directory from the one or more special file systems, (g) causing the EC to generate alternative metadata for the updated contents, and (h) configured to cause the EC to transmit the alternative metadata in the catalog. A system.

2. The system of claim 1, wherein each of the one or more combined notifications comprises a set of the plurality of event notifications affecting a specific directory.

3. The system according to claim 2, wherein each of the one or more merged notifications includes a status indicator for the specific directory.

4. The system according to claim 3, wherein each of the one or more merged notifications includes a status indicator with a timestamp for the specific directory.

5. The (d) further includes generating, by the MCP, an instruction to display "unsynchronized" along with a time timestamp for which metadata corresponding to the directory affected by the merged notification is not synchronized, and transmitting the instruction to the catalog. The system according to claim 4.

6. The (e) further includes receiving, by the EC, from the catalog, an ID of a directory (identified directory) having metadata displayed as "unsynchronized" that is greater than any synchronized time timestamp obtained by adding any tolerance to the unsynchronized timestamp for the identified "unsynchronized" directory. The system according to claim 5.

7. The (f) further includes receiving, by the EC, updated content of the identified "unsynchronized" directory from the one or more special file systems. The system according to claim 6.

8. The (h) further includes transmitting, by the EC, the alternative metadata to the catalog and causing the identified "unsynchronized" directory to be displayed with a synchronization time timestamp. The system according to claim 7.

9. The updated content of the "unsynchronized" directory in (f) further includes the content of any new directory. The method according to claim 1.

10. The (h) further includes replacing the metadata displayed as "unsynchronized" with corresponding alternative metadata, and transmitting the alternative metadata to the catalog together with an instruction to display the alternative metadata as "synchronized" at a specific time. The system according to claim 1.

11. The basic form includes one or more of real-time notification of events; a snapshot of events occurring during a predetermined time interval; a list of events occurring during a predetermined time interval, and / or combinations thereof. The system according to claim 1.

12. The system of claim 11, wherein the basic form includes one or more of a change log event processor; a snapshot-based event processor; a policy-based event processor; and / or a form generated by a combination thereof.

13. The system of claim 1, wherein the instruction to display metadata corresponding to the affected directory as "unsynchronized" further includes an instruction to display metadata corresponding to a tree including the affected directory as "tree unsynchronized".

14. A method for maintaining a comprehensive catalog of metadata records describing hierarchical file data stored in one or more file systems, comprising: (a) configuring a modular change processor (MCP) to communicatively couple to one or more special file systems containing hierarchical file data and a catalog containing metadata corresponding to the hierarchical file data; (b) configuring an enumerator computer (EC) to communicatively couple to the catalog and the special file system; (c) capturing, by the MCP, event notifications in one or more basic forms generated by the one or more special file systems, the event notifications being related to the contents of an affected directory within the hierarchical file data; (d) converting, by the MCP, the event notifications to a common form; (e) merging, by the MCP, the converted event notifications into one or more merged notifications for a particular directory among the affected directories; (f) generating, by the MCP, an instruction to display metadata corresponding to the affected directory as "unsynchronized" using the merged notifications and transmitting the instruction to the catalog; (g) receiving, by the EC, an ID of a directory (an "unsynchronized" directory) for which metadata is displayed as "unsynchronized" from the catalog; (h) receiving, by the EC, updated contents of the "unsynchronized" directory from the one or more special file systems. (i)generating, by the EC, alternative metadata for the updated content; (j)transmitting, by the EC, the alternative metadata to the catalog; A method comprising the above steps.

15. The method according to claim 14, wherein each of the one or more merged notifications includes a set of the plurality of event notifications that affect a specific directory.

16. The method according to claim 15, wherein each of the one or more merged notifications includes a status indicator for the specific directory.

17. The method according to claim 16, wherein each of the one or more merged notifications includes a status indicator with a timestamp for the specific directory.

18. The step (f) further includes generating, using the merged notification, an instruction to display "unsynchronized" together with an unsynchronized time timestamp for metadata corresponding to the directory affected, and transmitting the instruction to the catalog. The method according to claim 17.

19. The step (g) further includes receiving, from the catalog, an ID of a directory (identified directory) having metadata displayed as "unsynchronized" (identified "unsynchronized" directory) that is greater than any synchronized time timestamp by adding any tolerance to the unsynchronized timestamp. The method according to claim 18.

20. The step (h) further includes receiving updated content of the identified "unsynchronized" directory from the one or more special file systems. The method according to claim 19.

21. The step (j) further includes transmitting the alternative metadata to the catalog and displaying the identified "unsynchronized" directory with a synchronized time timestamp. The method according to claim 20.

22. The updated content of the "unsynchronized" directory in the step (j) further includes content of any new directory. The method according to claim 14.

23. The method according to claim 14, wherein step (j) further comprises replacing the metadata displayed as "not synchronized" with corresponding alternative metadata, and transmitting the alternative metadata to the catalog together with an instruction to display the alternative metadata as "synchronized" at a specific time.

24. The method according to claim 14, wherein the basic form includes one or more of: real-time notification of an event; a snapshot of events that occurred during a predetermined time interval; a list of events that occurred during a predetermined time interval, and / or combinations thereof.

25. The method according to claim 24, wherein the basic form includes one or more of the forms generated by a change log event processor; a snapshot-based event processor; a policy-based event processor; and / or combinations thereof.

26. The method according to claim 14, wherein the instruction to display the metadata corresponding to the affected directory as "not synchronized" further includes an instruction to display the metadata corresponding to the tree including the affected directory as "not tree synchronized".

Images