Custom metadata tag inheritance based on file system directory tree or object store bucket

Abstract

A method and / or system for managing metadata is disclosed, comprising connecting a source data storage system (DSS) that stores both data and metadata to a metadata management platform (MMP); scanning metadata records from the DSS onto the MMP; storing metadata attributes for at least one of a group of directories and buckets on the DSS in a lookup table on the MMP; and adding updated metadata attributes to the lookup table on the MMP upon each subsequent scan of the DSS.

Description

[Technical Field]

[0001] The present disclosure relates generally to improved data management, organization, and storage, and more particularly to data processing and storage systems in which large databases or vast amounts of data are processed and stored. Specifically, the present disclosure relates to systems and methods for storing, sorting, organizing, managing, and searching data and metadata, preferably large collections of data and metadata, in a computerized environment. [Background technology]

[0002] The use of electronic data storage is widespread. The adoption of network storage and the relatively rapid growth in the amount of electronic data created has created a need for the organization, storage, and management of large amounts of electronic data. The storage facilities available in large computer systems allow users to store, update, and retrieve data from large data collections. Users and organizations dealing with significant amounts of digital information often find it difficult to manage and search files in an efficient and intuitive manner. The inability to easily store, organize, search, locate, and manage documents and content can translate into significant inefficiencies and lost opportunities.

[0003] To make efficient use of computer resources, data should be intelligently organized to make the retrieval process efficient. Data can be collected and stored as files, objects, or both. Files, objects, or both typically contain data and may also contain metadata. Metadata is essentially data about data. Metadata refers to any descriptive or identifying information, in computer-processable form, associated with specific content, data, or a data set. Generally speaking, content will have metadata related to many characteristics of the content, the entire content collection, or both. Metadata tags or metadata attributes, also referred to as custom metadata tags or attributes, refer to any descriptive or identifying information, typically in computer-processable form, associated with specific metadata and indicate the actual information or content contained in various data storage systems with which the metadata is associated. Files are typically stored and organized in file systems or databases. Buckets are used to store objects containing data and metadata. Buckets are logical units of storage, for example, in IBM Cloud® Object Storage.

[0004] Many different types of data structures can be used in data processing and storage systems to store, sort, organize, manage, search, and locate data. These data structures include linked lists, hash tables, and tree structures. Tree structures, such as data trees, are often used as a technique to optimize the number of operations performed to find a particular item in an ordered data storage system or database. Large collections of data in a data storage system or database typically have associated directories, buckets, or indexes that speed searches by directing the search to the portion of the database or data storage system that contains the desired item, similar to labels on drawers and folders in a file cabinet. If a directory or bucket is large, another index, directory, or bucket may be built on top of it to further speed up searches. The resulting hierarchy is referred to as a tree. Organizing data, files, and objects in a tree helps to search and locate the data, files, or objects more efficiently.

[0005] Files, objects, and data can also be found by searching metadata. Users of databases, especially large databases, use metadata to search for files, objects, and data. Users prefer that metadata, such as custom metadata tags, be stored separately from the source storage system so that they can quickly and efficiently search the metadata independently of the source storage system. This is because searching on source storage system metadata can be slow and has performance implications on the source storage systems, requiring searchers to search each source storage system separately to search for metadata.

[0006] Existing systems, however, do not allow custom metadata tags set on directories in a source storage file system to be inherited when a file is created in a directory on a source storage system where the custom metadata tags are stored separately from the data on the source storage system (e.g., not stored inline). For example, if a source storage system stores metadata inline with the data and a directory of a source storage file has a custom metadata tag set in an extended attribute, when a new file is created in the directory on the source storage system, the existing system requires an application or user to retrieve the custom metadata tag from a higher-level directory and apply it manually or programmatically to the newly created file in the directory. Existing systems also do not allow custom metadata tags set at the bucket level to be inherited when an object is created in a bucket where the custom metadata tags are stored separately from the data on the source storage system (e.g., not stored inline). For example, if a source storage system stores metadata inline with data and a bucket in the source object storage system has custom metadata tags set in its extended attributes, when a source object is created in a bucket that already has the custom metadata tags set, a user or application must read the tags from the source storage system and apply them to the newly created object. Summary of the Invention

[0007] This summary of the present disclosure is provided to aid in understanding data storage systems, including metadata storage systems and their architectural structures, and methods for storing, organizing, managing, or searching data and metadata residing in data storage systems and separate metadata storage systems, or any combination thereof, and is not intended to limit the present disclosure or the present invention. This disclosure is directed to persons skilled in the art. It should be understood that various aspects and features of the present disclosure may be used separately in some examples or in combination with other aspects and features of the present disclosure in other examples. Accordingly, various changes and modifications may be made to the data storage systems, metadata storage systems, their architectural structures, and their methods of operation to achieve different effects.

[0008] A method, technique, process, system, or platform, or a combination thereof, for managing metadata is disclosed. In one or more embodiments, a method, process, or technique, or a combination thereof, for managing metadata is disclosed, including connecting a source data storage system (DSS) that stores both data and metadata to a metadata management platform (MMP); scanning metadata records from the DSS onto the MMP; storing metadata attributes for at least one of a group consisting of directories and buckets on the DSS in a lookup table on the MMP; and adding updated metadata attributes to the lookup table on the MMP for each subsequent scan of the DSS. In one aspect, the DSS stores the metadata separately from the data and stores the metadata attributes separately from the data in at least one of a group consisting of directories or buckets. According to one or more embodiments, the MMP is separate from and external to the DSS, and only the metadata is stored in the MMP. In one aspect, the method further includes scanning metadata from all files and all directories from the DSS onto the MMP, or scanning metadata from all objects and all buckets from the DSS onto the MMP. In one aspect, the lookup table includes a plurality of entries, each entry including at least one of a group of directories or buckets and its corresponding metadata attributes. Adding updates to the lookup table includes adding one or more new metadata attributes in the lookup table to at least one of a group of existing directories or existing buckets in one or more of the plurality of entries in the lookup table. In a further aspect, adding updates to the lookup table includes adding one or more new entries to the lookup table.

[0009] The method optionally further includes, in response to scanning the file or object metadata record on the MMP, examining the file path or object path and extracting a directory or bucket from the file path or object path. In one aspect, the MMP examines the file path and object path and extracts the directory or bucket. In response to extracting the directory or bucket from the file path or object path, the method in one embodiment further includes performing a lookup of the extracted directory or bucket in a lookup table. In response to matching the extracted directory or bucket to an entry in the lookup table, the method further includes applying any metadata attributes associated with the matching entry in the lookup table to the metadata record and storing the metadata record with the applied metadata attributes as an extended metadata record in the MMP with metadata attributes. In response to creating at least one group of files and objects on the connected DSS, the method may include pushing a notification from the DSS to the MMP containing metadata for at least one of the groups of files and objects. In response to the notification being pushed to the MMP, the method in one aspect further includes identifying an event type from the metadata as a creation instruction for at least one of a group of files and objects. The method may further include inspecting at least one of a group of file paths and object paths from the metadata pushed to the MMP, and extracting from the at least one of the group of file paths and object paths a directory or bucket corresponding to at least one of the files and objects created in the DSS.In one or more embodiments, the method includes, in response to extracting a directory or bucket from the metadata pushed to the MMP, performing a lookup to determine whether the extracted directory or bucket matches a directory or bucket in a lookup table, and, in response to matching the extracted directory or bucket to a directory or bucket in the lookup table, attaching metadata attributes corresponding to the matching directory or bucket in the lookup table to a metadata record to form an extended metadata record in the MMP with the metadata attributes. The extended metadata record with the metadata attributes is stored in the MMP.

[0010] In one or more embodiments, a system for managing metadata is provided that includes a metadata management platform (MMP) comprising circuitry and logic, the MMP configured to store data and metadata separate from the data, and to connect to a data storage system (DSS) having a metadata attribute set to at least one selected from the group consisting of a directory for a file system and a bucket for an object system. In one or more embodiments, the MMP includes a database for storing extended metadata records with metadata attributes; and a cache lookup table having a plurality of entries, each entry having at least one of a group consisting of a directory and a bucket and corresponding metadata attributes for the directory or bucket; the MMP is configured to receive the metadata records from the DSS and store the metadata attributes from the at least one of the group consisting of a directory or a bucket in the lookup table on the MMP; the MMP is configured to examine a file path or object path in the metadata record for a directory of a file or a bucket of an object, perform a search in the lookup table for an entry with a matching directory or bucket, apply the metadata attributes from the entry in the lookup table to the metadata record with the matching directory or bucket, create an extended metadata record in the MMP with the metadata attributes, and store the extended metadata record in the database on the MMP. In one aspect, the MMP is further configured to receive metadata records from traversals of the DSS and, with each additional traversal of the DSS, add updated metadata attributes to a lookup table on the MMP.An MMP according to an embodiment is further configured to: receive a notification containing metadata for at least one of the groups of files and objects in response to creation of at least one of the groups of files and objects on the DSS; examine at least one of the groups of file paths and object paths from the metadata in the received notification; extract a directory or bucket corresponding to at least one of the files and objects created on the DSS from the at least one group of file paths and object paths; and search a lookup table for an entry matching the extracted directory or bucket. In a further aspect, the MMP is configured to determine whether the extracted directory or bucket matches a directory or bucket in the lookup table; and, in response to the extracted directory or bucket matching one of the directories or buckets in the lookup table, add metadata attributes from the matched entry to the metadata record to form an extended metadata record; and store the extended metadata record in the database.

[0011] The above and other objects, features and advantages of the present invention will become apparent from the following more particular description of illustrative embodiments of the invention, as illustrated in the accompanying drawings, in which like reference numerals generally represent like parts of the illustrative embodiments of the invention.

[0012] Various aspects, features, and embodiments of data storage systems, data organizational and management structures, metadata storage and management systems and structures, and data organization trees, processing, sorting and organizing, managing, or searching data and metadata in data storage systems and metadata storage systems, or combinations thereof, will be better understood when read in conjunction with the provided drawings. Although embodiments are provided in the drawings for the purpose of illustrating various aspects, features, and embodiments of data storage systems, data organizational and management structures, metadata storage and management systems, and data organization trees, processing, sorting and organizing, managing, or searching data and metadata in data storage systems and metadata storage systems, or combinations thereof, the claims should not, however, be limited to the exact arrangements, structures, features, aspects, assemblies, subassemblies, systems, circuits, embodiments, or devices shown, and any arrangements, structures, assemblies, subassemblies, features, aspects, methods, processes, circuits, embodiments, or devices shown may be used alone or in combination with other arrangements, structures, assemblies, subassemblies, systems, features, aspects, circuits, embodiments, methods, and devices. [Brief explanation of the drawings]

[0013] [Figure 1] 1 illustrates an example of a computing environment according to an embodiment of the present disclosure. [Figure 2] FIG. 1 is a functional block diagram illustrating a computer system according to an embodiment of the present disclosure. [Figure 3] 1 illustrates an exemplary block diagram of an information and data storage and management system according to an embodiment of the present disclosure. [Figure 4] FIG. 1 is an exemplary block diagram of a data tree structure according to an embodiment of the present disclosure. [Figure 5] It is an example of a data storage file system that stores data and metadata, and a separate and external metadata storage system. [Figure 6] 1 is an example of a data storage system having a storage directory. [Figure 7] FIG. 1 illustrates an example of a data storage object system that stores data and metadata, and a separate and external metadata storage system. [Figure 8] 1 is an exemplary flowchart illustrating a method for scanning metadata, including custom metadata tags, into an external metadata management system that is separate from a data storage system that stores both data and metadata inline. [Figure 9] 1 is an exemplary flowchart illustrating a method for storing custom metadata tags in a metadata management system that is separate and external to a data storage system. DETAILED DESCRIPTION OF THE INVENTION

[0014] The following description is made for the purpose of illustrating the general principles of the present invention and is not intended to limit the inventive concepts claimed herein. In the following detailed description, numerous details are set forth to provide an understanding of the data storage system, metadata storage and management system, data and metadata organization and management structure, data tree, data and metadata processing, storing, organizing, managing, and / or searching, and combinations thereof. However, those skilled in the art will understand that a wide variety of embodiments of the data storage system, metadata storage system, data and metadata organization and management structure, data organization tree, data and metadata processing, storing, organizing, managing, and / or searching, and combinations thereof, may be realized without these specific details, and that the claims and disclosure should not be limited to the embodiments, assemblies, subassemblies, features, processes, methods, aspects, characteristics, or details specifically described and shown herein. Furthermore, particular features described herein may be used in combination with other features described in each of various potential combinations and permutations.

[0015] Unless expressly defined herein, all terms should be given their broadest interpretation, including the meaning implied by this specification, the meaning understood by a person skilled in the art, or as defined in dictionaries, treatises, etc., or any combination thereof. As used in this specification and the appended claims, the singular forms "a," "an," and "the" include plural referents unless otherwise expressly stated, and it should be noted that "comprises" and / or "comprising," when used herein, specify the presence of stated features, integers, steps, operations, elements, or components, or combinations thereof, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, or groups thereof, or combinations thereof.

[0016] As used herein, the terms "content" or "data" mean any computer-readable data, including, but not limited to, files simply named digital photographs, digitized analog photographs, music files, video clips, text documents, interactive programs, web pages, word processing documents, computer-aided design files, blueprints, flowcharts, invoices, database reports, database records, spreadsheets, charts, tables, graphs, video game assets, audio samples, transaction log files, electronic documents, other objects, etc. Content may be organized and stored in one or more data storage systems in objects, files, records, blocks, or any other suitable format, and may also include any hierarchy or organization of files, objects, records, directories, indexes, file system volumes, data blocks, buckets, extents, or other data blocks. As used herein, a "data set" may refer to (1) any file that is or was addressable by a file system, (2) a subset (e.g., a data block) of such a file, (3) any object that is or was addressable by a bucket, or (4) a subset of such an object. Data may include structured data (e.g., database files and objects), unstructured data (e.g., documents), or semi-structured data, or a combination thereof. Specifically, a data set may be a file, object, directory, bucket, block, share, volume, region within a volume, or embedded object. A data set may also be complex, containing other embedded objects. For example, a file or object may be a container that contains other files or objects, or a volume may have a file or object system on it that in turn contains files or objects.The system is able to recognize complex objects and track changes at the granularity of finer embedded objects.

[0017] As used herein, the term "metadata" refers to any descriptive or identifying information in a computer-processable format that is associated with particular content, data, or a data set. Generally speaking, content has metadata related to several characteristics relating to the content or the entire collection of content, or both, including, but not limited to, technical aspects of the content (format, bytes used, date of creation), the workflow the content is involved in (creator, owner, publisher, publication date, copyright information, etc.), and the subject matter of the content (the sonic qualities of an audio file such as music or sound effects, the subject matter of a photo or video clip, the summary of a longer text document, an excerpt from an invoice, or other data interchange format file).For example, the metadata items may be one or more of, but are not limited to, the content owner (e.g., the client or user who generated the content), last modified date and time (e.g., the time of the most recent modification of the data set), the name (file name) of the data set, the size (number of bytes in the data set), information about the content (indications regarding the presence of specific search terms), user-supplied or custom metadata tags, email to / from information (e.g., email sender, recipient, etc.), creation date, file type (e.g., format or application type), last accessed time, application type (e.g., the type of application that generated the data block), location / network (e.g., the current, past, or future location of the data set and the network route to or from the data block), geographic location (e.g., GPS coordinates), frequency of change (e.g., the period over which the data set will change), business unit (e.g., the group or department that generated, manages, or is associated with the set), aging information (e.g., the data This information may include, but is not limited to, the following: schedule, such as the period during which the set will be migrated to secondary or long-term storage; boot sector, partition layout, location of files within the file folder directory structure; user permissions, owner, group, access control lists (ACLs); system metadata (such as registry information); any combination of the above or other similar information associated with the data set.

[0018] The term "metadata tag" or "metadata attribute" (also referred to as a custom metadata tag or attribute) refers to any descriptive or identifying information in a computer-processable format that is associated with particular metadata and indicates the actual information or content contained in various data storage systems with which the metadata is associated. Metadata tags are also referred to as metadata attributes.

[0019] The following discussion omits or briefly describes conventional features of data storage systems and information processing systems, and metadata storage and management systems, including data and metadata organization and management structures, data trees, and data and metadata processing, storing, organizing, managing, or retrieving, or combinations thereof, that will be apparent to those skilled in the art. It is assumed that those skilled in the art are familiar with the general architecture of data storage systems, the operation of data storage systems or operations on stored data or combinations thereof, data and metadata storage systems, the operation of metadata storage systems or operations on stored metadata or combinations thereof, data and metadata organization and management structures, data trees, and data and metadata processing, storing, organizing, managing, or retrieving, or combinations thereof, in data organization structures. Note that numbered elements are numbered according to the figure in which they are introduced and are typically referenced by that number throughout subsequent figures.

[0020] Systems and methods are disclosed for storing, organizing, sorting, managing, processing, and searching collections of data and metadata. In one or more embodiments, systems, methods, and techniques are disclosed for storing metadata from a source storage file system (SSF) in a separate metadata management system external to the SSF. The separate metadata management system may be used to search for metadata independently of the SSF. In embodiments, systems, methods, and techniques are disclosed for inheriting custom metadata tags to a separate metadata storage and management system, where custom metadata tags are set on directories in the SSF when files are created in the directories of the SSF. In one aspect, the systems, methods, and techniques detect when a new file is created in a directory of the SSF whose directory has a custom metadata tag or attribute set, and in response to detecting the creation of the new file in the SSF, the custom metadata tag of the newly created file in the metadata storage system is updated with the custom metadata tag set at the directory level in the SSF.

[0021] In another aspect, in one or more embodiments, systems, methods, and techniques are disclosed for storing metadata from a source object storage system (SOS) in a separate metadata management system external to the SOS. In embodiments, systems, methods, and techniques are disclosed for inheriting custom metadata tags in a separate metadata storage and management system, where the custom metadata tag is set at the bucket level in the SOS when an object is created in a bucket in the SOS. In one aspect, the systems, methods, and techniques detect when an object is created in a bucket on the SOS that has a custom metadata tag or attribute set, and in response to detecting the creation of the new object in the SOS, the custom metadata tag of the newly created object in the metadata storage system is updated with the custom metadata tag set at the bucket level in the SOS.

[0022] Turning now to an environment in which the systems, methods, and techniques have potential application, Figure 1 illustrates a computer network system architecture 100 according to an embodiment. As shown in Figure 1, multiple remote networks 102 are provided, including a first remote network 104 and a second remote network 106. A gateway 101 may be coupled between the remote network 102 and a neighboring network 108. In the context of this architecture 100, the networks 104 and 106 may each take any form, including, but not limited to, a LAN, a WAN such as the Internet, a public switched telephone network (PSTN), or an internal telephone network.

[0023] In use, the gateway 101 acts as an entry point from the remote network 102 to the nearby network 108. In this way, the gateway 101 can function as a router that can direct a given packet of data arriving at the gateway 101, and as a switch that provides the actual path into and out of the gateway 101 for a given packet.

[0024] Additionally, at least one data server 114 is included, coupled to the proximate network 108 and accessible from the remote network 102 via the gateway 101. Note that the data server 114 may include any type of computing device / groupware. Coupled to each data server 114 are multiple user devices 116. The user devices 116 may also be directly connected through one of the networks 104, 106, and 108. Such user devices 116 may include desktop computers, laptop computers, handheld computers, printers, smartphones, or any other type of logic device. Note that in embodiments, the user device 116 may be directly coupled to one of the networks.

[0025] A peripheral device 120 or a series of peripheral devices 120, such as a facsimile machine, a printer, a network or local storage unit or system, or a combination thereof, may be coupled to one or more of the networks 104, 106, 108. Databases and / or additional components may be utilized with or integrated into any type of network element coupled to the networks 104, 106, 108. In the context of this description, a network element may refer to any component of a network.

[0026] According to some approaches, the methods and systems described herein may be implemented in conjunction with and / or on virtual systems and / or systems emulating one or more other systems, such as a UNIX system emulating an IBM® z / OS® environment, a UNIX system virtually hosting a Microsoft® Windows® environment, a Microsoft® Windows® system emulating an IBM® z / os environment, etc. This virtualization and / or emulation may be enhanced in some embodiments through the use of VMWARE® software.

[0027] In a more general approach, one or more networks 104, 106, 108 may represent a cluster of systems commonly referred to as a "cloud." In cloud computing, shared resources such as processing power, peripherals, software, data, servers, etc. are provided on an on-demand basis to any system in the cloud, thereby enabling access and delivery of services across multiple computing systems. Cloud computing typically involves internet connectivity between systems operating in the cloud, although other technologies for connecting systems may also be used.

[0028] Figure 2 illustrates a representative hardware environment associated with the user device 116 and / or server 114 of Figure 1, according to an embodiment. The diagram illustrates an exemplary hardware configuration of a workstation having a central processing unit 210, such as a microprocessor, and a number of other units interconnected via a system bus 212.

[0029] The workstation shown in FIG. 2 includes an I / O adapter 218 for connecting peripherals such as random access memory (RAM) 214, read-only memory (ROM) 216, and disk storage unit 220 to bus 212, a user interface adapter 222 for connecting a keyboard 224, a mouse 226, speakers 228, a microphone 232 and / or other user interface devices such as a touch screen or digital camera (not shown) to bus 212, a communications adapter 234 for connecting the workstation to a communications network 235 (e.g., a data processing network), and a display adapter 236 for connecting bus 212 to a display device 238.

[0030] The workstation may have an operating system resident thereon, such as the Microsoft® Windows® operating system (OS), Mac OS®, or UNIX® OS. It is understood that the preferred embodiment may also be practiced on platforms and operating systems other than those mentioned above. The preferred embodiment may be written using XML, C or C++, or other programming languages, along with object-oriented programming methodologies. Object-oriented programming (OOP), which is increasingly being used to develop complex applications, may also be used.

[0031] 3, an exemplary block diagram of an information management system 300 according to an embodiment of the present disclosure is shown, including a set of networked data storage systems 320A, 320B, ..., 320N and client devices 330A, 330B, ..., 330N connected via a data network 310. It should be understood that the implementations disclosed herein are not limited by the number of storage devices or data storage systems attached to the data network 310. It should also be understood that the storage devices or data storage systems attached to the data network 310 are not limited by communication protocols, storage environments, physical locations, etc.

[0032] In an embodiment, each data storage system 320A, 320B, ..., 320N may include a storage subsystem 321 and a storage device 322. The storage subsystem 321 may include a storage server or enterprise storage server, such as an IBM® Enterprise Storage Server® (IBM and Enterprise Storage Server are registered trademarks of IBM). The storage device 322 may include storage systems known in the art, such as a direct access storage device (DASD), a just bunch of disks (JBOD), a redundant array of independent disks (RAID), a virtualization device, tape storage, optical disk storage, or other data storage systems. The storage device 322 may contain content organized as object storage, file storage, or block storage, or a combination thereof. In certain embodiments, multiple storage subsystems may be implemented with one storage subsystem 321 and storage devices 322, or one storage subsystem may be implemented with one or more storage subsystems with attached storage devices.

[0033] In embodiments, metadata corresponding to the content of storage systems 320A, 320B, ..., 320N is collected and stored. The metadata may be stored, organized, protected, managed, manipulated, transferred, analyzed, or processed, or any combination thereof. Other types of information that provide insight into the content of storage systems 320A, 320B, ..., 320N may also be stored.

[0034] In certain embodiments, client devices 330A, 330B, ..., 330N may be general-purpose computers having multiple components. These components may include a central processing unit (CPU), main memory, I / O devices, and data storage devices (e.g., flash memory, hard drives, etc.). The main memory may be coupled to the CPU via a system bus or a local memory bus. The main memory may be used to provide the CPU access to data and / or program information stored in the main memory during execution. Typically, the main memory is comprised of random access memory (RAM) circuitry. A computer system having a CPU and main memory is often referred to as a host system. Client devices 330A, 330B, ..., 330N may have at least one operating system (e.g., Microsoft® Windows®, Mac® OS X®, iOS®, IBM® z / OS®, LINUX®, other UNIX®-based operating systems, etc.) installed thereon, which may support or host one or more file systems and other applications.

[0035] The data storage systems 320A, 320B, ..., 320N and client devices 330A, 330B, ..., 330N communicate according to well-known protocols, such as the Network File System (NFS) or Common Internet File System (CIFS) protocols, to make content stored in the data storage systems 320A, 320B, ..., 320N appear to users and / or application programs as if it were stored locally on the client systems 330A, 330B, ..., 330N. In a typical mode of operation, client devices 330A, 330B, ..., 330N send one or more input / output commands, such as NFS or CIFS requests, over computer network 310 to data storage systems 320A, 320B, ..., 320N, which in turn issue NFS or CIFS responses containing the requested content to each client device 330A, 330B, ..., 330N over network 310.

[0036] Client devices 330A, 330B, ..., 330N may execute (internally and / or externally) one or more applications that generate and manipulate content on one or more data storage systems 320A, 320B, ..., 320N. The applications generally facilitate the operation of an organization (or multiple affiliated organizations) and may include, without limitation, mail server applications (e.g., Microsoft Exchange Server), file server applications, mail client applications (e.g., Microsoft® Exchange Client), database applications (e.g., SQL, Oracle®, SAP®, Lotus Notes® databases), word processing applications (e.g., Microsoft® Word), spreadsheet applications, financial applications, presentation applications, browser applications, mobile applications, entertainment applications, etc. Applications may have the ability to access (eg, read and write) data storage systems 320A, 320B, . . . , 320N using a network file system protocol such as NFS or CIFS.

[0037] As shown, data storage systems 320A, 320B, ..., 320N, client devices 330A, 330B, ..., 330N, and other components within information management system 300 are connectable to one another via communications network 310. Communications network 310 may include one or more networks or other connection types, including the Internet, a wide area network (WAN), a local area network (LAN), a storage area network (SAN), a Fibre Channel connection, a small computer system interface (SCSI) connection, a virtual private network (VPN), a token ring or TCP / IP-based network, an intranet network, a point-to-point link, a cellular network, a wireless data transmission system, a two-way cable system, an interactive kiosk network, a satellite network, a broadband network, a baseband network, a neural network, a mesh network, an ad hoc network, other suitable wired, wireless, or partially wired / wireless computer or telecommunications network, combinations thereof, etc. The communications network 310 in some cases may include application programming interfaces (APIs), including, for example, cloud service provider APIs, virtual machine management APIs, and hosted service provider APIs.

[0038] Many different types of structures are used to store, sort, organize, and manage data and metadata in data storage systems (and metadata storage systems). Tree structures are one technique used to optimize the organization of large data and information collections and the ability to find specific items stored therein. Many different tree structures exist for organizing data (including metadata) and its indexes or directories. A B-tree is an example of a tree for organizing data stored in a data storage system. A B-tree is a self-balancing tree data structure that maintains sorted data and allows search, sequential access, insertion, and deletion in logarithmic time. A B-tree has a root node, internal nodes, and leaf nodes. The root may be either a leaf node or an internal node. Nodes in a B-tree may have more than two children. The order, or branching factor, "d," of a B-tree measures the capacity (number of child nodes) in the tree. Figure 4 shows an example B-tree 400 with a root node 410, internal nodes 420, and leaves 450. The nodes in tree 400 are shown as connected by pointers or references 430 and links 460. The lowest level 470 of the tree are leaf nodes 450 that are files or objects that contain the nodes, while the higher level or top portion 440 of tree 400, including interior nodes 420 and root 410, do not contain any files or objects and are directories or buckets (or indexes) to leaf nodes 450.

[0039] Referring to FIG. 5, an embodiment of a system 500 for managing metadata is illustrated, having a source data storage system (DSS) 510 and a metadata management system 530, also referred to as a metadata management platform (MMP) 530. The metadata storage management system 530 is external to and separate from the source data storage system 510. The source data storage system 510 is a source storage file system (SSF) 510, which includes both data 512 (e.g., data files) and metadata 514 and has a file directory 516, as shown in FIG. 6. The file directory 516, in one or more embodiments, has active or set custom metadata tags or metadata attributes 518. In an embodiment, the custom metadata tags 518 are stored separately from the data 512 (e.g., files) on the SSF 510. That is, the custom metadata tags 518 are not stored inline with the data (files) 512 on the SSF 510.

[0040] The metadata management system 530 includes a database 560 containing metadata records 540, a lookup table 550, and extended metadata records 570. The system 500 enables custom metadata tags set at the directory level in a source file system to be inherited by, e.g., applied to, a newly created metadata file in a separate metadata management system 530 when a file is created in a directory on the SSF. The system 500 has this capability in one or more embodiments where, for example, on the SSF, custom metadata tags are stored separately from the data, rather than stored inline with the data. The metadata management system 530 preferably maintains a lookup table 550 for each file system. The metadata lookup table 550 is preferably a cache structure and has a directory column 552 that identifies the file system directory and a custom metadata attribute or metadata tag column 554 that identifies the custom metadata attribute or custom metadata tag 518 for the corresponding directory. Each row or entry 556 corresponds to a single directory and its associated custom metadata attribute or tag. The lookup table 550 resides on the same system as the index of external metadata, for example, on the MMP 530.

[0041] In operation, a connection is established between the SSF 510 and the metadata management platform (MMP) 530. The MMP 530 begins scanning the attached SSF 510 to capture system metadata from the SSF's 510 files and directories. In one or more embodiments, scanning the SSF 510 captures metadata for all files and directories within the SSF 510 and generates metadata records 540. Custom metadata attributes or tags within directories on the SSF 510 are captured by the scan and stored (cached) in a lookup table 550 on the MMP 530. Scanning the SSF 510 to capture source metadata from SSF files and directories can be initiated on demand, set up to repeat on a scheduled or recurring basis, or set up to be initiated according to a trigger event, such as the volume or amount of data added to the SSF since the last scan. With each subsequent scan, the lookup table 550 is refreshed and updated. Any new custom metadata tags or attributes 518 added to existing directories in the SSF 510 are also added to the lookup table 550 in the MMP 530. Any new directories and their associated custom metadata tags or attributes 518 added to the SSF 510 since the last traversal are added to the lookup table 550 in the MMP 530.

[0042] During scanning of the SSF 510, when a file is ingested, the MMP 530, in one or more embodiments, examines the file path in the metadata record 540 and obtains the file's directory. In one aspect, the MMP 530 performs a lookup of the file's directory obtained from the file path in a lookup table 550. If there is a match (hit) in the lookup table 550, e.g., if the lookup is successful, then any custom metadata tags or attributes associated with the matching directory in the lookup table 550 are attached to the metadata record 540 to be indexed into the database 560. The metadata record 540 with the attached custom metadata tags is an extended metadata record 570, and the extended metadata record 570 is indexed into the database 560 on the MMP 530. An unsuccessful lookup of a file's directory in the lookup table 550 indicates that the directory custom metadata tag 518 has not yet been scanned by the MMP 530 and captured in the lookup table 550 on the MMP 530.

[0043] A reconciliation policy may be executed when a lookup or search during record ingestion is unsuccessful, meaning the directory does not have an associated entry in lookup table 550 because the information was not indexed in lookup table 550. When a row is added to lookup table 550, a subset of records in the metadata management system's index that have directories matching the custom metadata tag will be updated. A reconciliation policy may also be executed when directory entries are updated with new metadata and custom metadata tag updates are applied to those files in the directory already indexed in metadata management system 530. When an update is made to a row or set of rows in lookup table 550, a subset of records 570 in the metadata management system's index that have directories matching the updated metadata tag will be updated.

[0044] In one aspect, when a file is created in an SSF 510 that is connected or attached to an MMP 530, the SSF 510 pushes a notification containing SSF metadata 580 from the SSF 510 to the MMP 530. The MMP 530 identifies the event type from the metadata 580 pushed to the MMP 530 as the file was created. The file path in the metadata record 580 is inspected to extract the directory of the file, and the MMP 530 performs a lookup in the lookup table 550 for the directory extracted from the metadata 580. If the search for the extracted directory is successful in the lookup table, for example, if there is a matching entry in the lookup table 550, then any custom metadata tags or attributes associated with the matching directory from the lookup table 550 are attached to the metadata record to form an extended metadata record 570, which is indexed and stored in the database 560 within the MMP 530.

[0045] Referring to FIG. 7 , one embodiment of a system 700 for managing metadata is illustrated, including a source data storage system 710 and a metadata management system 730, also referred to as a metadata management platform (MMP) 730. The metadata storage management system 730 is external to and separate from the source data storage system 710. The source data storage system 710 in FIG. 7 is a source stored object system (SOS) 710, which contains both data 712 (e.g., objects) and metadata 714 and has buckets 716, as shown in FIG. 6. In one or more embodiments, the buckets 716 have active or set custom metadata tags or attributes 718 ( FIG. 6 ). In an embodiment, the custom metadata tags 718 are stored separately from the data 712 (e.g., objects) on the SOS 710. That is, the custom metadata tags 718 are not stored inline with the data (objects) 712 on the SOS 710.

[0046] The metadata management system 730 includes a database 760 containing metadata records 740, a lookup table 750, and extended metadata records 770. The system 700 enables custom metadata tags set at the bucket level in a source stored object system (SOS) to be inherited, e.g., applied to newly created metadata records, in the separate metadata management system 530 when objects are created in the SOS buckets. The system 700 has this capability in one or more embodiments where custom metadata tags are not stored inline with the data (objects) on the SOS; e.g., the custom metadata tags are stored separately from the data. The metadata management system 730 preferably maintains a lookup table 750 for each object system. The metadata lookup table 750 is preferably a cache structure having a bucket column 752 that identifies an object system bucket and a custom metadata attribute or tag column 754 that identifies a custom metadata attribute or tag 718 for the corresponding bucket. Each row or entry 756 corresponds to a single bucket and its associated custom metadata attribute or tag. The lookup table 750 resides on the same system as the index of external metadata, for example, the memory management system or platform 730.

[0047] In operation, a connection is established between the SOS 710 and the metadata management platform (MMP) 730. The MMP 730 initiates a scan of the attached SOS 710 to capture system metadata from the SOS 710's objects and buckets. In one or more embodiments, the SOS 710 scan captures metadata for all objects and buckets in the SOS 710, generating metadata records 740. Custom metadata attributes or tags in directories on the SOS 710 are captured by the scan and stored (cached) in a lookup table 750 on the MMP 730. The SOS 710 scan to capture SOS metadata from SOS objects and buckets can be initiated on demand, set up to repeat on a scheduled or recurring basis, or set up to be initiated according to a trigger event, such as the volume or amount of data added to the SOS since the last scan. With each subsequent scan, the lookup table 750 is refreshed and updated. Any new custom metadata tags or attributes 718 added to existing buckets in the SOS 710 will be added to the lookup table 750 in the MMP 730. Any new buckets and their associated custom metadata tags or attributes 718 added to the SOS 710 since the last scan will be added to the lookup table 750 in the MMP 730.

[0048] When an object is ingested during SOS 710 scanning, in one or more embodiments, MMP 730 examines the object path in metadata record 740 to obtain or extract the object's bucket. In one aspect, MMP 730 performs a lookup of the object's bucket in lookup table 750. If there is a match (hit) in lookup table 750, e.g., if the lookup is successful, then any custom metadata tags or attributes associated with the matching bucket in lookup table 750 are attached to metadata record 740. The metadata record 740 with the attached custom metadata tags is an extended metadata record 770, and the extended metadata record 770 is preferably indexed and stored in database 760 on MMP 730. An unsuccessful lookup of a file's bucket in the lookup table 750 indicates that the bucket custom metadata tag 718 has not yet been scanned by the MMP 730 and captured in the lookup table 750 on the MMP 730.

[0049] A reconciliation policy may be executed when a lookup or search during record ingestion is unsuccessful, meaning that a bucket does not have an associated entry in the lookup table 550 because the information is not indexed in the lookup table 550. When a row is added to the lookup table 550, a subset of records in the metadata management system's index that have a bucket matching the custom metadata tag will be updated. A reconciliation policy may be executed when a bucket entry is updated with new metadata and the custom metadata tag updates are applied to those objects in buckets already indexed in the metadata management system 530. When an update is made to a row or set of rows in the lookup table 550, a subset of records 570 in the metadata management system's index that have a bucket matching the updated metadata tag will be updated.

[0050] In one aspect, when an object is created in an SOS 710 that is connected or attached to an MMP 730, the SOS 710 pushes a notification containing metadata 780 from the SOS 710 to the MMP 730. The MMP 730 identifies the event type from the metadata pushed to the MMP 730 as the created file. The file path in the metadata record 780 is examined to extract a bucket, and the MMP 730 performs a lookup in the lookup table 750 for the bucket of the object extracted from the metadata 780. If the lookup in the lookup table for the extracted bucket is successful, for example, if there is a matching entry in the lookup table 750, then any custom metadata tags or attributes associated with the matching bucket from the lookup table 750 are attached to the metadata record to form an extended metadata record 770, which is indexed and stored in the database 760 within the MMP 730.

[0051] In one or more embodiments, the system and method detects, without user intervention or assistance, when a file or object is created in a directory or bucket that has a custom metadata attribute or tag set, and attaches the custom metadata attribute or tag to the newly created file or object in an index external metadata storage and management system.

[0052] 8, an exemplary flowchart according to various embodiments illustrates and describes a method for replicating or scanning metadata from a source data storage system (DSS) to a separate, external metadata storage and management system, preferably attaching any custom metadata tags or attributes to the newly created metadata records in the metadata storage system. The method 800 shown in FIG. 8 is described for convenience, but is not intended to limit the disclosure as including a series and / or number of steps, and it should be understood that the process need not be performed as a series of steps, or the steps need not be performed in the order shown and described in connection with FIG. 8, or both, but that the process may be integrated, or one or more steps may be performed together simultaneously, or both, or the steps may be performed in the order disclosed or in another order unless otherwise indicated.

[0053] In method 800, a connection is established between a source data storage system (DSS) and a metadata management platform or system at 802. Preferably, a metadata management platform (MMP) traverses the DSS to capture metadata from files, objects, directories, and / or buckets in the SSF to the MMP at 804. The traversal of the DSS preferably captures metadata for all files and directories in the source storage file system (SSF) or all objects and buckets in the source storage object system (SOS). The captured metadata on the MMP creates and / or forms metadata records. The traversal of directories or buckets on the DSS by the MMP captures any custom metadata tags or attributes set on directories or buckets on the DSS. The custom metadata tags or attributes captured by the traversal of the DSS directories or buckets at 806 are stored in a lookup table on the MMP. In one or more embodiments, the lookup table includes a directory or bucket column and a custom metadata attribute or tag column. In one embodiment, the MMP contains and maintains a lookup table for each data system whose metadata is stored in the external MMP.

[0054] In one aspect, for each subsequent scan of the DSS, any updates to custom metadata tags or attributes for new or existing directories or buckets in the DSS are added to the lookup table at 808. For example, if a new custom metadata tag or attribute is added to an existing directory or bucket in the DSS during a subsequent scan, the new tag is added to the lookup table for that directory or bucket. As a further example, if a new directory or bucket is added to the DSS during a subsequent scan, the new directory or bucket (and its corresponding custom metadata tag or attribute) is added to the lookup table. At 810, when a file or object from the DSS is ingested into the MMP, the MMP inspects the file path or object path in the metadata record to obtain or extract the file's directory or object's bucket and uses the extracted file's directory or object's bucket to perform a lookup or search in the lookup table for a matching directory or bucket. At 815, it is determined whether the search in the lookup table was successful. In other words, at 815, it is determined whether a hit or matching directory or bucket exists in the lookup table. If the search in the lookup table is successful (815:YES), then at 820, any custom metadata tags or attributes associated with the matching directory or bucket from the lookup table are attached to the metadata record to form an extended metadata record. At 820, the extended metadata record is also indexed and stored in a database on the MMP. In one or more embodiments, the custom metadata tags may be used to search for extended metadata records in the MMP independently of the DSS.

[0055] If the search in the lookup table for the directory or bucket obtained from the metadata file or object path is unsuccessful (815: NO), it means that the custom metadata attribute has not yet been scanned and captured in the lookup table. In this scenario, in embodiments, if the search in the lookup table is unsuccessful, a reconciliation policy may be executed or implemented at 825. In one aspect, the implemented reconciliation policy may add a row to the lookup table, or in a further aspect, may update a subset of records in the MMP's index that have directories or buckets that match the custom metadata tag. When a directory or bucket entry is updated with new metadata, an alternative or additional reconciliation policy may be implemented so that the updates to the custom metadata tag are applied to their associated files in directories or objects in buckets that are already indexed and stored in the MMP database. When an update is made to a row or set of rows in the lookup table, the subset of records indexed and stored in the MMP database with a matching directory or bucket will be updated with the updated custom metadata tag, unless otherwise indicated.

[0056] 9, an exemplary flowchart according to various embodiments shows and describes a method for applying custom metadata tags or attributes to a newly created file or object that is indexed and stored in an external metadata database when the file or object is created in a directory or bucket of a source data storage system (DSS) that has the custom metadata tags or attributes set. The method 900 shown in FIG. 9 is described for convenience and is not intended to limit the disclosure as including a series and / or number of steps, and it should be understood that the process need not be performed as a series of steps, or the steps need not be performed in the order shown and described in connection with FIG. 9, but rather the process may be integrated, or one or more steps may be performed together simultaneously, or the steps may be performed in the order disclosed or in another order.

[0057] In method 900, when a file or object is created in a source data storage system (DSS) connected to an external metadata management platform (MMP), the DSS pushes a notification containing the DSS's metadata to the MMP at 905. In one or more embodiments, the DSS has custom metadata tags or attributes set on directories or buckets, and in one or more embodiments, the custom metadata tags or attributes are not stored inline with the data on the DSS. At 910, the MMP identifies an event type from the metadata pushed to the MMP as a file or object creation instruction. The file path or object path of the metadata record is preferably inspected by the MMP at 915 to obtain or extract directory or bucket information / identification information. Using the extracted directory or bucket, the MMP performs a lookup or search in a lookup table for a matching directory or bucket at 920. If the extracted directory or bucket lookup matches an entry in a lookup table, e.g., if there is a hit in the lookup table, then at 925, any custom metadata tags or attributes associated with the matching directory or bucket are attached or applied to the metadata record to form an extended metadata record. At 930, the extended metadata record is indexed and stored in the MMP database. In one or more embodiments, the custom metadata tags can be used to search and find the extended metadata record within the MMP independent of the DSS.

[0058] While the exemplary embodiments described above are preferably implemented in hardware, such as processor units and circuits, various aspects of the exemplary embodiments may also be implemented in software. For example, it will be understood that each block of the flowcharts of FIGS. 8 and 9 and combinations of blocks in the flowchart diagrams may be implemented by computer program instructions. These computer program instructions may be provided to a processor or other programmable data processing device to produce a machine such that the instructions, executed on the processor or other programmable data processing device, produce means for performing the function(s) identified in the flowchart block(s). These computer program instructions may be stored in a computer-readable memory or storage medium that can direct the processor or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable memory or storage medium produce an article of manufacture including instruction means for performing the function(s) identified in the flowchart block(s).

[0059] In one or more embodiments, a computer program product is disclosed. The computer program product may be embodied on one or more computer-readable media and includes programming instructions that, when executed, cause a processor to (a) receive a metadata record from a data storage system (DSS) to a metadata management platform (MMP), (b) store metadata attributes from at least one of a group of directories or buckets in a lookup table on the MMP, (c) check a file path or object path in the metadata record for a directory of the file or a bucket of the object, (d) search the lookup table for an entry having a matching directory or bucket, (e) apply the metadata attributes to the metadata record from the lookup table entry having the matching directory or bucket to create an extended metadata record having the metadata attributes, and (f) store the extended metadata record in a database on the MMP. It will be understood that the instructions may cause a processor to operate in the order described or in an alternate order and may include alternate or additional instructions.

[0060] In one or more aspects, the computer program product further includes programming instructions that, when executed, cause the processor to receive metadata records from a scan of the DSS and add updated metadata attributes to a lookup table on the MMP for each additional scan of the DSS. In a further embodiment, the computer program product additionally or alternatively includes programming instructions that, when executed, cause the processor to: receive a notification containing metadata for at least one of the group of files and objects in response to creation of at least one of the group of files and objects on the DSS; examine at least one of the group of file paths and object paths from the metadata in the received notification; extract, from the at least one of the group of file paths and object paths, a directory or bucket corresponding to at least one of the files and objects created in the DSS; and search the lookup table for an entry matching the extracted directory or bucket. Optionally, in one or more embodiments, the computer program product further includes programming instructions that, when executed, cause the processor to determine whether the extracted directory or bucket matches a directory or bucket in the lookup table, and, in response to the extracted directory or bucket matching a directory or bucket of a plurality of entries in the lookup table, attach metadata attributes from the matched entry to the metadata record to form an extended metadata record, and store the extended metadata record in the database.

[0061] Thus, the blocks of the flowchart diagrams support means for performing a specified function, combinations of steps for performing a specified function, and combinations of program instruction means for performing a specified function. It will also be understood that each block of the flowchart diagrams and combinations of blocks of the flowchart diagrams may be implemented by a special-purpose hardware-based computer system that performs the specified function or step, or by a combination of special-purpose hardware and computer instructions.

[0062] There are many different data storage systems that store objects in buckets, including, for example, IBM Cloud® Object Storage, OpenStack® Swift, and Amazon S3®. For object storage, for example, in the case of Amazon S3® and OpenStack® Swift, a PUT or POST API call would be used to discover newly created objects in a bucket. For example, for object storage, the Spectrum® Discover metadata platform is separate and external to the source object storage system (SOS), and a PUT or POST REST API call would create an object in a bucket. In one or more embodiments, the metadata platform of this disclosure uses a PUT or POST REST API call to add custom metadata tags from the object bucket to the object in the metadata storage system or MMP (database). In an embodiment using Amazon S3®, the metadata storage platform (MMP) would be notified of the PUT or POST API call used by the source object storage system (SOS) to create an object in the bucket, and the MMP would apply the custom metadata tags from the object bucket to the object in the MMP database. In another embodiment using OpenStack® Swift, the Metadata Storage Platform (MMP) is notified of a PUT or POST API call used to create an object in a bucket by the Stored Object System (SOS), and the Metadata Storage Platform (MMP) applies custom metadata tags from the object bucket to the object in the Metadata Storage System (MMP) database. In another embodiment for a file system, a Virtual File System (VFS) layer in the source data storage file system would detect the create command and notify the Metadata Management Platform (MMP) of the create command.For example, in Spectrum® Discover, the VFS layer in the source data store file system detects the create command and notifies a metadata store system (a Spectrum® Discover instance) separate from the source data store file system of the create command, which, in an embodiment, applies custom metadata tags to the metadata record according to one or more embodiments described above.

[0063] One or more embodiments of the present disclosure may be a system, method, or computer program product, or a combination thereof. The computer program product may include a computer-readable storage medium (or media) having computer-readable program instructions thereon for causing a processor to perform aspects of the present disclosure.

[0064] A computer-readable storage medium may be a tangible device that holds and stores instructions for use by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the above. A list of more specific examples of computer-readable storage media includes portable computer diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disk read-only memory (CD-ROM), digital versatile disk (DVD), memory sticks, floppy disks, mechanically encoded devices such as punch cards or ridge structures in grooves with recorded instructions, and any suitable combination of the above. Computer-readable storage media, as used herein, is not to be construed as a transitory signal per se, such as an electric wave, a freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse passing through a fiber optic cable), or an electrical signal transmitted through a wire.

[0065] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device or to an external computer or external storage device over a network, such as the Internet, a local area network, a wide area network, or a wireless network, or a combination thereof. The network may include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, or edge servers, or a combination thereof. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and transfers the computer-readable program instructions for storage on a computer-readable storage medium within the respective computing / processing device.

[0066] Computer-readable program instructions for carrying out the operations of the present disclosure may be assembler instructions, instruction set architecture (ISA) instructions, machine language instructions, machine-dependent instructions, microcode, firmware instructions, state-setting data, or source or object code written in any combination of one or more programming languages, including object-oriented languages ​​such as Smalltalk®, C++, or the like, and conventional procedural languages ​​such as the C programming language or similar programming languages. The computer-readable program instructions may execute as a stand-alone software package entirely on the user's computer, partially on the user's computer, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be to an external computer (e.g., through the Internet using an Internet service provider). In some embodiments, the electrical circuitry may execute the computer-readable program instructions by utilizing state information of the computer-readable program instructions to individualize the electrical circuitry, which may include, for example, a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), to perform aspects of the present disclosure.

[0067] Aspects of the present disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0068] These computer-readable program instructions may be provided to a general-purpose computer, application-specific computer processor, or other programmable data processing apparatus to generate machine-generated instructions that, when executed by the computer processor or other programmable data processing apparatus, create means for implementing the functions / acts identified in the block or blocks of the flowchart diagrams and / or blocks. These computer-readable program instructions may also be stored on a computer-readable storage medium capable of directing a computer, programmable data processing apparatus, or other device, or combination thereof, to function in a particular manner, such that a computer-readable storage medium having instructions stored thereon includes an article of manufacture containing instructions that implement aspects of the functions / acts identified in the block or blocks of the flowchart diagrams and / or blocks.

[0069] The computer readable program instructions may also be loaded into a computer, other programmable data processing apparatus, or other device and cause the computer, other programmable data processing apparatus, or other device to perform a series of operational steps to generate a computer-implemented process such that the instructions executing on the computer, other programmable data processing apparatus, or other device implement aspects of the functions / actions identified in a block or blocks of the flowchart and / or blocks.

[0070] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowcharts or block diagrams may represent a module, segment, or portion of instructions, including one or more executable instructions for implementing a particular logical function. In some alternative implementations, the functions noted in the blocks may occur out of the order depicted in the figures. For example, two blocks shown in succession may be executed substantially concurrently, or the blocks may be executed in the reverse order, depending on the functionality involved. It should be noted that each block in the block diagrams and / or flowchart diagrams, and combinations of blocks in the block diagrams and / or flowchart diagrams, may be implemented by a special-purpose hardware-based system that performs a particular function or function or that implements a combination of special-purpose hardware and computer instructions.

[0071] Furthermore, systems according to various embodiments include a processor and logic integrated with and / or executable by the processor, the logic configured to perform one or more of the processing steps described herein. By integrated, it is meant that the processor has logic embedded in it as hardware logic, such as an application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), etc. By executable by the processor, it is meant that the logic is hardware logic, software logic, such as firmware, part of an operating system, or part of an application program, or some combination of hardware and software logic accessible by the processor and causing the processor to perform some function when executed by the processor. Software logic may be stored in any memory type, local or remote, or both, as known in the art. Any processor known in the art may be used, such as a software processor module, or a hardware processor, such as an ASIC, FPGA, central processing unit (CPU), integrated circuit (IC), graphics processing unit (GPU), or a combination thereof.

[0072] In the following claims, equivalents of corresponding structure, material, acts, and all means or step-plus-function elements are intended to include any structure, material, or acts for performing the function in combination with other claimed elements as explicitly claimed. The description of the embodiments of the present disclosure has been presented for purposes of illustration and description, but is not intended to be exhaustive or limited to the embodiments in the form disclosed. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the present disclosure. The embodiments and examples were chosen and described in order to best explain the principles and practical application of the present disclosure, and to enable those skilled in the art to appreciate the disclosure of various embodiments with various modifications suitable for the particular use contemplated.

[0073] The programs described herein are identified based on the application for which they are implemented in particular embodiments of the present disclosure. However, it should be understood that specific program nomenclature herein is used merely for convenience, and thus the present disclosure should not be limited to use only with the particular application identified or implied by such nomenclature.

[0074] It will be apparent that the various features of the systems and / or methods described above can be combined in any manner to create multiple combinations from the above description.

[0075] Furthermore, it will be appreciated that embodiments of the present disclosure may be provided in the form of a service that is deployed to provide services on demand on behalf of customers.

[0076] The description of various embodiments of the present disclosure has been presented for purposes of explanation, but is not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terms used herein have been chosen to best explain the principles of the embodiments, practical applications, or technical improvements to the technology found in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. 1. A method for managing metadata, comprising: connecting a source data storage system (DSS) that stores both data and metadata to a metadata management platform (MMP) having a lookup table, the DSS storing the metadata separately from the data and storing metadata attributes in directories or buckets separately from the data; scanning the DSS and capturing metadata for directories or buckets in the DSS and storing it as a first metadata record on the MMP; storing, in the lookup table on the MMP, a directory or bucket on the DSS related to the first metadata record acquired in the scanning and a corresponding metadata attribute in association with each other; and the method further comprises the processor: In response to a directory of the file or a bucket of the object extracted from a second metadata record, the second metadata record being metadata captured for the file or object in the DSS, matching an entry in the lookup table, applying metadata attributes associated with the matching entry in the lookup table to the second metadata record. How to perform.

2. The method further comprises: for each subsequent scan of the DSS following the scan, in response to updates regarding metadata attributes for new or existing directories or buckets on the DSS captured in the subsequent scan, adding the updated metadata attributes to the lookup table on the MMP. The method of claim 1 , comprising:

3. The method of claim 2 , wherein the MMP is separate from and external to the DSS, and only metadata is stored in the MMP.

4. the processor: Scanning all files and all directories on the DSS and capturing metadata for all files and all directories, or scanning all objects and all buckets on the DSS and capturing metadata for all objects and all buckets. The method of claim 2 , further comprising:

5. The method of claim 2 , wherein the lookup table includes a plurality of entries, each entry including a directory or bucket and its corresponding metadata attributes.

6. 6. The method of claim 5, wherein the adding step includes, in response to updates regarding metadata attributes for existing directories or buckets on the DSS captured in the subsequent scan, adding the one or more new metadata attributes related to the updates in the lookup table to the existing directory or existing bucket in one or more of a plurality of entries in the lookup table.

7. 6. The method of claim 5, wherein the adding step comprises, in response to updates regarding metadata attributes for new directories or buckets on the DSS captured in the subsequent scan, adding one or more new entries to the lookup table including one or more new metadata attributes related to the updates.

8. the processor: In response to storing the second metadata record corresponding to the file or object on the MMP, examining a file path or object path in the second metadata record; extracting a directory or bucket from the file path or object path; The method of claim 2 , further comprising:

9. The method of claim 8 , wherein the MMP inspects the file or object path and extracts the directory of the file or the bucket of the object.

10. the processor: In response to extracting a directory or bucket from the file path or object path, performing a lookup of the extracted directory or bucket in the lookup table.

9. The method of claim 8, further comprising: performing a lookup to determine whether the lookup matches an entry in the lookup table.

11. the processor: storing the second metadata record with the applied metadata attributes as an extended metadata record in the MMP having the metadata attributes. The method of claim 2 , further comprising:

12. After the step of associating and storing, the processor receiving a notification pushed from the connected DSS in response to a new file or object being created on the DSS, the notification including new metadata about the new file or object; inspecting file or object paths in the new metadata pushed to the MMP; extracting from the file path or object path in the new metadata the directory or bucket corresponding to the new file or object created in the DSS; in response to extracting a directory or bucket corresponding to the new file or object from the new metadata pushed to the MMP, performing a lookup to determine whether the extracted directory or bucket matches a directory or bucket in the lookup table; in response to the extracted directory or bucket matching a directory or bucket in the lookup table, appending the metadata attributes corresponding to the matching directory or bucket in the lookup table to a metadata record corresponding to the new metadata to form an extended metadata record in the MMP having the metadata attributes; The method of claim 2 , further comprising:

13. the processor: storing the extended metadata record with metadata attributes in the MMP. The method of claim 12 further comprising:

14. 1. A system for managing metadata, comprising: A metadata management platform (MMP) comprising circuitry and logic, the MMP configured to connect to a data storage system (DSS), the DSS storing data and metadata separate from the data, with metadata attributes set on directories for a file system or buckets for an object system, the MMP comprising: a database for storing extended metadata records having metadata attributes; a cache lookup table having a plurality of entries, each entry having a directory or bucket and corresponding metadata attributes for said directory or bucket; the MMP is configured to receive from the DSS metadata about files or objects and directories or buckets in the DSS, and to store metadata attributes from the directories or buckets related to the received metadata in the lookup table on the MMP in association with the directories or buckets related to the received metadata; the MMP is configured to check a file path or object path in the received metadata for the file or object against a directory or bucket of the file or object, perform a search in the lookup table for an entry having a matching directory or bucket, apply metadata attributes associated with the matching entry in the lookup table to the received metadata, create an extended metadata record in the MMP having the metadata attributes, and store the extended metadata record in the database on the MMP.

15. 15. The system of claim 14, wherein the MMP is further configured to receive metadata from the DSS in response to a scan, and, for each subsequent scan of the DSS following the scan, add to the lookup table on the MMP, in response to updates regarding metadata attributes for new or existing directories or buckets on the DSS captured in the subsequent scan, the updated metadata attributes.

16. The MMP may further comprise: receiving a notification in response to a new file or object being created on the DSS, the notification including new metadata about the new file or object; inspecting the file or object path in the new metadata in the received notification; extracting a directory or bucket corresponding to the new file or object created in the DSS from the file path or object path for the received notification; Searching the lookup table for an entry that matches the extracted directory or bucket for the received notification. The system of claim 14 configured to:

17. The MMP may further comprise: determining whether the extracted directory or bucket for the received notification matches a directory or bucket in the lookup table; responsive to the extracted directory or bucket for the received notification matching a directory or bucket of one of the plurality of entries in the lookup table, adding metadata attributes associated with the matching entry in the lookup table to the new metadata to form an extended new metadata record; Storing the new expanded metadata record in the database. The system of claim 16 configured to:

Images