Custom metadata tag inheritance based on file system directory tree or object storage bucket

Abstract

Methods and / or systems of managing metadata are disclosed, including connecting a source data storage system (DSS) that stores both data and metadata to a metadata management platform (MMP); scanning metadata records from the DSS onto the MMP; storing metadata attributes of at least one of a group consisting of directories and buckets on the DSS in a lookup table on the MMP; and for each subsequent scan of the DSS, adding updated metadata attributes to the lookup table on the MMP.

Description

Background Technology

[0001] This disclosure generally relates to improved data management, organization, and storage, particularly to data processing and storage systems that process and store large databases or large volumes of data. Specifically, this disclosure relates to systems and methods for storing, sorting, organizing, managing, and searching data and metadata, preferably large datasets and metadata collections, in a computerized environment.

[0002] The use of electronic data storage is widespread. The adoption of network storage and the relatively rapid increase in the amount of electronic data being created necessitate the organization, storage, and management of large volumes of electronic data. The storage facilities available on large computer systems allow users to store, update, and retrieve data from large datasets. Users and organizations dealing with large amounts of digital information often find it difficult to manage and search files in an efficient and intuitive way. The inability to easily store, organize, search, locate, and manage documents and content can translate into significant inefficiencies and lost opportunities.

[0003] To fully utilize computer resources, data should be organized intelligently to make the retrieval process efficient. Data can be collected and stored as files and / or objects. Files and / or objects typically contain data and may also include metadata. Metadata is essentially data about data. Metadata refers to any descriptive or identifying information in a computer-processable form associated with specific content, data, or datasets. Generally, content will have metadata associated with multiple characteristics of the content and / or the entire collection of content. Metadata tags or metadata attributes, also known as custom metadata tags or attributes, refer to any descriptive or identifying information, typically in a computer-processable form, that is associated with specific metadata and indicates the actual information or content included in various data storage systems and associated with the metadata. Files are typically stored and organized in file systems or databases. Buckets are used to store objects that include both data and metadata. A bucket is a logical storage unit, such as in an IBM Cloud Object Storage appliance.

[0004] In data processing and storage systems, various types of data structures can be used to store, sort, organize, manage, search, and locate data. These data structures include linked lists, hash tables, and tree structures. Tree structures, such as data trees, are frequently used as a technique to optimize the number of operations performed to locate specific items within an ordered data storage system or database. Typically associated with large collections of data in a data storage system or database are directories, buckets, or indexes, which are analogous to drawers in a filing cabinet and labels on folders, accelerating retrieval by directing searches to the section of the database or data storage system containing the desired item. If a directory or bucket is large, another index, directory, or bucket can be built on top of it to further accelerate retrieval, and so on. The resulting hierarchical structure is called a tree. Organizing data, files, and objects within a tree helps to search and locate data, files, or objects more efficiently.

[0005] Files, objects, and data can also be located by searching metadata. Database users, especially those using large databases, use metadata to search for files, objects, and data. Users want metadata, such as custom metadata tags, stored separately from the source storage system and want to be able to search for metadata quickly and efficiently, independently of the source storage system. This is because searching metadata on the source storage system can be slow, has performance implications for the source storage system, and the searcher must go to each source storage system separately to search for metadata.

[0006] However, existing systems do not allow the inheritance of custom metadata tags set at the directory level in the source storage file system when a file is created in a directory in the source storage system. These custom metadata tags are stored separately from the data on the source storage system (e.g., not consistently with it). For example, if the source storage system stores metadata consistently with the data and the directory in the source storage file system has custom metadata tags set in extended attributes, when a new file is created in the directory of the source storage file system, the existing system requires the application or user to obtain the custom metadata tags from a higher-level directory and apply them manually or programmatically to the newly created file in that directory. Similarly, when creating objects in buckets, existing systems do not allow the inheritance of custom metadata tags set at the bucket level, where the custom metadata tags are stored separately from the data on the source storage system (not consistently with it). For example, if the source storage system stores metadata consistently with the data and the bucket in the source object storage system has custom metadata tags set in extended attributes, when a source object is created in a bucket where custom metadata tags have been set, the user or application must read the tags from the source storage system and apply them to the newly created object. Summary of the Invention

[0007] The present disclosure is provided to aid in understanding data storage systems, including metadata storage systems and their architectures, as well as methods for storing, organizing, managing, and / or searching data and metadata residing in data storage systems and separate metadata storage systems, and is not intended to limit the scope of this disclosure or the invention. This disclosure is intended for those skilled in the art. It should be understood that various aspects and features of this disclosure may be used advantageously alone in some cases, or in combination with other aspects and features of this disclosure in others. Therefore, variations and modifications can be made to data storage systems, metadata storage systems, their architectures, and their methods of operation to achieve different effects.

[0008] Methods, techniques, processes, systems, and / or platforms for managing metadata are disclosed. In one or more embodiments, methods, processes, and / or techniques for managing metadata are disclosed, including connecting a source data storage system (DSS) storing both data and metadata to a metadata management platform (MMP); scanning metadata records from the DSS onto the MMP; storing metadata attributes of at least one of a group consisting of directories and buckets on the DSS in a lookup table on the MMP; and for each subsequent scan of the DSS, adding updated metadata attributes to the lookup table on the MMP. In one aspect, the DSS stores metadata separately from the data, and stores metadata attributes in at least one of a group consisting of directories or buckets separate from the data. According to one or more embodiments, the MMP is separate from the DSS and external to the DSS, and only metadata is stored on the MMP. In one aspect, the method further includes scanning metadata from all files and all directories from the DSS onto the MMP, or scanning metadata from all objects and all buckets from the DSS onto the MMP. In one aspect, the lookup table includes multiple entries, and each entry includes at least one of a group consisting of a directory or bucket and its corresponding metadata attributes. Adding an update to a lookup table may optionally include adding one or more new metadata attributes from the lookup table to at least one of a group consisting of one or more existing directories or existing buckets in the lookup table. Alternatively, adding an update to a lookup table may include adding one or more new entries to the lookup table.

[0009] The method optionally includes, in response to a file or object metadata record being scanned onto the MMP, examining the file path or object path and extracting a directory or bucket from the file path or object path. In one aspect, the MMP examines the file path and object path and extracts the directory or bucket. In response to extracting the directory or bucket from the file path or object path, the method in the embodiments includes performing a lookup of the extracted directory or bucket in a lookup table. In response to the extracted directory or bucket matching an entry in the lookup table, any metadata attributes associated with the matching entry in the lookup table are applied to the metadata record, and the metadata record having the applied metadata attributes is stored in the MMP as an enhanced metadata record with metadata attributes. In response to creating at least one of a group of files and objects on a connected DSS, the method may include pushing a notification containing metadata of at least one of the group of files and objects from the DSS to the MMP. In response to the notification being pushed to the MMP, the method, in one aspect, includes identifying an event type from the metadata as a creation instruction for at least one of the group of files and objects. The method may further include checking the metadata pushed to the MMP for at least one of a group consisting of file paths and object paths, and extracting a directory or bucket corresponding to at least one of the files and objects created in the DSS from the at least one of the group consisting of file paths and object paths. In response to the extraction of the directory or bucket from the metadata pushed to the MMP, the method, in one or more embodiments, includes performing a lookup to determine whether the extracted directory or bucket matches a directory or bucket in a lookup table, and in response to the extracted directory or bucket matching a directory or bucket in the lookup table, appending metadata attributes corresponding to the matching directory or bucket in the lookup table to the metadata record to form an enhanced metadata record with metadata attributes in the MMP. On one hand, the enhanced metadata record with metadata attributes is saved to the MMP.

[0010] A system for managing metadata, in one or more embodiments, includes a metadata management platform (MMP) comprising circuitry and logic, the MMP being configured to connect to a data storage system (DSS) that stores data and metadata separate from the data, and having metadata attributes set in at least one of a group consisting of directories of a file system and buckets of an object system. In one or more embodiments, the MMP includes: a database storing enhanced metadata records having metadata attributes; and a cache lookup table having multiple entries, each entry having at least one of a group consisting of directories and buckets and corresponding metadata attributes of directories or buckets, wherein the MMP is configured to receive metadata records from the DSS and store metadata attributes from at least one of the groups consisting of directories or buckets in the lookup table on the MMP, and wherein the MMP is further configured to examine file paths or object paths in the metadata records of directories or buckets of files, perform a search in the lookup table for entries with matching directories or buckets, apply metadata attributes to metadata records from entries in the lookup table with matching directories or buckets to create enhanced metadata records with metadata attributes, and store the enhanced metadata records in the database on the MMP. On one hand, the MMP is also configured to receive metadata records from scans of the DSS and add updated metadata attributes to a lookup table on the MMP for each additional scan of the DSS. According to an embodiment, the MMP is also configured to: receive a notification containing metadata of at least one of the groups of files and objects created on the DSS in response to the creation of at least one group of files and objects; check at least one of the groups of file paths and object paths from the metadata in the received notification; extract a directory or bucket corresponding to said at least one of the files and objects created on the DSS from said at least one group of file paths and object paths; and search the lookup table for entries that match the extracted directory or bucket. On the other hand, the MMP is configured to: determine whether the extracted directory or bucket matches a directory or bucket in the lookup table; in response to the extracted directory or bucket matching a directory or bucket in one of a plurality of entries in the lookup table, append metadata attributes from the matching entry to a metadata record to form an enhanced metadata record; and store the enhanced metadata record in a database.

[0011] The foregoing and other objects, features and advantages of the invention will become apparent from the following more detailed description of exemplary embodiments of the invention illustrated in the accompanying drawings, wherein the same reference numerals generally denote the same parts of exemplary embodiments of the invention. Attached Figure Description

[0012] When read in conjunction with the accompanying drawings, a better understanding will be gained of the various aspects, features, and embodiments of the data storage system, data organization and management structure, metadata storage and management system and structure, data organization tree, and the processing, storage, organization, management, and / or searching of data and metadata in the data storage system and metadata storage system. Embodiments are provided in the drawings to illustrate aspects, features, and / or various embodiments of the processing, storage, organization, management, and / or searching of data and metadata in the data storage system, data organization and management structure, metadata storage and management system, data organization tree, and data storage and organization structure. However, the claims should not be limited to the precise arrangements, structures, features, aspects, components, sub-components, systems, circuits, embodiments, or devices shown, and the shown arrangements, structures, components, sub-components, features, aspects, methods, processes, circuits, embodiments, and devices can be used alone or in combination with other arrangements, structures, components, sub-components, systems, features, aspects, circuits, embodiments, methods, and devices.

[0013] Figure 1 An example of a computing environment according to embodiments of the present disclosure is depicted.

[0014] Figure 2 This is a functional block diagram illustrating a computer system according to an embodiment of the present disclosure.

[0015] Figure 3 An example block diagram of an information and data storage / management system according to an embodiment of the present disclosure is shown.

[0016] Figure 4 This is an example block diagram of a data tree structure according to an embodiment of the present disclosure.

[0017] Figure 5 This is an example of a data storage file system that stores data and metadata, as well as a separate external metadata storage system.

[0018] Figure 6 It is an example data storage system with a storage directory.

[0019] Figure 7 It is an example of a data storage object system that stores data and metadata, as well as a separate external metadata storage system.

[0020] Figure 8 This is an exemplary flowchart illustrating a method for scanning metadata, including custom metadata tags, into an external metadata management system that is separate from a data storage system that consistently stores data and metadata.

[0021] Figure 9This is an exemplary flowchart illustrating and describing a method for storing custom metadata tags in a metadata management system that is separate from and outside the data storage system. Detailed Implementation

[0022] The following description is for illustrative purposes only and is not intended to limit the inventive concepts claimed herein. Numerous details are set forth in the following detailed description to provide an understanding of data storage systems, metadata storage and management systems, data and metadata organization and management structures, data trees, and the processing, storage, organization, management, and / or searching of data and metadata. However, those skilled in the art will understand that different and numerous embodiments of data storage systems, metadata storage systems, data and metadata organization and management structures, data organization trees, and the processing, storage, organization, management, and / or searching of data and metadata can be practiced without these specific details, and the claims and disclosure should not be limited to the embodiments, components, subcomponents, features, processes, methods, aspects, features, or details specifically described and illustrated herein. Furthermore, specific features described herein may be used in various possible combinations and permutations with other described features.

[0023] Unless otherwise specifically defined herein, all terms shall be given their broadest possible interpretation, including the meaning implied in the specification and the meaning as understood by those skilled in the art and / or as defined in dictionaries, papers, etc. It must also be noted that, as used in the specification and appended claims, unless otherwise stated, the singular forms “a,” “an,” and “the” include plural referents, and when used in this specification, the terms “comprising” and / or “including” specify the presence of the stated features, integers, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0024] As used herein, the terms “content” or “data” mean any computer-readable data, including but not limited to digital photographs, digitized analog photographs, music files, video clips, text documents, interactive programs, web pages, word processing documents, computer-aided design documents, blueprints, flowcharts, invoices, database reports, database records, spreadsheets, charts, tables, graphs, video game assets, sound samples, transaction log files, electronic documents, files simply named other objects, etc. Content may be organized and stored in one or more data storage systems in the form of objects, files, records, blocks, or any other suitable format, and may include any other hierarchy or organization of files, objects, records, directories, indexes, file system volumes, data blocks, buckets, extents, or data blocks. As used herein, “dataset” may mean (1) any file that is or was previously addressable by a file system; (2) a subset of such files (e.g., data blocks); (3) any object that is or was previously addressable by a bucket; or (4) a subset of such objects. Data may include structured data (e.g., database files and objects), unstructured data (e.g., documents), and / or semi-structured data. Specifically, a dataset can be a file, object, directory, bucket, block, share, volume, region within a volume, or embedded object. The dataset can be complex, containing other embedded objects. For example, a file or object can be a container containing other files or objects, or a volume can have a file or object system on top of it, which in turn contains files or objects. This system is capable of recognizing complex objects and tracking changes at a finer granularity of embedded objects.

[0025] As used herein, the term "metadata" refers to any descriptive or identifying information in a computer-processable form associated with specific content, data, or datasets. Generally, content will have metadata related to multiple characteristics of the content and / or the entire collection of content, including but not limited to the technical aspects of the content (format, bytes used, creation date), the workflow in which the content is involved (creator, owner, publication date, copyright information, etc.), and the subject matter of the content (the nature of the sound in an audio file, whether music or sound effects; the subject of a photo or video clip; a summary of a lengthy text document; excerpted details of an invoice; or other data-interchangeable format files). For example, metadata items may include, but are not limited to, one or more of the following: content owner (e.g., the client or user who generated the content), last modified time (e.g., the time when the dataset was last modified), dataset name (e.g., filename), dataset size (e.g., the number of bytes in the dataset), information about the content (e.g., an indication of the existence of a specific search term), user-provided or custom metadata tags, email sender / recipient information (e.g., email sender, recipient, etc.), creation date, file type (e.g., format or application type), last access time, and application type (e.g., the type of application that generated the data block). Location / network (e.g., the current, past, or future location of the dataset and network paths to / from data blocks), geographic location (e.g., GPS coordinates), frequency of change (e.g., the period during which the dataset was modified), business unit (e.g., the group or department that generates, manages, or otherwise associates with the dataset), aging information (e.g., schedules, such as the period during which the dataset was migrated to secondary or long-term storage), boot sector, partition layout, file locations within folder directory structure, user permissions, owners, groups, access control lists (ACLS), system metadata (e.g., registry information), and combinations of the same or other similar information associated with the dataset.

[0026] The terms "metadata tag" or "metadata attribute" (also known as a custom metadata tag or attribute) refer to any descriptive or identifying information in a computer-processable form associated with specific metadata, and it indicates the actual information or content included in and associated with metadata in various data storage systems. Metadata tags are also referred to as metadata attributes.

[0027] The following discussion omits or only briefly describes the general characteristics of data storage systems and information processing systems, metadata storage and management systems, including data and metadata organization and management structures, data trees, and the processing, storage, organization, management and / or retrieval of data and metadata, which will be apparent to those skilled in the art. It is assumed that those skilled in the art are familiar with the general architecture of data storage systems, the operation of data storage systems and / or the operation of stored data, metadata storage systems, the operation of metadata storage systems and / or the operation of stored metadata, data and metadata organization and management structures, data trees, and the processing, storage, organization, management and / or retrieval of data and metadata within data organization structures. It can be noted that numbered elements are numbered according to the accompanying drawings in which the element is introduced, and are generally referenced by that number in subsequent drawings.

[0028] Systems and methods for storing, organizing, sorting, managing, processing, and / or searching collections of data and metadata are disclosed. In one or more embodiments, systems, methods, and techniques for storing metadata from a source storage file system (SSF) to a separate metadata management system outside the SSF are disclosed. The separate metadata management system can be used to search metadata independently of the SSF. In embodiments, systems, methods, and techniques for inheriting custom metadata tags into a separate metadata storage and management system are disclosed, wherein a custom metadata tag is set at the directory in the SSF when a file is created in the SSF. In one aspect, the system, methods, and techniques detect when a new file is created in a directory of the SSF, wherein the directory has a set custom metadata tag or attribute, and in response to detecting the creation of a new file in the SSF, update the custom metadata tag of the newly created file in the metadata storage system with a directory-level custom metadata tag set in the SSF.

[0029] In another aspect, in one or more embodiments, systems, methods, and techniques for storing metadata from a source object storage system (SOS) to a separate metadata management system outside the SOS are disclosed. In these embodiments, systems, methods, and techniques for inheriting custom metadata tags into the separate metadata storage and management system are disclosed, wherein when an object is created in a bucket in the SOS, the custom metadata tag is set at the bucket level in the SOS. In one aspect, the system, methods, and techniques detect when an object is created in a bucket on the SOS, wherein the bucket has a set custom metadata tag or attribute, and in response to detecting the creation of a new object in the SOS, update the custom metadata tag of the newly created object in the metadata storage system with the custom metadata tag set at the bucket level in the SOS.

[0030] Move to an environment where the systems, methods, and technologies have potential applications. Figure 1An architecture 100 of a computer network system according to an embodiment is shown. For example... Figure 1 As shown, multiple remote networks 102 are provided, including a first remote network 104 and a second remote network 106. A gateway 101 may be coupled between the remote network 102 and the neighboring network 108. In the context of this architecture 100, networks 104 and 106 may each take any form, including but not limited to LANs, WANs such as the Internet, public switched telephone networks (PSTN), internal telephone networks, etc.

[0031] In use, gateway 101 serves as the entry point from remote network 102 to neighboring network 108. Thus, gateway 101 can function as both a router and a switch; a router can guide a given data packet to gateway 101, and a switch can provide the actual path for a given packet to and from gateway 101.

[0032] It also includes at least one data server 114 coupled to a neighboring network 108, and which is accessible from a remote network 102 via a gateway 101. It should be noted that the data server 114 may include any type of computing device / component. Coupled to each data server 114 are multiple user devices 116. User devices 116 may also be directly connected via one of networks 104, 106, and 108. Such user devices 116 may include desktop computers, laptop computers, handheld computers, printers, smartphones, or any other type of logical device. It should be noted that in this embodiment, user devices 111 may also be directly coupled to any network.

[0033] Peripheral device 120, or a series of peripheral devices 120, such as fax machines, printers, networking and / or local storage units or systems, may be coupled to one or more of networks 104, 106, and 108. It should be noted that databases and / or additional components may be used with or integrated into any type of network element coupled to networks 104, 106, and 108. In the context of this specification, network element may refer to any component of the network.

[0034] According to some methods, the methods and systems described herein can be implemented using and / or on a virtual system and / or a system emulating one or more other systems, such as a UNIX system emulating an IBM z / OS environment, a UNIX system virtually hosting a Microsoft Windows environment, a Microsoft Windows system emulating an IBM z / OS environment, etc. In some embodiments, this virtualization and / or emulation can be enhanced by the use of VMware software.

[0035] In other methods, one or more networks 104, 106, 108 can represent a cluster of systems commonly referred to as a "cloud." In cloud computing, shared resources such as processing power, peripherals, software, data, servers, etc., are provided to any system in the cloud on an on-demand basis, allowing access to and distribution of services across many computing systems. Cloud computing typically involves internet connectivity between systems operating in the cloud, but other technologies for connecting systems can also be used.

[0036] Figure 2 The illustration shows the relationship with the embodiment. Figure 1 The figure illustrates a representative hardware environment associated with user equipment 116 and / or server 114. It also shows an example hardware configuration for a workstation having a central processing unit 210, such as a microprocessor, and several other units interconnected via a system bus 212.

[0037] Figure 2 The workstation shown includes random access memory (RAM) 214, read-only memory (ROM) 216, I / O adapter 218 for connecting peripheral devices such as disk storage unit 220 to bus 212, user interface adapter 222 for connecting keyboard 224, mouse 226, speaker 228, microphone 232 and / or other user interface devices such as touch screen and digital camera (not shown) to bus 212, communication adapter 234 for connecting the workstation to communication network 235 (e.g., data processing network), and display adapter 236 for connecting bus 212 to display device 238.

[0038] The workstation can host an operating system, such as Microsoft Windows. Operating systems (OS), macOS, UNIX OS, etc. It should be understood that preferred embodiments can also be implemented on platforms and operating systems other than those mentioned. Preferred embodiments can be written using XML, C and / or C++ or other programming languages ​​and object-oriented programming methodologies. Object-oriented programming (OOP), which has become increasingly used for developing complex applications, can be used.

[0039] Now for reference Figure 3 This diagram illustrates an example block diagram of an information management system 300 according to an embodiment of the present disclosure, comprising a group of networked data storage systems 320a, 320b...320n communicating via a data network 310 and client devices 330a, 330b...330n. It is understood that the implementations disclosed herein are not limited by the number of storage devices or data storage systems attached to the data network 310. It is also understood that the storage devices or data storage systems attached to the data network 310 are not limited by communication protocols, storage environments, physical locations, etc.

[0040] In one embodiment, each data storage system 320a, 320b...320n may include a storage subsystem 321 and a storage device 322. The storage subsystem 321 may include a storage server or an enterprise storage server, such as an IBM enterprise storage server. (IBM and Enterprise Storage Server are registered trademarks of IBM). Storage device 322 may include storage systems known in the art, such as Direct Access Storage Device (DASD), Just a Bunch of Disks (JBOD), Redundant Array of Independent Disks (RAID), virtualization devices, tape storage, optical disc storage, or any other data storage system. Storage device 322 may include content organized as object storage, file storage, and / or block storage. In some embodiments, multiple storage subsystems may be implemented in a single storage subsystem 321 and storage device 322, or a storage subsystem may be implemented using one or more storage subsystems with attached storage devices.

[0041] In this embodiment, metadata corresponding to the content of storage systems 320a, 320b...320n is collected and stored. The metadata can be stored, organized, protected, managed, manipulated, moved, analyzed, and / or processed. Other types of information that typically provide insights into the content of storage systems 320a, 320b...320n may also be stored.

[0042] In some embodiments, client devices 330a, 330b...330n may be general-purpose computers with multiple components. These components may include a central processing unit (CPU), main memory, I / O devices, and data storage devices (e.g., flash memory, hard disk drives, etc.). Main memory may be coupled to the CPU via a system bus or a local memory bus. Main memory can be used to provide the CPU with access to data and / or program information stored in main memory during execution. Typically, main memory consists of random access memory (RAM) circuitry. A computer system with a CPU and main memory is generally referred to as a host system. Client devices 330a, 330b...330n may have at least one operating system installed thereon (e.g., Microsoft Windows, Mac OS X, iOS, IBM z / OS, Linux, other Unix-based operating systems, etc.), which may support or host one or more file systems and other applications.

[0043] Data storage systems 320a, 320b...320n and client devices 330a, 330b...330n communicate according to known protocols (such as Network File System (NFS) or Common Internet File System (CIFS) protocols) to make content stored on data storage systems 320a, 320b...320n appear to users and / or applications as if the content were stored locally on client systems 330a, 330b...330n. In a typical operating mode, client devices 330a, 330b...330n send one or more input / output commands (such as NFS or CIFS requests) to data storage systems 320a, 320b...320n via computer network 310, which in turn send an NFS or CIFS response containing the requested content to the corresponding client devices 330a, 330b...330n via network 310.

[0044] Client devices 330a, 330b…330n can execute (internal and / or external) one or more applications that generate and manipulate content on one or more data storage systems 320a, 320b…320n. Applications generally facilitate the operation of an organization (or multiple affiliated organizations) and may include, but are not limited to, mail server applications (e.g., Microsoft Exchange Server), file server applications, mail client applications (e.g., Microsoft Exchange Client), database applications (e.g., SQL, Oracle, SAP, Lotus Notes databases), word processing applications (e.g., Microsoft Word), spreadsheet applications, financial applications, presentation applications, browser applications, mobile applications, entertainment applications, etc. Applications may also have the ability to access (e.g., read and write) data storage systems 320a, 320b…320n using network file system protocols (such as NFS or CIFS).

[0045] As shown in the figure, data storage systems 320a, 320b...320n, client devices 330a, 330b...330n, and other components in the information management system 300 can be interconnected via communication network 310. Communication network 310 may include one or more networks or other connection types, including but not limited to any of the following: the Internet, wide area network (WAN), local area network (LAN), storage area network (SAN), Fibre Channel connection, small computer system interface (SCSI) connection, virtual private network (VPN), token ring or TCP / IP-based network, intranet, point-to-point link, cellular network, wireless data transmission system, two-way cable system, interactive kiosk network, satellite network, broadband network, baseband network, neural network, mesh network, self-organizing network, other suitable wired, wireless, or partially wired / wireless computer or telecommunications network, or combinations thereof. In some cases, communication network 310 may also include application programming interfaces (APIs), including, for example, cloud service provider APIs, virtual machine management APIs, and managed service provider APIs.

[0046] Many different types of structures are used to store, sort, organize, and manage data and metadata in data storage systems (and metadata storage systems). A tree structure is a technique used to optimize the organization and location of specific items stored within large collections of data and information. Several different tree structures exist for organizing data (including metadata) and its index or directory. A B-tree is an example of a tree used to organize data stored on a data storage system. A B-tree is a self-balancing tree data structure that maintains ordered data and allows for search, sequential access, insertion, and deletion in logarithmic time. A B-tree has a root node, internal nodes, and leaf nodes. The root can be a leaf node or an internal node. A node in a B-tree can have more than two child nodes. The order or branching factor "d" of a B-tree measures the capacity of an internal node in the tree (i.e., the number of child nodes). Figure 4 An example B-tree 400 with a root node 410, internal nodes 420, and leaves 450 is shown. Nodes in tree 400 are shown connected by pointers or references 430 and chains 460. The lowest level 470 of the tree is the leaf nodes 450, which are files or objects containing nodes, and the higher levels or top portion 440 of tree 400, including internal nodes 420 and the root 410, do not contain any files or objects and are directories or buckets (or indexes) leading to leaf nodes 450.

[0047] refer to Figure 5An embodiment of a system 500 for managing metadata is shown, which has a source data storage system (DSS) 510 and a metadata management system 530, also referred to as a metadata management platform (MMP) 530. The metadata storage and management system 530 is external to and separate from the source data storage system 510. The source data storage system 510 is a source storage file system (SSF) 510 that contains both data 512 (e.g., data files) and metadata 514, and has, for example... Figure 6 The file directory 516 is shown. In one or more embodiments, the file directory 516 has active or set custom metadata tags or metadata attributes 518. In an embodiment, the custom metadata tags 518 are stored separately from the data 512 (e.g., files) on the SSF 510. That is, the custom metadata tags 518 are not stored concurrently with the data (files) 512 on the SSF 510.

[0048] Metadata management system 530 includes metadata records 540, lookup tables 550, and a database 560 containing enhanced metadata records 570. When a file is created in a directory of the source file system (SSF), system 500 allows custom metadata tags set at the directory level in the source file system (SSF) to be inherited by, for example, applied to a newly created metadata file in a separate metadata management system 530. System 500 has this capability in one or more embodiments where custom metadata tags are not stored consistently with the data (e.g., custom metadata tags are stored separately from the data) on the SSF. Metadata management system 530 preferably maintains a lookup table 550 for each file system. Metadata lookup table 550 is preferably a cache structure and has a directory column 552 that identifies the file system directory and a custom metadata attribute or metadata tag column 554 that identifies the custom metadata attribute or custom metadata tag 518 of the corresponding directory. Each row or entry 556 corresponds to a single directory and its associated custom metadata attribute or tag. Lookup table 550 resides on the same system as external metadata indexes (such as MMP 530).

[0049] In operation, a connection is established between SSF 510 and Metadata Management Platform (MMP) 530. MMP 530 initiates a scan of the attached SSF 510 and captures system metadata from the files and directories of SSF 510. In one or more embodiments, the scan of SSF 510 captures metadata for all files and directories in SSF 510 and creates metadata records 540. Custom metadata attributes or tags in directories on SSF 510 are captured by the scan and stored (cached) in lookup table 550 on MMP 530. Scans of SSF 510 can be initiated on demand to capture source metadata from SSF files and directories, built to repeat on a scheduled or cyclical basis, or built based on a triggering event (e.g., the amount or quantity of data added to the SSF since the last scan). For each subsequent scan, lookup table 550 is refreshed and updated. Any new custom metadata tags or attributes 518 added to existing directories in SSF 510 are added to lookup table 550 in MMP 530. Any new directories added to SSF 510 since the last scan, along with their associated custom metadata tags or attributes 518, will also be added to lookup table 550 in MMP 530.

[0050] During the scanning of SSF 510, when a file is ingested, in one or more embodiments, MMP 530 examines the file path in metadata record 540 and obtains the directory of the file. In one aspect, MMP 530 performs a lookup of the directory of the file obtained from the file path in lookup table 550. If a match (hits) exists in lookup table 550, for example, a successful lookup, any custom metadata tags or attributes associated with the matching directory in lookup table 550 are appended to metadata record 540 for indexing in database 560. Metadata record 540 with the appended custom metadata tags is enhanced metadata record 570, which is then indexed in database 560 on MMP 530. An unsuccessful lookup of the directory of the file in lookup table 550 indicates that the directory custom metadata tag 518 has not yet been scanned by MMP 530 and captured in lookup table 550 on MMP 530.

[0051] If a lookup or search during record ingestion is unsuccessful, a reconciliation strategy can be executed. This means that the directory has no associated entries in lookup table 550 because the information has not yet been indexed into lookup table 550. After a row is added to lookup table 550, a subset of the records in the metadata management system index that contain directories matching the custom metadata tags will be updated. A reconciliation strategy can also be executed if directory entries are updated with new metadata so that the custom metadata tag updates are applied to those files within the directory that are already indexed in the metadata management system 530. When a row or set of rows in lookup table 550 is updated, a subset of the records 570 in the metadata management system index that contain directories matching the updated metadata tags will be updated.

[0052] On one hand, when a file is created in an SSF 510 connected to or attached to MMP 530, SSF 510 pushes a notification containing SSF metadata 580 from SSF 510 to MMP 530. When a file is created, MMP 530 identifies the event type from the metadata 580 pushed to MMP 530. The file path in metadata record 580 is checked to extract the directory of the file, and MMP 530 performs a lookup (search) in lookup table 550 against the directory extracted from metadata 580. If the search for the extracted directory in the lookup table is successful, for example, if a matching entry exists in lookup table 550 (hit), then any custom metadata tags or attributes associated with the matching directory from lookup table 550 are appended to the metadata record to form an enhanced metadata record 570, and this enhanced metadata record 570 is indexed and stored in database 560 in MMP 530.

[0053] refer to Figure 7 An embodiment of a system 700 for managing metadata is shown, which has a source data storage system 710 and a metadata management system 730, also referred to as a metadata management platform (MMP) 730. The metadata storage and management system 730 is external to and separate from the source data storage system 710. Figure 7 The source data storage system 710 is a source storage object system (SOS) 710, and contains both data 712 (e.g., objects) and metadata 714, and has the following characteristics: Figure 6 The bucket 716 shown. In one or more embodiments, bucket 716 has a custom metadata tag or attribute 718 that is active or set. Figure 6 In this embodiment, the custom metadata tag 718 is stored separately from the data 712 (e.g., the object) on the SOS 710. That is, the custom metadata tag 718 is not stored concurrently with the data (object) 712 on the SOS 710.

[0054] Metadata management system 730 includes metadata records 740, lookup tables 750, and a database 760 containing enhanced metadata records 770. System 700 allows custom metadata tags set at the bucket level in the source storage object system (SOS) to be inherited, for example, applied to newly created metadata records in a separate metadata management system 530, when an object is created in a bucket in the source storage object system (SOS). System 700 has this capability in one or more embodiments, wherein the custom metadata tags are not stored co-located with the data (object), for example, the custom metadata tags are stored separately on the SOS from the data. Metadata management system 730 preferably maintains a lookup table 750 for each object system. Metadata lookup table 750 is preferably a cache structure and has a bucket column 752 identifying the object system bucket, and a custom metadata attribute or tag column 754 identifying the custom metadata attribute or custom metadata tag 718 of the corresponding bucket. Each row or entry 756 corresponds to a single bucket and its associated custom metadata attribute or tag. Lookup table 750 resides on the same system as an external metadata index, such as a storage management system or platform 730.

[0055] In operation, a connection is established between SOS 710 and Metadata Management Platform (MMP) 730. MMP 730 initiates a scan of the attached SOS 710 and captures system metadata from the objects and buckets of SOS 710. In one or more embodiments, the scan of SOS 710 captures metadata for all objects and buckets in SOS 710 and creates metadata records 740. Custom metadata attributes or tags in the catalog on SOS 710 are captured by the scan and stored (cached) in a lookup table 750 on MMP 730. Scans of SOS 710 can be initiated as needed to capture SOS metadata from SOS objects and buckets, established to repeat on a scheduled or loop basis, or established to be initiated based on a triggering event, such as the amount or quantity of data added to SOS since the last scan. For each subsequent scan, lookup table 750 is refreshed and updated. Any new custom metadata tags or attributes 718 added to existing buckets in SOS 710 are added to lookup table 750 in MMP 730. Any new buckets added to SOS 710 since the last scan, along with their associated custom metadata tags or attributes 718, will be added to lookup table 750 in MMP 730.

[0056] During the SOS 710 scan, when an object is ingested, in one or more embodiments, the MMP 730 examines the object path in the metadata record 740 and obtains or retrieves the object's bucket. In one aspect, the MMP 730 performs a bucket lookup for the object in a lookup table 750. If a match (hits) exists in the lookup table 750, for example, a successful lookup, any custom metadata tags or attributes associated with the matching bucket in the lookup table 750 are appended to the metadata record 740. The metadata record 740 with the appended custom metadata tags is an enhanced metadata record 770, and the enhanced metadata record 770 is preferably indexed and stored in a database 760 on the MMP 730. An unsuccessful bucket lookup for a file in the lookup table 750 indicates that the bucket custom metadata tag 718 has not yet been scanned by the MMP 730 and captured in the lookup table 750 on the MMP 730.

[0057] If a lookup or search during record ingestion is unsuccessful, a reconciliation strategy can be executed. This means that the bucket has no associated entries in lookup table 550 because the information has not yet been indexed into lookup table 550. After a row is added to lookup table 550, a subset of records in the metadata management system index that have buckets matching the custom metadata tags will be updated. A reconciliation strategy can also be executed if updating bucket entries with new metadata causes the custom metadata tag update to be applied to those objects within the bucket that are already indexed in the metadata management system 530. When updating a row or a set of rows in lookup table 550, a subset of records in the metadata management system index that have buckets matching the updated metadata tags will be updated.

[0058] On one hand, when an object is created in an SOS 710 connected to or attached to an external MMP 730, the SOS 710 pushes a notification containing a metadata record 780 from the SOS 710 to the MMP 730. Upon object creation, the MMP 730 identifies the event type from the metadata pushed to the MMP 730. The file path in the metadata record 780 is checked to extract buckets, and the MMP 730 performs a lookup (search) in a lookup table 750 for the bucket of the object extracted from the metadata record 780. If the search for the extracted bucket in the lookup table is successful, for example, if a matching entry (hit) exists in the lookup table 750, any custom metadata tags or attributes associated with the matching bucket from the lookup table 750 are appended to the metadata record to form an enhanced metadata record 770, which is preferably indexed and stored in a database 760 within the MMP 730.

[0059] In one or more embodiments, without user intervention or assistance, the system and method detect when a file or object is created in a directory or bucket with a set of custom metadata attributes or tags, and attach the custom metadata attributes or tags to the newly created file or object in the index external metadata storage and management system.

[0060] Now for reference Figure 8 Exemplary flowcharts according to various embodiments illustrate and describe a method for copying or scanning metadata from a source data storage system (DSS) to a separate external metadata storage and management system, and preferably attaching any custom metadata tags or attributes to newly created metadata records in the metadata storage system. While for convenience and not to limit this disclosure... Figure 8 The method 800 shown is described as comprising a series and / or multiple steps; however, it should be understood that the process does not need to be performed as a series of steps, and / or these steps do not need to be related to... Figure 8 The process may be performed in the order shown and described, but the process may be integrated and / or one or more steps may be performed together or simultaneously, or these steps may be performed in the disclosed order or an alternative order, unless otherwise indicated.

[0061] At 802 in method 800, a connection is established between the source data storage system (DSS) and the metadata management platform or system. At 804, metadata is preferably captured from files or objects and / or directories or buckets in the SSF and transferred to the MMP by the metadata management platform (MMP). The DSS scan preferably captures metadata for all files and directories in the source storage file system (SSF) or all objects and buckets in the source storage object system (SOS). The metadata captured on the MMP creates and / or forms metadata records. The scan of directories or buckets on the DSS by the MMP captures any custom metadata tags or attributes set for directories or buckets on the DSS. At 806, the custom metadata tags or attributes captured by the scan of the DSS directories or buckets are stored in a lookup table on the MMP. In one or more embodiments, the lookup table contains a directory or bucket column and a custom metadata attribute or tag column. In embodiments, the MMP has and maintains a lookup table for each data system to which its metadata is saved to an external MMP.

[0062] On one hand, for each subsequent scan of the DSS, at 808, any updates to the custom metadata tags or attributes for new or existing directories or buckets in the DSS are added to the lookup table. For example, if a new custom metadata tag or attribute is added to an existing directory or bucket in the DSS during a subsequent scan, the new tag is added to the lookup table for that directory or bucket. As another example, if a new directory or bucket is added to the DSS during a subsequent scan, the new directory or bucket (and its corresponding custom metadata tag or attribute) is added to the lookup table. At 810, when a file or object from the DSS is ingested into the MMP, the MMP checks the file or object path in the metadata record to obtain or extract the directory or bucket of the file, and performs a lookup or search in the lookup table against the matching directory or bucket using the bucket of the extracted file's directory or object. At 815, it is determined whether the search in the lookup table was successful. In other words, at 815, it is determined whether a hit or matching directory or bucket exists in the lookup table. If the search in the lookup table is successful (815: Yes), then at 820, any custom metadata tags or attributes associated with the matching directory or bucket from the lookup table are appended to the metadata record to form an enhanced metadata record. The enhanced metadata record at 820 is also indexed and stored in a database on the MMP. In one or more embodiments, custom metadata tags can be used to search for enhanced metadata records in the MMP independently of the DSS.

[0063] If a search for a directory or bucket obtained from a metadata file or object path in the lookup table is unsuccessful (815: No), it means that the custom metadata attribute has not yet been scanned and captured in the lookup table. In the event of an unsuccessful search in the lookup table, in the embodiment of 825, a reconciliation strategy can be executed or implemented. On one hand, the implemented reconciliation strategy may add a row to the lookup table, and on the other hand, it may update a subset of records in the MMP index that have directories or buckets matching the custom metadata tag. Alternative or additional reconciliation strategies may be implemented where updating directory or bucket entries with new metadata so that the update to the custom metadata tag is applied to associated files or objects within buckets already indexed and stored in the MMP database. When updating a row or set of rows in the lookup table, the subset of records indexed and stored in the MMP database that have matching directories or buckets will be updated with the updated custom metadata tag, unless otherwise specified.

[0064] Now for reference Figure 9Exemplary flowcharts according to various embodiments illustrate and describe a method for applying custom metadata tags or attributes to newly created files or objects that are indexed and stored in an external metadata database when a file or object is created in a directory or bucket of a source data storage system (DSS) with custom metadata tags or attributes set. While for convenience and not for limiting this disclosure, Figure 9 The method 900 shown is described as comprising a series and / or multiple steps; however, it should be understood that the process does not need to be performed as a series of steps, and / or these steps do not need to be related to... Figure 9 The process may not be performed in the order shown and described, but the process may be integrated and / or one or more steps may be performed together, simultaneously, or these steps may be performed in the disclosed order or an alternative order.

[0065] In method 900, when a file or object is created in a source data storage system (DSS) connected to an external metadata management platform (MMP), at 905, the DSS pushes a notification containing the DSS's metadata to the MMP. In one or more embodiments, the DSS has custom metadata tags or attributes set in a directory or bucket, and in one or more embodiments, the custom metadata tags or attributes are not stored consistently on the DSS with the data. At 910, the MMP identifies the event type from the metadata pushed to the MMP as a file or object creation instruction. At 915, preferably, the MMP examines the file or object path of the metadata record to obtain or extract directory or bucket information / identifiers. At 920, using the extracted directory or bucket, the MMP performs a lookup or search of a lookup table against the matching directory or bucket. If the lookup using the extracted directory or bucket matches an entry in the lookup table, for example, a hit exists in the lookup table, then at 925, any custom metadata tags or attributes associated with the matching directory or bucket are appended or applied to the metadata record, forming an enhanced metadata record. At point 930, the enhanced metadata record is indexed and stored in the MMP database. In one or more embodiments, custom metadata tags can be used to search for and locate the enhanced metadata record in the MMP independently of the DSS.

[0066] While the illustrative embodiments described above are preferably implemented in hardware (such as processor units and circuitry), various aspects of the illustrative embodiments may also be implemented in software. For example, it will be understood that... Figures 8-9Each block of the flowchart illustration, and combinations of blocks in the flowchart illustration, can be implemented by computer program instructions. These computer program instructions can be provided to a processor or other programmable data processing apparatus to produce a machine, such that the instructions, which execute on the processor or other programmable data processing apparatus, create parts for implementing the functions specified in one or more blocks of the flowchart. These computer program instructions can also be stored in a computer-readable storage medium or storage medium that can direct a processor or other programmable data processing apparatus to operate in a particular manner, such that the instructions stored in the computer-readable storage medium or storage medium produce an article of writing comprising instruction parts that implement the functions specified in one or more blocks of the flowchart.

[0067] In one or more embodiments, a computer program product is disclosed. The computer program product may be contained on one or more computer-readable media and includes programming instructions that, when executed, cause a processor to: (a) receive metadata records from a data storage system (DSS) to a metadata management platform (MMP); (b) store metadata attributes of at least one of a group of future free directories or buckets in a lookup table on the MMP; (c) examine the file path or object path of a directory or bucket of a file in the metadata record; (d) perform a search in the lookup table for entries with matching directories or buckets; (e) apply the metadata attributes to the metadata record from entries in the lookup table with matching directories or buckets to create an enhanced metadata record with metadata attributes; and (f) store the enhanced metadata record in a database on the MMP. It will be understood that the instructions may cause the processor to operate in the stated order or an alternative order, and may include alternative or additional instructions.

[0068] The computer program product in one or more aspects further includes programming instructions that, when executed, cause the processor to: receive metadata records from a scan of the DSS; and for each additional scan of the DSS, add updated metadata attributes to a lookup table on the MMP. In a further embodiment, the computer program product may additionally or alternatively include programming instructions that, when executed, cause the processor to: receive a notification containing metadata of at least one of the groups of files and objects created on the DSS in response to at least one of the groups of files and objects being created; check at least one of the groups of file paths and object paths from the metadata in the received notification; extract a directory or bucket corresponding to at least one of the files and objects created on the DSS from at least one of the groups of file paths and object paths; and search the lookup table for entries matching the extracted directory or bucket. Optionally, in one or more embodiments, the computer program product may further include programming instructions that, when executed, cause the processor to: determine whether the extracted directory or bucket matches a directory or bucket in a lookup table; in response to the extracted directory or bucket matching a directory or bucket in one of a plurality of entries in the lookup table, append metadata attributes from the matching entry to a metadata record to form an enhanced metadata record; and store the enhanced metadata record in a database.

[0069] Therefore, the blocks in a flowchart illustration support combinations of components for performing a specified function, combinations of steps for performing a specified function, and program instruction components for performing a specified function. It will also be understood that each block in a flowchart illustration, and combinations of blocks in a flowchart illustration, can be implemented by a dedicated hardware-based computer system that performs the specified function or step, or by a combination of dedicated hardware and computer instructions.

[0070] There are multiple different data storage systems that store objects in buckets, including, for example, IBM Cloud Object Storage, OpenStack Swift, and Amazon S3. For object storage, such as in Amazon S3 and OpenStack Swift, a PUT or API call is used to detect newly created objects in the bucket. For example, in object storage, the spectrum discovery metadata platform is separate from and outside the source object storage system (SOS), and a PUT or POST REST API call creates objects in the bucket. In one or more embodiments, the metadata platform of this disclosure is used, along with a PUT or POST REST API call, and a custom metadata tag from the object bucket is added to the object in the metadata storage system or MMP (database). In an embodiment with Amazon S3, the metadata storage platform (MMP) is notified of the PUT or POST API call used by the storage object system (SOS) to create objects in the bucket, and the MMP applies the custom metadata tag from the object bucket to the object in the MMP database. In another embodiment with OpenStack Swift, a PUT or POST API call for creating an object in a bucket by the Storage Object System (SOS) is notified to the Metadata Storage Platform (MMP), and the MMP applies custom metadata tags from the object bucket to the object in the MMP database. In another embodiment of the file system, the Virtual File System (VFS) layer in the source data storage file system detects the create command and notifies the Metadata Management Platform (MMP). For example, in Spectrum Discover, the VFS layer in the source data storage file system detects the create command and notifies the metadata storage system (Spectrum Discover instance), which is decoupled from the source data storage file system. In this embodiment, according to one or more embodiments disclosed above, custom metadata tags are applied to the metadata record based on the create command.

[0071] One or more embodiments of this disclosure may be a system, method, and / or computer program product. A computer program product may include a computer-readable storage medium (or media) having computer-readable program instructions thereon for causing a processor to perform aspects of this disclosure.

[0072] Computer-readable storage media can be tangible devices capable of retaining and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example, but not limited to, electronic storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of computer-readable storage media includes the following: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable optical disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices such as punch cards or recessed structures with instructions recorded thereon, and any suitable combination of the foregoing. As used herein, computer-readable storage media should not be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0073] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to a suitable computing / processing device, or via a network, such as the Internet, a local area network (LAN), a wide area network (WAN), and / or a wireless network, to an external computer or external storage device. The network may include copper cables, optical fibers, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to a computer-readable storage medium within the respective computing / processing device.

[0074] Computer-readable program instructions used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​(such as Smalltalk, C++, etc.) and conventional procedural programming languages ​​(such as the "C" programming language or similar programming languages). The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter case, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry including, for example, programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs) may execute computer-readable program instructions by utilizing state information from the computer-readable program instructions to personalize the electronic circuitry in order to perform aspects of this disclosure.

[0075] This document describes aspects of the disclosure with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the disclosure. It will be understood that each block of the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer-readable program instructions.

[0076] These computer-readable program instructions may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create parts for implementing the functions / actions specified in one or more blocks of a flowchart and / or block diagram. These computer-readable program instructions may also be stored in a computer-readable storage medium that can instruct a computer, programmable data processing apparatus, and / or other device to operate in a particular manner, such that the computer-readable storage medium in which the instructions are stored includes an article of writing comprising instructions for implementing aspects of the functions / actions specified in one or more blocks of a flowchart and / or block diagram.

[0077] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer-implemented process, such that the instructions, which execute on the computer, other programmable apparatus or other device, perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0078] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of instructions comprising one or more executable instructions for implementing a specified logical function. In some alternative embodiments, the functions mentioned in the blocks may occur in a non-linear order as shown in the figures. For example, depending on the functions involved, two blocks shown consecutively may be executed substantially simultaneously, or these blocks may sometimes be executed in reverse order. It will also be noted that each block illustrated in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented by a dedicated hardware-based system that performs the specified function or action or executes a combination of dedicated hardware and computer instructions.

[0079] Furthermore, systems according to various embodiments may include a processor and logic integrated with and / or integrated by the processor, the logic being configured to perform one or more of the process steps described herein. Integration means that the processor has logic embedded therein as hardware logic, such as an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), etc. Processor-executable means that the logic is hardware logic; software logic, such as firmware, part of an operating system, part of an application; etc., or some combination of hardware and software logic that is accessible to the processor and configured to cause the processor to perform a certain function when executed by the processor. Software logic may be stored in local and / or remote memory of any memory type, as is known in the art. Any processor known in the art may be used, such as software processor modules and / or hardware processors, such as ASICs, FPGAs, central processing units (CPUs), integrated circuits (ICs), graphics processing units (GPUs), etc.

[0080] All means or steps plus functional elements in the following claims are intended to include any structure, material, action, and equivalent for performing a function in combination with other claimed elements as specifically claimed. Descriptions of embodiments of this disclosure have been presented for illustrative and descriptive purposes, but are not intended to be exhaustive or limited to the forms disclosed. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of this disclosure. The embodiments and examples were chosen and described in order to best explain the principles and practical application of this disclosure and to enable others skilled in the art to understand the various embodiments of this disclosure with various modifications suitable for the particular purpose contemplated.

[0081] The programs described herein are identified based on applications that implement them in specific embodiments of this disclosure. However, it should be understood that any particular program names herein are used for convenience only, and therefore the invention should not be limited to use only in any particular application identified and / or implied by such names.

[0082] It will be clear that the various features of the aforementioned systems and / or methods can be combined in any way, resulting in multiple combinations from the description presented above.

[0083] It will also be understood that embodiments of this disclosure may be provided in the form of services deployed on behalf of customers to provide services on demand.

[0084] Various embodiments of this disclosure have been described for illustrative purposes, but are not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein has been chosen to best explain the principles of the embodiments, their practical application, or improvements to existing technologies in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A method for managing metadata, comprising: The source data storage system DSS, which stores both data and metadata, is connected to the metadata management platform MMP, wherein the DSS stores the metadata separately from the data and stores the metadata attributes in at least one of a group consisting of directories or buckets separate from the data; Scanning each metadata record from the DSS to the MMP, wherein scanning each metadata record from the DSS to the MMP includes storing each metadata record in the MMP; In response to a metadata record being scanned onto the MMP, the MMP examines the file path or object path of the metadata record and extracts a directory from the file path or a bucket from the object path; In response to retrieving the directory from the file path or the bucket from the object path, a lookup of the retrieved directory or bucket is performed in a lookup table, wherein the lookup table comprises multiple entries, and each entry includes at least one of a group consisting of directories or buckets and a corresponding metadata attribute; and In response to a retrieved directory or bucket matching an entry in the lookup table, any metadata attributes associated with the matching entry in the lookup table are applied to the metadata record, and the metadata record and the applied metadata attributes are stored in the MMP.

2. The method according to claim 1, wherein, The MMP is separate from the DSS and is located outside the DSS, and only metadata is stored on the MMP.

3. The method according to claim 1, wherein, Scanning metadata records includes scanning metadata from all files and all directories from the DSS onto the MMP, or scanning metadata from all objects and all buckets from the DSS onto the MMP.

4. The method according to claim 1, wherein, Adding updated metadata attributes to the lookup table includes adding one or more new metadata attributes from the lookup table to at least one of a group consisting of existing directories or existing buckets from one or more of the plurality of entries in the lookup table.

5. The method according to claim 1, wherein, Adding updated metadata attributes to the lookup table includes adding one or more new entries to the lookup table.

6. The method of claim 1, further comprising, in response to creating at least one of a group of files and objects on the connected DSS, pushing a notification containing the metadata of the at least one of the groups of files and objects from the DSS to the MMP.

7. The method of claim 6, further comprising, in response to the notification being pushed to the MMP, identifying an event type from the metadata as a creation instruction for at least one of the group consisting of the files and objects.

8. The method of claim 6, further comprising checking at least one of the group consisting of the file path and the object path from the metadata pushed to the MMP, and extracting from at least one of the group consisting of the file path and the object path a directory or bucket corresponding to at least one of the files and objects created in the DSS.

9. The method of claim 8, further comprising, in response to retrieving the directory or bucket from the metadata pushed to the MMP, performing a lookup to determine whether the retrieved directory or bucket matches a directory or bucket in the lookup table, and in response to the retrieved directory or bucket matching a directory or bucket in the lookup table, applying the metadata attribute corresponding to the matching directory or bucket in the lookup table to the metadata record to form an enhanced metadata record with metadata attributes in the MMP.

10. The method of claim 9, further comprising storing the enhanced metadata record having metadata attributes in the MMP.

11. A system for managing metadata, comprising: Metadata management platform MMP includes: Circuitry containing a processor; A database used to store enhanced metadata records with metadata attributes; and A cache lookup table with multiple entries, each entry having at least one of a group consisting of directories, buckets, and combinations thereof, as well as a corresponding metadata attribute for said group. The MMP is configured as follows: Connected to a data storage system (DSS) that stores data and metadata separate from the data, and has metadata attributes set in at least one of the groups; Each metadata record from the DSS is scanned and stored on the MMP; In response to scanning metadata records onto the MMP, the file path or object path of the metadata records is checked; In response to inspecting the file path or object path of the metadata record, extract the directory from the file path or the bucket from the object path; In response to extracting the directory from the file path or the bucket from the object path, a search for the extracted directory or bucket is performed in the lookup table; and In response to a retrieved directory or bucket matching an entry in the lookup table, any metadata attributes associated with the matching entry in the lookup table are applied to the metadata record, and the metadata record and the applied metadata attributes are stored in the MMP.

12. The system according to claim 11, wherein, Configuring the MMP to add updated metadata attributes to the lookup table includes configuring the MMP to add one or more new metadata attributes from the lookup table to at least one of a group consisting of existing directories or existing buckets in one or more of the plurality of entries in the lookup table.

13. The system according to claim 11, wherein, The MMP is also configured to: In response to the creation of at least one of a group consisting of files and objects on the DSS, a notification pushed from the DSS containing the metadata of the at least one of the groups consisting of files and objects is received from the DSS. Check at least one of the group consisting of the file path and the object path from the metadata pushed to the MMP; as well as Extract from at least one of the groups consisting of the file path and the object path a directory or bucket corresponding to at least one of the files and objects created in the DSS.

14. The system according to claim 13, wherein, The MMP is also configured to: In response to extracting the directory or bucket from the metadata pushed to the MMP, a lookup is performed to determine whether the extracted directory or bucket matches a directory or bucket in the lookup table; as well as In response to a retrieved directory or bucket matching a directory or bucket in the lookup table, the metadata attribute corresponding to the matching directory or bucket in the lookup table is applied to the metadata record to form an enhanced metadata record with metadata attributes in the MMP.

Images