Metadata federation management system and method under uniform namespace

Abstract

The application provides a metadata federation management system and method under a unified namespace, which comprises a data storage unit, a data directory unit, a metadata federation unit and a data gateway unit. The data storage unit comprises a plurality of heterogeneous storage systems, and the types of the plurality of heterogeneous storage systems are different. The data directory unit is used for managing business metadata of each heterogeneous storage system; the business metadata is used for describing data content, data source and data correlation. The metadata federation unit is used for managing technical metadata of each heterogeneous storage system; the technical metadata is used for describing logical-to-physical mapping information, access protocol and endpoint information, authentication and authorization information of data. The data gateway unit is used for providing protocol conversion and is responsible for data routing to read and cache data. Through unified management of metadata by the metadata federation unit, the data island problem in the heterogeneous storage environment can be avoided.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of information storage, and in particular to a metadata federation management system and method under a uniform namespace. BACKGROUND

[0002] Large model applications, for example, enterprises or units train large-scale multi-modal large models, and the data is mainly obtained through massive multi-source heterogeneous data collection (such as Internet public text, professional corpus, synthetic data) and enhanced generation through cross-modal alignment (such as picture-text synthesis based on semantic association). In the data full life cycle, the storage system needs to provide PB (Petabytes, petabytes) level or even EB (Exabytes, exabytes) level high-concurrency throughput, cross-modal data consistency guarantee, and long-period fault recovery functions.

[0003] For units with massive data management needs, storage resources can be used to create an automated and scalable global data environment to provide users with local access experience for data that may exist on various storage types from one or more locations of different vendors. However, for the management of various types of storage systems, it is usually necessary to move massive amounts of metadata to independent metadata servers (or metadata clusters), and metadata migration is very time-consuming, especially for PB-level or even EB-level storage systems, which increases the time overhead of system management, and the storage systems being managed can only provide performance-limited storage access during migration. SUMMARY

[0004] Therefore, the present application provides a metadata federation management system and method under a uniform namespace to at least solve the technical problems existing in the related art.

[0005] According to a first aspect of the present application, a metadata federation management system under a uniform namespace is provided, and the system comprises:

[0006] A data storage unit comprising a plurality of heterogeneous storage systems, the types of the plurality of heterogeneous storage systems being different;

[0007] A data directory unit for managing business metadata of each of the heterogeneous storage systems; the business metadata is used to describe data content, data source, and data correlation relationship;

[0008] A metadata federation unit connected with the data directory unit, for managing technical metadata of each of the heterogeneous storage systems, the technical metadata being used to describe logical-to-physical mapping information of data, access protocol and endpoint information, authentication and authorization information; for the heterogeneous storage system with an independent metadata server, the metadata federation unit manages the corresponding technical metadata by registering information of the independent metadata server;

[0009] A data gateway unit, connected with the metadata federation unit, is configured to provide protocol conversion and is responsible for data routing to read and cache data.

[0010] In an optional embodiment, the data federation unit manages the plurality of heterogeneous storage systems according to different integration strategies based on the types of the plurality of heterogeneous storage systems.

[0011] In an optional embodiment, for a network-attached storage type of the heterogeneous storage systems, the metadata federation unit acquires corresponding technical metadata information thereof by full directory tree crawling.

[0012] In an optional embodiment, the data directory unit is further configured to provide a globally unified index.

[0013] In an optional embodiment, the data gateway unit is dynamically scalable according to load; and / or, the data gateway unit is controlled by the metadata federation unit.

[0014] In an optional embodiment, the plurality of heterogeneous storage systems include two types of standard NFS protocol and non-standard NFS protocol; wherein, for the heterogeneous storage system using the standard NFS protocol, the access path is direct access; and for the heterogeneous storage system using the non-standard NFS protocol, the access path is determined according to whether the heterogeneous storage system has an independent metadata server.

[0015] In an optional embodiment, for the heterogeneous storage system using the non-standard NFS protocol and without an independent metadata server, the access path is accessed via the data gateway unit; and for the heterogeneous storage system using the non-standard NFS protocol and with an independent metadata server, the access path is accessed via a private protocol of a target runtime environment.

[0016] According to a second aspect of the present application, a metadata federation management method under a unified namespace is provided, which is applied to the management system in any of the embodiments of the first aspect; the method comprises:

[0017] The data directory unit acquires target business metadata of target data in response to a user business query request, and returns the target business metadata to the user; the target business metadata is used to describe data content, data source and data correlation of the target data.

[0018] The metadata federation unit determines target technical metadata matching the target business metadata in response to a user data query instruction; the target technical metadata is used to describe logical-to-physical mapping information, access protocol and endpoint information, authentication and authorization information of the target data.

[0019] The metadata federation unit returns the target technical metadata to the user; the target technical metadata is used to determine the data layout of the target heterogeneous storage system and the type of storage access protocol of the target heterogeneous storage system;

[0020] The target heterogeneous storage system accepts the access operation of the user according to the type of storage access protocol.

[0021] In an optional embodiment, the method further comprises:

[0022] In the case where the target heterogeneous storage system adopts a standard NFS protocol, the target access path is determined to be direct access;

[0023] In the case where the target heterogeneous storage system adopts a non-standard NFS protocol, the corresponding access mode is determined according to whether the target heterogeneous storage system has an independent metadata server.

[0024] In an optional embodiment, in the case where the target heterogeneous storage system adopts a non-standard NFS protocol, the corresponding access mode is determined according to whether the target heterogeneous storage system has an independent metadata server, comprising:

[0025] For a heterogeneous storage system that adopts a non-standard NFS protocol and has no independent metadata server, the access path is accessed via the data gateway unit;

[0026] For a heterogeneous storage system that adopts a non-standard NFS protocol and has an independent metadata server, the access path is accessed via the private protocol of the target running environment.

[0027] The metadata federation management system and method under the unified namespace provided by the present application comprises a data storage unit 10, a data directory unit 20, a metadata federation unit 30 and a data gateway unit 40. The metadata federation unit 30 is used to manage the technical metadata of heterogeneous storage systems of different types, and for a heterogeneous storage system with an independent metadata server, the metadata federation unit manages the corresponding technical metadata by registering the information of the independent metadata server. In this way, the data in the data storage unit 10 can be included in the unified namespace without migration, and the metadata federation unit 30 separates the metadata and the data, which can realize the management of storage systems from different manufacturers, different regions and different access protocols, provide parallel access of data under the unified namespace, and include global management without modifying the existing storage, thereby solving the data island problem in the heterogeneous storage environment.

[0028] In order to make the above objectives, features and advantages of the present disclosure more obvious and comprehensible, below a preferred embodiment is specifically described, and the accompanying drawings are described in detail as follows. BRIEF DESCRIPTION OF DRAWINGS

[0029] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the drawings needed to be used in the embodiments will be briefly introduced as follows. The drawings herein are incorporated into the description and form a part of the description, which show the embodiments consistent with the present disclosure, and are used to explain the technical solutions of the present disclosure together with the description. It should be understood that the following drawings only show some embodiments of the present disclosure, and therefore should not be regarded as a limitation on the scope, and for those skilled in the art, other related drawings can also be obtained without creative labor.

[0030] Figure 1 is a structural schematic diagram of a metadata federation management system under a unified namespace according to an example embodiment of the present application;

[0031] Figure 2 is a schematic diagram of an access process of a metadata federation management system under a unified namespace according to an example embodiment of the present application;

[0032] Figure 3 is a flowchart of a metadata federation management method under a unified namespace according to an example embodiment of the present application;

[0033] Figure 4 is a flowchart of a method for determining storage access according to a protocol type according to an example embodiment of the present application;

[0034] Figure 5 is a process schematic diagram of a metadata federation management method under a unified namespace according to an example embodiment of the present application. DETAILED DESCRIPTION

[0035] The example embodiments will be described in detail herein, and the examples are shown in the drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following example embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of apparatuses and methods consistent with some aspects of the present application as detailed in the appended claims.

[0036] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and / or "comprising," when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0037] It is to be understood that the singular forms "a", "an", and "the" include plural referents unless the context clearly dictates otherwise. It is to be understood that the term "and / or" as used herein encompasses all possible combinations of particular items listed apart from disjunctively one from another. It is further understood that the terms "comprise" and / or "comprising", when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0038] The term "and / or", as used herein, merely describes associated items, and can mean that three conditions exist, for example, A and / or B, can mean that A exists alone, A and B exist together, and B exists alone. In addition, the term "at least one" herein means any one of a plurality or at least two of a plurality in any combination, for example, including at least one of A, B, and C, can mean including any one or more elements selected from the set consisting of A, B, and C.

[0039] The user-related data involved in the technical solutions of the present application are all obtained after obtaining the authorization of the user, such as through a pop-up window or by inviting the individual to upload authorization by himself or herself.

[0040] It is found through research that a unit with massive data management needs uses its existing storage resources to create an automated and scalable global data environment, providing users served thereby with a local access experience to data that can exist on various storage types of different manufacturers in one or more locations. However, across different manufacturers, different storage platforms, and different geographical locations, great challenges will be faced. In particular, for storage systems that manage files or objects using Lustre, GPFS (General Parallel FileSystem), BeeGFS, etc., a non-standard NFSv4.2 / pNFS (Parallel NFS) protocol is used, and the current method needs to move massive metadata to an independent metadata server (or metadata cluster), and since metadata migration is very time-consuming, especially for PB-level or even EB-level storage systems, the time overhead of system management is increased, and the storage system being managed can only provide performance-limited storage access during migration.

[0041] In addition, the existing metadata management scheme only processes efficient storage and accurate query of structured attributes (such as permissions, paths, modification times, etc.), and is difficult to efficiently cope with complex queries in multi-modal application scenarios, such as the need for semantic-level retrieval and analysis of unstructured data (such as file content, image features, text description, etc.) (for example, "find all files similar to the theme of a certain document").

[0042] Based on the above research, the present application provides a metadata federation management system and method under a unified namespace, which can realize parallel access of data under a unified namespace for multiple independent storage systems using non-standard protocols. The data of the storage systems managed by the system does not need to be moved, and the metadata of the storage systems with independent metadata servers also does not need to be moved, thereby avoiding the formation of data islands due to the storage of data in incompatible storage types. In addition, the management system can also provide efficient data access and retrieval of data content. That is, the management system can solve the problem of data islands in a multi-source heterogeneous storage environment and provide a unified data view and high-performance parallel data access capability without migrating data (the metadata of the storage systems with independent metadata servers does not need to be migrated).

[0043] The metadata server can include a standalone server or a server cluster composed of multiple servers, and the specific implementation is not limited.

[0044] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the accompanying drawings in the embodiments of the present application.

[0045] Referring to Figure 1 As shown in the figure, Figure 1 A structure diagram of a metadata federation management system under a unified namespace is shown. The metadata federation management system under the namespace 100 includes a data storage unit 10, a data directory unit 20, a metadata federation unit 30, and a data gateway unit 40. The data storage unit 10 includes multiple heterogeneous storage systems. The data directory unit 20 is used to manage the business metadata of each of the heterogeneous storage systems. The business metadata is used to describe the data content, data source, and data correlation relationship. The metadata federation unit is used to manage the technical metadata of each of the heterogeneous storage systems. The technical metadata is used to describe the logical-to-physical mapping information of the data, the access protocol and endpoint information, the authentication and authorization information. The data gateway unit 40 is used to provide protocol conversion and is responsible for specific data routing, reading and caching data.

[0046] The logical-to-physical mapping information refers to a mapping relationship between a logical library and a physical library, and is used to correspond a logical concept to an actual storage location. The access protocol is used to indicate a type of storage access protocol adopted by the heterogeneous storage system. The endpoint information refers to a set of target service addresses and protocol entries required for accessing the storage unit. The authentication and authorization information includes user authentication information and data access permissions (such as read, write, and view permissions), wherein the user authentication information is used to indicate that the authenticated user has the permission to access.

[0047] The present application provides a metadata federation management system 100 under a unified namespace, which includes a data storage unit 10, a data directory unit 20, a metadata federation unit 30, and a data gateway unit 40. The metadata federation unit 30 is used to manage technical metadata of different types of heterogeneous storage systems, and for the heterogeneous storage systems with independent metadata servers, the metadata federation unit manages the corresponding technical metadata by registering information of the independent metadata servers. In this way, the data in the data storage unit 10 can be included in the unified namespace without migration, and the metadata and data are separated by the metadata federation unit 30, which can manage storage systems from different manufacturers, different regions, and different access protocols, and provide parallel data access under a unified namespace. The existing storage can be included in global management without modification, and the data island problem in the heterogeneous storage environment is solved.

[0048] The metadata and data paths are separated, that is, the metadata is placed in the metadata federation unit 30, and the data is placed in the data storage unit 10. In a large-scale storage scenario, each unit is stored in an independent server. In this way, the metadata and data are independently stored and separated.

[0049] It should be noted that each layer unit can be connected (including direct connection or indirect connection) to each other for communication, for example, data flow and control flow can be interacted through a defined clear interface.

[0050] Specifically, the data storage unit 10 includes all managed heterogeneous storage systems. The types of the plurality of heterogeneous storage systems are different, and can be divided into two categories of standard NFS and non-standard NFS according to their protocol support. In the present application, for the heterogeneous storage system using the standard NFS protocol, the access path is direct access; for the heterogeneous storage system using the non-standard NFS protocol, the access path is determined according to whether the heterogeneous storage system has an independent metadata server.

[0051] Exemplarily, for a heterogeneous storage system employing a non-standard NFS protocol and without an independent metadata server, the access path is accessed via the data gateway unit 40 (indirect access); for a heterogeneous storage system employing a non-standard NFS protocol and with an independent metadata server, the access path is accessed via a private (native) protocol of the target runtime environment, i.e., accessed via the private protocol of the heterogeneous storage system. For example, for a heterogeneous storage system supporting a standard NFS / pNFS protocol, the storage can be directly accessed; for a heterogeneous storage system employing a non-standard NFS protocol such as SMB, S3 or NAS, the heterogeneous storage system does not have an independent metadata server (cluster), and the access path is accessed via the data gateway unit 40; and for a heterogeneous storage system such as Lustre having an independent metadata server, the heterogeneous storage system is accessed via a private protocol of the Lustre, for example, by deploying a Lustre client in the form of a Serverless Runtime or a Stateless Container to access the heterogeneous storage system via the private protocol of the Lustre, so as to realize data parallel access under a unified namespace while ensuring performance.

[0052] The data catalog unit 20 is configured to provide a global unified index, and can provide a user with a unified data view and semantic query capability. Exemplarily, the data catalog unit 20 can be configured to construct a global unified business metadata (also referred to as semantic metadata) index library, and use a vector database and other technologies to organize information such as business tags, owners, blood relationship and access heat of data, so that a user can quickly query through high-level semantic attributes, and understand the business context of data without traversing underlying storage units. The blood relationship is used to reflect the association relationship between upstream and downstream data.

[0053] The metadata federation unit 30 is configured to implement federated management of technical metadata. Specifically, the metadata federation unit 30 can manage the plurality of heterogeneous storage systems according to different integration strategies according to types of the plurality of heterogeneous storage systems, i.e., can adopt a differentiated integration strategy according to types of the managed heterogeneous storage systems.

[0054] Exemplarily, for a network-attached storage type of heterogeneous storage system, the metadata federation unit 30 obtains its corresponding metadata information by full directory tree crawling, that is, for the network-attached storage system, the complete directory tree structure metadata can be obtained by the crawler; for a storage system with an independent metadata server (cluster), the metadata server (cluster) is registered in the data directory unit 20 and forms a metadata federation management with the metadata federation unit 30, and the data directory unit 20 records the information of the newly added metadata server (cluster), that is, for the storage system with an independent metadata server (cluster), only the metadata server information needs to be federated, for example, the metadata information in the independent metadata server can be directly registered in the data directory unit 20 without migrating (copying) the metadata. In this way, the technical metadata of various types of heterogeneous storage systems can be uniformly managed, a consistent technical view is provided for the upper layer, and the physical storage space for storing metadata can be saved.

[0055] The data gateway unit 40 is a protocol adaptation layer, providing multi-protocol access capability and can perform protocol conversion. In addition, it can also perform specific data paths and can linearly improve data throughput bandwidth through horizontal expansion. In some embodiments, the data gateway unit 40 can be dynamically expanded according to the load, and if the number of users served by the data gateway unit 40 reaches a certain threshold or the storage space for caching reaches a certain threshold, a new data gateway needs to be added. In other embodiments, the data gateway unit 40 is also controlled by the metadata federation unit 30, which cooperates with the metadata federation unit 30, receives instructions from the metadata federation unit 30, and directly exchanges data with each heterogeneous storage system in the data storage unit 10.

[0056] Referring to Figure 2 shown, Figure 2 An access process of a metadata federation management system under a unified namespace is shown. The data flow and control flow between each unit can interact through an interface. The access process starts from a user querying business metadata, then obtains technical metadata, and finally routes to the corresponding path according to the protocol type indicated by the technical metadata.

[0057] Specifically, the application interaction side serves as the access starting point of the system, and can generate a user query request according to the output content of the user. The user query request first reaches the data directory unit 20. The data directory unit 20 is used for managing global business metadata, and can return corresponding business metadata. Then, the user can input a data query instruction according to the business metadata returned by the data directory unit 20. The data query instruction is transmitted to the metadata federation unit 30. The metadata federation unit 30 finds corresponding technical metadata, and can determine a corresponding access path according to the protocol type indicated by the technical metadata. In the case of indirect access, the access path can be routed to the data gateway unit 40, and then guided to the underlying data storage unit 10 by the data gateway unit 40. Through the architecture of separating metadata from data paths, the system realizes the logical unified management and parallel access of physically dispersed storage resources, and effectively solves the data island problem in the multi-source heterogeneous storage environment.

[0058] It should be noted that the system embodiments described above are only schematic, and the units described as separate components can or can not be physically separated, and the components displayed as units can or can not be physical units, i.e. they can be located in one place or distributed on multiple network units. Part or all of the units can be selected to achieve the purpose of the application scheme according to actual needs. Those skilled in the art can understand and implement without creative labor.

[0059] For example, each of the above units can be a server, which can be a standalone physical server, a server cluster or a distributed system composed of multiple physical servers, a cloud server providing cloud services, cloud databases, cloud computing, cloud storage, big data and artificial intelligence platforms, and basic cloud computing services.

[0060] Referring to Figure 3 The flowchart of the metadata federation management method under the unified namespace provided by the embodiment of the application is shown, and the method is applied to the metadata federation management system 100 under the unified namespace in any of the foregoing embodiments. The method comprises the following S101-S104:

[0061] S101, the data directory unit acquires target business metadata of target data in response to a user business query request, and returns the target business metadata to the user. The target business metadata is used to describe the data content, data source and data correlation of the target data.

[0062] S102, the metadata federation unit determines target technical metadata matching the target service metadata in response to a user data query instruction; the target technical metadata is used to describe logical-to-physical mapping information, access protocol and endpoint information, authentication and authorization information of the target data.

[0063] S103, the metadata federation unit returns the target technical metadata to the user; the target technical metadata is used to determine data layout of the target heterogeneous storage system and storage access protocol type of the target heterogeneous storage system.

[0064] S104, the target heterogeneous storage system accepts access operation of the user according to the storage access protocol type.

[0065] The access operation includes read and write operations. Illustratively, the metadata federation unit returns technical metadata of target data of the user; the user end can determine protocol type adopted by the target heterogeneous storage system according to the technical metadata, and determine corresponding target access path according to the protocol type, and the user end can perform access operation on the target heterogeneous storage system according to other information provided by the technical metadata. Illustratively, the metadata federation unit can also perform the above operation based on the technical metadata, and return the target access path to the user end.

[0066] In some embodiments, referring to Figure 4 The method can further include S1031-S1032.

[0067] S1031, in the case where the target heterogeneous storage system adopts standard NFS protocol, determining that the target access path is direct access.

[0068] S1032, in the case where the target heterogeneous storage system adopts non-standard NFS protocol, determining corresponding access mode according to whether the target heterogeneous storage system has an independent metadata server.

[0069] In some embodiments, in the case where the target heterogeneous storage system adopts non-standard NFS protocol, determining corresponding access mode according to whether the target heterogeneous storage system has an independent metadata server includes:

[0070] For the heterogeneous storage system adopting non-standard NFS protocol and without independent metadata server, the access path is accessed via the data gateway unit;

[0071] For the heterogeneous storage system adopting non-standard NFS protocol and with independent metadata server, the access path is accessed via private protocol of the target running environment.

[0072] Specifically, referring toFigure 5 As shown, Figure 5 A process diagram of a metadata federation management method under a unified namespace is shown. The process starts with a user initiating a query to the data directory unit 20 to obtain the business metadata of the target data, which enables the user to understand the data content, source and association from the business semantic level. Subsequently, the user accesses the metadata federation unit 30 according to the business metadata to obtain detailed technical metadata, thereby clarifying the underlying information such as the physical storage location, organization structure and access interface of the data.

[0073] Further, the method can also judge the access protocol. Illustratively, the protocol type supported by the storage unit (target heterogeneous storage system) can be identified according to the obtained technical metadata; if the storage unit supports standard NFS / pNFS protocol, the user can directly access the storage unit to realize efficient data read / write operation; if the storage unit uses non-standard NFS / pNFS protocol, the access path is further subdivided; for storage resources without independent metadata server, such as SMB, S3 or NAS using non-standard NFS protocol, the access request will access the storage unit after protocol conversion via the data gateway unit 40; and for storage cluster systems with independent metadata server, such as Lustre, GPFS and other types of storage cluster systems, the private access protocol is embedded in the form of Serverless Runtime or Stateless Containers, and the storage unit is directly accessed using the native protocol of the cluster.

[0074] The present application realizes seamless and parallel access to heterogeneous storage resources under a unified namespace through multi-layer metadata coordination and protocol adaptive routing mechanism. The entire process embodies the architecture advantage of separating metadata and data path, ensuring the simplicity of the user access interface and the effective isolation of the complexity of the backend storage protocol, and finally providing the user with transparent and efficient data access experience.

[0075] Those skilled in the art can understand that in the above method of the specific implementation, the writing order of each step does not mean a strict execution order and does not constitute any limitation on the implementation process, and the specific execution order of each step should be determined by its function and possible internal logic.

[0076] Moreover, embodiments of the subject matter described in this specification can be implemented in digital electronic circuitry, in tangibly-embodied computer software or firmware, in computer hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible non-transitory program carrier for execution by, or to control the operation of, data processing apparatus. Alternatively or additionally, the program instructions can be encoded on an artificially generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal that is generated to encode information for transmission to suitable receiver apparatus for execution by a data processing apparatus. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them.

[0077] The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit), and the apparatus can be implemented as special purpose logic circuitry.

[0078] Computers suitable for the execution of a computer program include, by way of example, general and / or special purpose microprocessors, or any other kind of central processing unit. Generally, a central processing unit will receive instructions and data from a read-only memory and / or a random access memory. The essential elements of a computer are a central processing unit for performing instructions and one or more memory devices for storing instructions and data. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto-optical disks, or optical disks. However, a computer need not have such devices. Moreover, a computer can be embedded in another device, e.g., a mobile telephone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a Global Positioning System (GPS) receiver, or a portable storage device (e.g., a universal serial bus (USB) flash drive), to name just a few.

[0079] Computer readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.

[0080] While the specification contains many specifics, these should not be construed as limiting the scope of any invention or of what can be claimed, but as merely providing illustrations of some of the embodiments of the inventions. Certain features that are described in this specification in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Moreover, although features can be described above as acting in certain combinations and even initially claimed as such, one or more features from a claimed combination can in some cases be excised from the combination and the claimed combination can be directed to a subcombination or variation of a subcombination.

[0081] Similarly, while operations are depicted in the drawings in a particular order, this should not be understood as requiring such order, nor that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing can be advantageous. Moreover, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0082] Accordingly, particular embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. In some cases, actions recited in the claims can be performed in a different order and still achieve desirable results. In addition, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve the desired results. In some implementations, multitasking and parallel processing can be advantageous.

[0083] The above description is merely illustrative of the application and not restrictive. Since certain changes can be made in the application as described without departing from the spirit and scope of the application, it is intended that all of the possible modifications of the application be included within the scope of the application as defined in the claims.

Claims

1. A metadata federation management system under a uniform namespace, characterized in that, include: The data storage unit includes multiple heterogeneous storage systems of different types. A data catalog unit is used to manage the business metadata of each of the heterogeneous storage systems; the business metadata is used to describe data content, data source, and data association relationships. The metadata federation unit, connected to the data directory unit, is used to manage the technical metadata of each heterogeneous storage system. The technical metadata is used to describe the logical-to-physical mapping information of the data, access protocol and endpoint information, and authentication and authorization information. Specifically, for heterogeneous storage systems with independent metadata servers, the metadata federation unit manages their corresponding technical metadata by registering the information of the independent metadata servers. The data gateway unit, connected to the metadata federation unit, is used to provide protocol conversion and is responsible for data routing to read and cache data.

2. The system of claim 1, wherein, The data federation unit manages the multiple heterogeneous storage systems according to different integration strategies based on their types.

3. The system of claim 2, wherein, For heterogeneous storage systems of the network-attached storage type, the metadata federation unit obtains its corresponding technical metadata information by crawling the full directory tree.

4. The system of claim 1, wherein, The data catalog unit is also used to provide a globally unified index.

5. The system of claim 1, wherein, The data gateway unit can be dynamically expanded according to the load; and / or, the data gateway unit is controlled by the metadata federation unit.

6. The system of claim 1, wherein, The heterogeneous storage systems include two types: those using the standard NFS protocol and those using non-standard NFS protocols. For heterogeneous storage systems using the standard NFS protocol, the access path is direct access. For heterogeneous storage systems using non-standard NFS protocols, the access path is determined based on whether the heterogeneous storage system has an independent metadata server.

7. The system of claim 6, wherein, For heterogeneous storage systems that use a non-standard NFS protocol and do not have an independent metadata server, the access path is through the data gateway unit; for heterogeneous storage systems that use a non-standard NFS protocol and have an independent metadata server, the access path is through the private protocol of the target operating environment.

8. A metadata federation management method under a uniform namespace, characterized by, The method is applied to a metadata federated management system under a unified namespace as described in any one of claims 1-7; the method includes: The data catalog unit responds to a user's business query request by obtaining the target business metadata of the target data and returning the target business metadata to the user; the target business metadata is used to describe the data content, data source, and data association of the target data. In response to a user data query command, the metadata federation unit determines the target technical metadata that matches the target business metadata; the target technical metadata is used to describe the logical-to-physical mapping information, access protocol and endpoint information, and authentication and authorization information of the target data; The metadata federation unit returns the target technology metadata to the user; the target technology metadata is used to determine the data layout of the target heterogeneous storage system and the storage access protocol type of the target heterogeneous storage system; The target heterogeneous storage system accepts access operations performed by users according to the storage access protocol type.

9. The method of claim 8, wherein, The method further comprises: in the case that the target heterogeneous storage system adopts a standard NFS protocol, determining a target access path as direct access; in the case that the target heterogeneous storage system adopts a non-standard NFS protocol, determining a corresponding access mode according to whether the target heterogeneous storage system has an independent metadata server.

10. The method of claim 9, wherein, The determining a corresponding access mode according to whether the target heterogeneous storage system has an independent metadata server in the case that the target heterogeneous storage system adopts a non-standard NFS protocol comprises: for a heterogeneous storage system adopting a non-standard NFS protocol and having no independent metadata server, an access path accesses via the data gateway unit; for a heterogeneous storage system adopting a non-standard NFS protocol and having an independent metadata server, an access path accesses via a private protocol of a target running environment.

Images