Metadata processing method and system and computing equipment

Abstract

The invention discloses a metadata processing method and system and computing equipment, and the method comprises the steps that a data production cluster generates metadata of shared data, the data production cluster stores the shared data and the metadata to a shared storage, and the shared storage generates path information of the metadata; the data consumption cluster obtains the path information of the metadata from the shared storage, obtains the metadata from the shared storage based on the path information of the metadata, and obtains the shared data from the shared storage based on the metadata. The method can reduce the storage cost of the metadata of the shared data.

Description

Technical Field

[0001] The present application relates to the field of data storage, and more specifically, to a metadata processing method, system and computing device. Background Art

[0002] Data warehouses are carriers for data storage and analysis. Currently, all major cloud vendors have launched data warehouse services. Data sharing has become a property of data warehouse services. Its purpose is to eliminate data silos, realize data transactions and sharing between users, and achieve data intercommunication and sharing between users. Data sharing means that data producers provide shared data, and data consumers subscribe to or purchase certain shared data produced by data producers. After purchasing this shared data, data consumers can obtain the shared data and perform data analysis and calculation based on the shared data.

[0003] In the process of data producers producing and storing shared data, metadata of the shared data is also generated and stored. When data consumers obtain shared data from shared storage, they need to obtain the shared data from the shared storage based on the metadata of the shared data. In the related prior art solutions, the metadata of shared data is stored separately in the metadata cluster. Since the metadata cluster is an independent cluster for storing metadata, the storage cost of the metadata cluster is relatively high.

[0004] Therefore, how to reduce the storage cost of metadata of shared data has become a technical problem that needs to be solved urgently. Summary of the invention

[0005] The present application provides a metadata processing method, system and computing device, which can reduce the storage cost of metadata of shared data.

[0006] In a first aspect, a metadata processing method is provided, the method comprising: a data production cluster generates metadata for shared data, the data production cluster includes multiple data nodes that provide the shared data; the data production cluster stores the shared data and the metadata in a shared storage, the metadata is used by a data consumption cluster to obtain the shared data from the shared storage, the data consumption cluster includes multiple data nodes that use the shared data; the shared storage generates path information for the metadata; the data consumption cluster obtains the path information for the metadata from the shared storage; the data consumption cluster obtains the metadata from the shared storage based on the path information for the metadata; the data consumption cluster obtains the shared data from the shared storage based on the metadata.

[0007] In the above technical solution, by reusing the coexistence storage for storing shared data, the metadata of the shared data is stored in the coexistence storage. In this way, there is no need to introduce a separate cluster to store metadata, which can reduce the storage cost of the metadata of the shared data and enable large-scale, multi-cluster data sharing.

[0008] In combination with the first aspect, in certain implementations of the first aspect, the method further includes: the data production cluster sets access rights to the path information of the metadata; and the data consumption cluster uses the access rights to obtain the path information of the metadata.

[0009] In the above technical solution, the data production cluster sets access rights to the path information of the metadata, so that only the data consumption cluster with the access rights can obtain the path information of the metadata.

[0010] In combination with the first aspect, in certain implementations of the first aspect, the data consumption cluster periodically obtains the path information of the metadata from the shared storage; or, the data consumption cluster obtains the path information of the metadata from the shared storage when the target event occurs.

[0011] In combination with the first aspect, in certain implementations of the first aspect, multiple data nodes in the data production cluster store metadata of their respective shared data in the shared storage.

[0012] In combination with the first aspect, in some implementations of the first aspect, the method further includes: after the data consumption cluster reads the shared data, the shared storage deletes the shared data or metadata of the shared data.

[0013] In the above technical solution, after the data consumption cluster reads the shared data, the data production cluster deletes the shared data or the metadata of the shared data in the shared storage. In this way, the storage space of the shared storage is further saved by garbage collection of the shared data or the metadata of the shared data used by the data consumption cluster.

[0014] In a second aspect, a metadata processing system is provided, which includes: a data production cluster, a shared storage and a data consumption cluster, wherein the data production cluster is used to generate metadata for shared data, and the data production cluster includes multiple data nodes that provide the shared data; the data production cluster is also used to store the shared data and the metadata in the shared storage, and the metadata is used by the data consumption cluster to obtain the shared data from the shared storage, and the data consumption cluster includes multiple data nodes that use the shared data; the shared storage is used to generate path information for the metadata; the data consumption cluster is used to obtain the path information for the metadata from the shared storage; the data consumption cluster is also used to obtain the metadata from the shared storage based on the path information for the metadata; the data consumption cluster is also used to obtain the shared data from the shared storage based on the metadata.

[0015] In combination with the second aspect, in certain implementations of the second aspect, the data production cluster is further used to set access rights for the path information of the metadata; and the data consumption cluster is further used to obtain the path information of the metadata using the access rights.

[0016] In combination with the second aspect, in certain implementations of the second aspect, the data consumption cluster is specifically used to: periodically obtain the path information of the metadata from the shared storage; or, when a target event occurs, obtain the path information of the metadata from the shared storage.

[0017] In combination with the second aspect, in some implementations of the second aspect, the data production cluster is specifically used for: multiple data nodes in the data production cluster store metadata of their respective shared data in the shared storage.

[0018] In combination with the second aspect, in some implementations of the second aspect, the shared storage is further used to delete the shared data or metadata of the shared data after the data consumption cluster reads the shared data.

[0019] It should be noted that the beneficial effects in the second aspect can be referred to the beneficial effects in the first aspect, which will not be repeated here.

[0020] In a third aspect, a computing device cluster is provided, comprising at least one computing device, each computing device comprising a processor and a memory; the processor of the at least one computing device is used to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster executes the method in the first aspect or any possible implementation of the first aspect.

[0021] Optionally, the processor may be a general-purpose processor, which may be implemented by hardware or software. When implemented by hardware, the processor may be a logic circuit, an integrated circuit, etc.; when implemented by software, the processor may be a general-purpose processor implemented by reading software code stored in a memory, which may be integrated in the processor or located outside the processor and exist independently.

[0022] In a fourth aspect, a computer program product including instructions is provided. When the instructions are executed by a computing device cluster, the computing device cluster executes the method in the first aspect and any one of the implementations of the first aspect.

[0023] In a fifth aspect, a computer-readable storage medium is provided, comprising computer program instructions. When the computer program instructions are executed by a computing device cluster, the computing device cluster executes the method in the above-mentioned first aspect and any one of the implementations of the first aspect.

[0024] As an example, computer-readable storage media include, but are not limited to, one or more of the following: read-only memory (ROM), programmable ROM (PROM), erasable PROM (EPROM), Flash memory, electrically EPROM (EEPROM), and hard drive.

[0025] Optionally, as an implementation manner, the above-mentioned storage medium may specifically be a non-volatile storage medium. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 It is a schematic block diagram of the data sharing service provided by the data warehouse.

[0027] Figure 2 It is a schematic block diagram of a cloud scenario applicable to an embodiment of the present application.

[0028] Figure 3 It is a schematic flowchart of a metadata processing method provided in an embodiment of the present application.

[0029] Figure 4 It is a schematic flowchart of a method for a data production cluster to store metadata of shared data provided in an embodiment of the present application.

[0030] Figure 5 It is a schematic block diagram of a data production cluster provided in an embodiment of the present application.

[0031] Figure 6This is an example of the correspondence between data nodes and data shards in a data production cluster provided in an embodiment of the present application.

[0032] Figure 7 It is a schematic block diagram of the data segments and metadata corresponding to shard1 of the shared data table1 of the example of this application.

[0033] Figure 8 It is a schematic diagram of the information recorded in a manifest1 file provided in an embodiment of the present application.

[0034] Fig. 9 It is a schematic flowchart of another method for storing metadata of shared data in a data production cluster provided in an embodiment of the present application.

[0035] Fig.10 It is a schematic block diagram of newly added shared data of a data production cluster and metadata corresponding to the shared data provided in an embodiment of the present application.

[0036] Fig.11 It is a schematic diagram of the information recorded in a manifest2 file provided in an embodiment of the present application.

[0037] Fig.12 It is a schematic flowchart of a method provided by an embodiment of the present application for a data consumption cluster to obtain metadata and shared data corresponding to the metadata from shared storage.

[0038] Fig.13 It is a schematic flowchart of another method provided by an embodiment of the present application for a data consumption cluster to obtain metadata and shared data corresponding to the metadata from a shared storage.

[0039] Fig.14 It is a schematic block diagram of a metadata processing system 1400 provided in an embodiment of the present application.

[0040] Fig.15 It is a schematic diagram of the architecture of a computing device 1500 provided in an embodiment of the present application.

[0041] Fig.16 It is a schematic diagram of the architecture of a computing device cluster provided in an embodiment of the present application.

[0042] Fig.17 It is a schematic diagram of the connection between computing devices 1500A and 1500B via a network provided in an embodiment of the present application. DETAILED DESCRIPTION

[0043] The technical solution in this application will be described below in conjunction with the accompanying drawings.

[0044] The present application will present various aspects, embodiments or features around a system including multiple devices, components, modules, etc. It should be understood and appreciated that each system may include additional devices, components, modules, etc., and / or may not include all devices, components, modules, etc. discussed in conjunction with the figures. In addition, combinations of these schemes may also be used.

[0045] In addition, in the embodiments of the present application, words such as "exemplary" and "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "exemplary" in the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of the word "exemplary" is intended to present concepts in a concrete way.

[0046] In the embodiments of the present application, "corresponding (corresponding, relevant)" and "corresponding (corresponding)" can sometimes be used interchangeably. It should be pointed out that when the distinction between them is not emphasized, the meanings they intend to express are consistent.

[0047] The business scenarios described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. A person of ordinary skill in the art can appreciate that, with the evolution of network architecture and the emergence of new business scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0048] References to "one embodiment" or "some embodiments" etc. described in this specification mean that a particular feature, structure or characteristic described in conjunction with the embodiment is included in one or more embodiments of the present application. Thus, the phrases "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear at different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0049] In the present application, "at least one" means one or more, and "plurality" means two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: including the existence of A alone, the existence of A and B at the same time, and the existence of B alone, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b, or c can mean: a, b, c, ab, ac, bc, or abc, where a, b, c can be single or multiple.

[0050] Data warehouses are the carriers of data storage and analysis. Currently, all major cloud vendors have launched data warehouse services. Data sharing has become a property of data warehouse services, and its purpose is to eliminate data silos, realize data transactions and sharing between users, and achieve data intercommunication and sharing between users.

[0051] It should be understood that the data sharing service provided by the above data warehouse refers to the data producer providing shared data, and the data consumer subscribing to or purchasing certain shared data produced by the data producer. After the data consumer purchases this shared data, the data consumer can obtain the shared data and perform data analysis and calculation based on the shared data.

[0052] As an example, the above-mentioned shared data may include but is not limited to: a table produced by a data producer, or a materialized view, or certain column data in a table, or certain row data in a table, etc.

[0053] For ease of understanding, let's first combine Figure 1 , the basic principles of the data sharing services provided by the data warehouse are described in detail.

[0054] like Figure 1 As shown in the figure, data producers continuously produce data in the cloud data warehouse and write the produced data to the shared storage. Data producers can also authorize data consumers to read which part of the data stored in the shared storage based on the data subscription or purchase of the data consumer. Data consumers can read the part of the data they purchased or subscribed from the shared storage, and perform data analysis and calculation based on the data to execute the data consumption process.

[0055] It should be understood that data stored in the shared storage may be referred to as shared data.

[0056] Specifically, in the process of the data producer producing and storing the shared data, the metadata of the shared data will also be generated and stored. When the data consumer obtains the shared data from the shared storage, it needs to obtain the shared data from the shared storage according to the metadata of the shared data.

[0057] It should be understood that the metadata of shared data is used to describe the shared data, and is information that describes the attributes of the shared data. For example, the metadata of the shared data may include, but is not limited to, information such as the location and address of the shared data stored in the shared storage.

[0058] In the related prior art solutions, the metadata of the shared data is stored separately in a metadata cluster (ie, Metadata cluster). Since the Metadata cluster is an independent cluster for storing metadata, the storage cost of the Metadata cluster is relatively high.

[0059] In view of this, an embodiment of the present application provides a metadata processing method, which can reduce the storage cost of metadata of shared data and can also perform large-scale, multi-cluster data sharing.

[0060] In a possible implementation, the method provided in the embodiment of the present application can be applied to a cloud service scenario, and the method is executed by a cloud management platform in the cloud service scenario. Figure 2 , describe the cloud service scenarios in detail.

[0061] Figure 2 is a schematic block diagram of a cloud scenario applicable to an embodiment of the present application. Figure 2 As shown, the cloud scenario may include: a cloud management platform 110 , the Internet 120 , and a client 130 .

[0062] like Figure 2 As shown, the cloud management platform 110 is used to manage the infrastructure that provides multiple cloud services. The infrastructure includes multiple cloud data centers, each of which includes multiple servers, each of which includes cloud service resources to provide corresponding cloud services for tenants.

[0063] The cloud management platform 110 may be located in a cloud data center, which may provide an access interface (such as an interface or an application program interface (API)). The tenant may operate the client 130 to remotely access the access interface to register a cloud account and password on the cloud management platform 110, and log in to the cloud management platform 110. After the cloud management platform 110 successfully authenticates the cloud account and password, the tenant may further pay to select and purchase a virtual machine of specific specifications (processor, memory, disk) on the cloud management platform 110. After the paid purchase is successful, the cloud management platform 110 provides the remote login account and password of the purchased virtual machine, and the client 130 may remotely log in to the virtual machine, install and run the tenant's application in the virtual machine. Therefore, the tenant may create, manage, log in and operate a virtual machine in the cloud data center through the cloud management platform 110. Among them, the virtual machine may also be called a cloud server (elastic compute service, ECS) or an elastic instance (different cloud service providers have different names).

[0064] It should be understood that tenants of cloud services can be individuals, enterprises, schools, hospitals, administrative agencies, etc.

[0065] The functions of the cloud management platform 110 include, but are not limited to, user console, computing management service, network management service, storage management service, authentication service, and image management service. The user console provides an interface or API to interact with tenants, the computing management service is used to manage servers running virtual machines and containers and bare metal servers, the network management service is used to manage network services (such as gateways, firewalls, etc.), the storage management service is used to manage storage services (such as data bucket services), the authentication service is used to manage tenant accounts and passwords, and the image management service is used to manage virtual machine images. Tenants can use the client 130 to log in to the cloud management platform 110 through the Internet 120 to manage the rented cloud services.

[0066] Figure 3 FIG. 1 is a schematic flow chart of a metadata processing method provided in an embodiment of the present application. Figure 3 As shown, the method may include steps 310 to 360, and steps 310 to 360 are described in detail below.

[0067] Step 310: The data production cluster generates metadata for the shared data.

[0068] In the embodiment of the present application, after the data production cluster produces the shared data, it can also generate metadata of the shared data. For specific metadata of the shared data, please refer to the above description, which will not be repeated here.

[0069] As an example, the data production cluster may include a plurality of data nodes (DN), and the plurality of data nodes are used to produce or provide the shared data.

[0070] Step 320: The data production cluster stores the shared data and metadata in the shared storage.

[0071] In the embodiment of the present application, the data production cluster can store both the shared data and the metadata of the shared data in the shared storage, wherein the metadata is used by the data consumption cluster to obtain the corresponding shared data from the shared storage.

[0072] For example, taking a data production cluster including multiple data nodes as an example, the multiple data nodes in the data production cluster can store metadata of their respective shared data in the shared storage.

[0073] There are many ways to implement the data production cluster storing metadata to shared storage, and the embodiments of the present application do not specifically limit this. In one possible implementation, the data production cluster can write the metadata of the shared data to the shared storage based on a certain consistency point (checkpoint). Taking into account the performance of metadata writing and the amount of data written, it can be divided into a full checkpoint process and an incremental checkpoint process. Among them, the full checkpoint process refers to writing the full metadata of the current shared data to the shared storage at a certain consistency point. The incremental checkpoint process refers to writing the metadata of the incremental shared data to the shared storage at a certain consistency point based on the consistency point of the previous checkpoint. These two different methods will be described in detail below in conjunction with specific embodiments, and will not be described in detail here.

[0074] It should be understood that in order to ensure the consistency of shared data in business logic, the data producer uniformly constructs a consistency point (checkpoint, which can also be understood as a snapshot at a certain moment) for the metadata on the distributed nodes. Since the data itself has been written to the shared storage during the data generation process, the consistency of metadata is particularly important in the data sharing process. On the one hand, in a distributed system, all nodes have a unified consistency point. At this consistency point, the data read through the metadata is consistent in business logic and can meet the ACID requirements of the database system. On the other hand, the metadata of the consistency point written by all nodes is in a certain path in the shared storage and cannot overwrite each other.

[0075] As an example, the above-mentioned shared storage can be an object storage service (OBS) in cloud storage or other storage services. As long as the function of shared storage can be achieved, the embodiments of the present application do not make specific limitations on this.

[0076] As an example, the data consumption cluster may include a plurality of data nodes (DN), and the plurality of data nodes are used to use the shared data stored in the shared storage by the data production cluster.

[0077] Step 330: Shared storage generates path information of metadata.

[0078] In an embodiment of the present application, after the data production cluster stores the metadata in the shared storage, the shared storage may allocate storage space for the metadata, thereby generating path information of the metadata.

[0079] In some embodiments, the data production cluster may also set access rights to the path information of the metadata, so that the data consumption cluster having the access rights may use the access rights to obtain the path information of the metadata.

[0080] Step 340: The data consumption cluster obtains path information of the metadata from the shared storage.

[0081] In an embodiment of the present application, the data consumption cluster may obtain the path information of the metadata from the shared storage. In one possible implementation, the data consumption cluster may periodically obtain the path information of the metadata from the shared storage. In another possible implementation, the data consumption cluster may also obtain the path information of the metadata from the shared storage when the target event occurs.

[0082] In some embodiments, if the path information of the metadata is set with an access permission, the data consumption cluster can also use the access permission to obtain the path information of the metadata.

[0083] Step 350: The data consumption cluster obtains metadata from the shared storage based on the path information of the metadata.

[0084] In the embodiment of the present application, the data consumption cluster can obtain the stored metadata from the shared storage according to the path information of the obtained metadata.

[0085] Step 360: The data consumption cluster obtains shared data from the shared storage based on the metadata.

[0086] In an embodiment of the present application, the data consumption cluster can obtain the shared data corresponding to the metadata stored in the shared storage based on the obtained metadata, thereby performing data analysis based on the obtained shared data and executing the data consumption process.

[0087] For example, taking a data consumption cluster including multiple data nodes as an example, the multiple data nodes in the data consumption cluster can obtain shared data from shared storage in parallel according to the metadata of the shared data, and perform data analysis based on the obtained shared data.

[0088] In some embodiments, the shared storage may also delete the shared data or delete the metadata of the shared data after the data consumption cluster reads the shared data.

[0089] In the above technical solution, by reusing the shared storage for storing shared data, the metadata of the shared data is stored in the shared storage, thereby realizing metadata sharing without introducing a separate cluster to store metadata. This can reduce the storage cost of the metadata of the shared data and enable large-scale, multi-cluster data sharing.

[0090] Combine the following Figures 4 to 8 , taking the full checkpoint process as an example, a detailed description is given of an implementation method in which the data production cluster stores the metadata of shared data in shared storage. It should be understood that Figures 4 to 8 The examples are only intended to help those skilled in the art understand the embodiments of the present application, and are not intended to limit the embodiments of the present application to Figures 4 to 8 The specific numerical values ​​or specific scenarios shown in the examples. Figures 4 to 8 The examples given below are obviously susceptible to various equivalent modifications or changes, and such modifications and changes also fall within the scope of the embodiments of the present application.

[0091] Figure 4 1 is a schematic flow chart of a method for storing metadata of shared data in a data production cluster provided in an embodiment of the present application. Figure 4 As shown, the method may include steps 410 to 460, and steps 410 to 460 are described in detail below.

[0092] Step 410: the coordinator node 1 (CN1) queries the timestamp oracle (TSO) and obtains the commit sequence number (CSN) of the latest successfully committed transaction from the TSO.

[0093] As an example, Figure 5As shown, the data production cluster includes multiple coordinator nodes (CN). The functions of the CN of the data production cluster may include but are not limited to: transaction management, query management, permission management, access control, sharing control, etc. For example, the CN of the data production cluster records which data needs to be shared (for example, a specific table, a materialized view, which columns of the table, which rows of the table), which data is shared with which tenant's consumption cluster, and other information.

[0094] In an embodiment of the present application, the data production cluster can select a CN from the multiple CNs to execute the metadata storage process. There are many specific selection methods, which are not limited in the embodiment of the present application. For example, a CN can be selected from multiple CNs for execution by voting. For another example, a CN with the largest ID can be selected from multiple CNs for execution. For another example, a CN with the smallest ID can be selected from multiple CNs for execution.

[0095] It should be noted that the embodiment of the present application does not specifically limit the number of CNs and DNs included in the data production cluster. For the convenience of description, Figure 5 In the description, two CNs (CN1, CN2) and three DNs (DN0, DN1, DN2) are used as examples.

[0096] For example, take the storage of metadata executed by CN1. CN1 triggers the data sharing module. In order to ensure logical consistency, CN1 needs to interact with the system's TSO. Specifically, the data sharing module in CN1 queries TSO and finally submits the successful CSN. For example, the latest successfully submitted CSN is CSN0.

[0097] It should be understood that the core of the database is to sort the operations of each transaction. In the traditional stand-alone architecture, the sorting of transactions can be easily achieved through log sequence numbers or transaction IDs. However, in a distributed architecture, the database runs on multiple servers, and each database instance has an independent clock or log. The clock points between servers vary in speed, so the clock under the distributed database cannot reflect the global order of things. As one of the key technologies of distributed databases, clock synchronization technology exists to solve the problem of the order of events in distributed databases. TSO is a clock synchronization technology that provides time services and uses centralized control services to assign monotonically increasing timestamps, such as monotonically increasing integer CSNs. TSO ensures correct linear consistency and good performance.

[0098] Step 420: CN1 constructs a plan for writing metadata and sends the plan to each DN.

[0099] In one example, at the initial stage of data sharing, the full metadata corresponding to the shared data of the current checkpoint needs to be written into the shared storage. Therefore, the metadata writing plan constructed by CN1 instructs each DN to write the full metadata of the current checkpoint stored in its memory into the shared storage.

[0100] For ease of description, the following takes table 1, the shared data written to the shared storage, as an example.

[0101] Step 430: Each DN writes the full metadata of the current checkpoint into the shared storage according to the metadata plan synchronization.

[0102] It should be understood that in a distributed system, the distribution of table data in each DN may include but is not limited to: Hash distribution, RoundRobin distribution, Replica distribution, etc. The corresponding data is called a Shard / Segment / Bucket / Partition, etc. The names are different in different product systems. The term used in this embodiment is data shard. Shard is to split a table data into multiple parts, and each DN manages and stores a part of the table data.

[0103] For example, Figure 6 As shown in the figure, taking table1 as an example, table1 is split into 9 shards for management, namely shard1 to shard9. Among them, DN0 manages and stores shard1, shard4, and shard7; DN1 manages and stores shard2, shard5, and shard8; and DN2 manages and stores shard3, shard6, and shard9.

[0104] In the embodiment of the present application, CN can manage and record the correspondence between the above DN and shard. If a DN node is scaled, only the correspondence between the shard and the DN node needs to be changed. The purpose of this is to make the distribution of data independent of the number of nodes.

[0105] Since the data stored and managed by each shard may be very large, the data in the shard needs to be further segmented (data segments). Each DN synchronously writes the part of table1 it stores to the shared storage, for example, Figure 5As shown in the figure, the data of shard1 table1, shard4 table1 and shard7 table1 are written to the shared storage from DN0; the data of shard2 table1, shard5 table1 and shard8 table1 are written to the shared storage from DN1; the data of shard3 table1, shard6 table1 and shard9 table1 are written to the shared storage from DN2. Taking the data of shard1 table1 as an example, Figure 7 As shown, the data includes data segment 1 to data segment m. For the rest of the data structure, please refer to Figure 7 , I will not go into details here.

[0106] It should be understood that in the embodiment of the present application, the data obtained after further segmenting the data in the shard is called a data segment. Different systems have different names, for example, Mirco-Partition, SSTFile, Compress Unit, etc. This application does not make specific limitations on this.

[0107] In the embodiment of the present application, since each data segment has its own corresponding metadata and the metadata is relatively small in size, the metadata of multiple data segments in a shard can be written into one metadata file. Figure 7 As shown, taking the metadata of shard1 table1 as an example, the metadata includes the metadata of data segment 1 to the metadata of data segment m.

[0108] Step 440: Each DN records the current CSN number and the path information of the currently stored metadata in the shared storage in the shared storage.

[0109] In one possible implementation, Figure 8 As shown, each DN can write a manifest1 file in the shared storage, and the manifest1 file records the path information of CSN0 and each metadata in the shared storage.

[0110] Step 450: Set CSN_Mainfest and record the current CSN number.

[0111] In the embodiment of the present application, the CSN number recorded in CSN_Mainfest is CSN0.

[0112] Step 460: Set the metadata path permissions and the shared data permissions.

[0113] In an embodiment of the present application, the data producer can set permissions for the metadata path and permissions for shared data. In this way, by setting permissions, only users who have purchased or subscribed to the data can access it, while users who have not purchased or subscribed to the data have no access.

[0114] Combine the following Figures 9 to 11 , taking the incremental checkpoint process as an example, a detailed description is given of an implementation method in which a data production cluster stores the metadata of shared data in shared storage. It should be understood that Figures 9 to 11 The examples are only intended to help those skilled in the art understand the embodiments of the present application, and are not intended to limit the embodiments of the present application to Figures 9 to 11 The specific numerical values ​​or specific scenarios shown in the examples. Figures 9 to 11 The examples given below are obviously susceptible to various equivalent modifications or changes, and such modifications and changes also fall within the scope of the embodiments of the present application.

[0115] Fig. 9 FIG. 1 is a schematic flow chart of another method for storing metadata of shared data in a data production cluster provided in an embodiment of the present application. Fig. 9 As shown, the method may include steps 910 to 960, and steps 910 to 960 are described in detail below.

[0116] Step 910: CN1 queries TSO and obtains the latest successfully submitted CSN from TSO.

[0117] The method for the data production cluster to select a CN from multiple CNs to execute the metadata storage process is the same as that in step 410. For the specific selection method, please refer to the description in step 410, which will not be repeated here.

[0118] For example, take the storage of metadata performed by CN1. CN1 triggers the data sharing module. In order to ensure logical consistency, CN1 needs to interact with the system's TSO. Specifically, the data sharing module in CN1 queries the TSO for the latest successfully submitted CSN. For example, CSN1.

[0119] Step 920: CN1 constructs a plan for writing metadata and sends the plan to each DN.

[0120] In one example, the initial full checkpoint and subsequent metadata checkpoint processes will write consistent metadata in an incremental manner. Therefore, the metadata writing plan constructed by CN1 instructs each DN to write the metadata added to the current checkpoint stored in its memory compared to the previous checkpoint to the shared storage.

[0121] Step 930: Each DN writes the metadata newly added to the current checkpoint compared to the previous checkpoint into the shared storage according to the metadata plan synchronization.

[0122] As an example, each DN executes the above plan and identifies the data segments that are increased or decreased relative to the last checkpoint through metadata. It can be compared through CSN or through the last Manifest comparison. The embodiments of the present application do not make specific limitations on this.

[0123] For example, Fig.10 As shown in the figure, the newly added data in the shared storage is the data of shard1 table2, the data of shard2 table2, and the data of shard4 table2. Specifically, the data of shard1 table2 and the data of shard4 table2 are written to the shared storage from DN0; the data of shard2 table2 is written to the shared storage from DN1.

[0124] like Fig.10 As shown, in the embodiment of the present application, DN0 can also write the metadata corresponding to the newly added data of shard1 table2 (called metadata of shard1table2) and the metadata corresponding to the data of shard4 table2 (called metadata of shard4table2) into the shared storage; DN1 writes the metadata corresponding to the newly added data of shard2 table2 (called metadata of shard2 table2). Taking the data of shard1table4 as an example, the data includes multiple data segments, so the metadata of shard1 table2 includes metadata of multiple data segments.

[0125] Step 940: Each DN records the current CSN number and the path information of the currently stored metadata in the shared storage in the shared storage.

[0126] In one possible implementation, since shard2 has no new data, there is no shard metadata under the table2_CSN1 path. Since the complete data can be read with any consistency, this information and CSN0 metadata information need to be supplemented in manifest2. The corresponding information in manifest2 is as follows: Fig.11 The manifest2 not only records CSN1, but also records the path information of the newly added metadata in the shared storage and the path information of each metadata of the last CSN0 in the shared storage.

[0127] In some embodiments, if the current checkpoint has fewer data segments than the previous checkpoint, the path information of the metadata of the shared data to be deleted in the shared storage may also be recorded in manifest2. After the data consumption cluster reads the shared data to be deleted from the shared storage, the data production cluster may delete the shared data to be deleted and / or the metadata of the shared data to be deleted from the shared storage based on the path information of the metadata of the shared data to be deleted in the shared storage.

[0128] Step 950: Update CSN_Mainfest and record the current CSN number.

[0129] In the embodiment of the present application, CSN_Mainfest is updated, and the CSN numbers recorded in CSN_Mainfest are CSN0 and CSN1.

[0130] Step 960: Set permissions for the newly added metadata path and permissions for the newly added shared data.

[0131] In some embodiments, if the data producer wants to cancel the sharing process, it is only necessary to cancel the access rights to the metadata and shared data.

[0132] It should be noted that the timing for the above-mentioned data producer to trigger the incremental checkpoint can be periodic, or it can be when a query transaction actively triggers it. The embodiment of the present application does not make specific limitations on this.

[0133] Combine the following Fig.12 , taking the full checkpoint process as an example, a detailed description is given of an implementation method in which a data consumption cluster obtains metadata from shared storage and the shared data corresponding to the metadata. It should be understood that Fig.12 The examples are only intended to help those skilled in the art understand the embodiments of the present application, and are not intended to limit the embodiments of the present application to Fig.12 The specific numerical values ​​or specific scenarios shown in the examples. Fig.12 The examples given below are obviously susceptible to various equivalent modifications or changes, and such modifications and changes also fall within the scope of the embodiments of the present application.

[0134] Fig.12 1 is a schematic flow chart of a method for a data consumption cluster to obtain metadata and shared data corresponding to the metadata from a shared storage provided by an embodiment of the present application. Fig.12 As shown, the method may include steps 1210 to 1250, and steps 1210 to 1250 are described in detail below.

[0135] Step 1210: The data consumption cluster selects CN1 from multiple CNs.

[0136] The method for the data consumption cluster to select a CN from multiple CNs is the same as that in step 410. For the specific selection method, please refer to the description in step 410, which will not be repeated here. For ease of description, the following description is made by taking CN1 as an example.

[0137] Step 1220: CN1 triggers the shared capability module, loads CSN_mainfest, and obtains the CSN number CSN0.

[0138] As an example, CN1 triggers the shared capability module, and the shared capability module obtains CSN_mainfest from the shared storage and obtains the CSN number recorded in CSN_mainfest. In this embodiment, the CSN number recorded in CSN_mainfest is CSN0.

[0139] Step 1230: CN1 builds a metadata loading plan and sends the plan to each DN.

[0140] Step 1240: Each DN executes the Plan synchronously and loads the metadata of the checkpoint whose CSN is CSN0.

[0141] It should be understood that the data consumption cluster follows the same data distribution as the data generation cluster. For example, if the shared data follows the hash distribution, then the data consumption in the consumption cluster must also follow the hash distribution. Therefore, each DN follows Figure 6 The corresponding relationship between DN and shard is shown, and the corresponding data is obtained from the shared storage synchronously.

[0142] For example, each DN is based on Figure 8 The path information of each metadata recorded in the mainfest1 shown is used to obtain the corresponding metadata, and the obtained metadata is stored in the respective memories.

[0143] Step 1250: Each DN synchronously obtains the corresponding shared data from the shared storage according to the obtained metadata, and performs data analysis based on the shared data.

[0144] In the embodiment of the present application, each DN synchronously obtains the corresponding shared data from the shared storage according to the obtained metadata. For example, DN0 obtains the data of shard1 table1, the data of shard4 table1, and the data of shard7 table1 from the shared storage according to the stored metadata. DN1 obtains the data of shard2table1, the data of shard5 table1, and the data of shard8table1 from the shared storage according to the stored metadata. DN2 obtains the data of shard3 table1, the data of shard6 table1, and the data of shard9 table1 from the shared storage according to the stored metadata. And data analysis is performed based on the obtained shared data to execute the process of data consumption.

[0145] Combine the following Fig.13 , taking the incremental checkpoint process as an example, another implementation method for the data consumption cluster to obtain metadata from shared storage and the shared data corresponding to the metadata is described in detail. It should be understood that Fig.13 The examples are only intended to help those skilled in the art understand the embodiments of the present application, and are not intended to limit the embodiments of the present application to Fig.13 The specific numerical values ​​or specific scenarios shown in the examples. Fig.13 The examples given below are obviously susceptible to various equivalent modifications or changes, and such modifications and changes also fall within the scope of the embodiments of the present application.

[0146] Fig.13 FIG. 1 is a schematic flow chart of another method for a data consumption cluster to obtain metadata and shared data corresponding to the metadata from a shared storage provided by an embodiment of the present application. Fig.13 As shown, the method may include steps 1310 to 1350, and steps 1310 to 1350 are described in detail below.

[0147] Step 1310: The data consumption cluster selects CN1 from multiple CNs.

[0148] Similar to step 410, the specific selection method is described in step 410, which will not be described here. For ease of description, the following description is made by taking CN1 as an example for the CN selected by the data consumption cluster.

[0149] Step 1320: CN1 triggers the shared capability module, loads CSN_mainfest, and obtains the CSN number CSN1.

[0150] Step 1330: CN1 builds a metadata loading plan and sends the plan to each DN.

[0151] Step 1340: Each DN executes the Plan synchronously and loads the metadata of the checkpoint whose CSN is CSN1.

[0152] For example, each DN reads Fig.11 The mainfest2 shown in the figure records the path information of the newly added metadata in the shared storage and the path information of the metadata of the last CSN0 in the shared storage. Since the metadata of the last CSN0 has been stored in the memory of each DN, each DN only needs to Fig.11 The path information of the newly added metadata recorded in the mainfest2 shown is used to obtain the newly added metadata and store the newly added metadata in their respective memories.

[0153] For example, DN0 obtains metadata corresponding to the newly added shard1 table2 data (called shard1 table2 metadata) and metadata corresponding to the shard4 table2 data (called shard4table2 metadata) from the shared storage, and DN1 obtains metadata corresponding to the newly added shard2 table2 data (called shard2table2 metadata) from the shared storage, and stores them in their respective memories.

[0154] In some embodiments, if mainfest2 also records the metadata of the shared data that needs to be deleted compared to the last CSN0, then the corresponding DN in the data consumption cluster can delete the corresponding metadata from the metadata saved in the memory last time. The data consumption cluster can also write CSN0 to the shared storage, so that the data production cluster can determine that the data consumption cluster has read the shared data corresponding to CSN0 from the shared storage according to CSN0 written by the data consumption cluster, so that the data production cluster can delete the shared data and / or the metadata of the shared data that need to be deleted from the shared storage, realize garbage collection, and save the storage space of the shared storage.

[0155] Step 1350: Each DN synchronously obtains the newly added shared data from the shared storage according to the newly added metadata and performs the data analysis process.

[0156] In the embodiment of the present application, each DN synchronously obtains the newly added shared data from the shared storage according to the newly added metadata. For example, DN0 obtains the data of shard1 table2 and shard4 table2 from the shared storage according to the newly added metadata. DN1 obtains the data of shard2 table2 from the shared storage according to the newly added metadata.

[0157] It should be noted that the timing for the data producer to trigger the loading of incremental metadata may be periodic, or may be actively triggered by a query transaction, and the embodiments of the present application do not specifically limit this.

[0158] Combination of the above Figures 1 to 13 , describes in detail the method provided by the embodiment of the present application, and will be combined with Figure 14-17 , describes in detail the embodiment of the system of the present application. It should be understood that the description of the method embodiment corresponds to the description of the system embodiment, so the parts not described in detail can refer to the previous method embodiment.

[0159] Fig.14 1 is a schematic block diagram of a metadata processing system 1400 provided in an embodiment of the present application. The system 1400 can be implemented by software, hardware, or a combination of the two. The metadata processing system 1400 provided in an embodiment of the present application can implement the method flow shown in the embodiment of the present application, and the system 1400 includes: a data production cluster 1410, a shared storage 1420, and a data consumption cluster 1430, wherein the data production cluster 1410 is used to generate metadata of shared data, and the data production cluster includes multiple data nodes that provide the shared data; the data production cluster 1410 is also used to store the shared data and the metadata to the shared storage, and the metadata is used for the data consumption cluster to obtain the shared data from the shared storage, and the data consumption cluster includes multiple data nodes that use the shared data; the shared storage 1420 is used to generate path information of the metadata; the data consumption cluster 1430 is used to obtain the path information of the metadata from the shared storage; the data consumption cluster 1430 is also used to obtain the metadata from the shared storage based on the path information of the metadata; the data consumption cluster 1430 is also used to obtain the shared data from the shared storage based on the metadata.

[0160] Optionally, the data production cluster 1410 is further used to set access rights for the path information of the metadata; the data consumption cluster 1430 is specifically used to obtain the path information of the metadata using the access rights.

[0161] Optionally, the data production cluster 1410 is specifically used to: periodically obtain the path information of the metadata from the shared storage; or, when a target event occurs, obtain the path information of the metadata from the shared storage.

[0162] Optionally, the data production cluster 1410 is specifically used for: multiple data nodes in the data production cluster 1410 store metadata of their respective shared data in the shared storage.

[0163] Optionally, the shared storage 1420 is also used to delete the shared data or metadata of the shared data after the data consumption cluster 1430 reads the shared data.

[0164] The system 1400 here can be embodied in the form of functional modules. The functional modules here can be implemented in the form of software and / or hardware, which is not specifically limited.

[0165] For example, a "data node" may be a software program, a hardware circuit, or a combination of the two that implements the above functions. Exemplarily, the implementation of the data production cluster 1410 is described below by taking the data node in the data production cluster 1410 as an example. Similarly, for example, the implementation of the shared storage 1420 and the data consumption cluster 1430 may refer to the implementation of the data production cluster 1410.

[0166] The data node may include code running on a computing instance. The computing instance may include at least one of a physical host (computing device), a virtual machine, and a container. Furthermore, the computing instance may be one or more. For example, a data node may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed in the same region or in different regions. Furthermore, the multiple hosts / virtual machines / containers used to run the code may be distributed in the same availability zone (AZ) or in different AZs, each AZ including one data center or multiple data centers with close geographical locations. Generally, a region may include multiple AZs.

[0167] Similarly, multiple hosts / virtual machines / containers used to run the code can be distributed in the same virtual private cloud (VPC) or in multiple VPCs. Usually, a VPC is set up in a region. For cross-region communication between two VPCs in the same region and between VPCs in different regions, a communication gateway needs to be set up in each VPC to achieve interconnection between VPCs through the communication gateway.

[0168] As an example of a hardware functional unit, a data node may include at least one computing device, such as a server, etc. Alternatively, a data node may also be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0169] The multiple computing devices included in the data node can be distributed in the same region or in different regions. The multiple computing devices included in the data node can be distributed in the same AZ or in different AZs. Similarly, the multiple computing devices included in the data node can be distributed in the same VPC or in multiple VPCs. The multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0170] Therefore, each example described in the embodiments of the present application can be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0171] In addition, the system and method embodiments provided in the above embodiments belong to the same concept, and their specific implementation processes are detailed in the method embodiments above, which will not be repeated here.

[0172] The method provided in the embodiment of the present application can be performed by a computing device, which can also be referred to as a computer system. It includes a hardware layer, an operating system layer running on the hardware layer, and an application layer running on the operating system layer. The hardware layer includes hardware such as a processing unit, a memory and a memory control unit, and then the function and structure of the hardware are described in detail. The operating system is any one or more computer operating systems that implement business processing through a process, for example, a Linux operating system, a Unix operating system, an Android operating system, an iOS operating system, or a windows operating system. The application layer includes applications such as a browser, an address book, a word processing software, and an instant messaging software. In addition, optionally, the computer system is a handheld device such as a smart phone, or a terminal device such as a personal computer, and the present application is not particularly limited, as long as the method provided by the embodiment of the present application can be used. The execution subject of the method provided in the embodiment of the present application can be a computing device, or a functional module in a computing device that can call a program and execute a program.

[0173] Combine the following Fig.15 , a computing device provided in an embodiment of the present application is described in detail.

[0174] Fig.15 1 is a schematic diagram of the architecture of a computing device 1500 provided in an embodiment of the present application. The computing device 1500 may be a server or a computer or other device with computing capabilities. Fig.15 The computing device 1500 shown includes at least one processor 1510 and a memory 1520 .

[0175] It should be understood that the present application does not limit the number of processors and memories in the computing device 1500 .

[0176] The processor 1510 executes the instructions in the memory 1520, so that the computing device 1500 implements the method provided by the present application. Alternatively, the processor 1510 executes the instructions in the memory 1520, so that the computing device 1500 implements the functional modules provided by the present application, thereby implementing the method provided by the present application.

[0177] Optionally, the computing device 1500 further includes a communication interface 1530. The communication interface 1530 uses a transceiver module such as, but not limited to, a network interface card or a transceiver to implement communication between the computing device 1500 and other devices or a communication network.

[0178] Optionally, the computing device 1500 further includes a system bus 1540, wherein the processor 1510, the memory 1520, and the communication interface 1530 are respectively connected to the system bus 1540. The processor 1510 can access the memory 1520 through the system bus 1540. For example, the processor 1510 can read and write data or execute code in the memory 1520 through the system bus 1540. The system bus 1540 is a peripheral component interconnect express (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The system bus 1540 is divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig.15 It is represented by only one thick line, but it does not mean that there is only one bus or one type of bus.

[0179] In a possible implementation, the function of the processor 1510 is mainly to interpret the instructions (or codes) of the computer program and process the data in the computer software. The instructions of the computer program and the data in the computer software can be stored in the memory 1520 or the cache 1516.

[0180] Optionally, the processor 1510 may be an integrated circuit chip with signal processing capabilities. As an example and not a limitation, the processor 1510 is a general-purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component. Among them, the general-purpose processor is a microprocessor, etc. For example, the processor 1510 is a central processing unit (CPU).

[0181] Optionally, each processor 1510 includes at least one processing unit 1512 and a memory control unit 1514 .

[0182] Optionally, the processing unit 1512 is also called a core or kernel, which is the most important component of the processor. The processing unit 1512 is manufactured from single crystal silicon using a certain production process, and all calculations, command reception, command storage, and data processing of the processor are performed by the core. The processing units run program instructions independently and use the ability of parallel computing to speed up the running speed of the program. Various processing units have a fixed logical structure. For example, the processing unit includes logical units such as a first-level cache, a second-level cache, an execution unit, an instruction-level unit, and a bus interface.

[0183] In one implementation example, the memory control unit 1514 is used to control data interaction between the memory 1520 and the processing unit 1512. Specifically, the memory control unit 1514 receives a memory access request from the processing unit 1512, and controls access to the memory based on the memory access request. As an example and not a limitation, the memory control unit is a device such as a memory management unit (MMU).

[0184] In one implementation example, each memory control unit 1514 addresses the memory 1520 through the system bus. And an arbiter ( Fig.15 ), which is responsible for handling and coordinating competing accesses of multiple processing units 1512.

[0185] In an implementation example, the processing unit 1512 and the memory control unit 1514 are connected to each other through connection lines inside the chip, such as address lines, so as to achieve communication between the processing unit 1512 and the memory control unit 1514.

[0186] Optionally, each processor 1510 also includes a cache 1516, wherein the cache is a buffer for data exchange (called cache). When the processing unit 1512 wants to read data, it will first search for the required data from the cache. If it is found, it will be executed directly. If it is not found, it will be searched from the memory. Since the running speed of the cache is much faster than the memory, the role of the cache is to help the processing unit 1512 run faster.

[0187] The memory 1520 can provide a running space for the processes in the computing device 1500. For example, the computer program (specifically, the program code) used to generate the process is stored in the memory 1520. After the computer program is executed by the processor to generate the process, the processor allocates a corresponding storage space for the process in the memory 1520. Furthermore, the above storage space further includes a text segment, an initialized data segment, a bit initialized data segment, a stack segment, a heap segment, etc. The memory 1520 stores the data generated during the running of the process in the storage space corresponding to the above process, such as intermediate data, process data, etc.

[0188] Optionally, the storage is also called memory, and its function is to temporarily store the operation data in the processor 1510 and the data exchanged with the external storage such as the hard disk. As long as the computer is running, the processor 1510 will transfer the data to be calculated to the memory for calculation, and when the calculation is completed, the processing unit 1512 will transmit the result.

[0189] As an example and not limitation, memory 1520 is a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory is a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory is a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchlink DRAM (SLDRAM), and direct rambus RAM (DRRAM). It should be noted that the memory 1520 of the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0190] The structure of the computing device 1500 listed above is only an example, and the present application is not limited thereto. The computing device 1500 of the embodiment of the present application includes various hardware in the computer system in the prior art. For example, the computing device 1500 also includes other memories besides the memory 1520, such as disk storage, etc. Those skilled in the art should understand that the computing device 1500 may also include other devices necessary for normal operation. At the same time, according to specific needs, those skilled in the art should understand that the above-mentioned computing device 1500 may also include hardware devices for implementing other additional functions. In addition, those skilled in the art should understand that the above-mentioned computing device 1500 may also include only the devices necessary to implement the embodiment of the present application, and does not necessarily include Fig.15 All devices shown in .

[0191] The embodiment of the present application also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device may be a server. In some embodiments, the computing device may also be a terminal device such as a desktop computer, a laptop computer, or a smart phone.

[0192] like Fig.16 As shown, the computing device cluster includes at least one computing device 1500. The memory 1520 in one or more computing devices 1500 in the computing device cluster may store the same instructions for executing the above method.

[0193] In some possible implementations, the memory 1520 in one or more computing devices 1500 in the computing device cluster may also store some instructions for executing the above method. In other words, the combination of one or more computing devices 1500 may jointly execute the instructions of the above method.

[0194] It should be noted that the memory 1520 in different computing devices 1500 in the computing device cluster may store different instructions, which are respectively used to execute part of the functions of the above system. That is, the instructions stored in the memory 1520 in different computing devices 1500 may implement one or more functions of the above system.

[0195] In some possible implementations, one or more computing devices in the computing device cluster may be connected via a network, which may be a wide area network or a local area network. Fig.17 A possible implementation is shown. Fig.17 As shown, two computing devices 1500A and 1500B are connected via a network. Specifically, they are connected to the network via a communication interface in each computing device.

[0196] It should be understood that Fig.17The functionality of the computing device 1500A shown in FIG. 1 may also be implemented by multiple computing devices 1500. Similarly, the functionality of the computing device 1500B may also be implemented by multiple computing devices 1500.

[0197] In this embodiment, a computer program product including instructions is also provided, and the computer program product may be software or a program product including instructions that can be run on a computing device cluster or stored in any available medium. When the computer program product is run by a computing device cluster, the computing device cluster executes the method provided above, or the computing device cluster implements the function of the system provided above.

[0198] In this embodiment, a computer-readable storage medium is also provided, and the computer-readable storage medium can be any available medium that can be stored by a computing device or a data storage device such as a data center that includes one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk), etc. The computer-readable storage medium includes instructions, and when the instructions in the computer-readable storage medium are executed by a computing device cluster, the computing device cluster executes the method provided above.

[0199] It should be understood that in the various embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0200] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0201] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-described system can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0202] In the several embodiments provided in this application, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are only schematic, for example, the division of the units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed.

[0203] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0204] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0205] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage media include: various media that can store program codes, such as USB flash drives, mobile hard disks, read-only memories (ROM), random access memories (RAM), magnetic disks or optical disks.

[0206] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A method for processing metadata, characterized in that: The method comprises: A data production cluster generates metadata of shared data, wherein the data production cluster includes a plurality of data nodes that provide the shared data; The data production cluster stores the shared data and the metadata in a shared storage, the metadata is used by a data consumption cluster to obtain the shared data from the shared storage, and the data consumption cluster includes a plurality of data nodes that use the shared data; The shared storage generates path information of the metadata; The data consumption cluster obtains the path information of the metadata from the shared storage; The data consumption cluster obtains the metadata from the shared storage based on the path information of the metadata; The data consumption cluster obtains the shared data from the shared storage based on the metadata.

2. The method according to claim 1, characterized in that The method further comprises: The data production cluster sets access rights to the path information of the metadata; The data consumption cluster uses the access permission to obtain the path information of the metadata.

3. The method according to claim 1 or 2, characterized in that: The data consumption cluster obtains the path information of the metadata from the shared storage, including: The data consumption cluster periodically obtains the path information of the metadata from the shared storage; or, When a target event occurs, the data consumption cluster obtains the path information of the metadata from the shared storage.

4. The method according to any one of claims 1 to 3, characterized in that The data production cluster stores the metadata in a shared storage, including: Multiple data nodes in the data production cluster store metadata of their respective shared data in the shared storage.

5. The method according to any one of claims 1 to 4, characterized in that The method further comprises: The shared storage deletes the shared data or metadata of the shared data after the data consumption cluster reads the shared data.

6. A metadata processing system, characterized in that: The system comprises: A data production cluster, used to generate metadata of shared data, wherein the data production cluster includes a plurality of data nodes that provide the shared data; The data production cluster is further used to store the shared data and the metadata in a shared storage, the metadata is used by a data consumption cluster to obtain the shared data from the shared storage, and the data consumption cluster includes a plurality of data nodes that use the shared data; Shared storage, used to generate path information of the metadata; The data consumption cluster is used to obtain the path information of the metadata from the shared storage; The data consumption cluster is further used to obtain the metadata from the shared storage based on the path information of the metadata; The data consumption cluster is further used to obtain the shared data from the shared storage based on the metadata.

7. The system according to claim 6, characterized in that The data production cluster is further used to set access rights to the path information of the metadata; The data consumption cluster is further used to obtain the path information of the metadata using the access permission.

8. The system according to claim 6 or 7, characterized in that: The data consumption cluster is specifically used for: periodically obtaining the path information of the metadata from the shared storage; or, When a target event occurs, path information of the metadata is obtained from the shared storage.

9. The system according to any one of claims 6 to 8, characterized in that The data production cluster is specifically used for: Multiple data nodes in the data production cluster store metadata of their respective shared data in the shared storage.

10. The system according to any one of claims 6 to 9, characterized in that The shared storage is further used to delete the shared data or metadata of the shared data after the data consumption cluster reads the shared data.

11. A computing device cluster, characterized in that: comprising at least one computing device, each computing device comprising a processor and a memory; The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster executes the method according to any one of claims 1 to 5.

12. A computer program product comprising instructions, characterized in that When the instructions are executed by a computing device cluster, the computing device cluster is caused to perform the method according to any one of claims 1 to 5.

13. A computer-readable storage medium, characterized in that: The method comprises computer program instructions, and when the computer program instructions are executed by a computing device cluster, the computing device cluster performs the method as claimed in any one of claims 1 to 5.

Images