Metadata processing method and system, and computing device

Abstract

A metadata processing method and system, and a computing device. The method comprises: a data production cluster generating metadata of shared data, and storing the shared data and the metadata in a shared memory, and the shared memory generating path information of the metadata; and a data consumption cluster acquiring the path information of the metadata from the shared memory, acquiring the metadata from the shared memory on the basis of the path information of the metadata, and acquiring the shared data from the shared memory on the basis of the metadata. The method can reduce the storage cost of metadata of shared data.

Description

Metadata processing method, system and computing device

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office on June 29, 2023, with application number 202310786187.1 and invention name “Metadata processing method, system and computing device”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of data storage, and more specifically, to a metadata processing method, system, and computing device. Background Art

[0003] Data warehouses serve as a platform for data storage and analysis. Major cloud vendors have launched data warehouse services. Data sharing has become a key feature of data warehouse services. Its purpose is to eliminate data silos, enable data transactions and sharing between users, and promote interoperability and sharing of data. Data sharing involves data producers providing shared data, and data consumers subscribing to or purchasing certain shared data produced by the data producers. After purchasing this shared data, the data consumer can access it and perform data analysis and computation based on it.

[0004] When data producers produce and store shared data, they also generate and store metadata for that shared data. When data consumers retrieve shared data from shared storage, they need to retrieve the shared data based on the metadata. In related existing solutions, the metadata for shared data is stored separately in a metadata cluster. Because the metadata cluster is a separate cluster for storing metadata, its storage costs are high.

[0005] Therefore, how to reduce the storage cost of metadata of shared data has become a technical problem that needs to be solved urgently.

[0006] Summary of the Invention

[0007] The present application provides a metadata processing method, system, and computing device, which can reduce the storage cost of metadata of shared data.

[0008] In a first aspect, a metadata processing method is provided, which includes: a data production cluster generates metadata for shared data, the data production cluster includes multiple data nodes that provide the shared data; the data production cluster stores the shared data and the metadata in shared storage, the metadata is used by a data consumption cluster to obtain the shared data from the shared storage, the data consumption cluster includes multiple data nodes that use the shared data; the shared storage generates path information for the metadata; the data consumption cluster obtains the path information for the metadata from the shared storage; the data consumption cluster obtains the metadata from the shared storage based on the path information for the metadata; the data consumption cluster obtains the shared data from the shared storage based on the metadata.

[0009] In the above technical solution, by reusing the coexistence storage for storing shared data, the metadata of the shared data is stored in the coexistence storage. In this way, there is no need to introduce a separate cluster to store metadata, which can reduce the storage cost of the metadata of the shared data and enable large-scale, multi-cluster data sharing.

[0010] In combination with the first aspect, in certain implementations of the first aspect, the method further includes: the data production cluster sets access rights to the path information of the metadata; and the data consumption cluster uses the access rights to obtain the path information of the metadata.

[0011] In the above technical solution, the data production cluster sets access rights to the path information of the metadata, so that only the data consumption cluster with the access rights can obtain the path information of the metadata.

[0012] In combination with the first aspect, in certain implementations of the first aspect, the data consumption cluster periodically obtains the path information of the metadata from the shared storage; or, the data consumption cluster obtains the path information of the metadata from the shared storage when a target event occurs.

[0013] In combination with the first aspect, in certain implementations of the first aspect, multiple data nodes in the data production cluster store metadata of their respective shared data in the shared storage.

[0014] In combination with the first aspect, in some implementations of the first aspect, the method further includes: after the data consumption cluster reads the shared data, the shared storage deletes the shared data or metadata of the shared data.

[0015] In the above technical solution, after the data consumption cluster reads the shared data, the data production cluster deletes the shared data or the metadata of the shared data in the shared storage. In this way, the shared data or the metadata of the shared data used by the data consumption cluster is garbage collected, thereby further saving the storage space of the shared storage.

[0016] In a second aspect, a metadata processing system is provided, which includes: a data production cluster, shared storage and a data consumption cluster, wherein the data production cluster is used to generate metadata for shared data, and the data production cluster includes multiple data nodes that provide the shared data; the data production cluster is also used to store the shared data and the metadata in the shared storage, and the metadata is used for the data consumption cluster to obtain the shared data from the shared storage, and the data consumption cluster includes multiple data nodes that use the shared data; the shared storage is used to generate path information for the metadata; the data consumption cluster is used to obtain the path information for the metadata from the shared storage; the data consumption cluster is also used to obtain the metadata from the shared storage based on the path information for the metadata; the data consumption cluster is also used to obtain the shared data from the shared storage based on the metadata.

[0017] In combination with the second aspect, in certain implementations of the second aspect, the data production cluster is further used to set access rights for the path information of the metadata; and the data consumption cluster is further used to obtain the path information of the metadata using the access rights.

[0018] In combination with the second aspect, in certain implementations of the second aspect, the data consumption cluster is specifically used to: periodically obtain the path information of the metadata from the shared storage; or, when a target event occurs, obtain the path information of the metadata from the shared storage.

[0019] In combination with the second aspect, in some implementations of the second aspect, the data production cluster is specifically configured to: multiple data nodes in the data production cluster store metadata of their respective shared data in the shared storage.

[0020] In combination with the second aspect, in certain implementations of the second aspect, the shared storage is further used to delete the shared data or metadata of the shared data after the data consumption cluster reads the shared data.

[0021] It should be noted that for the beneficial effects in the second aspect, please refer to the beneficial effects in the first aspect, and no further details will be given here.

[0022] In a third aspect, a computing device cluster is provided, comprising at least one computing device, each computing device comprising a processor and a memory; the processor of the at least one computing device is used to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster executes the method in the first aspect or any possible implementation of the first aspect.

[0023] Optionally, the processor may be a general-purpose processor, which may be implemented in hardware or software. When implemented in hardware, the processor may be a logic circuit, an integrated circuit, or the like; when implemented in software, the processor may be a general-purpose processor implemented by reading software code stored in a memory, which may be integrated into the processor or located independently of the processor.

[0024] In a fourth aspect, a computer program product comprising instructions is provided. When the instructions are executed by a computing device cluster, the computing device cluster executes the method in the first aspect and any one of the implementations of the first aspect.

[0025] In a fifth aspect, a computer-readable storage medium is provided, comprising computer program instructions. When the computer program instructions are executed by a computing device cluster, the computing device cluster executes the method according to the first aspect and any one of the implementations of the first aspect.

[0026] As an example, computer-readable storage media include, but are not limited to, one or more of the following: read-only memory (ROM), programmable ROM (PROM), erasable PROM (EPROM), Flash memory, electrically EPROM (EEPROM), and a hard drive.

[0027] Optionally, as an implementation manner, the above-mentioned storage medium may specifically be a non-volatile storage medium. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] FIG1 is a schematic block diagram of a data sharing service provided by a data warehouse.

[0029] FIG2 is a schematic block diagram of a cloud scenario applicable to an embodiment of the present application.

[0030] FIG3 is a schematic flowchart of a metadata processing method provided in an embodiment of the present application.

[0031] FIG4 is a schematic flowchart of a method for storing metadata of shared data in a data production cluster provided in an embodiment of the present application.

[0032] FIG5 is a schematic block diagram of a data production cluster provided in an embodiment of the present application.

[0033] FIG6 is an example of the correspondence between data nodes and data shards in a data production cluster provided in an embodiment of the present application.

[0034] FIG7 is a schematic block diagram of data segments and metadata corresponding to shard1 of the shared data table1 in the example of this application.

[0035] FIG8 is a schematic diagram of information recorded in a manifest1 file provided in an embodiment of the present application.

[0036] FIG9 is a schematic flowchart of another method for storing metadata of shared data in a data production cluster provided in an embodiment of the present application.

[0037] FIG10 is a schematic block diagram of newly added shared data of a data production cluster and metadata corresponding to the shared data provided in an embodiment of the present application.

[0038] FIG11 is a schematic diagram of information recorded in a manifest2 file provided in an embodiment of the present application.

[0039] FIG12 is a schematic flowchart of a method provided by an embodiment of the present application for a data consumption cluster to obtain metadata and shared data corresponding to the metadata from shared storage.

[0040] FIG13 is a schematic flowchart of another method provided by an embodiment of the present application for a data consumption cluster to obtain metadata and shared data corresponding to the metadata from shared storage.

[0041] FIG14 is a schematic block diagram of a metadata processing system 1400 provided in an embodiment of the present application.

[0042] FIG15 is a schematic diagram of the architecture of a computing device 1500 provided in an embodiment of the present application.

[0043] FIG16 is a schematic diagram of the architecture of a computing device cluster provided in an embodiment of the present application.

[0044] FIG17 is a schematic diagram of a network connection between computing devices 1500A and 1500B provided in an embodiment of the present application. DETAILED DESCRIPTION

[0045] The technical solution in this application will be described below with reference to the accompanying drawings.

[0046] This application will present various aspects, embodiments, or features around systems including multiple devices, components, modules, etc. It should be understood and appreciated that each system may include additional devices, components, modules, etc., and / or may not include all of the devices, components, modules, etc. discussed in conjunction with the figures. Furthermore, combinations of these aspects may also be used.

[0047] Additionally, in the embodiments of this application, words such as "exemplary" and "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described in this application as "exemplary" should not be construed as being preferred or advantageous over other embodiments or designs. Rather, the use of the word "exemplary" is intended to present concepts in a concrete manner.

[0048] In the embodiments of the present application, “corresponding” and “relevant” may sometimes be used interchangeably. It should be noted that when the distinction between them is not emphasized, the meanings they intend to express are consistent.

[0049] The business scenarios described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. A person skilled in the art will appreciate that, with the evolution of network architecture and the emergence of new business scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0050] References to "one embodiment" or "some embodiments" in this specification mean that a particular feature, structure, or characteristic described in conjunction with that embodiment is included in one or more embodiments of the present application. Thus, phrases such as "in one embodiment," "in some embodiments," "in other embodiments," and "in yet other embodiments" appearing in various places in this specification do not necessarily refer to the same embodiment, but rather mean "one or more but not all embodiments," unless otherwise specifically emphasized. The terms "including," "comprising," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0051] In this application, "at least one" means one or more, and "more" means two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: including the existence of A alone, the existence of A and B at the same time, and the existence of B alone, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can mean: a, b, c, ab, ac, bc, or abc, where a, b, c can be single or multiple.

[0052] Data warehouses serve as a platform for data storage and analysis. Currently, major cloud vendors have launched data warehouse services. Data sharing has become a key feature of data warehouse services, aiming to eliminate data silos, enable data transactions and sharing between users, and achieve data interoperability and sharing among users.

[0053] It should be understood that the data sharing service provided by the data warehouse mentioned above refers to data producers providing shared data, and data consumers subscribing to or purchasing certain shared data produced by the data producers. After purchasing this shared data, the data consumer can access the shared data and perform data analysis and calculations based on it.

[0054] As an example, the shared data may include but is not limited to: a table produced by a data producer, or a materialized view, or certain column data in a table, or certain row data in a table, etc.

[0055] For ease of understanding, the basic principles of the data sharing service provided by the data warehouse are described in detail below with reference to FIG1 .

[0056] As shown in Figure 1, data producers continuously produce data in the cloud data warehouse and write it to shared storage. Data producers can also authorize data consumers to read certain data stored in shared storage based on their data subscriptions or purchases. Data consumers can then read the data they purchased or subscribed to from shared storage and perform data analysis and computation based on that data, completing the data consumption process.

[0057] It should be understood that data stored in the shared storage may be referred to as shared data.

[0058] Specifically, when the data producer generates and stores shared data, it also generates and stores metadata for the shared data. When the data consumer obtains the shared data from the shared storage, it needs to obtain the shared data from the shared storage based on the metadata of the shared data.

[0059] It should be understood that the metadata of shared data is used to describe the shared data and is information that describes the attributes of the shared data. For example, the metadata of shared data may include but is not limited to information such as the location and address of the shared data stored in the shared storage.

[0060] In the related existing technical solutions, the metadata of the shared data is stored separately in a metadata cluster (ie, a Metadata cluster). Since the Metadata cluster is an independent cluster for storing metadata, the storage cost of the Metadata cluster is relatively high.

[0061] In view of this, an embodiment of the present application provides a metadata processing method, which can reduce the storage cost of metadata of shared data and can also perform large-scale, multi-cluster data sharing.

[0062] In one possible implementation, the method provided in the embodiment of the present application can be applied to a cloud service scenario, and the method is executed by a cloud management platform in the cloud service scenario. For ease of description, the cloud service scenario is described in detail below with reference to FIG2 .

[0063] FIG2 is a schematic block diagram of a cloud scenario applicable to an embodiment of the present application. As shown in FIG2 , the cloud scenario may include: a cloud management platform 110 , the Internet 120 , and a client 130 .

[0064] As shown in Figure 2, the cloud management platform 110 is used to manage the infrastructure that provides multiple cloud services. The infrastructure includes multiple cloud data centers, each of which includes multiple servers, each of which includes cloud service resources to provide corresponding cloud services to tenants.

[0065] The cloud management platform 110 can be located in a cloud data center, which can provide an access interface (such as an interface or an application program interface (API)). The tenant can operate the client 130 to remotely access the access interface to register a cloud account and password on the cloud management platform 110, and log in to the cloud management platform 110. After the cloud management platform 110 successfully authenticates the cloud account and password, the tenant can further pay to select and purchase a virtual machine with specific specifications (processor, memory, disk) on the cloud management platform 110. After the payment is successful, the cloud management platform 110 provides the remote login account and password of the purchased virtual machine, and the client 130 can remotely log in to the virtual machine, install and run the tenant's application in the virtual machine. Therefore, the tenant can create, manage, log in and operate virtual machines in the cloud data center through the cloud management platform 110. Among them, the virtual machine can also be called a cloud server (elastic compute service, ECS) or an elastic instance (different cloud service providers have different names).

[0066] It should be understood that tenants of cloud services can be individuals, enterprises, schools, hospitals, administrative agencies, etc.

[0067] The functions of the cloud management platform 110 include, but are not limited to, a user console, computing management services, network management services, storage management services, authentication services, and image management services. The user console provides an interface or API for interacting with tenants. The computing management service is used to manage servers running virtual machines and containers, as well as bare metal servers. The network management service is used to manage network services (such as gateways and firewalls). The storage management service is used to manage storage services (such as data bucket services). The authentication service is used to manage tenant accounts and passwords. The image management service is used to manage virtual machine images. Tenants can use the client 130 to log in to the cloud management platform 110 via the Internet 120 to manage the rented cloud services.

[0068] FIG3 is a schematic flow chart of a metadata processing method provided by an embodiment of the present application. As shown in FIG3 , the method may include steps 310 to 360 , which are described in detail below.

[0069] Step 310: The data production cluster generates metadata for the shared data.

[0070] In the embodiment of the present application, after the data production cluster produces the shared data, it can also generate metadata for the shared data. For details about the metadata of the shared data, please refer to the description above and will not be repeated here.

[0071] As an example, the data production cluster may include multiple data nodes (DN), which are used to produce or provide the shared data.

[0072] Step 320: The data production cluster stores the shared data and metadata in the shared storage.

[0073] In the embodiment of the present application, the data production cluster can store both the shared data and the metadata of the shared data in the shared storage, wherein the metadata is used by the data consumption cluster to obtain the corresponding shared data from the shared storage.

[0074] For example, taking a data production cluster including multiple data nodes as an example, the multiple data nodes in the data production cluster can store metadata of their respective shared data in the shared storage.

[0075] There are many ways for the above-mentioned data production cluster to store metadata in shared storage, and the embodiments of the present application do not specifically limit this. In one possible implementation method, the data production cluster can write the metadata of the shared data to the shared storage based on a certain consistency point (checkpoint). Taking into account the performance of metadata writing and the amount of data written, it can be divided into a full checkpoint process and an incremental checkpoint process. Among them, the full checkpoint process refers to writing the full metadata of the current shared data to the shared storage at a certain consistency point. The incremental checkpoint process refers to writing the metadata of the incremental shared data to the shared storage at a certain consistency point based on the consistency point of the previous checkpoint. These two different methods will be described in detail below in conjunction with specific embodiments, and will not be described in detail here.

[0076] It should be understood that to ensure the consistency of shared data within business logic, data producers uniformly construct a consistency point (also known as a snapshot at a specific moment) for the metadata on distributed nodes. Since the data itself is already written to shared storage during the data generation process, metadata consistency is particularly important during data sharing. On the one hand, in a distributed system, all nodes have a unified consistency point. At this consistency point, data read through metadata is consistent with business logic, meeting the ACID requirements of the database system. On the other hand, the metadata written to the consistency point by all nodes is located in a certain path in shared storage and cannot overwrite each other.

[0077] As an example, the above-mentioned shared storage can be an object storage service (OBS) in cloud storage or other storage services. As long as the shared storage function can be achieved, the embodiments of the present application do not make specific limitations on this.

[0078] As an example, the data consumption cluster may include multiple data nodes (DN), which are used to use the shared data stored in the shared storage by the data production cluster.

[0079] Step 330: Share the path information of the generated metadata.

[0080] In an embodiment of the present application, after the data production cluster stores the metadata in the shared storage, the shared storage can allocate storage space for the metadata, thereby generating path information of the metadata.

[0081] In some embodiments, the data production cluster may further set access rights to the path information of the metadata, so that the data consumption cluster having the access rights may use the access rights to obtain the path information of the metadata.

[0082] Step 340: The data consumption cluster obtains metadata path information from the shared storage.

[0083] In an embodiment of the present application, the data consumption cluster may obtain metadata path information from shared storage. In one possible implementation, the data consumption cluster may periodically obtain metadata path information from the shared storage. In another possible implementation, the data consumption cluster may also obtain metadata path information from the shared storage when a target event occurs.

[0084] In some embodiments, if the path information of the metadata is set with an access permission, the data consumption cluster may also use the access permission to obtain the path information of the metadata.

[0085] Step 350: The data consumption cluster obtains metadata from the shared storage based on the path information of the metadata.

[0086] In the embodiment of the present application, the data consumption cluster may obtain the stored metadata from the shared storage according to the path information of the obtained metadata.

[0087] Step 360: The data consumption cluster obtains shared data from the shared storage based on the metadata.

[0088] In an embodiment of the present application, the data consumption cluster can obtain the shared data corresponding to the metadata stored in the shared storage based on the obtained metadata, thereby performing data analysis based on the obtained shared data and executing the data consumption process.

[0089] For example, taking a data consumption cluster including multiple data nodes as an example, the multiple data nodes in the data consumption cluster can obtain shared data from shared storage in parallel according to the metadata of the shared data, and perform data analysis based on the obtained shared data.

[0090] In some embodiments, the shared storage may further delete the shared data or delete metadata of the shared data after the data consumption cluster reads the shared data.

[0091] In the above technical solution, by reusing the shared storage for storing shared data, the metadata of the shared data is stored in the shared storage, and metadata sharing is achieved without introducing a separate cluster to store metadata. This can reduce the storage cost of the metadata of the shared data and can also carry out large-scale, multi-cluster data sharing.

[0092] In conjunction with Figures 4 to 8, taking the full checkpoint process as an example, a detailed description is given of an implementation method in which a data production cluster stores metadata of shared data in shared storage. It should be understood that the examples of Figures 4 to 8 are only intended to help those skilled in the art understand the embodiments of the present application, and are not intended to limit the application embodiments to the specific numerical values ​​or specific scenarios illustrated in Figures 4 to 8. Those skilled in the art can obviously make various equivalent modifications or changes based on the following examples given in Figures 4 to 8, and such modifications and changes also fall within the scope of the embodiments of the present application.

[0093] Figure 4 is a schematic flow chart of a method for storing metadata of shared data in a data production cluster according to an embodiment of the present application. As shown in Figure 4, the method may include steps 410 to 460, which are described in detail below.

[0094] Step 410: Coordinator node 1 (CN1) queries the timestamp oracle (TSO) and obtains the commit sequence number (CSN) of the latest successfully committed transaction from TSO.

[0095] As an example, as shown in Figure 5, a data production cluster includes multiple coordinator nodes (CNs). The functions of a data production cluster's CNs may include, but are not limited to, transaction management, query management, permission management, access control, and sharing control. For example, the data production cluster's CN records information such as which data needs to be shared (e.g., a specific table, a materialized view, table columns, or table rows), which data is shared with which tenant's consumer cluster, and so on.

[0096] In an embodiment of the present application, the data production cluster may select a CN from the multiple CNs to execute the metadata storage process. There are various specific selection methods, which are not limited in the present embodiment. For example, a CN may be selected from multiple CNs through voting to execute the process. In another example, the CN with the largest ID may be selected from multiple CNs to execute the process. In another example, the CN with the smallest ID may be selected from multiple CNs to execute the process.

[0097] It should be noted that the embodiment of the present application does not specifically limit the number of CNs and DNs included in the data production cluster. For the sake of ease of description, Figure 5 uses two CNs (CN1, CN2) and three DNs (DN0, DN1, DN2) as an example.

[0098] For example, consider the case where CN1 performs metadata storage. CN1 triggers the data sharing module, which interacts with the system's TSO to ensure logical consistency. Specifically, the data sharing module in CN1 queries the TSO for the CSN of the last submitted successful request. For example, the most recently submitted CSN is CSN0.

[0099] It should be understood that the core of a database is to sort the operations of each transaction. In a traditional stand-alone architecture, transaction sorting can be easily achieved through log sequence numbers or transaction IDs. However, in a distributed architecture, the database runs on multiple servers, and each database instance has an independent clock or log. The clock points between servers vary in speed, so the clocks in a distributed database cannot reflect the global order of things. As one of the key technologies of distributed databases, clock synchronization technology exists to solve the problem of the order in which events occur in distributed databases. TSO is a clock synchronization technology that provides time services and uses a centralized control service to assign monotonically increasing timestamps, such as monotonically increasing integer CSNs. TSO ensures correct linearizability and good performance.

[0100] Step 420: CN1 constructs a plan for writing metadata and sends the plan to each DN.

[0101] For example, at the initial stage of data sharing, the full metadata corresponding to the shared data at the current checkpoint needs to be written to the shared storage. Therefore, the metadata write plan constructed by CN1 instructs each DN to write the full metadata of the current checkpoint stored in its memory to the shared storage.

[0102] For ease of description, the following takes table 1 as an example of shared data written to shared storage.

[0103] Step 430: Each DN writes the full metadata of the current checkpoint to the shared storage according to the metadata plan synchronization.

[0104] It should be understood that in a distributed system, table data can be distributed across various DNs in various ways, including, but not limited to, Hash distribution, RoundRobin distribution, and Replica distribution. The corresponding data is referred to as a Shard, Segment, Bucket, or Partition. While different product systems use different names, the term "data shard" is used in this embodiment. Shards split table data into multiple parts, with each DN managing and storing a portion of the table data.

[0105] For example, as shown in Figure 6, taking table 1 as an example, table 1 is split into nine shards for management, namely shard 1 to shard 9. Among them, DN0 manages and stores shard 1, shard 4, and shard 7; DN1 manages and stores shard 2, shard 5, and shard 8; and DN2 manages and stores shard 3, shard 6, and shard 9.

[0106] In the embodiment of the present application, the CN can manage and record the correspondence between the above DN and shard. If a DN node is scaled up or down, only the correspondence between the shard and the DN node needs to be changed. The purpose of this is to make the data distribution independent of the number of nodes.

[0107] Because each shard can store and manage a large amount of data, the data in the shard needs to be further segmented (data segments). Each DN synchronously writes its portion of table1 to shared storage. For example, as shown in Figure 5, the data for shard1, table1, shard4, and table1 are written to shared storage by DN0; the data for shard2, table1, shard5, and table1 are written to shared storage by DN1; and the data for shard3, table1, shard6, and table1 are written to shared storage by DN2. Taking the data for shard1, table1 as an example, as shown in Figure 7, this data includes data segments 1 to m. The remaining data structures are described in Figure 7 and are not detailed here.

[0108] It should be understood that in the embodiment of the present application, the data obtained after further segmenting the data in the shard is called a data segment. Different systems have different names, for example, Mirco-Partition, SSTFile, Compress Unit, etc. This application does not make specific limitations on this.

[0109] In this embodiment of the present application, since each data segment has its own corresponding metadata, and the metadata is relatively small relative to the data volume, the metadata of multiple data segments in a shard can be written into a metadata file. For example, as shown in Figure 7, taking the metadata of shard1 table1 as an example, the metadata includes the metadata of data segments 1 to data segments m.

[0110] Step 440: Each DN records the current CSN number and the path information of the currently stored metadata in the shared storage in the shared storage.

[0111] In a possible implementation, as shown in FIG8 , each DN may write a manifest1 file into the shared storage. The manifest1 file records CSN0 and path information of each metadata in the shared storage.

[0112] Step 450: Set CSN_Mainfest and record the current CSN number.

[0113] In the embodiment of the present application, the CSN number recorded in CSN_Mainfest is CSN0.

[0114] Step 460: Set the permissions for the metadata path and the permissions for sharing data.

[0115] In an embodiment of the present application, the data producer can set permissions for the metadata path and permissions for sharing data. In this way, by setting permissions, only users who have purchased or subscribed to the data can access it, while users who have not purchased or subscribed to the data have no access.

[0116] In conjunction with Figures 9 to 11, the following describes in detail an implementation method in which a data production cluster stores metadata of shared data in shared storage, taking the incremental checkpoint process as an example. It should be understood that the examples of Figures 9 to 11 are merely to help those skilled in the art understand the embodiments of the present application, and are not intended to limit the application embodiments to the specific numerical values ​​or specific scenarios illustrated in Figures 9 to 11. Those skilled in the art can obviously make various equivalent modifications or changes based on the following examples given in Figures 9 to 11, and such modifications and changes also fall within the scope of the embodiments of the present application.

[0117] Figure 9 is a schematic flow chart of another method for storing metadata of shared data in a data production cluster provided by an embodiment of the present application. As shown in Figure 9, the method may include steps 910 to 960, which are described in detail below.

[0118] Step 910: CN1 queries TSO and obtains the latest successfully submitted CSN from TSO.

[0119] The method for the data production cluster to select a CN from multiple CNs to execute the metadata storage process is the same as that in step 410. Please refer to the description in step 410 for the specific selection method, which will not be repeated here.

[0120] For example, consider CN1 performing metadata storage. CN1 triggers the data sharing module, which interacts with the system's TSO to ensure logical consistency. Specifically, the data sharing module in CN1 queries the TSO for the most recently submitted CSN, for example, CSN1.

[0121] Step 920: CN1 constructs a plan for writing metadata and sends the plan to each DN.

[0122] For example, the initial full checkpoint and subsequent metadata checkpoints will write consistent metadata incrementally. Therefore, the metadata write plan constructed by CN1 instructs each DN to write the metadata added to the current checkpoint stored in its memory compared to the previous checkpoint to shared storage.

[0123] Step 930: Each DN writes the metadata newly added to the current checkpoint compared to the previous checkpoint into the shared storage according to the metadata plan synchronization.

[0124] As an example, each DN executes the above plan and identifies the data segments that have increased or decreased relative to the last checkpoint through metadata. This can be done through CSN comparison or through the last Manifest comparison. The embodiments of this application do not make specific limitations on this.

[0125] For example, as shown in Figure 10, the newly added data in shared storage is the data in shard1 table2, shard2 table2, and shard4 table2. Specifically, the data in shard1 table2 and shard4 table2 are written to the shared storage from DN0, and the data in shard2 table2 is written to the shared storage from DN1.

[0126] As shown in Figure 10, in this embodiment of the present application, DN0 can also write the metadata corresponding to the newly added data of shard1 table2 (referred to as shard1 table2 metadata) and the metadata corresponding to the data of shard4 table2 (referred to as shard4table2 metadata) to the shared storage; DN1 writes the metadata corresponding to the newly added data of shard2 table2 (referred to as shard2 table2 metadata). Taking the data of shard1table4 as an example, the data includes multiple data segments, so the metadata of shard1 table2 includes metadata of multiple data segments.

[0127] Step 940: Each DN records the current CSN number and the path information of the currently stored metadata in the shared storage in the shared storage.

[0128] In one possible implementation, since shard2 has no new data, there is no shard metadata in the table2_CSN1 path. Since complete data can be read with any consistency, this information and the metadata for CSN0 need to be supplemented in manifest2. The corresponding information in manifest2 is shown in Figure 11. Manifest2 not only records CSN1 but also the path information for the newly added metadata in shared storage and the path information for the previous CSN0 metadata in shared storage.

[0129] In some embodiments, if the current checkpoint has fewer data segments than the previous checkpoint, the path information of the metadata of the shared data to be deleted in the shared storage may also be recorded in manifest2. After the data consumption cluster reads the shared data to be deleted from the shared storage, the data production cluster may delete the shared data to be deleted and / or the metadata of the shared data to be deleted from the shared storage based on the path information of the metadata of the shared data to be deleted in the shared storage.

[0130] Step 950: Update CSN_Mainfest and record the current CSN number.

[0131] In the embodiment of the present application, CSN_Mainfest is updated, and the CSN numbers recorded in CSN_Mainfest are CSN0 and CSN1.

[0132] Step 960: Set permissions for the newly added metadata path and the newly added shared data.

[0133] In some embodiments, if the data producer wants to cancel the sharing process, it is only necessary to cancel the access rights to the metadata and shared data.

[0134] It should be noted that the timing for the above-mentioned data producer to trigger the incremental checkpoint can be periodic, or it can be when a query transaction actively triggers it. The embodiment of the present application does not make specific limitations on this.

[0135] In conjunction with Figure 12, taking the full checkpoint process as an example, a detailed description is given of an implementation method in which a data consumption cluster obtains metadata from shared storage and the shared data corresponding to the metadata. It should be understood that the example of Figure 12 is only to help those skilled in the art understand the embodiments of the present application, and is not intended to limit the application embodiments to the specific numerical values ​​or specific scenarios illustrated in Figure 12. Those skilled in the art can obviously make various equivalent modifications or changes based on the following examples given in Figure 12, and such modifications and changes also fall within the scope of the embodiments of the present application.

[0136] Figure 12 is a schematic flow chart of a method for a data consumption cluster to obtain metadata and shared data corresponding to the metadata from shared storage, as provided in an embodiment of the present application. As shown in Figure 12, the method may include steps 1210 to 1250, each of which is described in detail below.

[0137] Step 1210: The data consumption cluster selects CN1 from multiple CNs.

[0138] The method for the data consumption cluster to select a CN from multiple CNs is the same as that in step 410. For the specific selection method, please refer to the description of step 410 and will not be repeated here. For ease of description, the following description will take CN1 as an example.

[0139] Step 1220: CN1 triggers the shared capability module, loads CSN_mainfest, and obtains the CSN number CSN0.

[0140] As an example, CN1 triggers the shared capability module, and the shared capability module obtains CSN_mainfest from the shared storage and the CSN number recorded in CSN_mainfest. In this embodiment, the CSN number recorded in CSN_mainfest is CSN0.

[0141] Step 1230: CN1 builds a metadata loading plan and sends the plan to each DN.

[0142] Step 1240: Each DN executes the plan synchronously and loads the metadata of the checkpoint whose CSN is CSN0.

[0143] It should be understood that data consumption clusters follow the same data distribution as data generation clusters. For example, if shared data follows a hash distribution, then data consumption in the consumption cluster must also follow this hash distribution. Therefore, each DN synchronously retrieves the corresponding data from shared storage based on the corresponding relationship between DN and shard as shown in Figure 6.

[0144] For example, each DN obtains corresponding metadata according to the path information of each metadata recorded in mainfest1 shown in FIG8 , and stores the obtained metadata in its own memory.

[0145] Step 1250: Each DN synchronously obtains the corresponding shared data from the shared storage according to the obtained metadata, and performs data analysis based on the shared data.

[0146] In an embodiment of the present application, each DN synchronously obtains the corresponding shared data from the shared storage based on the obtained metadata. For example, DN0 obtains the data of shard1 table1, shard4 table1, and shard7 table1 from the shared storage based on the stored metadata. DN1 obtains the data of shard2 table1, shard5 table1, and shard8 table1 from the shared storage based on the stored metadata. DN2 obtains the data of shard3 table1, shard6 table1, and shard9 table1 from the shared storage based on the stored metadata. Data analysis is performed based on the obtained shared data to execute the data consumption process.

[0147] In conjunction with Figure 13, taking the incremental checkpoint process as an example, another implementation method for the data consumption cluster to obtain metadata from shared storage and the shared data corresponding to the metadata is described in detail. It should be understood that the example of Figure 13 is only to help those skilled in the art understand the embodiment of the present application, and is not intended to limit the application embodiment to the specific numerical values ​​or specific scenarios illustrated in Figure 13. Those skilled in the art can obviously make various equivalent modifications or changes based on the following example given in Figure 13, and such modifications and changes also fall within the scope of the embodiment of the present application.

[0148] Figure 13 is a schematic flow chart of another method for a data consumption cluster to obtain metadata and shared data corresponding to the metadata from shared storage, provided in an embodiment of the present application. As shown in Figure 13, the method may include steps 1310 to 1350, each of which is described in detail below.

[0149] Step 1310: The data consumption cluster selects CN1 from multiple CNs.

[0150] Similar to step 410, the specific selection method is described in step 410 and will not be repeated here. For ease of description, the following description is made by taking CN1 as an example for the data consumption cluster to select CN.

[0151] Step 1320: CN1 triggers the shared capability module, loads CSN_mainfest, and obtains the CSN number CSN1.

[0152] Step 1330: CN1 builds a metadata loading plan and sends the plan to each DN.

[0153] Step 1340: Each DN executes the plan synchronously and loads the metadata of the checkpoint whose CSN is CSN1.

[0154] For example, each DN reads mainfest2 shown in Figure 11. Mainfest2 records the path information of the newly added metadata in shared storage and the path information of the metadata of the previous CSN0 in shared storage. Since the metadata of the previous CSN0 is already stored in the memory of each DN, each DN only needs to obtain the newly added metadata based on the path information of the newly added metadata recorded in mainfest2 shown in Figure 11 and store the newly added metadata in its own memory.

[0155] For example, DN0 obtains metadata corresponding to the newly added shard1 table2 data (called shard1 table2 metadata) and metadata corresponding to the shard4 table2 data (called shard4table2 metadata) from the shared storage, and DN1 obtains metadata corresponding to the newly added shard2 table2 data (called shard2 table2 metadata) from the shared storage and stores them in their respective memories.

[0156] In some embodiments, if mainfest2 also records metadata of shared data that needs to be deleted compared to the last CSN0, the corresponding DN in the data consumption cluster can delete the corresponding metadata from the metadata stored in the memory last time. The data consumption cluster can also write CSN0 to the shared storage, so that the data production cluster can determine that the data consumption cluster has read the shared data corresponding to CSN0 from the shared storage based on the CSN0 written by the data consumption cluster. In this way, the data production cluster can delete the shared data and / or the metadata of the shared data that needs to be deleted from the shared storage, thereby implementing garbage collection and saving storage space of the shared storage.

[0157] Step 1350: Each DN synchronously obtains the newly added shared data from the shared storage according to the newly added metadata and performs data analysis.

[0158] In this embodiment of the present application, each DN synchronously obtains the newly added shared data from the shared storage based on the newly added metadata. For example, DN0 obtains the data of shard1 table2 and shard4 table2 from the shared storage based on the newly added metadata. DN1 obtains the data of shard2 table2 from the shared storage based on the newly added metadata.

[0159] It should be noted that the timing for the above-mentioned data producer to trigger the loading of incremental metadata can be periodic, or it can be when a query transaction actively triggers it. The embodiment of the present application does not make specific limitations on this.

[0160] The above describes in detail the method provided by the embodiment of the present application in conjunction with Figures 1 to 13. The following describes in detail the embodiment of the system of the present application in conjunction with Figures 14 to 17. It should be understood that the description of the method embodiment corresponds to the description of the system embodiment. Therefore, for parts not described in detail, reference can be made to the above method embodiment.

[0161] 14 is a schematic block diagram of a metadata processing system 1400 provided in an embodiment of the present application. The system 1400 can be implemented through software, hardware, or a combination of both. The metadata processing system 1400 provided in the embodiment of the present application can implement the method flow shown in the embodiment of the present application. The system 1400 includes: a data production cluster 1410, a shared storage 1420 and a data consumption cluster 1430, wherein the data production cluster 1410 is used to generate metadata for shared data, and the data production cluster includes multiple data nodes that provide the shared data; the data production cluster 1410 is also used to store the shared data and the metadata in the shared storage, and the metadata is used by the data consumption cluster to obtain the shared data from the shared storage, and the data consumption cluster includes multiple data nodes that use the shared data; the shared storage 1420 is used to generate path information for the metadata; the data consumption cluster 1430 is used to obtain path information for the metadata from the shared storage; the data consumption cluster 1430 is also used to obtain the metadata from the shared storage based on the path information of the metadata; the data consumption cluster 1430 is also used to obtain the shared data from the shared storage based on the metadata.

[0162] Optionally, the data production cluster 1410 is further configured to set access rights for the path information of the metadata; and the data consumption cluster 1430 is specifically configured to use the access rights to obtain the path information of the metadata.

[0163] Optionally, the data production cluster 1410 is specifically configured to: periodically obtain the path information of the metadata from the shared storage; or, when a target event occurs, obtain the path information of the metadata from the shared storage.

[0164] Optionally, the data production cluster 1410 is specifically configured to: enable multiple data nodes in the data production cluster 1410 to store metadata of their respective shared data in the shared storage.

[0165] Optionally, the shared storage 1420 is further configured to delete the shared data or metadata of the shared data after the data consumption cluster 1430 reads the shared data.

[0166] The system 1400 can be implemented in the form of functional modules, which can be implemented in software and / or hardware, without specific limitation.

[0167] For example, a "data node" can be a software program, hardware circuit, or a combination of the two that implements the aforementioned functions. For example, the implementation of data production cluster 1410 will be described below using the data nodes in data production cluster 1410 as an example. Similarly, the implementation of shared storage 1420 and data consumption cluster 1430 can refer to the implementation of data production cluster 1410.

[0168] A data node may include code running on a computing instance. The computing instance may include at least one of a physical host (computing device), a virtual machine, and a container. Furthermore, the computing instance may be one or more. For example, a data node may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed in the same region or in different regions. Furthermore, the multiple hosts / virtual machines / containers used to run the code may be distributed in the same availability zone (AZ) or in different AZs, each AZ including one data center or multiple geographically close data centers. Typically, a region may include multiple AZs.

[0169] Similarly, multiple hosts / virtual machines / containers used to run the code can be distributed within the same virtual private cloud (VPC) or across multiple VPCs. Typically, a VPC is set up within a region. Cross-region communication between two VPCs within the same region, or between VPCs in different regions, requires a communication gateway within each VPC to interconnect the VPCs.

[0170] As an example of a hardware functional unit, a data node may include at least one computing device, such as a server. Alternatively, a data node may be implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0171] The multiple computing devices included in a data node can be distributed in the same region or in different regions. They can also be distributed in the same AZ or in different AZs. Similarly, they can be distributed in the same VPC or across multiple VPCs. These multiple computing devices can be any combination of servers, ASICs, PLDs, CPLDs, FPGAs, GALs, and other computing devices.

[0172] Therefore, each example described in the embodiments of this application can be implemented using electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0173] In addition, the system and method embodiments provided in the above embodiments belong to the same concept, and their specific implementation processes are detailed in the method embodiments above, which will not be repeated here.

[0174] The method provided in the embodiment of the present application can be performed by a computing device, which can also be referred to as a computer system. It includes a hardware layer, an operating system layer running on the hardware layer, and an application layer running on the operating system layer. The hardware layer includes hardware such as a processing unit, a memory and a memory control unit, and the function and structure of the hardware are subsequently described in detail. The operating system is any one or more computer operating systems that implement business processing through a process, for example, a Linux operating system, a Unix operating system, an Android operating system, an iOS operating system or a Windows operating system. The application layer includes applications such as a browser, an address book, a word processing software, and an instant messaging software. Furthermore, optionally, the computer system is a handheld device such as a smart phone, or a terminal device such as a personal computer, and this application is not particularly limited, as long as the method provided in the embodiment of the present application can be used. The execution subject of the method provided in the embodiment of the present application can be a computing device, or a functional module in a computing device that can call a program and execute a program.

[0175] 15 , a computing device provided in an embodiment of the present application is described in detail below.

[0176] FIG15 is a schematic diagram of the architecture of a computing device 1500 provided in an embodiment of the present application. The computing device 1500 may be a server, a computer, or other device with computing capabilities. The computing device 1500 shown in FIG15 includes: at least one processor 1510 and a memory 1520.

[0177] It should be understood that this application does not limit the number of processors and memories in the computing device 1500.

[0178] The processor 1510 executes the instructions in the memory 1520 so that the computing device 1500 implements the method provided in this application. Alternatively, the processor 1510 executes the instructions in the memory 1520 so that the computing device 1500 implements the functional modules provided in this application, thereby implementing the method provided in this application.

[0179] Optionally, the computing device 1500 further includes a communication interface 1530. The communication interface 1530 uses a transceiver module such as, but not limited to, a network interface card or a transceiver to implement communication between the computing device 1500 and other devices or a communication network.

[0180] Optionally, the computing device 1500 further includes a system bus 1540, wherein the processor 1510, the memory 1520, and the communication interface 1530 are respectively connected to the system bus 1540. The processor 1510 can access the memory 1520 through the system bus 1540. For example, the processor 1510 can read and write data or execute code in the memory 1520 through the system bus 1540. The system bus 1540 is a peripheral component interconnect express (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The system bus 1540 is divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 15 is represented by only one thick line, but it does not mean that there is only one bus or one type of bus.

[0181] In one possible implementation, the processor 1510 is primarily responsible for interpreting computer program instructions (or codes) and processing data in the computer software. The computer program instructions and the data in the computer software may be stored in the memory 1520 or the cache 1516.

[0182] Optionally, the processor 1510 may be an integrated circuit chip having signal processing capabilities. By way of example and not limitation, the processor 1510 is a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component. The general-purpose processor is a microprocessor, etc. For example, the processor 1510 is a central processing unit (CPU).

[0183] Optionally, each processor 1510 includes at least one processing unit 1512 and a memory control unit 1514 .

[0184] Processing unit 1512, also known as a core, is the most important component of a processor. Processing unit 1512 is manufactured from single-crystal silicon using a specific production process. All processor calculations, command reception, command storage, and data processing are performed by the core. Each processing unit independently executes program instructions, leveraging parallel computing capabilities to accelerate program execution. Each processing unit has a fixed logical structure. For example, a processing unit includes logical units such as a level 1 cache, a level 2 cache, an execution unit, an instruction-level unit, and a bus interface.

[0185] In one implementation example, the memory control unit 1514 is configured to control data exchange between the memory 1520 and the processing unit 1512. Specifically, the memory control unit 1514 receives memory access requests from the processing unit 1512 and controls access to the memory based on the memory access requests. By way of example and not limitation, the memory control unit is a device such as a memory management unit (MMU).

[0186] In one implementation example, each memory control unit 1514 addresses the memory 1520 via the system bus. An arbiter (not shown in FIG. 15 ) is configured in the system bus to handle and coordinate competing accesses by multiple processing units 1512 .

[0187] In an implementation example, the processing unit 1512 and the memory control unit 1514 are connected to each other via connection lines inside the chip, such as address lines, so as to achieve communication between the processing unit 1512 and the memory control unit 1514.

[0188] Optionally, each processor 1510 also includes a cache 1516, which is a buffer for data exchange (called a cache). When a processing unit 1512 needs to read data, it first searches the cache for the required data. If the data is found, it executes the request directly; if not, it searches the memory. Because the cache runs much faster than the memory, the cache helps the processing unit 1512 run faster.

[0189] Memory 1520 can provide runtime space for processes in computing device 1500. For example, memory 1520 stores computer programs (specifically, program code) used to generate processes. After the computer program is executed by the processor to generate a process, the processor allocates corresponding storage space for the process in memory 1520. Furthermore, the aforementioned storage space further includes a text segment, an initialized data segment, a bit-initialized data segment, a stack segment, a heap segment, and the like. Memory 1520 stores data generated during the execution of the process, such as intermediate data or process data, in the storage space corresponding to the aforementioned process.

[0190] Optionally, the memory is also called the internal memory. Its function is to temporarily store the data processed by the processor 1510 and the data exchanged with external storage such as a hard disk. As long as the computer is running, the processor 1510 will load the data to be calculated into the internal memory for calculation. When the calculation is completed, the processing unit 1512 will transmit the result.

[0191] By way of example and not limitation, memory 1520 is a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Non-volatile memory is read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory is random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), and direct RAM bus RAM (DR RAM). It should be noted that the memory 1520 of the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0192] The structure of the computing device 1500 listed above is only for illustrative purposes, and the present application is not limited thereto. The computing device 1500 of the embodiment of the present application includes various hardware in the computer system in the prior art. For example, the computing device 1500 also includes other memories in addition to the memory 1520, such as disk storage, etc. Those skilled in the art should understand that the computing device 1500 may also include other devices necessary to achieve normal operation. At the same time, according to specific needs, those skilled in the art should understand that the above-mentioned computing device 1500 may also include hardware devices that implement other additional functions. In addition, those skilled in the art should understand that the above-mentioned computing device 1500 may also include only the devices necessary to implement the embodiment of the present application, and does not necessarily include all the devices shown in Figure 15.

[0193] The present application also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device may be a server. In some embodiments, the computing device may also be a terminal device such as a desktop computer, a laptop computer, or a smartphone.

[0194] As shown in Figure 16, the computing device cluster includes at least one computing device 1500. The memory 1520 in one or more computing devices 1500 in the computing device cluster may store the same instructions for executing the above method.

[0195] In some possible implementations, the memory 1520 of one or more computing devices 1500 in the computing device cluster may also store some instructions for executing the above method. In other words, the combination of one or more computing devices 1500 can jointly execute the instructions of the above method.

[0196] It should be noted that the memory 1520 in different computing devices 1500 in the computing device cluster can store different instructions, each for executing part of the functions of the above system. In other words, the instructions stored in the memory 1520 in different computing devices 1500 can implement one or more functions of the above system.

[0197] In some possible implementations, one or more computing devices in a computing device cluster can be connected via a network. The network can be a wide area network (WAN) or a local area network (LAN), among others. FIG. 17 illustrates one possible implementation. As shown in FIG. 17 , two computing devices 1500A and 1500B are connected via a network. Specifically, each computing device is connected to the network via a communication interface within the computing device.

[0198] It should be understood that the functionality of the computing device 1500A shown in FIG17 may also be implemented by multiple computing devices 1500. Similarly, the functionality of the computing device 1500B may also be implemented by multiple computing devices 1500.

[0199] This embodiment also provides a computer program product containing instructions. The computer program product may be software or a program product containing instructions that can be executed on a computing device cluster or stored on any available medium. When executed by a computing device cluster, the computer program product causes the computing device cluster to perform the method provided above, or causes the computing device cluster to implement the functions of the system provided above.

[0200] This embodiment also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium capable of storing data on a computing device, or a data storage device such as a data center that contains one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, or a magnetic tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive). The computer-readable storage medium includes instructions that, when executed by a computing device cluster, cause the computing device cluster to perform the method provided above.

[0201] It should be understood that in the various embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0202] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0203] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the system described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0204] In the several embodiments provided in this application, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative. For example, the division of the units described is merely a logical functional division. In actual implementation, other division methods may be used, such as combining or integrating multiple units or components into another system, or ignoring or not implementing certain features.

[0205] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0206] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0207] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0208] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. A method for processing metadata, characterized in that: The method comprises: A data production cluster generates metadata of shared data, wherein the data production cluster includes a plurality of data nodes that provide the shared data; The data production cluster stores the shared data and the metadata in a shared storage, the metadata is used by a data consumption cluster to obtain the shared data from the shared storage, and the data consumption cluster includes a plurality of data nodes that use the shared data; The shared storage generates path information of the metadata; The data consumption cluster obtains the path information of the metadata from the shared storage; The data consumption cluster obtains the metadata from the shared storage based on the path information of the metadata; The data consumption cluster obtains the shared data from the shared storage based on the metadata.

2. The method according to claim 1, characterized in that: The method further comprises: The data production cluster sets access rights to the path information of the metadata; The data consumption cluster uses the access permission to obtain the path information of the metadata.

3. The method according to claim 1 or 2, characterized in that: The data consumption cluster obtains the path information of the metadata from the shared storage, including: The data consumption cluster periodically obtains the path information of the metadata from the shared storage; or, When a target event occurs, the data consumption cluster obtains the path information of the metadata from the shared storage.

4. The method according to any one of claims 1 to 3, characterized in that The data production cluster stores the metadata in a shared storage, including: Multiple data nodes in the data production cluster store metadata of their respective shared data in the shared storage.

5. The method according to any one of claims 1 to 4, characterized in that The method further comprises: The shared storage deletes the shared data or metadata of the shared data after the data consumption cluster reads the shared data.

6. A metadata processing system, characterized in that: The system comprises: A data production cluster, used to generate metadata of shared data, wherein the data production cluster includes a plurality of data nodes that provide the shared data; The data production cluster is further used to store the shared data and the metadata in a shared storage, the metadata is used by a data consumption cluster to obtain the shared data from the shared storage, and the data consumption cluster includes a plurality of data nodes that use the shared data; Shared storage, used to generate path information of the metadata; The data consumption cluster is used to obtain the path information of the metadata from the shared storage; The data consumption cluster is further used to obtain the metadata from the shared storage based on the path information of the metadata; The data consumption cluster is further used to obtain the shared data from the shared storage based on the metadata.

7. The system according to claim 6, characterized in that The data production cluster is further used to set access rights to the path information of the metadata; The data consumption cluster is further used to obtain the path information of the metadata using the access permission.

8. The system according to claim 6 or 7, characterized in that: The data consumption cluster is specifically used for: periodically obtaining the path information of the metadata from the shared storage; or, When a target event occurs, path information of the metadata is obtained from the shared storage.

9. The system according to any one of claims 6 to 8, characterized in that The data production cluster is specifically used for: Multiple data nodes in the data production cluster store metadata of their respective shared data in the shared storage.

10. The system according to any one of claims 6 to 9, characterized in that The shared storage is further used to delete the shared data or metadata of the shared data after the data consumption cluster reads the shared data.

11. A computing device cluster, characterized in that: comprising at least one computing device, each computing device comprising a processor and a memory; The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster executes the method according to any one of claims 1 to 5.

12. A computer program product comprising instructions, characterized in that When the instructions are executed by a computing device cluster, the computing device cluster is caused to perform the method according to any one of claims 1 to 5.

13. A computer-readable storage medium, characterized in that: The method comprises computer program instructions, and when the computer program instructions are executed by a computing device cluster, the computing device cluster performs the method as claimed in any one of claims 1 to 5.

Images