Metadata processing method and system, and computing device

Abstract

A metadata processing method and system, and a computing device. The method comprises: a data production cluster generates metadata of shared data; the data production cluster stores the shared data into a shared memory; the data production cluster receives a metadata operation instruction and determines target metadata on the basis of the metadata operation instruction, the target metadata being metadata of target shared data; the data production cluster sends the target metadata to a data consumption cluster; and the data consumption cluster reads the target shared data from the shared memory on the basis of the target metadata. In the method, the data consumption cluster can acquire the metadata of the shared data in real time, so that the user experience is better, and the storage cost of the metadata can be reduced.

Description

Metadata processing method, system and computing device

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office on June 29, 2023, with application number 202310786272.8 and invention name “Metadata processing method, system and computing device”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of data storage, and more specifically, to a metadata processing method, system, and computing device. Background Art

[0003] Data warehouses serve as a platform for data storage and analysis. Major cloud vendors have launched data warehouse services. Data sharing has become a key feature of data warehouse services. Its purpose is to eliminate data silos, enable data transactions and sharing between users, and promote interoperability and sharing of data. Data sharing involves data producers providing shared data, and data consumers subscribing to or purchasing certain shared data produced by the data producers. After purchasing this shared data, the data consumer can access it and perform data analysis and computation based on it.

[0004] During the process of generating and storing shared data, data producers also generate and store metadata for this shared data. When data consumers retrieve shared data from shared storage, they must use the metadata to retrieve the shared data. In related technical solutions, the metadata for shared data is stored separately in a metadata cluster. Because the metadata cluster is a separate cluster dedicated to metadata storage, its storage costs are high, making large-scale, multi-cluster data sharing impossible. Furthermore, data consumption clusters cannot access shared data in real time, resulting in a poor user experience.

[0005] Therefore, how to save the storage cost of metadata of shared data while improving the user experience has become a technical problem that needs to be solved urgently.

[0006] Summary of the Invention

[0007] The present application provides a metadata processing method, system, and computing device, in which a data consumption cluster can obtain metadata of shared data in real time, providing a better user experience while also saving metadata storage costs.

[0008] In a first aspect, a metadata processing method is provided, which includes: a data production cluster generates metadata for shared data, the data production cluster includes multiple data nodes that provide the shared data; the data production cluster stores the shared data in shared storage, the shared storage is shared by the data production cluster and a data consumption cluster, the data consumption cluster includes multiple data nodes that use the shared data; the data production cluster receives a metadata operation instruction; the data production cluster determines target metadata according to the metadata operation instruction, the target metadata is metadata of target shared data, the shared data includes the target shared data; the data production cluster sends the target metadata to the data consumption cluster; the data consumption cluster reads the target shared data from the shared storage according to the target metadata.

[0009] In this technical solution, data consumption clusters can obtain metadata from data production clusters in real time. This not only reduces the storage overhead associated with storing metadata, but also ensures stable data analysis results from the data consumption clusters, providing a better user experience. Furthermore, the smaller metadata size reduces communication traffic between the data consumption cluster and the data production cluster, allowing for linear scaling of shared clusters, ensuring that storage costs do not increase with the number of shared clusters.

[0010] In combination with the first aspect, in some implementations of the first aspect, the method further includes: the data consumption cluster receiving a data processing instruction for the target shared data; and the data consumption cluster processing the target shared data according to the data processing instruction.

[0011] In combination with the first aspect, in certain implementations of the first aspect, the method also includes: the coordination node formulates a global execution plan for the data production cluster and the data consumption cluster based on the user's query request, and the global execution plan includes the metadata operation instruction and the data processing instruction.

[0012] In combination with the first aspect, in certain implementations of the first aspect, the data production cluster scans the index of the shared data to obtain the index of the target shared data according to the metadata operation instruction; the data production cluster determines the target metadata according to the index of the target shared data.

[0013] In the above technical solution, the data consumption cluster can obtain a corresponding data block based on metadata. The index is used to indicate a specific data in the data block. In this way, after obtaining the data block, the data consumption cluster can directly find the desired data in the data block based on the index instead of sequentially scanning.

[0014] In combination with the first aspect, in certain implementations of the first aspect, the method further includes: the data production cluster sends the index of the target shared data to the data consumption cluster; the data consumption cluster reads the target shared data from the shared storage based on the target metadata and the index of the target shared data.

[0015] In combination with the first aspect, in certain implementations of the first aspect, the data production cluster determines one or more destination data nodes in the data consumption cluster based on the correspondence between data shards and data nodes in the data consumption cluster, and the shared data belongs to the data shards corresponding to the one or more destination data nodes; the data production cluster sends the target metadata to the one or more destination data nodes.

[0016] In a second aspect, a metadata processing system is provided, which includes: a data production cluster and a data consumption cluster, wherein the data production cluster is used to generate metadata for shared data, and the data production cluster includes multiple data nodes that provide the shared data; the data production cluster is also used to store the shared data in a shared storage, which is shared by the data production cluster and the data consumption cluster, and the data consumption cluster includes multiple data nodes that use the shared data; the data production cluster is also used to receive metadata operation instructions; the data production cluster is also used to determine target metadata according to the metadata operation instructions, and the target metadata is metadata of target shared data, and the shared data includes the target shared data; the data production cluster is also used to send the target metadata to the data consumption cluster; the data consumption cluster is used to read the target shared data from the shared storage according to the target metadata.

[0017] In combination with the second aspect, in certain implementations of the second aspect, the data consumption cluster is further used to receive a data processing instruction for the target shared data; the data consumption cluster is further used to process the target shared data according to the data processing instruction.

[0018] In combination with the second aspect, in certain implementations of the second aspect, the system also includes a coordination node for formulating a global execution plan for the data production cluster and the data consumption cluster based on the user's query request, and the global execution plan includes the metadata operation instruction and the data processing instruction.

[0019] In combination with the second aspect, in certain implementations of the second aspect, the data production cluster is further used to: scan the index of the shared data to obtain the index of the target shared data according to the metadata operation instruction; and determine the target metadata according to the index of the target shared data.

[0020] In combination with the second aspect, in certain implementations of the second aspect, the data production cluster is also used to send the index of the target shared data to the data consumption cluster; the data consumption cluster is specifically used to read the target shared data from the shared storage based on the target metadata and the index of the target shared data.

[0021] In combination with the second aspect, in certain implementations of the second aspect, the data production cluster is specifically used to: determine one or more destination data nodes in the data consumption cluster based on the correspondence between data shards and data nodes in the data consumption cluster, and the shared data belongs to the data shards corresponding to the one or more destination data nodes; the data production cluster sends the target metadata to the one or more destination data nodes.

[0022] It should be noted that for the beneficial effects in the second aspect, please refer to the beneficial effects in the first aspect, and no further details will be given here.

[0023] In a third aspect, a computing device cluster is provided, comprising at least one computing device, each computing device comprising a processor and a memory; the processor of the at least one computing device is used to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster executes the method in the first aspect or any possible implementation of the first aspect.

[0024] Optionally, the processor may be a general-purpose processor, which may be implemented in hardware or software. When implemented in hardware, the processor may be a logic circuit, an integrated circuit, or the like; when implemented in software, the processor may be a general-purpose processor implemented by reading software code stored in a memory, which may be integrated into the processor or located independently of the processor.

[0025] In a fourth aspect, a computer program product comprising instructions is provided. When the instructions are executed by a computing device cluster, the computing device cluster executes the method in the first aspect and any one of the implementations of the first aspect.

[0026] In a fifth aspect, a computer-readable storage medium is provided, comprising computer program instructions. When the computer program instructions are executed by a computing device cluster, the computing device cluster executes the method according to the first aspect and any one of the implementations of the first aspect.

[0027] As an example, computer-readable storage media include, but are not limited to, one or more of the following: read-only memory (ROM), programmable ROM (PROM), erasable PROM (EPROM), Flash memory, electrically EPROM (EEPROM), and a hard drive.

[0028] Optionally, as an implementation manner, the above-mentioned storage medium may specifically be a non-volatile storage medium. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] FIG1 is a schematic block diagram of a data sharing service provided by a data warehouse.

[0030] FIG2 is a schematic block diagram of a cloud scenario applicable to an embodiment of the present application.

[0031] FIG3 is a schematic flowchart of a metadata processing method provided in an embodiment of the present application.

[0032] FIG4 is a schematic block diagram of a metadata processing system provided in an embodiment of the present application.

[0033] FIG5 is a schematic block diagram of another metadata processing system 500 provided in an embodiment of the present application.

[0034] FIG6 is a schematic diagram of the architecture of a computing device 1500 provided in an embodiment of the present application.

[0035] FIG7 is a schematic diagram of the architecture of a computing device cluster provided in an embodiment of the present application.

[0036] FIG8 is a schematic diagram of a network connection between computing devices 1500A and 1500B provided in an embodiment of the present application. DETAILED DESCRIPTION

[0037] The technical solution in this application will be described below with reference to the accompanying drawings.

[0038] This application will present various aspects, embodiments, or features around systems including multiple devices, components, modules, etc. It should be understood and appreciated that each system may include additional devices, components, modules, etc., and / or may not include all of the devices, components, modules, etc. discussed in conjunction with the figures. Furthermore, combinations of these aspects may also be used.

[0039] Additionally, in the embodiments of this application, words such as "exemplary" and "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described in this application as "exemplary" should not be construed as being preferred or advantageous over other embodiments or designs. Rather, the use of the word "exemplary" is intended to present concepts in a concrete manner.

[0040] In the embodiments of the present application, “corresponding” and “relevant” may sometimes be used interchangeably. It should be noted that when the distinction between them is not emphasized, the meanings they intend to express are consistent.

[0041] The business scenarios described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. A person skilled in the art will appreciate that, with the evolution of network architecture and the emergence of new business scenarios, the technical solutions provided in the embodiments of the present application are equally applicable to similar technical problems.

[0042] References to "one embodiment" or "some embodiments" in this specification mean that a particular feature, structure, or characteristic described in conjunction with that embodiment is included in one or more embodiments of the present application. Thus, phrases such as "in one embodiment," "in some embodiments," "in other embodiments," and "in yet other embodiments" appearing in various places in this specification do not necessarily refer to the same embodiment, but rather mean "one or more but not all embodiments," unless otherwise specifically emphasized. The terms "including," "comprising," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0043] In this application, "at least one" means one or more, and "more" means two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: including the existence of A alone, the existence of A and B at the same time, and the existence of B alone, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can mean: a, b, c, ab, ac, bc, or abc, where a, b, c can be single or multiple.

[0044] Data warehouses serve as a platform for data storage and analysis. Currently, major cloud vendors have launched cloud data warehouse services. In the context of cloud computing, the storage-computing separation architecture of data warehouses (cloud data warehouses) decouples computing and storage, allowing large amounts of user data to be stored in shared storage (for example, the cloud-based object storage service, OBS) for data sharing. Therefore, data sharing has become a key attribute of cloud data warehouse services. Its purpose is to eliminate data silos, enable inter-user data transactions and sharing, and achieve data interoperability and sharing. Multiple clusters can share a single piece of data in real time, enabling simple and effective concurrency improvements, isolating workloads by business, and rapidly expanding the concurrent carrying capacity of multiple clusters.

[0045] It should be understood that the data sharing service provided by the data warehouse mentioned above refers to data producers providing shared data, and data consumers subscribing to or purchasing certain shared data from the data producers. After purchasing this shared data, the data consumer can access the shared data and perform data analysis and calculations based on it.

[0046] As an example, the shared data may include but is not limited to: a table of a data producer, or a materialized view, or certain column data in a table, or certain row data in a table, etc.

[0047] For ease of understanding, the basic principles of the data sharing service provided by the data warehouse are described in detail below with reference to FIG1 .

[0048] As shown in Figure 1, data producers continuously generate data in the cloud data warehouse and write the generated data to shared storage. Data producers can also authorize data consumers to read data stored in shared storage based on their subscriptions or purchases. Data consumers can read the purchased or subscribed data from shared storage and perform data analysis and computation based on that data, completing the data consumption process.

[0049] It should be understood that data stored in the shared storage may be referred to as shared data.

[0050] Specifically, as data producers generate and store shared data, they also generate and store metadata about that shared data. When one computing cluster needs to access data produced by another cluster, it needs to consider how to obtain this metadata so that it can retrieve the required user data from shared storage based on the metadata.

[0051] It should be understood that the metadata of shared data is used to describe the shared data and is information that describes the attributes of the shared data. For example, the metadata of shared data may include but is not limited to information such as the location and address of the shared data stored in the shared storage.

[0052] In related technical solutions, metadata for shared data is stored separately in a metadata cluster. Since the metadata cluster is a separate cluster used to store metadata, its storage costs are high, making large-scale, multi-cluster data sharing impossible. Furthermore, data consumption clusters cannot access shared data in real time, resulting in a poor user experience.

[0053] In view of this, an embodiment of the present application provides a metadata processing method. On the one hand, the data consumption cluster can obtain the metadata of shared data in real time, so that the data consumption cluster can stabilize the results of data analysis and have a better user experience. On the other hand, it can also save storage costs and carry out large-scale, multi-cluster data sharing.

[0054] In one possible implementation, the method provided in the embodiment of the present application can be applied to a cloud service scenario, and the method is executed by a cloud management platform in the cloud service scenario. For ease of description, the cloud service scenario is described in detail below with reference to FIG2 .

[0055] FIG2 is a schematic block diagram of a cloud scenario applicable to an embodiment of the present application. As shown in FIG2 , the cloud scenario may include: a cloud management platform 110 , the Internet 120 , and a client 130 .

[0056] As shown in Figure 2, the cloud management platform 110 is used to manage the infrastructure that provides multiple cloud services. The infrastructure includes multiple cloud data centers, each of which includes multiple servers, each of which includes cloud service resources to provide corresponding cloud services to tenants.

[0057] The cloud management platform 110 can be located in a cloud data center, which can provide an access interface (such as an interface or an application program interface (API)). The tenant can operate the client 130 to remotely access the access interface to register a cloud account and password on the cloud management platform 110, and log in to the cloud management platform 110. After the cloud management platform 110 successfully authenticates the cloud account and password, the tenant can further pay to select and purchase a virtual machine with specific specifications (processor, memory, disk) on the cloud management platform 110. After the payment is successful, the cloud management platform 110 provides the remote login account and password of the purchased virtual machine, and the client 130 can remotely log in to the virtual machine, install and run the tenant's application in the virtual machine. Therefore, the tenant can create, manage, log in and operate virtual machines in the cloud data center through the cloud management platform 110. Among them, the virtual machine can also be called a cloud server (elastic compute service, ECS) or an elastic instance (different cloud service providers have different names).

[0058] It should be understood that tenants of cloud services can be individuals, enterprises, schools, hospitals, administrative agencies, etc.

[0059] The functions of the cloud management platform 110 include, but are not limited to, a user console, computing management services, network management services, storage management services, authentication services, and image management services. The user console provides an interface or API for interacting with tenants. The computing management service is used to manage servers running virtual machines and containers, as well as bare metal servers. The network management service is used to manage network services (such as gateways and firewalls). The storage management service is used to manage storage services (such as data bucket services). The authentication service is used to manage tenant accounts and passwords. The image management service is used to manage virtual machine images. Tenants can use the client 130 to log in to the cloud management platform 110 via the Internet 120 to manage the rented cloud services.

[0060] FIG3 is a schematic flow chart of a metadata processing method provided by an embodiment of the present application. As shown in FIG3 , the method may include steps 310 to 360 , which are described in detail below.

[0061] Step 310: The data production cluster generates metadata for the shared data.

[0062] In the embodiment of the present application, after the data production cluster produces the shared data, it can also generate metadata for the shared data. For details about the metadata of the shared data, please refer to the description above and will not be repeated here.

[0063] As an example, the data production cluster may include multiple data nodes (DN), which are used to produce or provide the shared data.

[0064] As an example, the above-mentioned shared storage can be cloud storage or other storage devices, as long as the shared storage function can be realized. The embodiments of the present application do not make specific limitations on this.

[0065] Step 320: The data production cluster stores the shared data in the shared storage.

[0066] In an embodiment of the present application, the data production cluster may store shared data in a shared storage, wherein the shared storage is shared by the data production cluster and the data consumption cluster.

[0067] As an example, the data consumption cluster may include multiple data nodes (DN) that use shared data stored in a shared storage.

[0068] Step 330: The data production cluster receives the metadata operation instruction and determines target metadata according to the metadata operation instruction.

[0069] As an example, in a specific implementation, as shown in FIG4 , the data sharing system includes: a data production cluster, a data consumption cluster, and a coordinator node (CN). CN, as a global coordinator node, can formulate a global execution plan for the data production cluster and the data consumption cluster based on the user's query request. The global execution plan may include metadata operation instructions and data processing instructions. The metadata operation instruction is used by the data production cluster to determine the target metadata according to the metadata operation instruction. The target metadata is the metadata of the target shared data, and the shared data includes the target shared data. The data processing instruction is used by the data consumption cluster to process the target shared data according to the data processing instruction.

[0070] Taking the data production cluster as an example, after receiving the metadata operation instruction included in the global execution plan issued by the CN, it can determine the target metadata according to the metadata operation instruction. Specifically, in one possible implementation method, the above-mentioned query request sent by the user may include the target shared data required for the data consumption cluster to perform data analysis. The CN can formulate a metadata operation instruction for the data production cluster based on the user's query request. The metadata operation instruction is used to instruct the data production cluster to determine the metadata of the target shared data required for the data consumption cluster to perform data analysis. After receiving the metadata operation instruction, the data production cluster scans the metadata of the locally stored shared data according to the metadata operation instruction to obtain the metadata of the target shared data (target metadata).

[0071] In some embodiments, the query request sent by the user may also include location information of the target shared data in the shared data. The data production cluster scans the index of the shared data according to the metadata operation instruction to obtain the index of the target shared data, and determines the target metadata based on the index of the target shared data.

[0072] Step 340: The data production cluster sends the target metadata to the data consumption cluster.

[0073] In the embodiment of the present application, after determining the target metadata, the data production cluster may send the target metadata to the data consumption cluster.

[0074] In some embodiments, before the data production cluster sends the target metadata to the data consumption cluster, it can also process the target metadata, for example, by performing coarse filtering, serialization, and packaging. In this implementation, the data production cluster sends serialized target metadata to the data consumption cluster. Specifically, a new operator responsible for scanning and processing metadata can be added. This operator executes on the DN of the data production cluster and is primarily responsible for reading metadata, performing coarse filtering, and serialization and packaging.

[0075] As an example, in a specific implementation method, when sending target metadata, the data production cluster can use the remote channel between the data production cluster and the data consumption cluster, for example, compression unit description (CUDesc) streaming, to transmit the target metadata from the data production cluster to the data consumption cluster.

[0076] In an embodiment of the present application, taking a data consumption cluster comprising multiple data nodes as an example, before the data production cluster sends target metadata to the data consumption cluster, it needs to determine one or more destination data nodes in the data consumption cluster and send the target metadata to the one or more destination data nodes. In one example, the data production cluster can determine the one or more destination data nodes in the data consumption cluster based on the correspondence between data shards and data nodes in the data consumption cluster.

[0077] It should be understood that the above-mentioned shared data belongs to the data shards corresponding to the one or more destination data nodes.

[0078] In the embodiment of the present application, the CN can also manage and record the correspondence between the above DN and shard. If a DN node is scaled up or down, only the correspondence between the shard and the DN node needs to be changed. The purpose of this is to make the data distribution independent of the number of nodes.

[0079] In some embodiments, if the data production cluster determines the target metadata based on the index of the target shared data, the data production cluster may also send the index of the target shared data to the data consumption cluster.

[0080] It should be noted that the method by which the data production cluster sends the index of the target shared data is similar to the method by which it sends the target metadata. For details, please refer to the above-mentioned process of the data production cluster sending the target metadata, which will not be repeated here.

[0081] Step 350: The data consumption cluster receives target metadata from the data production cluster.

[0082] In an embodiment of the present application, the data consumption cluster may receive target metadata from the data production cluster. Optionally, if the data production cluster also sends an index of target shared data to the data consumption cluster, the data consumption cluster may also receive the index of target shared data from the data production cluster.

[0083] For example, a data consumption cluster includes multiple data nodes. Each of the multiple data nodes in the data consumption cluster receives target metadata from the data production cluster. Optionally, if the data production cluster also sends an index of target shared data to one or more destination data nodes in the data consumption cluster, each of the multiple data nodes in the data consumption cluster receives an index of the target shared data from the data production cluster.

[0084] In some embodiments, if the data consumption cluster receives serialized and packaged target metadata from the data production cluster, the data consumption cluster may further deserialize and parse the serialized and packaged target metadata to obtain the target metadata.

[0085] Step 360: The data consumption cluster reads the corresponding target shared data from the shared storage according to the obtained target metadata.

[0086] In an embodiment of the present application, after receiving the target metadata sent by the data production cluster, the data consumption cluster can perform a sequential scan based on the target metadata and read the corresponding target shared data from the shared storage.

[0087] In some embodiments, if the data consumption cluster also receives the index of the target shared data sent by the data production cluster, the data consumption cluster can directly read the target shared data indicated by the index from the shared storage based on the target metadata and the index of the target shared data.

[0088] In some embodiments, the data consumption cluster may also receive a data processing instruction for target shared data issued by the CN, and process the acquired target shared data according to the data processing instruction, thereby executing the data consumption process.

[0089] In this technical solution, data consumption clusters can obtain metadata from data production clusters in real time. This not only reduces the storage overhead associated with storing metadata, but also ensures stable data analysis results from the data consumption clusters, providing a better user experience. Furthermore, the smaller metadata size reduces communication traffic between the data consumption cluster and the data production cluster, allowing for linear scaling of shared clusters, ensuring that storage costs do not increase with the number of shared clusters.

[0090] The above describes in detail the method provided by the embodiment of the present application in conjunction with Figures 1 to 4 . The following describes in detail the embodiment of the system of the present application in conjunction with Figures 5 to 8 . It should be understood that the description of the method embodiment corresponds to the description of the system embodiment. Therefore, for portions not described in detail, reference can be made to the above method embodiment.

[0091] Figure 5 is a schematic block diagram of a metadata processing system 500 provided in an embodiment of the present application. The metadata processing system 500 can be implemented by software, hardware, or a combination of both. The metadata processing system 500 provided in an embodiment of the present application can implement the method flow shown in the embodiment of the present application. The system 500 includes: a data production cluster 510 and a data consumption cluster 520, wherein the data production cluster 510 is used to generate metadata for shared data, and the data production cluster includes multiple data nodes that provide the shared data; the data production cluster 510 is also used to store the shared data in a shared storage, which is shared by the data production cluster and the data consumption cluster, and the data consumption cluster includes multiple data nodes that use the shared data; the data production cluster 510 is also used to receive metadata operation instructions; the data production cluster 510 is also used to determine target metadata according to the metadata operation instruction, and the target metadata is the metadata of the target shared data, and the shared data includes the target shared data; the data production cluster 510 is also used to send the target metadata to the data consumption cluster; the data consumption cluster 520 is used to read the target shared data from the shared storage according to the target metadata.

[0092] Optionally, the data consumption cluster 510 is further configured to receive a data processing instruction for the target shared data; the data consumption cluster 510 is further configured to process the target shared data according to the data processing instruction.

[0093] Optionally, the system 500 further includes a coordination node 530 for formulating a global execution plan for the data production cluster and the data consumption cluster according to the user's query request, and the global execution plan includes the metadata operation instruction and the data processing instruction.

[0094] Optionally, the data production cluster 510 is specifically configured to: scan the index of the shared data to obtain the index of the target shared data according to the metadata operation instruction; and determine the target metadata according to the index of the target shared data.

[0095] Optionally, the data production cluster 510 is further configured to send an index of the target shared data to the data consumption cluster; the data consumption cluster 520 is specifically configured to read the target shared data from the shared storage according to the target metadata and the index of the target shared data.

[0096] Optionally, the data production cluster 510 is specifically used to: determine one or more destination data nodes in the data consumption cluster based on the correspondence between data shards and data nodes in the data consumption cluster, and the shared data belongs to the data shards corresponding to the one or more destination data nodes; and send the target metadata to the one or more destination data nodes.

[0097] The system 500 herein may be embodied in the form of functional modules, which may be implemented in the form of software and / or hardware, without specific limitation.

[0098] For example, a "data node" can be a software program, hardware circuit, or a combination of the two that implements the aforementioned functions. For example, the implementation of data production cluster 510 will be described below using the data nodes in data production cluster 510 as an example. Similarly, the implementation of coordination node 530 in data consumption cluster 520 can refer to the implementation of data production cluster 510.

[0099] As an example of a software functional unit, a data node may include code running on a computing instance. The computing instance may include at least one of a physical host (computing device), a virtual machine, and a container. Furthermore, the computing instance may be one or more. For example, a data node may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed in the same region or in different regions. Furthermore, the multiple hosts / virtual machines / containers used to run the code may be distributed in the same availability zone (AZ) or in different AZs, each AZ including one data center or multiple geographically close data centers. Typically, a region may include multiple AZs.

[0100] Similarly, multiple hosts / virtual machines / containers used to run the code can be distributed within the same virtual private cloud (VPC) or across multiple VPCs. Typically, a VPC is set up within a region. Cross-region communication between two VPCs within the same region, or between VPCs in different regions, requires a communication gateway within each VPC to interconnect the VPCs.

[0101] As an example of a hardware functional unit, a data node may include at least one computing device, such as a server. Alternatively, a data node may be implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0102] The multiple computing devices included in a data node can be distributed in the same region or in different regions. They can also be distributed in the same AZ or in different AZs. Similarly, they can be distributed in the same VPC or across multiple VPCs. These multiple computing devices can be any combination of servers, ASICs, PLDs, CPLDs, FPGAs, GALs, and other computing devices.

[0103] Therefore, the modules of each example described in the embodiments of this application can be implemented with electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0104] In addition, the system and method embodiments provided in the above embodiments belong to the same concept, and their specific implementation processes are detailed in the method embodiments above and will not be repeated here.

[0105] The method provided in the embodiment of the present application can be performed by a computing device, which can also be referred to as a computer system. It includes a hardware layer, an operating system layer running on the hardware layer, and an application layer running on the operating system layer. The hardware layer includes hardware such as a processing unit, a memory and a memory control unit, and the function and structure of the hardware are subsequently described in detail. The operating system is any one or more computer operating systems that implement business processing through a process, for example, a Linux operating system, a Unix operating system, an Android operating system, an iOS operating system or a Windows operating system. The application layer includes applications such as a browser, an address book, a word processing software, and an instant messaging software. Furthermore, optionally, the computer system is a handheld device such as a smart phone, or a terminal device such as a personal computer, and this application is not particularly limited, as long as the method provided in the embodiment of the present application can be used. The execution subject of the method provided in the embodiment of the present application can be a computing device, or a functional module in a computing device that can call a program and execute a program.

[0106] 6 , a computing device provided in an embodiment of the present application is described in detail.

[0107] FIG6 is a schematic diagram of the architecture of a computing device 1500 provided in an embodiment of the present application. The computing device 1500 may be a server, a computer, or other device with computing capabilities. The computing device 1500 shown in FIG6 includes: at least one processor 1510 and a memory 1520.

[0108] It should be understood that this application does not limit the number of processors and memories in the computing device 1500.

[0109] The processor 1510 executes the instructions in the memory 1520 so that the computing device 1500 implements the method provided in this application. Alternatively, the processor 1510 executes the instructions in the memory 1520 so that the computing device 1500 implements the functional modules provided in this application, thereby implementing the method provided in this application.

[0110] Optionally, the computing device 1500 further includes a communication interface 1530. The communication interface 1530 uses a transceiver module such as, but not limited to, a network interface card or a transceiver to implement communication between the computing device 1500 and other devices or a communication network.

[0111] Optionally, the computing device 1500 further includes a system bus 1540, wherein the processor 1510, the memory 1520, and the communication interface 1530 are respectively connected to the system bus 1540. The processor 1510 can access the memory 1520 through the system bus 1540. For example, the processor 1510 can read and write data or execute code in the memory 1520 through the system bus 1540. The system bus 1540 is a peripheral component interconnect express (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The system bus 1540 is divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in Figure 6, but this does not mean that there is only one bus or one type of bus.

[0112] In one possible implementation, the processor 1510 is primarily responsible for interpreting computer program instructions (or codes) and processing data in the computer software. The computer program instructions and the data in the computer software may be stored in the memory 1520 or the cache 1516.

[0113] Optionally, the processor 1510 may be an integrated circuit chip having signal processing capabilities. By way of example and not limitation, the processor 1510 is a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component. The general-purpose processor is a microprocessor, etc. For example, the processor 1510 is a central processing unit (CPU).

[0114] Optionally, each processor 1510 includes at least one processing unit 1512 and a memory control unit 1514 .

[0115] Processing unit 1512, also known as a core, is the most important component of a processor. Processing unit 1512 is manufactured from single-crystal silicon using a specific production process. All processor calculations, command reception, command storage, and data processing are performed by the core. Each processing unit independently executes program instructions, leveraging parallel computing capabilities to accelerate program execution. Each processing unit has a fixed logical structure. For example, a processing unit includes logical units such as a level 1 cache, a level 2 cache, an execution unit, an instruction-level unit, and a bus interface.

[0116] In one implementation example, the memory control unit 1514 is configured to control data exchange between the memory 1520 and the processing unit 1512. Specifically, the memory control unit 1514 receives memory access requests from the processing unit 1512 and controls access to the memory based on the memory access requests. By way of example and not limitation, the memory control unit is a device such as a memory management unit (MMU).

[0117] In one implementation example, each memory control unit 1514 addresses the memory 1520 via a system bus. An arbiter (not shown in FIG. 6 ) is configured in the system bus to handle and coordinate competing accesses by multiple processing units 1512 .

[0118] In an implementation example, the processing unit 1512 and the memory control unit 1514 are connected to each other via connection lines inside the chip, such as address lines, so as to achieve communication between the processing unit 1512 and the memory control unit 1514.

[0119] Optionally, each processor 1510 also includes a cache 1516, which is a buffer for data exchange (called a cache). When a processing unit 1512 needs to read data, it first searches the cache for the required data. If the data is found, it executes the request directly; if not, it searches the memory. Because the cache runs much faster than the memory, the cache helps the processing unit 1512 run faster.

[0120] Memory 1520 can provide runtime space for processes in computing device 1500. For example, memory 1520 stores computer programs (specifically, program code) used to generate processes. After the computer program is executed by the processor to generate a process, the processor allocates corresponding storage space for the process in memory 1520. Furthermore, the aforementioned storage space further includes a text segment, an initialized data segment, a bit-initialized data segment, a stack segment, a heap segment, and the like. Memory 1520 stores data generated during the execution of the process, such as intermediate data or process data, in the storage space corresponding to the aforementioned process.

[0121] Optionally, the memory is also called the internal memory. Its function is to temporarily store the data processed by the processor 1510 and the data exchanged with external storage such as a hard disk. As long as the computer is running, the processor 1510 will load the data to be calculated into the internal memory for calculation. When the calculation is completed, the processing unit 1512 will transmit the result.

[0122] By way of example and not limitation, memory 1520 is a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Non-volatile memory is read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory is random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), and direct RAM bus RAM (DR RAM). It should be noted that the memory 1520 of the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0123] The structure of the computing device 1500 listed above is only an example, and the present application is not limited thereto. The computing device 1500 of the embodiment of the present application includes various hardware in the computer system in the prior art. For example, the computing device 1500 also includes other memories in addition to the memory 1520, such as disk storage, etc. Those skilled in the art should understand that the computing device 1500 may also include other devices necessary to achieve normal operation. At the same time, according to specific needs, those skilled in the art should understand that the above-mentioned computing device 1500 may also include hardware devices that implement other additional functions. In addition, those skilled in the art should understand that the above-mentioned computing device 1500 may also include only the devices necessary to implement the embodiment of the present application, and does not necessarily include all the devices shown in Figure 6.

[0124] The present application also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device may be a server. In some embodiments, the computing device may also be a terminal device such as a desktop computer, a laptop computer, or a smartphone.

[0125] As shown in Figure 7, the computing device cluster includes at least one computing device 1500. The memory 1520 in one or more computing devices 1500 in the computing device cluster may store the same instructions for executing the above method.

[0126] In some possible implementations, the memory 1520 of one or more computing devices 1500 in the computing device cluster may also store some instructions for executing the above method. In other words, the combination of one or more computing devices 1500 can jointly execute the instructions of the above method.

[0127] It should be noted that the memory 1520 in different computing devices 1500 in the computing device cluster can store different instructions, each for executing part of the functions of the above system. In other words, the instructions stored in the memory 1520 in different computing devices 1500 can implement one or more functions of the above system.

[0128] In some possible implementations, one or more computing devices in a computing device cluster can be connected via a network. The network can be a wide area network (WAN), a local area network (LAN), or the like. FIG8 illustrates one possible implementation. As shown in FIG8 , two computing devices 1500A and 1500B are connected via a network. Specifically, each computing device is connected to the network via a communication interface within the computing device.

[0129] It should be understood that the functionality of the computing device 1500A shown in FIG8 may also be implemented by multiple computing devices 1500. Similarly, the functionality of the computing device 1500B may also be implemented by multiple computing devices 1500.

[0130] This embodiment also provides a computer program product including instructions. The computer program product may be software or a program product including instructions that can be run on a computing device or stored in any available medium. When the computer program product is run on a computing device, it causes the computing device to perform the method provided above, or causes the computing device to implement the functions of the system provided above.

[0131] This embodiment also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium capable of storing data on a computing device, or a data storage device such as a data center that contains one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, or a magnetic tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive). The computer-readable storage medium includes instructions that, when executed by a computing device cluster, cause the computing device cluster to perform the method provided above.

[0132] It should be understood that in the various embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0133] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0134] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the system described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0135] In the several embodiments provided in this application, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative. For example, the division of the units described is merely a logical functional division. In actual implementation, other division methods may be used, such as combining or integrating multiple units or components into another system, or ignoring or not implementing certain features.

[0136] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0137] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0138] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0139] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. A method for processing metadata, characterized in that: The method comprises: A data production cluster generates metadata of shared data, wherein the data production cluster includes a plurality of data nodes that provide the shared data; The data production cluster stores the shared data in a shared storage, the shared storage is shared by the data production cluster and a data consumption cluster, and the data consumption cluster includes a plurality of data nodes that use the shared data; The data production cluster receives a metadata operation instruction; The data production cluster determines target metadata according to the metadata operation instruction, where the target metadata is metadata of target shared data, and the shared data includes the target shared data; The data production cluster sends the target metadata to the data consumption cluster; The data consumption cluster reads the target shared data from the shared storage according to the target metadata.

2. The method according to claim 1, characterized in that The method further comprises: The data consumption cluster receives a data processing instruction for the target shared data; The data consumption cluster processes the target shared data according to the data processing instruction.

3. The method according to claim 2, characterized in that The method further comprises: The coordination node formulates a global execution plan for the data production cluster and the data consumption cluster according to the user's query request, and the global execution plan includes the metadata operation instruction and the data processing instruction.

4. The method according to any one of claims 1 to 3, characterized in that The data production cluster determines the target metadata according to the metadata operation instruction, including: The data production cluster scans the index of the shared data to obtain the index of the target shared data according to the metadata operation instruction; The data production cluster determines the target metadata according to the index of the target shared data.

5. The method according to claim 4, characterized in that The method further includes: the data production cluster sending the index of the target shared data to the data consumption cluster; The data consumption cluster reads the target shared data from the shared storage according to the target metadata, including: The data consumption cluster reads the target shared data from the shared storage according to the target metadata and the index of the target shared data.

6. The method according to any one of claims 1 to 5, characterized in that The data production cluster sends the target metadata to the data consumption cluster, including: The data production cluster determines one or more destination data nodes in the data consumption cluster according to the correspondence between the data shards and the data nodes in the data consumption cluster, and the shared data belongs to the data shards corresponding to the one or more destination data nodes; The data production cluster sends the target metadata to the one or more destination data nodes.

7. A metadata processing system, characterized in that: The system comprises: A data production cluster, used to generate metadata of shared data, wherein the data production cluster includes a plurality of data nodes that provide the shared data; The data production cluster is further used to store the shared data in a shared storage, where the shared storage is shared by the data production cluster and a data consumption cluster, and the data consumption cluster includes a plurality of data nodes that use the shared data; The data production cluster is also used to receive metadata operation instructions; The data production cluster is further used to determine target metadata according to the metadata operation instruction, where the target metadata is metadata of target shared data, and the shared data includes the target shared data; The data production cluster is further used to send the target metadata to the data consumption cluster; The data consumption cluster is used to read the target shared data from the shared storage according to the target metadata.

8. The system according to claim 7, characterized in that The data consumption cluster is further used to receive a data processing instruction for the target shared data; The data consumption cluster is further used to process the target shared data according to the data processing instruction.

9. The system according to claim 8, characterized in that The system further comprises: The coordination node is used to formulate a global execution plan for the data production cluster and the data consumption cluster according to the user's query request. The global execution plan includes the metadata operation instructions and the data processing instructions.

10. The system according to any one of claims 7 to 9, characterized in that The data consumption cluster is further used to scan the index of the shared data to obtain the index of the target shared data according to the metadata operation instruction; The data production cluster is further used to determine the target metadata according to the index of the target shared data.

11. The system according to claim 10, characterized in that The data production cluster is further used to send the index of the target shared data to the data consumption cluster; The data consumption cluster is specifically configured to read the target shared data from the shared storage according to the target metadata and the index of the target shared data.

12. The system according to any one of claims 7 to 11, characterized in that The data production cluster is specifically used for: Determine one or more destination data nodes in the data consumption cluster according to the correspondence between data shards and data nodes in the data consumption cluster, wherein the shared data belongs to the data shards corresponding to the one or more destination data nodes; The data production cluster sends the target metadata to the one or more destination data nodes.

13. A computing device cluster, characterized in that: comprising at least one computing device, each computing device comprising a processor and a memory; The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster executes the method according to any one of claims 1 to 6.

14. A computer program product comprising instructions, characterized in that When the instructions are executed by a computing device cluster, the computing device cluster is caused to perform the method according to any one of claims 1 to 6.

15. A computer-readable storage medium, characterized in that: The method comprises computer program instructions, and when the computer program instructions are executed by a computing device cluster, the computing device cluster performs the method according to any one of claims 1 to 6.

Images