Data processing method and system and computing device cluster

Abstract

The invention provides a data processing method and system, and belongs to the technical field of cloud computing. The method comprises the steps that a data processing system receives a first write request, the data processing system comprises a first computing cluster, a second computing cluster and a shared storage, the shared storage is used for storing data of a first data table, the first write request is used for writing first data into the first data table, and the first data table belongs to the first computing cluster; the second computing cluster writes the first data belonging to the first data table into the shared storage; the second computing cluster sends first metadata corresponding to the first data to the first computing cluster; the first computing cluster stores first metadata corresponding to the first data. According to the method, the requirement that the extension computing cluster supports data writing can be met, meanwhile, the computing cluster to which the table belongs is still responsible for writing of the metadata corresponding to the table data, an independent metadata management cluster does not need to be deployed, and the performance bottleneck problem of the metadata management cluster does not exist while the system cost is reduced.

Description

Technical Field

[0001] The present application relates to the field of cloud computing technology, and in particular to a data processing method, system, computing device cluster, computer-readable storage medium, and computer program product. Background Art

[0002] With the development of cloud computing technology, more and more users choose to store large amounts of data in the cloud and use the data processing systems provided by cloud vendors to operate the data in the cloud storage. For users with greater business pressure, in order to improve the overall read and write concurrency capabilities of the data processing system, a multi-cluster data processing system is often used, and data is shared among multiple clusters. How to provide a low-cost, high-performance data writing solution for a multi-cluster data processing system has become an important technical problem that major cloud vendors need to solve. Summary of the invention

[0003] The present application provides a data processing method and a corresponding data processing system, a computing device cluster, a computer-readable storage medium, and a computer program product.

[0004] In a first aspect, the present application provides a data processing method. The method includes: a data processing system receives a first write request, the data processing system includes a first computing cluster, a second computing cluster and a shared storage, the shared storage is used to store data of a first data table, the first data table belongs to the first computing cluster, and the first write request is used to write the first data to the first data table; the second computing cluster writes the first data belonging to the first data table to the shared storage; the second computing cluster sends the first metadata corresponding to the first data to the first computing cluster; the first computing cluster stores the first metadata corresponding to the first data.

[0005] The data processing method provided in this application can meet the requirements of expanding the computing cluster to support data writing. At the same time, the computing cluster to which the table belongs is still responsible for writing the metadata corresponding to the table data. There is no need to deploy a separate metadata management cluster. While reducing system costs, there is no performance bottleneck problem of the metadata management cluster.

[0006] In a possible implementation manner, the data processing method further includes:

[0007] The data processing system receives a second write request, where the second write request is used to write second data into the first data table;

[0008] The first computing cluster writes the second data belonging to the first data table into the shared storage;

[0009] The first computing cluster stores metadata corresponding to the second data.

[0010] Through the above implementation, the first computing cluster also supports data writing operations, which can improve the write concurrency carrying capacity of the data processing system.

[0011] In a possible implementation manner, the data processing method further includes:

[0012] The data processing system generates an execution plan for responding to the first write request, the execution plan comprising an operator for transmitting metadata across clusters;

[0013] The second computing cluster sends first metadata corresponding to the first data to the first computing cluster, including:

[0014] The second computing cluster executes the execution plan to send the first metadata corresponding to the first data to the first computing cluster through the operator for transmitting metadata across clusters.

[0015] In a possible implementation manner, the data processing method further includes: determining a computing cluster with the smallest load in the data processing system as the second computing cluster.

[0016] Through the above implementation, the data processing system has flexible scheduling capabilities and can select a suitable cluster to perform data writing operations according to the load pressure of each computing cluster, thereby achieving load balancing of the entire data processing system and avoiding excessive writing pressure on a single computing cluster.

[0017] In a possible implementation manner, the shared storage is further used to store data of a second data table, the second data table belongs to the first computing cluster, the first data comes from the second data table, and before the second computing cluster writes the first data belonging to the first data table into the shared storage, the data processing method further includes:

[0018] The first computing cluster sends second metadata corresponding to the first data to the second computing cluster;

[0019] The second computing cluster reads the first data from the second data table from the shared storage according to the second metadata corresponding to the first data.

[0020] In a possible implementation, the data processing system further includes a third computing cluster, the shared storage is further used to store data of a third data table, the third data table belongs to the third computing cluster, the first data comes from the third data table, and before the second computing cluster writes the first data belonging to the first data table into the shared storage, the data processing method further includes:

[0021] The third computing cluster sends third metadata corresponding to the first data to the second computing cluster;

[0022] The second computing cluster reads the first data from the third data table from the shared storage according to the third metadata corresponding to the first data.

[0023] Through the above implementation, the computing cluster that performs the data write operation can read the data from other computing clusters to which the table data belongs.

[0024] In a possible implementation, the data processing method further includes: the second computing cluster sending the index data of the first data table to the first computing cluster; and the first computing cluster storing the index data of the first data table.

[0025] In a second aspect, the present application provides a data processing system, which includes: a cluster management module, used to receive a first write request, the first write request is used to write first data into a first data table; a shared storage, used to store data in the first data table; a first computing cluster, the first data table belongs to the first computing cluster; a second computing cluster, used to write the first data belonging to the first data table into the shared storage; the second computing cluster is also used to send first metadata corresponding to the first data to the first computing cluster; the first computing cluster is used to store the first metadata corresponding to the first data.

[0026] In an embodiment of the present application, the cluster management module is also used to receive a second write request, which is used to write the second data into the first data table; the first computing cluster is also used to write the second data belonging to the first data table into the shared storage and store metadata corresponding to the second data.

[0027] In a possible embodiment, the cluster management module is also used to generate an execution plan for responding to the first write request, and the execution plan includes an operator for transmitting metadata across clusters; the second computing cluster is also used to send the first metadata corresponding to the first data to the first computing cluster, including: the second computing cluster is used to execute the execution plan to send the first metadata corresponding to the first data to the first computing cluster through the operator for transmitting metadata across clusters.

[0028] In a possible embodiment, the cluster management module is further used to determine a computing cluster with the smallest load in the data processing system as the second computing cluster.

[0029] In a possible embodiment, the shared storage is also used to store data of a second data table, the second data table belongs to the first computing cluster, and the first data comes from the second data table; before the second computing cluster writes the first data belonging to the first data table to the shared storage, the first computing cluster is also used to send second metadata corresponding to the first data to the second computing cluster; and the second computing cluster is also used to read the first data from the second data table from the shared storage based on the second metadata corresponding to the first data.

[0030] In a possible embodiment, the data processing system also includes a third computing cluster; the shared storage is also used to store data of a third data table, the third data table belongs to the third computing cluster, and the first data comes from the third data table; before the second computing cluster writes the first data belonging to the first data table into the shared storage, the third computing cluster is used to send third metadata corresponding to the first data to the second computing cluster; and the second computing cluster is also used to read the first data from the third data table from the shared storage according to the third metadata corresponding to the first data.

[0031] In a possible embodiment, the second computing cluster is further used to send the index data of the first data table to the first computing cluster; and the first computing cluster is further used to store the index data of the first data table.

[0032] In a third aspect, the present application provides a computing device cluster, comprising at least one computing device, each computing device comprising a processor and a memory; the processor of the at least one computing device is used to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster performs a data processing method provided in the first aspect or any possible implementation of the first aspect.

[0033] In a fourth aspect, the present application provides a computer program product comprising instructions, which, when executed by a computing device cluster, enables the computing device cluster to execute a data processing method as provided in the first aspect or any possible implementation of the first aspect.

[0034] In a fifth aspect, the present application provides a computer-readable storage medium comprising computer program instructions. When the computer program instructions are executed by a computing device cluster, the computing device cluster executes the data processing method provided in the above-mentioned first aspect or any possible implementation of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 It is a schematic diagram of the architecture of a data processing system provided in an embodiment of the present application.

[0036] Figure 2 It is a structural diagram of a computing device provided in an embodiment of the present application.

[0037] Figure 3 It is a structural diagram of a computing device cluster provided in an embodiment of the present application.

[0038] Figure 4 It is a schematic diagram of an implementation method of a computing device cluster provided in an embodiment of the present application. DETAILED DESCRIPTION

[0039] The technical solution provided by the present application will be described in detail below in conjunction with the accompanying drawings. Although certain embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be interpreted as being limited to the embodiments described herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present application. It should be understood that the drawings and embodiments of the present application are only for exemplary purposes and are not intended to limit the scope of protection of the present application.

[0040] In the description of the embodiments of the present application, the term "including" and similar terms should be understood as open inclusion, that is, "including but not limited to". The term "based on" should be understood as "based at least in part on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The terms "first", "second", etc. may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0041] In the present application, "at least one" means one or more, and "plurality" means two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: including the existence of A alone, the existence of A and B at the same time, and the existence of B alone, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b, or c can mean: a, b, c, ab, ac, bc, or abc, where a, b, c can be single or multiple.

[0042] With the development of cloud computing technology, more and more users choose to store large amounts of data in the cloud and use the data processing systems provided by cloud vendors to operate the data in the cloud storage. For users with greater business pressure, in order to improve the overall read and write concurrency capabilities of the data processing system, a multi-cluster data processing system is often used, and data is shared among multiple clusters. How to provide a low-cost, high-performance data writing solution for a multi-cluster data processing system has become an important technical problem that major cloud vendors need to solve.

[0043] The following introduces several existing data writing solutions for multi-cluster data processing systems.

[0044] The multi-cluster data processing systems provided by some cloud vendors only support data sharing among multiple clusters, and cannot meet the requirement that the extended cluster also supports data writing. Specifically, when the cluster to which the data table belongs (that is, the original computing cluster) has a large business pressure, the newly added extended cluster only supports reading the data in the data table, but does not support writing data to the data table. Therefore, the newly added extended cluster cannot share the writing pressure of the original computing cluster, and the write concurrency processing capability of the entire data processing system still has a performance bottleneck.

[0045] Another part of the multi-cluster data processing systems provided by cloud vendors can meet the requirements of supporting data writing in the extended cluster, but the metadata management cluster is used to uniformly write the metadata generated during the data writing process of each computing cluster into the metadata storage space. This multi-cluster writing solution based on unified metadata management has the following problems: metadata writing for all computing clusters is done in the metadata management cluster, which will become the performance bottleneck of the entire data processing system when business pressure is high; table permission control information is all in the metadata management cluster, and when the metadata management cluster fails, the business of the entire data processing system is widely affected; an independent metadata management cluster needs to be deployed, and the computing cluster alone cannot complete data writing, and the system initialization cost is high.

[0046] In order to solve at least one or more of the above problems, the present application provides a data processing method, which can meet the requirements of expanding the computing cluster to support data writing. At the same time, the computing cluster to which the table belongs is still responsible for writing the metadata corresponding to the table data. There is no need to deploy a separate metadata management cluster, which reduces the system cost and does not have the performance bottleneck problem of the metadata management cluster.

[0047] Figure 1 A schematic diagram of the architecture of a data processing system 200 provided in an embodiment of the present application is given below. Figure 1 The application scenarios of the data processing method provided in this application are introduced.

[0048] like Figure 1As shown, a first computing cluster 204 is initially deployed in the data processing system 200, and the user can use the first computing cluster 204 to perform table data write operations, wherein the first computing cluster 204 is the home cluster of the data table 210, and the data of the data table 210 is stored in the shared storage 208. For example, when the user's business pressure is relatively small, after the data processing system 200 receives a write request Q1 for writing data A into the data table 210 and a write request Q2 for writing data B into the data table 210, the write operations of data A and data B are both completed by the first computing cluster 204. In the embodiment of the present application, the data write operation can be an Insert operation, an Update operation, or a Delete operation of the table data.

[0049] As the user's business pressure increases, a second computing cluster can be added to the data processing system 200 as an extended cluster to be responsible for the writing of new business-related data. For example, the write request Q1 for writing data A to the data table 210 is responded to by the first computing cluster 204, and the write request Q2 for writing data B to the data table 210 is responded to by the second computing cluster 206. Specifically, the second computing cluster 206 is responsible for writing data B to the shared storage, and sending the metadata 212 generated during the writing of data B to the first computing cluster 204; the first computing cluster 204 is responsible for writing data A to the shared storage, and storing the metadata 214 generated during the writing of data A, and is also responsible for storing the metadata 212 corresponding to the received data B.

[0050] In the data processing system 200, the first computing cluster 204 and the second computing cluster 206 may include at least one data node, and the shared storage 208 may be a cloud storage such as an object storage service (OBS). In addition, the data processing system 200 also includes a cluster management module 202, which is used to generate an execution plan corresponding to a write request and send the execution plan to the first computing cluster 204 and the second computing cluster 206. In actual applications, the data management system 200 may include more than two computing clusters, and the cluster management module may be deployed on a coordinator node (CN) in a data warehouse system or a database system.

[0051] Combine the following Figure 1 The specific implementation of the data processing method provided by the present application is introduced. The method includes the following steps S301 to S305:

[0052] S301: A data processing system receives a first write request, where the first write request is used to write first data into a first data table.

[0053] The data processing system includes a first computing cluster, a second computing cluster and a shared storage, wherein the shared storage is used to store data of a first data table, and the first data table belongs to the first computing cluster.

[0054] For example, the data processing system 200 receives a write request Q2 regarding data B.

[0055] S302: The data processing system generates an execution plan for responding to a first write request, wherein the execution plan includes an operator for transmitting metadata across clusters, an operator for writing data, and an operator for storing metadata.

[0056] For example, after receiving the write request Q2, the cluster management module 202 in the data processing system 200 will generate a cross-cluster execution plan for the write request Q2, which includes an operator for writing data B, an operator for transmitting metadata 212 corresponding to data B across clusters, and an operator for storing metadata 212 corresponding to data B.

[0057] Specifically, the cluster management module 202 generates an execution plan corresponding to the write request Q2 and sends it to the first computing cluster 204 and the second computing cluster 206 .

[0058] In an embodiment of the present application, the cluster management module 202 has flexible scheduling capabilities and can select a computing cluster with a load less than other computing clusters or a load that does not exceed a preset threshold to perform data writing operations based on the load conditions of each computing cluster in the data management system 200, thereby achieving load balancing of the entire data processing system and avoiding excessive writing pressure on a single computing cluster.

[0059] In a possible embodiment, the cluster management module 202 may determine the computing cluster with the smallest load in the data management system 200 as the second computing cluster 206 to perform the data write operation according to the load pressure of each computing cluster in the data management system 200. Accordingly, the cluster management module 202 will send the execution plan corresponding to the write request to the data's home cluster and the computing cluster with the smallest load.

[0060] In a possible embodiment, the data processing system supports load isolation, different users are bound to different computing clusters, and the cluster management module 202 can select the computing cluster corresponding to the user to execute the execution plan corresponding to the user's write request based on the binding relationship between the user and the computing cluster. For example, the data processing system 202 binds the user M to the second computing cluster 206 in advance, then when the data processing system 202 receives the write request Q2 sent by the user M, the cluster management module 202 will send the execution plan corresponding to the write request Q2 to the second computing cluster 206, so that the second computing cluster 206 is responsible for writing the data B belonging to the data table 210 into the shared storage.

[0061] S303: The second computing cluster writes the first data belonging to the first data table into the shared storage.

[0062] Specifically, the second computing cluster executes the execution plan corresponding to the first write request, so as to write the first data belonging to the first data table into the shared storage through the operator for writing data.

[0063] In a possible embodiment, when writing the first data into the shared storage, the second computing cluster may organize the first data into column storage compression units according to a certain format and then write the first data.

[0064] S304: The second computing cluster sends the first metadata corresponding to the first data to the first computing cluster.

[0065] The first metadata corresponding to the first data is metadata generated by the second computing cluster when writing the first data into the shared storage.

[0066] In a possible embodiment, the first metadata includes but is not limited to the location of the column storage compression unit in the shared storage, the size of the column storage compression unit, the maximum value or minimum value of the data in the column storage compression unit, and transaction information.

[0067] Specifically, the second computing cluster executes the execution plan corresponding to the first write request, so as to send the first metadata corresponding to the first data to the first computing cluster through the operator for transmitting metadata across clusters.

[0068] In a possible embodiment, after the second computing cluster writes the first data belonging to the first data table into the shared storage, it also generates index data of the first data table. At this time, the second computing cluster also sends the index data of the first data table to the first computing cluster through the above-mentioned operator for transmitting metadata across clusters.

[0069] S305: The first computing cluster stores first metadata corresponding to the first data.

[0070] Specifically, the first computing cluster writes the first metadata corresponding to the first data into a local computing node of the first computing cluster.

[0071] In a possible embodiment, the first computing cluster also receives the index data of the first data table from the second computing cluster, and the first computing cluster also stores the index data corresponding to the first data table. Specifically, the first computing cluster writes the index data corresponding to the first data table to the computing node local to the first computing cluster.

[0072] In the embodiment of the present application, the data writing performance of the first computing cluster is not lower than the data writing performance of the second computing cluster. The first computing cluster can also process the data writing operation and store the metadata generated during the data writing operation. For example, the first computing cluster 204 writes data A to the shared storage in response to the write request Q1. Therefore, the data processing method provided by the present application also includes the following steps S401 to S403:

[0073] S401: The data processing system receives a second write request, where the second write request is used to write second data into a first data table.

[0074] S402: The first computing cluster writes second data belonging to the first data table into the shared storage.

[0075] S403: The first computing cluster stores metadata corresponding to the second data.

[0076] The metadata corresponding to the second data is metadata generated by the first computing cluster when writing the second data belonging to the first data table into the shared storage.

[0077] In practical applications, the data processing method provided in the present application also includes a process of obtaining the first data. The following examples introduce several application scenarios.

[0078] In a possible application scenario, the shared storage is also used to store data of a second data table, the second data table belongs to the first computing cluster, and the first write request is specifically used to insert first data from the second data table into the first data table. In this scenario, before the second computing cluster writes the first data belonging to the first data table into the shared storage, the process of obtaining the first data from the second data table includes:

[0079] The first computing cluster sends second metadata corresponding to the first data to the second computing cluster;

[0080] The second computing cluster reads the first data from the second data table from the shared storage according to the second metadata corresponding to the first data.

[0081] In another possible application scenario, the data processing system further includes a third computing cluster, the shared storage is further used to store data of a third data table, the third data table belongs to the third computing cluster, and the first write request is specifically used to insert first data from the third data table into the first data table. In this scenario, before the second computing cluster writes the first data belonging to the first data table into the shared storage, the process of obtaining the first data from the third data table includes:

[0082] The third computing cluster sends the third metadata corresponding to the first data to the second computing cluster;

[0083] The second computing cluster reads the first data from the third data table from the shared storage according to the third metadata corresponding to the first data.

[0084] In the data processing method provided in the present application, by having the computing cluster that performs the data writing operation send the metadata generated during the data writing process to the cluster to which the data belongs, each computing cluster of the data processing system can support data writing, which helps to improve the write concurrency carrying capacity of the multi-cluster data processing system.

[0085] In addition, the data processing method provided by the present application can enable the data processing system to have more flexible elastic expansion capabilities. When the user's business pressure increases, an extended cluster can be temporarily added to the data processing system to perform the write operation of the table data belonging to the original computing cluster. At the same time, the cluster to which the table data belongs is still responsible for writing the metadata corresponding to the table data. The extended cluster only needs to remotely transmit the metadata generated during the table data writing process to the cluster to which the table data belongs. Since the extended cluster shares the write operation of the table data, the metadata of the table data is still written by the original cluster to which the table data belongs. In this way, when the business pressure decreases and the extended cluster needs to be removed, there is no need to redistribute the metadata. The cluster to which the table data belongs can directly take over and handle all the business.

[0086] like Figure 1 As shown, the present application also provides a data processing system 200, which includes: a cluster management module 202, a first computing cluster 204, a second computing cluster 206, and a shared storage 208. Among them,

[0087] The cluster management module 202 is configured to receive a first write request, where the first write request is configured to write first data into a first data table;

[0088] Shared storage 208, used to store data in the first data table;

[0089] The second computing cluster 206 is configured to write first data belonging to a first data table into a shared storage 208 , where the first data table belongs to the first computing cluster 204 ;

[0090] The second computing cluster 206 is further configured to send first metadata corresponding to the first data to the first computing cluster 204;

[0091] The first computing cluster 204 is used to store first metadata corresponding to the first data.

[0092] In an embodiment of the present application, the cluster management module 202 is further used to receive a second write request, where the second write request is used to write the second data into the first data table; the first computing cluster 204 is further used to write the second data belonging to the first data table into the shared storage 208 and store metadata corresponding to the second data.

[0093] In a possible embodiment, the cluster management module 202 is also used to generate an execution plan for responding to the first write request, and the execution plan includes an operator for transmitting metadata across clusters; the second computing cluster 206 is also used to send the first metadata corresponding to the first data to the first computing cluster 204, including: the second computing cluster 206 is used to execute the execution plan to send the first metadata corresponding to the first data to the first computing cluster 204 through the operator for transmitting metadata across clusters.

[0094] In a possible embodiment, the cluster management module 202 is further configured to determine a computing cluster with the smallest load in the data processing system 200 as the second computing cluster 206 .

[0095] In a possible embodiment, the shared storage 208 is also used to store data of a second data table, the second data table belongs to the first computing cluster 204, and the first data comes from the second data table; before the second computing cluster writes the first data belonging to the first data table into the shared storage 208, the first computing cluster 204 is also used to send second metadata corresponding to the first data to the second computing cluster 206; and the second computing cluster 206 is also used to read the first data from the second data table from the shared storage 208 according to the second metadata corresponding to the first data.

[0096] In a possible embodiment, the data processing system 200 also includes a third computing cluster; the shared storage 208 is also used to store data of a third data table, the third data table belongs to the third computing cluster, and the first data comes from the third data table; before the second computing cluster 206 writes the first data belonging to the first data table into the shared storage 208, the third computing cluster is used to send third metadata corresponding to the first data to the second computing cluster 206; and the second computing cluster 206 is also used to read the first data from the third data table from the shared storage 208 according to the third metadata corresponding to the first data.

[0097] In a possible embodiment, the second computing cluster 206 is further configured to send the index data of the first data table to the first computing cluster 204; the first computing cluster 204 is further configured to store the index data of the first data table.

[0098] The data processing system 200 provided in this application can be used to implement the data processing method provided by any possible implementation in the above method embodiments of this application. Specifically, the specific implementation of the various operations of the data processing method performed by the data processing system 200 can refer to the description of the relevant content in the above method embodiments, which will not be repeated here.

[0099] In the embodiment of the present application, the data processing system 200 can be implemented by software or hardware. The following is an exemplary implementation of the cluster management module 202, and the implementation of the first computing cluster 204, the second computing cluster 206, and the shared storage 208 can be used as a reference.

[0100] As an example of a software functional unit, a module may include code running on a computing instance. Among them, the computing instance may be at least one of computing devices such as a physical host (computing device), a virtual machine, and a container. Furthermore, the computing device may be one or more. For example, a node may include code running on multiple hosts, or virtual machines, or containers. It should be noted that multiple hosts, or virtual machines, or containers for running the code may be distributed in the same AZ or in different AZs, and each AZ includes a data center or multiple data centers with similar geographical locations. Multiple hosts, or virtual machines, or containers for running the code may be distributed in the same region or in different regions. Among them, a region may usually include multiple AZs, and VPCs are set in a region. Cross-region communication between two VPCs in the same region and between VPCs in different regions requires the setting of a communication gateway in each VPC, and the interconnection between VPCs is realized through the communication gateway.

[0101] Similarly, multiple hosts or virtual machines or containers for running the code may be distributed in the same VPC or in multiple VPCs, wherein usually a region may include multiple AZs.

[0102] As an example of a hardware functional unit, a module may include at least one computing device, such as a server, etc. Alternatively, a module may also be a device implemented using ASIC or PLD, etc. The PLD may be implemented using CPLD, FPGA, GAL or any combination thereof.

[0103] The multiple computing devices included in the module can be distributed in the same AZ or in different AZs. The multiple computing devices included in the module can be distributed in the same region or in different regions. Similarly, the multiple computing devices included in the module can be distributed in the same VPC or in multiple VPCs. The multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0104] The present application also provides a computing device 500, Figure 2 5 is a schematic diagram of the structure of a computing device 500. The computing device 500 includes: a bus 502, a processor 504, a memory 506, and a communication interface 508. The processor 504, the memory 506, and the communication interface 508 communicate with each other through the bus 502. The computing device 500 can be a server or a terminal device. It should be understood that the present application does not limit the number of processors and memories in the computing device 500.

[0105] The bus 502 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 2 The bus 502 is represented by only one line, but it does not mean that there is only one bus or one type of bus. The bus 502 may include a path for transmitting information between various components of the computing device 500 (eg, the memory 506, the processor 504, and the communication interface 508).

[0106] The processor 504 may include any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0107] The memory 506 may include a volatile memory, such as a random access memory (RAM). The memory 506 may also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD), or a solid state drive (SSD).

[0108] The memory 506 stores executable program codes, and the processor 504 executes the executable program codes to respectively implement the functions of the cluster management module 202, the first computing cluster 204, the second computing cluster 206, and the shared storage 208 in the aforementioned data processing system 200, thereby implementing the data processing method. That is, the memory 506 stores instructions for executing the data processing method.

[0109] The communication interface 508 uses a transceiver module such as, but not limited to, a network interface card or a transceiver to implement communication between the computing device 500 and other devices or a communication network.

[0110] The present application also provides a computing device cluster, Figure 3 is a schematic diagram of the computing device cluster, which can implement the data processing method in the above embodiment. Figure 3 As shown, the computing device cluster includes at least one Figure 2 The computing device 500 shown may be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device may also be a terminal device such as a desktop computer, a laptop computer, or a smart phone.

[0111] The memory 506 in one or more computing devices 500 in the computing device cluster may store instructions for executing the data processing method. When at least one computing device in the computing device cluster executes the instructions, the computing device cluster may implement the data processing method described in the embodiment.

[0112] In some possible implementations, the memory 506 of one or more computing devices 500 in the computing device cluster may also store partial instructions for executing the data processing method. In other words, the combination of one or more computing devices 500 may jointly execute instructions for executing the data processing method.

[0113] It should be noted that the memory 506 in different computing devices 500 in the computing device cluster can store different instructions, which are respectively used to execute part of the functions of the data processing system. That is, the instructions stored in the memory 506 in different computing devices 500 can respectively implement the functions of the cluster management module 202, the first computing cluster 204, the second computing cluster 206, or the shared storage 208 in the data processing system.

[0114] In some possible implementations, one or more computing devices in the computing device cluster may be connected via a network, which may be a wide area network or a local area network. Figure 4 A possible implementation is shown. Figure 4 As shown, two computing devices 500A and 500B are connected via a network. Specifically, the network is connected via a communication interface in each computing device. In this type of possible implementation, the memory 506 in the computing device 500A stores instructions for executing the functions of the cluster management module 202 and the shared storage 208 in the data processing system. At the same time, the memory 506 in the computing device 500B stores instructions for executing the functions of the first computing cluster 204 and the second computing cluster 206 in the data processing system.

[0115] It should be understood that Figure 4 The functions of the computing device 500A shown in FIG. 5A may also be completed by multiple computing devices 500. Similarly, the functions of the computing device 500B may also be completed by multiple computing devices 500.

[0116] The present application also provides a computer program product comprising instructions. The computer program product may be software or a program product comprising instructions that can be run on a computing device or stored in any available medium. When the computer program product is run on at least one computing device, the at least one computing device is caused to execute a data processing method.

[0117] The present application also provides a computer-readable storage medium. The computer-readable storage medium may be any available medium that can be stored by a computing device or a data storage device such as a data center that includes one or more available media. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk). The computer-readable storage medium includes instructions that instruct a computing device to execute a data processing method, or instructs a computing device to execute a data processing method.

[0118] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of the present invention.

Claims

1. A data processing method, characterized in that: The method comprises: A data processing system receives a first write request, the data processing system includes a first computing cluster, a second computing cluster and a shared storage, the shared storage is used to store data of a first data table, the first data table belongs to the first computing cluster, and the first write request is used to write first data into the first data table; The second computing cluster writes the first data belonging to the first data table into the shared storage; The second computing cluster sends first metadata corresponding to the first data to the first computing cluster; The first computing cluster stores first metadata corresponding to the first data.

2. The method according to claim 1, characterized in that The method further comprises: The data processing system receives a second write request, where the second write request is used to write second data into the first data table; The first computing cluster writes the second data belonging to the first data table into the shared storage; The first computing cluster stores metadata corresponding to the second data.

3. The method according to claim 1 or 2, characterized in that: The method further comprises: The data processing system generates an execution plan for responding to the first write request, the execution plan comprising an operator for transmitting metadata across clusters; The second computing cluster sends first metadata corresponding to the first data to the first computing cluster, including: The second computing cluster executes the execution plan to send the first metadata corresponding to the first data to the first computing cluster through the operator for transmitting metadata across clusters.

4. The method according to any one of claims 1 to 3, characterized in that The method further comprises: A computing cluster with the smallest load in the data processing system is determined as the second computing cluster.

5. The method according to any one of claims 1 to 4, characterized in that The shared storage is further used to store data of a second data table, the second data table belongs to the first computing cluster, the first data comes from the second data table, and before the second computing cluster writes the first data belonging to the first data table into the shared storage, the method further includes: The first computing cluster sends second metadata corresponding to the first data to the second computing cluster; The second computing cluster reads the first data from the second data table from the shared storage according to the second metadata corresponding to the first data.

6. The method according to any one of claims 1 to 4, characterized in that The data processing system further includes a third computing cluster, the shared storage is further used to store data of a third data table, the third data table belongs to the third computing cluster, the first data comes from the third data table, and before the second computing cluster writes the first data belonging to the first data table into the shared storage, the method further includes: The third computing cluster sends third metadata corresponding to the first data to the second computing cluster; The second computing cluster reads the first data from the third data table from the shared storage according to the third metadata corresponding to the first data.

7. The method according to any one of claims 1 to 6, characterized in that The method further comprises: The second computing cluster sends the index data of the first data table to the first computing cluster; The first computing cluster stores index data of the first data table.

8. A data processing system, characterized in that: The system comprises: A cluster management module, configured to receive a first write request, wherein the first write request is configured to write first data into a first data table; Shared storage, used to store data in the first data table; a first computing cluster, the first data table belongs to the first computing cluster; a second computing cluster, configured to write first data belonging to the first data table into the shared storage; The second computing cluster is further configured to send first metadata corresponding to the first data to the first computing cluster; The first computing cluster is used to store first metadata corresponding to the first data.

9. The system according to claim 8, characterized in that The cluster management module is further used to receive a second write request, where the second write request is used to write second data into the first data table; The first computing cluster is further configured to write the second data belonging to the first data table into the shared storage; The first computing cluster is further used to store metadata corresponding to the second data.

10. The system according to claim 8 or 9, characterized in that The cluster management module is further used to generate an execution plan for responding to the first write request, wherein the execution plan includes an operator for transmitting metadata across clusters; The second computing cluster is further configured to send first metadata corresponding to the first data to the first computing cluster, including: The second computing cluster is used to execute the execution plan, so as to send the first metadata corresponding to the first data to the first computing cluster through the operator for transmitting metadata across clusters.

11. The system according to any one of claims 8 to 10, characterized in that The cluster management module is further configured to determine a computing cluster with the smallest load in the data processing system as the second computing cluster.

12. The system according to any one of claims 8 to 11, characterized in that The shared storage is further used to store data of a second data table, where the second data table belongs to the first computing cluster, and the first data comes from the second data table; Before the second computing cluster writes the first data belonging to the first data table into the shared storage, the first computing cluster is further configured to send second metadata corresponding to the first data to the second computing cluster; and The second computing cluster is further configured to read the first data from the second data table from the shared storage according to second metadata corresponding to the first data.

13. The system according to any one of claims 8 to 11, characterized in that The data processing system also includes a third computing cluster; The shared storage is further used to store data in a third data table, the third data table belongs to the third computing cluster, and the first data comes from the third data table; Before the second computing cluster writes the first data belonging to the first data table into the shared storage, the third computing cluster is used to send third metadata corresponding to the first data to the second computing cluster; as well as The second computing cluster is further configured to read the first data from the third data table from the shared storage according to third metadata corresponding to the first data.

14. The system according to any one of claims 8 to 13, characterized in that The second computing cluster is further configured to send the index data of the first data table to the first computing cluster; The first computing cluster is further used to store index data of the first data table.

15. A computing device cluster, characterized in that: comprising at least one computing device, each computing device comprising a processor and a memory; The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster executes the data processing method according to any one of claims 1 to 7.

16. A computer program product comprising instructions, characterized in that When the instructions are executed by a computing device cluster, the computing device cluster executes the data processing method according to any one of claims 1 to 7.

17. A computer-readable storage medium, characterized in that: The method comprises computer program instructions. When the computer program instructions are executed by a computing device cluster, the computing device cluster performs the data processing method according to any one of claims 1 to 7.

Images