Method for processing data, system, and computing device cluster

Abstract

The present application belongs to the technical field of cloud computing, and provides a method for processing data and a system. The method comprises: a data processing system receives a first write request, the data processing system comprising a first computing cluster, a second computing cluster, and shared storage, the shared storage being used for storing data of a first data table, the first write request being used for writing first data into the first data table, and the first data table belonging to the first computing cluster; the second computing cluster writes the first data belonging to the first data table into the shared storage; the second computing cluster sends to the first computing cluster first metadata corresponding to the first data; and the first computing cluster stores the first metadata corresponding to the first data. According to the described method, the requirement for extending a computing cluster to support data writing can be met, while the computing cluster to which a table belongs is responsible for writing metadata corresponding to data in the table, so that a separate metadata management cluster does not need to be deployed, thereby eliminating the problem of performance bottleneck of a metadata management cluster while reducing system costs.

Description

Data processing method, system and computing device cluster

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office of China on October 27, 2023, with application number 202311412603.8 and invention name “A data processing method, system and computing device cluster”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of cloud computing technology, and in particular to a data processing method, system, computing device cluster, computer-readable storage medium, and computer program product. Background Art

[0003] With the development of cloud computing technology, more and more users are choosing to store large amounts of data in the cloud and use the data processing systems provided by cloud vendors to operate on this data. For users with high business pressure, multi-cluster data processing systems are often used to improve the overall read and write concurrency of the data processing system, with data shared across multiple clusters. Providing low-cost, high-performance data writing solutions for multi-cluster data processing systems has become a key technical challenge for major cloud vendors.

[0004] Summary of the Invention

[0005] The present application provides a data processing method and a corresponding data processing system, a computing device cluster, a computer-readable storage medium, and a computer program product.

[0006] In a first aspect, the present application provides a data processing method. The method includes: a data processing system receiving a first write request, the data processing system including a first computing cluster, a second computing cluster, and shared storage, the shared storage being used to store data of a first data table, the first data table belonging to the first computing cluster, and the first write request being used to write first data to the first data table; the second computing cluster writing the first data belonging to the first data table to the shared storage; the second computing cluster sending first metadata corresponding to the first data to the first computing cluster; and the first computing cluster storing the first metadata corresponding to the first data.

[0007] The data processing method provided in this application can meet the requirements of expanding the computing cluster to support data writing. At the same time, the computing cluster to which the table belongs is still responsible for writing the metadata corresponding to the table data. There is no need to deploy a separate metadata management cluster. While reducing system costs, there is no performance bottleneck problem of the metadata management cluster.

[0008] In a possible implementation, the data processing method further includes:

[0009] The data processing system receives a second write request, where the second write request is used to write second data into the first data table;

[0010] The first computing cluster writes the second data belonging to the first data table into the shared storage;

[0011] The first computing cluster stores metadata corresponding to the second data.

[0012] Through the above implementation, the first computing cluster also supports data writing operations, which can improve the write concurrency carrying capacity of the data processing system.

[0013] In a possible implementation, the data processing method further includes:

[0014] The data processing system generates an execution plan for responding to the first write request, the execution plan including an operator for transmitting metadata across clusters;

[0015] The second computing cluster sending first metadata corresponding to the first data to the first computing cluster includes:

[0016] The second computing cluster executes the execution plan to send the first metadata corresponding to the first data to the first computing cluster through the operator for transmitting metadata across clusters.

[0017] In a possible implementation, the data processing method further includes: determining a computing cluster with the smallest load in the data processing system as the second computing cluster.

[0018] Through the above implementation, the data processing system has flexible scheduling capabilities and can select the appropriate cluster to perform data writing operations according to the load pressure of each computing cluster, thereby achieving load balancing of the entire data processing system and avoiding excessive writing pressure on a single computing cluster.

[0019] In one possible implementation, the shared storage is further used to store data in a second data table, the second data table belongs to the first computing cluster, and the first data comes from the second data table. Before the second computing cluster writes the first data belonging to the first data table to the shared storage, the data processing method further includes:

[0020] The first computing cluster sends second metadata corresponding to the first data to the second computing cluster;

[0021] The second computing cluster reads the first data from the second data table from the shared storage according to the second metadata corresponding to the first data.

[0022] In one possible implementation, the data processing system further includes a third computing cluster, the shared storage is further used to store data in a third data table, the third data table belongs to the third computing cluster, the first data comes from the third data table, and before the second computing cluster writes the first data belonging to the first data table to the shared storage, the data processing method further includes:

[0023] The third computing cluster sends third metadata corresponding to the first data to the second computing cluster;

[0024] The second computing cluster reads the first data from the third data table from the shared storage according to the third metadata corresponding to the first data.

[0025] Through the above implementation, the computing cluster that performs the data write operation can read the data from other computing clusters to which the table data belongs.

[0026] In a possible implementation, the data processing method further includes: the second computing cluster sending the index data of the first data table to the first computing cluster; and the first computing cluster storing the index data of the first data table.

[0027] In a second aspect, the present application provides a data processing system, which includes: a cluster management module for receiving a first write request, the first write request being used to write first data into a first data table; shared storage for storing data in the first data table; a first computing cluster, the first data table belonging to the first computing cluster; a second computing cluster for writing the first data belonging to the first data table into the shared storage; the second computing cluster is also used to send first metadata corresponding to the first data to the first computing cluster; the first computing cluster is used to store the first metadata corresponding to the first data.

[0028] In an embodiment of the present application, the cluster management module is further used to receive a second write request, which is used to write the second data into the first data table; the first computing cluster is also used to write the second data belonging to the first data table into the shared storage and store metadata corresponding to the second data.

[0029] In a possible embodiment, the cluster management module is also used to generate an execution plan for responding to the first write request, and the execution plan includes an operator for transmitting metadata across clusters; the second computing cluster is also used to send the first metadata corresponding to the first data to the first computing cluster, including: the second computing cluster is used to execute the execution plan to send the first metadata corresponding to the first data to the first computing cluster through the operator for transmitting metadata across clusters.

[0030] In a possible embodiment, the cluster management module is further configured to determine a computing cluster with the smallest load in the data processing system as the second computing cluster.

[0031] In one possible embodiment, the shared storage is also used to store data of a second data table, the second data table belongs to the first computing cluster, and the first data comes from the second data table; before the second computing cluster writes the first data belonging to the first data table to the shared storage, the first computing cluster is also used to send second metadata corresponding to the first data to the second computing cluster; and the second computing cluster is also used to read the first data from the second data table from the shared storage based on the second metadata corresponding to the first data.

[0032] In one possible embodiment, the data processing system also includes a third computing cluster; the shared storage is also used to store data of a third data table, the third data table belongs to the third computing cluster, and the first data comes from the third data table; before the second computing cluster writes the first data belonging to the first data table into the shared storage, the third computing cluster is used to send third metadata corresponding to the first data to the second computing cluster; and the second computing cluster is also used to read the first data from the third data table from the shared storage based on the third metadata corresponding to the first data.

[0033] In a possible embodiment, the second computing cluster is further configured to send index data of the first data table to the first computing cluster; and the first computing cluster is further configured to store the index data of the first data table.

[0034] In a third aspect, the present application provides a computing device cluster comprising at least one computing device, each computing device comprising a processor and a memory; the processor of the at least one computing device is used to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster performs a data processing method as provided in the first aspect or any possible implementation of the first aspect.

[0035] In a fourth aspect, the present application provides a computer program product comprising instructions, which, when executed by a computing device cluster, enables the computing device cluster to execute the data processing method provided in the first aspect or any possible implementation of the first aspect.

[0036] In a fifth aspect, the present application provides a computer-readable storage medium comprising computer program instructions. When the computer program instructions are executed by a computing device cluster, the computing device cluster executes the data processing method provided in the first aspect or any possible implementation of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] FIG1 is a schematic diagram of the architecture of a data processing system provided in an embodiment of the present application.

[0038] FIG2 is a schematic diagram of the structure of a computing device provided in an embodiment of the present application.

[0039] FIG3 is a schematic diagram of the structure of a computing device cluster provided in an embodiment of the present application.

[0040] FIG4 is a schematic diagram of an implementation method of a computing device cluster provided in an embodiment of the present application. DETAILED DESCRIPTION

[0041] The technical solutions provided by this application will be described in detail below in conjunction with the accompanying drawings. Although certain embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be construed as being limited to the embodiments described herein. Instead, these embodiments are provided to provide a more thorough and complete understanding of the present application. It should be understood that the drawings and embodiments of the present application are for illustrative purposes only and are not intended to limit the scope of protection of the present application.

[0042] In the description of the embodiments of this application, the term "including" and similar terms should be understood as open inclusion, that is, "including but not limited to." The term "based on" should be understood as "based at least in part on." The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment." The terms "first," "second," etc. can refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0043] In this application, "at least one" means one or more, and "more" means two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: including the existence of A alone, the existence of A and B at the same time, and the existence of B alone, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can mean: a, b, c, ab, ac, bc, or abc, where a, b, c can be single or multiple.

[0044] With the development of cloud computing technology, more and more users are choosing to store large amounts of data in the cloud and use the data processing systems provided by cloud vendors to operate on this data. For users with high business pressure, to improve the overall read and write concurrency of the data processing system, they often adopt multi-cluster data processing systems, where data is shared across multiple clusters. Providing low-cost, high-performance data writing solutions for multi-cluster data processing systems has become a key technical challenge for major cloud vendors.

[0045] The following introduces several existing data writing solutions for multi-cluster data processing systems.

[0046] Some cloud vendors offer multi-cluster data processing systems that only support data sharing across multiple clusters and fail to meet the requirement for extended clusters to also support data writing. Specifically, when a data table's home cluster (i.e., the original compute cluster) experiences heavy workloads, the newly added extended cluster only supports reading data from that table, not writing to it. Consequently, the newly added extended cluster cannot share the write pressure of the original compute cluster, and the overall data processing system's concurrent write processing capabilities still face performance bottlenecks.

[0047] Some cloud vendors offer multi-cluster data processing systems that can support both extended clusters and data writing. However, they use a metadata management cluster to centrally write the metadata generated by each compute cluster to the metadata storage space. This multi-cluster writing solution based on unified metadata management has the following issues: metadata writing for all compute clusters occurs in the metadata management cluster, which can become a performance bottleneck for the entire data processing system under high business pressure; table permission control information is stored in the metadata management cluster, and a failure of the metadata management cluster can have a wide-ranging impact on the entire data processing system; a separate metadata management cluster must be deployed, as the compute cluster alone cannot complete data writing, resulting in high system initialization costs.

[0048] In order to solve at least one or more of the above problems, the present application provides a data processing method that can meet the requirements of expanding the computing cluster to support data writing. At the same time, the computing cluster to which the table belongs is still responsible for writing the metadata corresponding to the table data. There is no need to deploy a separate metadata management cluster, which reduces system costs and eliminates the performance bottleneck problem of the metadata management cluster.

[0049] FIG1 shows a schematic diagram of the architecture of a data processing system 200 provided in an embodiment of the present application. The following first introduces the application scenario of the data processing method provided in the present application in conjunction with FIG1 .

[0050] As shown in Figure 1, a first computing cluster 204 is initially deployed in data processing system 200. Users can use first computing cluster 204 to write table data. First computing cluster 204 is the home cluster of data table 210, and the data in data table 210 is stored in shared storage 208. For example, when user business pressure is relatively low, after data processing system 200 receives a write request Q1 for writing data A to data table 210 and a write request Q2 for writing data B to data table 210, the write operations for both data A and data B are completed by first computing cluster 204. In this embodiment of the present application, the data write operation can be an Insert operation, an Update operation, or a Delete operation on table data.

[0051] As user business pressure increases, a second computing cluster can be added to data processing system 200 as an extended cluster to handle write operations for newly added business-related data. For example, write request Q1 for writing data A to data table 210 is responded to by first computing cluster 204, while write request Q2 for writing data B to data table 210 is responded to by second computing cluster 206. Specifically, second computing cluster 206 is responsible for writing data B to shared storage and sending metadata 212 generated during the writing of data B to first computing cluster 204. First computing cluster 204 is responsible for writing data A to shared storage and storing metadata 214 generated during the writing of data A. It is also responsible for storing metadata 212 corresponding to received data B.

[0052] In data processing system 200, first computing cluster 204 and second computing cluster 206 can each include at least one data node, and shared storage 208 can be cloud storage such as an Object Storage Service (OBS). Furthermore, data processing system 200 includes a cluster management module 202, which is used to generate execution plans corresponding to write requests and distribute the execution plans to first computing cluster 204 and second computing cluster 206. In practical applications, data management system 200 may include more than two computing clusters, and the cluster management module can be deployed on a coordinator node (CN) in a data warehouse system or database system.

[0053] The following describes a specific implementation of the data processing method provided by the present application in conjunction with FIG1 . The method includes the following steps S301 to S305 :

[0054] S301: A data processing system receives a first write request, where the first write request is used to write first data into a first data table.

[0055] The data processing system includes a first computing cluster, a second computing cluster and a shared storage. The shared storage is used to store data of a first data table. The first data table belongs to the first computing cluster.

[0056] For example, the data processing system 200 receives a write request Q2 regarding data B.

[0057] S302: The data processing system generates an execution plan for responding to a first write request, where the execution plan includes an operator for transmitting metadata across clusters, an operator for writing data, and an operator for storing metadata.

[0058] For example, after receiving the write request Q2, the cluster management module 202 in the data processing system 200 will generate a cross-cluster execution plan for the write request Q2, which includes an operator for writing data B, an operator for transmitting metadata 212 corresponding to data B across clusters, and an operator for storing metadata 212 corresponding to data B.

[0059] Specifically, the cluster management module 202 generates an execution plan corresponding to the write request Q2 and sends it to the first computing cluster 204 and the second computing cluster 206 .

[0060] In an embodiment of the present application, the cluster management module 202 has flexible scheduling capabilities and can select a computing cluster with a load less than that of other computing clusters or a load that does not exceed a preset threshold to perform data writing operations based on the load conditions of each computing cluster in the data management system 200, thereby achieving load balancing of the entire data processing system and avoiding excessive writing pressure on a single computing cluster.

[0061] In one possible embodiment, cluster management module 202 may determine, based on the load pressure of each computing cluster in data management system 200, the computing cluster with the least load in data management system 200 as second computing cluster 206 to execute the data write operation. Accordingly, cluster management module 202 will send the execution plan corresponding to the write request to the data's home cluster and the computing cluster with the least load.

[0062] In one possible embodiment, the data processing system supports load isolation, where different users are bound to different computing clusters. Cluster management module 202 can select the computing cluster corresponding to a user to execute the execution plan corresponding to the user's write request based on the binding relationship between the user and the computing cluster. For example, if data processing system 202 pre-binds user M to second computing cluster 206, then when data processing system 202 receives write request Q2 from user M, cluster management module 202 will send the execution plan corresponding to write request Q2 to second computing cluster 206, which will then be responsible for writing data B belonging to data table 210 to shared storage.

[0063] S303: The second computing cluster writes the first data belonging to the first data table into the shared storage.

[0064] Specifically, the second computing cluster executes the execution plan corresponding to the first write request, so as to write the first data belonging to the first data table into the shared storage through the operator for writing data.

[0065] In a possible embodiment, when writing the first data into the shared storage, the second computing cluster may organize the first data into column storage compression units according to a certain format and then write the data.

[0066] S304: The second computing cluster sends the first metadata corresponding to the first data to the first computing cluster.

[0067] The first metadata corresponding to the first data is metadata generated by the second computing cluster when writing the first data into the shared storage.

[0068] In a possible embodiment, the first metadata includes but is not limited to the location of the column storage compression unit in the shared storage, the size of the column storage compression unit, the maximum value or minimum value of the data in the column storage compression unit, and transaction information.

[0069] Specifically, the second computing cluster executes the execution plan corresponding to the first write request, so as to send the first metadata corresponding to the first data to the first computing cluster through the operator for transmitting metadata across clusters.

[0070] In one possible embodiment, after the second computing cluster writes the first data belonging to the first data table into the shared storage, it also generates index data of the first data table. At this time, the second computing cluster also sends the index data of the first data table to the first computing cluster through the above-mentioned operator for transmitting metadata across clusters.

[0071] S305: The first computing cluster stores first metadata corresponding to the first data.

[0072] Specifically, the first computing cluster writes the first metadata corresponding to the first data into a local computing node of the first computing cluster.

[0073] In one possible embodiment, the first computing cluster also receives index data for the first data table from the second computing cluster. In this case, the first computing cluster also stores the index data corresponding to the first data table. Specifically, the first computing cluster writes the index data corresponding to the first data table to a local computing node of the first computing cluster.

[0074] In this embodiment of the present application, the data writing performance of the first computing cluster is not lower than that of the second computing cluster. The first computing cluster can also process data writing operations and store metadata generated during the data writing operations. For example, the first computing cluster 204 writes data A to the shared storage in response to the write request Q1. Therefore, the data processing method provided by the present application also includes the following steps S401 to S403:

[0075] S401: The data processing system receives a second write request, where the second write request is used to write second data into a first data table.

[0076] S402: The first computing cluster writes the second data belonging to the first data table into the shared storage.

[0077] S403: The first computing cluster stores metadata corresponding to the second data.

[0078] The metadata corresponding to the second data is metadata generated by the first computing cluster when writing the second data belonging to the first data table into the shared storage.

[0079] In practical applications, the data processing method provided in this application also includes a process of obtaining the first data. The following examples introduce several application scenarios.

[0080] In one possible application scenario, the shared storage is also used to store data in a second data table, which belongs to the first computing cluster. The first write request is specifically used to insert first data from the second data table into the first data table. In this scenario, before the second computing cluster writes the first data belonging to the first data table to the shared storage, the process of obtaining the first data from the second data table includes:

[0081] The first computing cluster sends the second metadata corresponding to the first data to the second computing cluster;

[0082] The second computing cluster reads the first data from the second data table from the shared storage according to the second metadata corresponding to the first data.

[0083] In another possible application scenario, the data processing system further includes a third computing cluster, and the shared storage is further used to store data in a third data table. The third data table belongs to the third computing cluster, and the first write request is specifically used to insert first data from the third data table into the first data table. In this scenario, before the second computing cluster writes the first data belonging to the first data table to the shared storage, the process of obtaining the first data from the third data table includes:

[0084] The third computing cluster sends the third metadata corresponding to the first data to the second computing cluster;

[0085] The second computing cluster reads the first data from the third data table from the shared storage according to the third metadata corresponding to the first data.

[0086] In the data processing method provided in the present application, the computing cluster that performs the data writing operation sends the metadata generated during the data writing process to the cluster to which the data belongs. Each computing cluster of the data processing system can support data writing, which helps to improve the write concurrency carrying capacity of the multi-cluster data processing system.

[0087] In addition, the data processing method provided by the present application can make the data processing system have more flexible elastic expansion capabilities. When the user's business pressure becomes greater, an extended cluster can be temporarily added to the data processing system to perform the write operation of the table data belonging to the original computing cluster. At the same time, the cluster to which the table data belongs is still responsible for writing the metadata corresponding to the table data. The extended cluster only needs to remotely transmit the metadata generated during the table data writing process to the cluster to which the table data belongs. Since the extended cluster shares the write operation of the table data, the metadata of the table data is still written by the cluster to which the original table data belongs. In this way, when the business pressure becomes smaller and the extended cluster needs to be removed, there is no need to redistribute the metadata. The cluster to which the table data belongs can directly take over to handle all the business.

[0088] As shown in FIG1 , the present application also provides a data processing system 200, which includes: a cluster management module 202, a first computing cluster 204, a second computing cluster 206, and a shared storage 208.

[0089] The cluster management module 202 is configured to receive a first write request, where the first write request is configured to write first data into a first data table;

[0090] Shared storage 208, used to store data in the first data table;

[0091] The second computing cluster 206 is configured to write first data belonging to a first data table belonging to the first computing cluster 204 into the shared storage 208 ;

[0092] The second computing cluster 206 is further configured to send first metadata corresponding to the first data to the first computing cluster 204 ;

[0093] The first computing cluster 204 is configured to store first metadata corresponding to the first data.

[0094] In an embodiment of the present application, the cluster management module 202 is further used to receive a second write request, where the second write request is used to write the second data into the first data table; the first computing cluster 204 is further used to write the second data belonging to the first data table into the shared storage 208 and store metadata corresponding to the second data.

[0095] In a possible embodiment, the cluster management module 202 is also used to generate an execution plan for responding to the first write request, and the execution plan includes an operator for transmitting metadata across clusters; the second computing cluster 206 is also used to send the first metadata corresponding to the first data to the first computing cluster 204, including: the second computing cluster 206 is used to execute the execution plan to send the first metadata corresponding to the first data to the first computing cluster 204 through the operator for transmitting metadata across clusters.

[0096] In a possible embodiment, the cluster management module 202 is further configured to determine a computing cluster with the smallest load in the data processing system 200 as the second computing cluster 206 .

[0097] In one possible embodiment, the shared storage 208 is also used to store data of a second data table, where the second data table belongs to the first computing cluster 204, and the first data comes from the second data table; before the second computing cluster writes the first data belonging to the first data table to the shared storage 208, the first computing cluster 204 is further used to send second metadata corresponding to the first data to the second computing cluster 206; and the second computing cluster 206 is further used to read the first data from the second data table from the shared storage 208 based on the second metadata corresponding to the first data.

[0098] In one possible embodiment, the data processing system 200 also includes a third computing cluster; the shared storage 208 is further used to store data of a third data table, the third data table belongs to the third computing cluster, and the first data comes from the third data table; before the second computing cluster 206 writes the first data belonging to the first data table into the shared storage 208, the third computing cluster is used to send third metadata corresponding to the first data to the second computing cluster 206; and the second computing cluster 206 is further used to read the first data from the third data table from the shared storage 208 based on the third metadata corresponding to the first data.

[0099] In a possible embodiment, the second computing cluster 206 is further configured to send the index data of the first data table to the first computing cluster 204 ; the first computing cluster 204 is further configured to store the index data of the first data table.

[0100] The data processing system 200 provided in this application can be used to implement the data processing method provided by any possible implementation method in the above method embodiments of this application. Specifically, the specific implementation of the various operations of the data processing method performed by the data processing system 200 can refer to the description of the relevant content in the above method embodiments, and will not be repeated here.

[0101] In the embodiment of the present application, the data processing system 200 can be implemented by software or hardware. The following is an exemplary implementation of the cluster management module 202, which can be used as a reference for the implementation of the first computing cluster 204, the second computing cluster 206, and the shared storage 208.

[0102] A module, as an example of a software functional unit, can include code running on a compute instance. A compute instance can be at least one of a physical host (computing device), a virtual machine, a container, or other computing devices. Furthermore, these computing devices can be one or more. For example, a node can include code running on multiple hosts, virtual machines, or containers. It should be noted that the multiple hosts, virtual machines, or containers running the code can be distributed in the same AZ or different AZs, with each AZ comprising a data center or multiple geographically close data centers. The multiple hosts, virtual machines, or containers running the code can be distributed in the same region or different regions. Typically, a region can include multiple AZs, and a VPC is set up within a region. Cross-region communication between two VPCs within the same region, as well as between VPCs in different regions, requires a communication gateway within each VPC to interconnect the VPCs.

[0103] Similarly, multiple hosts, virtual machines, or containers running the code can be distributed in the same VPC or in multiple VPCs. Typically, a region can include multiple AZs.

[0104] As an example of a hardware functional unit, a module may include at least one computing device, such as a server. Alternatively, a module may be implemented using an ASIC or a PLD. The PLD may be implemented using a CPLD, FPGA, GAL, or any combination thereof.

[0105] The multiple computing devices included in a module can be distributed in the same AZ or different AZs. They can also be distributed in the same region or different regions. Similarly, they can be distributed in the same VPC or across multiple VPCs. These multiple computing devices can be any combination of servers, ASICs, PLDs, CPLDs, FPGAs, GALs, and other computing devices.

[0106] This application also provides a computing device 500. Figure 2 is a schematic diagram of the structure of computing device 500. Computing device 500 includes a bus 502, a processor 504, a memory 506, and a communication interface 508. Processor 504, memory 506, and communication interface 508 communicate with each other via bus 502. Computing device 500 can be a server or a terminal device. It should be understood that this application does not limit the number of processors and memories in computing device 500.

[0107] Bus 502 may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, among others. Buses may be classified as address buses, data buses, control buses, and the like. For ease of illustration, FIG2 illustrates a single bus line, but this does not imply a single bus or type of bus. Bus 502 may include a path for transmitting information between various components of computing device 500 (e.g., memory 506, processor 504, and communication interface 508).

[0108] The processor 504 may include any one or more processors such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0109] The memory 506 may include volatile memory, such as random access memory (RAM). The memory 506 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD).

[0110] Memory 506 stores executable program code. Processor 504 executes this executable program code to implement the functions of cluster management module 202, first computing cluster 204, second computing cluster 206, and shared storage 208 in data processing system 200, thereby implementing the data processing method. In other words, memory 506 stores instructions for executing the data processing method.

[0111] The communication interface 508 uses a transceiver module such as, but not limited to, a network interface card or a transceiver to implement communication between the computing device 500 and other devices or a communication network.

[0112] The present application also provides a computing device cluster, FIG3 is a schematic diagram of the computing device cluster, which can implement the data processing method in the above embodiment. As shown in FIG3, the computing device cluster includes at least one computing device 500 as shown in FIG2. The computing device 500 can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smartphone.

[0113] The memory 506 of one or more computing devices 500 in the computing device cluster may store instructions for executing the data processing method. When at least one computing device in the computing device cluster executes the instructions, the computing device cluster may implement the data processing method described in the embodiment.

[0114] In some possible implementations, the memory 506 of one or more computing devices 500 in the computing device cluster may also store partial instructions for executing the data processing method. In other words, the combination of one or more computing devices 500 can jointly execute the instructions for executing the data processing method.

[0115] It should be noted that the memory 506 in different computing devices 500 in the computing device cluster can store different instructions, each used to execute a portion of the functions of the data processing system. In other words, the instructions stored in the memory 506 in different computing devices 500 can respectively implement the functions of the cluster management module 202, the first computing cluster 204, the second computing cluster 206, or the shared storage 208 in the data processing system.

[0116] In some possible implementations, one or more computing devices in a computing device cluster may be connected via a network. The network may be a wide area network (WAN) or a local area network (LAN), among others. FIG. 4 illustrates one possible implementation. As shown in FIG. 4 , two computing devices 500A and 500B are connected via a network. Specifically, the connection to the network is achieved via a communication interface within each computing device. In this type of possible implementation, the memory 506 within the computing device 500A stores instructions for executing the functions of the cluster management module 202 and the shared storage 208 within the data processing system. Simultaneously, the memory 506 within the computing device 500B stores instructions for executing the functions of the first computing cluster 204 and the second computing cluster 206 within the data processing system.

[0117] It should be understood that the functionality of the computing device 500A shown in FIG4 may also be implemented by multiple computing devices 500. Similarly, the functionality of the computing device 500B may also be implemented by multiple computing devices 500.

[0118] The present application also provides a computer program product comprising instructions. The computer program product may be software or a program product comprising instructions that can be run on a computing device or stored in any available medium. When the computer program product is run on at least one computing device, the computer program product causes the at least one computing device to perform a data processing method.

[0119] The present application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium capable of being stored by a computing device, or a data storage device such as a data center that contains one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a magnetic tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive). The computer-readable storage medium includes instructions that instruct a computing device to execute a data processing method, or instructs a computing device to execute a data processing method.

[0120] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the protection scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A data processing method, characterized in that: The method comprises: A data processing system receives a first write request, the data processing system includes a first computing cluster, a second computing cluster and a shared storage, the shared storage is used to store data of a first data table, the first data table belongs to the first computing cluster, and the first write request is used to write first data into the first data table; The second computing cluster writes the first data belonging to the first data table into the shared storage; The second computing cluster sends first metadata corresponding to the first data to the first computing cluster; The first computing cluster stores first metadata corresponding to the first data.

2. The method according to claim 1, characterized in that The method further comprises: The data processing system receives a second write request, where the second write request is used to write second data into the first data table; The first computing cluster writes the second data belonging to the first data table into the shared storage; The first computing cluster stores metadata corresponding to the second data.

3. The method according to claim 1 or 2, characterized in that: The method further comprises: The data processing system generates an execution plan for responding to the first write request, the execution plan comprising an operator for transmitting metadata across clusters; The second computing cluster sends first metadata corresponding to the first data to the first computing cluster, including: The second computing cluster executes the execution plan to send the first metadata corresponding to the first data to the first computing cluster through the operator for transmitting metadata across clusters.

4. The method according to any one of claims 1 to 3, characterized in that The method further comprises: A computing cluster with the smallest load in the data processing system is determined as the second computing cluster.

5. The method according to any one of claims 1 to 4, characterized in that The shared storage is further used to store data of a second data table, the second data table belongs to the first computing cluster, the first data comes from the second data table, and before the second computing cluster writes the first data belonging to the first data table into the shared storage, the method further includes: The first computing cluster sends second metadata corresponding to the first data to the second computing cluster; The second computing cluster reads the first data from the second data table from the shared storage according to the second metadata corresponding to the first data.

6. The method according to any one of claims 1 to 4, characterized in that The data processing system further includes a third computing cluster, the shared storage is further used to store data of a third data table, the third data table belongs to the third computing cluster, the first data comes from the third data table, and before the second computing cluster writes the first data belonging to the first data table into the shared storage, the method further includes: The third computing cluster sends third metadata corresponding to the first data to the second computing cluster; The second computing cluster reads the first data from the third data table from the shared storage according to the third metadata corresponding to the first data.

7. The method according to any one of claims 1 to 6, characterized in that The method further comprises: The second computing cluster sends the index data of the first data table to the first computing cluster; The first computing cluster stores index data of the first data table.

8. A data processing system, characterized in that: The system comprises: A cluster management module, configured to receive a first write request, wherein the first write request is configured to write first data into a first data table; Shared storage, used to store data in the first data table; a first computing cluster, the first data table belongs to the first computing cluster; a second computing cluster, configured to write first data belonging to the first data table into the shared storage; The second computing cluster is further configured to send first metadata corresponding to the first data to the first computing cluster; The first computing cluster is used to store first metadata corresponding to the first data.

9. The system according to claim 8, characterized in that The cluster management module is further used to receive a second write request, where the second write request is used to write second data into the first data table; The first computing cluster is further configured to write the second data belonging to the first data table into the shared storage; The first computing cluster is further used to store metadata corresponding to the second data.

10. The system according to claim 8 or 9, characterized in that The cluster management module is further used to generate an execution plan for responding to the first write request, wherein the execution plan includes an operator for transmitting metadata across clusters; The second computing cluster is further configured to send first metadata corresponding to the first data to the first computing cluster, including: The second computing cluster is used to execute the execution plan, so as to send the first metadata corresponding to the first data to the first computing cluster through the operator for transmitting metadata across clusters.

11. The system according to any one of claims 8 to 10, characterized in that The cluster management module is further configured to determine a computing cluster with the smallest load in the data processing system as the second computing cluster.

12. The system according to any one of claims 8 to 11, characterized in that The shared storage is further used to store data of a second data table, where the second data table belongs to the first computing cluster, and the first data comes from the second data table; Before the second computing cluster writes the first data belonging to the first data table into the shared storage, the first computing cluster is further configured to send second metadata corresponding to the first data to the second computing cluster; and The second computing cluster is further configured to read the first data from the second data table from the shared storage according to second metadata corresponding to the first data.

13. The system according to any one of claims 8 to 11, characterized in that The data processing system also includes a third computing cluster; The shared storage is further used to store data in a third data table, the third data table belongs to the third computing cluster, and the first data comes from the third data table; Before the second computing cluster writes the first data belonging to the first data table into the shared storage, the third computing cluster is used to send third metadata corresponding to the first data to the second computing cluster; as well as The second computing cluster is further configured to read the first data from the third data table from the shared storage according to third metadata corresponding to the first data.

14. The system according to any one of claims 8 to 13, characterized in that The second computing cluster is further configured to send the index data of the first data table to the first computing cluster; The first computing cluster is further used to store index data of the first data table.

15. A computing device cluster, characterized in that: comprising at least one computing device, each computing device comprising a processor and a memory; The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster executes the data processing method according to any one of claims 1 to 7.

16. A computer program product comprising instructions, characterized in that When the instructions are executed by a computing device cluster, the computing device cluster executes the data processing method according to any one of claims 1 to 7.

17. A computer-readable storage medium, characterized in that: The method comprises computer program instructions. When the computer program instructions are executed by a computing device cluster, the computing device cluster performs the data processing method according to any one of claims 1 to 7.

Images