Distributed database data table processing method, device, equipment and storage medium

Abstract

The present application provides a data table processing method, device, equipment and storage medium for a distributed database; the distributed database includes multiple nodes; the method includes: obtaining a first data table and a second data table to be connected in the distributed database, wherein the data in the first data table and the data in the second data table are dispersed and stored in multiple nodes; obtaining a first data distribution of the first data table and a second data distribution of the second data table; based on the first data distribution and the second data distribution, performing partial replication, partial redistribution and partial unchanged operations on the full amount of data, wherein the full amount of data includes the data in the first data table and the data in the second data table; performing a connection operation on the operated data in the first data table and the operated data in the second data table to obtain a connection result. Through the present application, the data skew problem can be solved, thereby improving the data processing efficiency.

Description

Technical Field

[0001] The present application relates to the field of data processing and database technology, and in particular to a data table processing method, device, equipment and storage medium for a distributed database. Background Art

[0002] Distributed databases usually use smaller computer systems. Each computer (also called a node) can be placed in a separate place. Each computer may have a complete copy or a partial copy of the database management system (DBMS) and its own local database. Many computers located in different locations are interconnected through a network to form a complete, global, logically centralized, and physically distributed large database.

[0003] Data skew is a difficult problem in distributed databases. Under certain data distribution conditions, data skew can cause certain nodes in the distributed database to process hundreds or even thousands of times more data than other nodes, thus slowing down overall processing.

[0004] However, there is no effective solution in related technologies on how to solve the data skew problem in distributed databases. Summary of the invention

[0005] The embodiments of the present application provide a data table processing method, device, electronic device, computer-readable storage medium and computer program product for a distributed database, which can solve the problem of data skew and thereby improve data processing efficiency.

[0006] The technical solution of the embodiment of the present application is implemented as follows:

[0007] The present application embodiment provides a data table processing method of a distributed database, wherein the distributed database includes a plurality of nodes; including:

[0008] Acquire a first data table and a second data table to be connected in the distributed database, wherein the data in the first data table and the data in the second data table are dispersedly stored in the plurality of nodes;

[0009] Acquire a first data distribution of the first data table and a second data distribution of the second data table;

[0010] Based on the first data distribution and the second data distribution, performing operations of partially copying, partially redistributing, and partially keeping unchanged on the full amount of data, wherein the full amount of data includes data in the first data table and data in the second data table;

[0011] A connection operation is performed on the data in the first data table after the operation and the data in the second data table after the operation to obtain a connection result.

[0012] The present application embodiment provides a data table processing device for a distributed database, wherein the distributed database includes a plurality of nodes; including:

[0013] An acquisition module, configured to acquire a first data table and a second data table to be connected in the distributed database, wherein the data in the first data table and the data in the second data table are dispersedly stored in the plurality of nodes;

[0014] The acquisition module is further used to acquire a first data distribution of the first data table and a second data distribution of the second data table;

[0015] an operation module, configured to perform operations of partially copying, partially redistributing, and partially keeping unchanged on the full amount of data based on the first data distribution and the second data distribution, wherein the full amount of data includes data in the first data table and data in the second data table;

[0016] The connection module is used to perform a connection operation on the data in the first data table after the operation and the data in the second data table after the operation to obtain a connection result.

[0017] An embodiment of the present application provides an electronic device, including:

[0018] A memory for storing executable instructions;

[0019] The processor is used to implement the data table processing method of the distributed database provided in the embodiment of the present application when executing the executable instructions stored in the memory.

[0020] An embodiment of the present application provides a computer-readable storage medium storing computer-executable instructions for implementing a data table processing method of a distributed database provided in an embodiment of the present application when executed by a processor.

[0021] An embodiment of the present application provides a computer program product, including a computer program or computer executable instructions, which are used to implement the data table processing method of the distributed database provided in the embodiment of the present application when executed by a processor.

[0022] The embodiments of the present application have the following beneficial effects:

[0023] For the first data table and the second data table to be connected in the distributed database, based on the first data distribution of the first data table and the second data distribution of the second data table, partial replication, partial redistribution and partial unchanged operations are performed on the full data. In this way, the distribution of the full data after the operation in multiple nodes is basically uniform, avoiding the data skew problem caused by copying a large amount of data to the same node when redistributing the full data in the related technology, thereby improving data processing efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 1 is a schematic diagram of the architecture of a data table processing system 100 of a distributed database provided in an embodiment of the present application;

[0025] Figure 2 is a schematic diagram of the structure of an electronic device 600 provided in an embodiment of the present application;

[0026] Figure 3 It is a flowchart of a data table processing method of a distributed database provided in an embodiment of the present application;

[0027] Figure 4 It is a flowchart of a data table processing method of a distributed database provided in an embodiment of the present application;

[0028] Figure 5 It is a schematic diagram of the corresponding relationship between the execution time and the data skew provided in the embodiment of the present application;

[0029] Figure 6 It is a schematic diagram of the principle of a data table processing method of a distributed database provided by the related technology;

[0030] Figure 7 It is a schematic diagram of the principle of a data table processing method of a distributed database provided in an embodiment of the present application;

[0031] Figure 8 It is a schematic diagram of the principle of a data table processing method of a distributed database provided in an embodiment of the present application;

[0032] Fig. 9 It is a principle schematic diagram of a data table processing method of a distributed database provided by related technology;

[0033] Fig.10 It is a schematic diagram of the principle of a data table processing method of a distributed database provided in an embodiment of the present application;

[0034] Fig.11 It is a schematic diagram of the principle of a data table processing method of a distributed database provided in an embodiment of the present application;

[0035] Fig.12It is a schematic diagram of the principle of a data table processing method of a distributed database provided in an embodiment of the present application;

[0036] Fig.13 It is a schematic diagram of the principle of a data table processing method of a distributed database provided in an embodiment of the present application;

[0037] Fig.14 It is a schematic diagram for comparing the effects provided by the embodiments of the present application. DETAILED DESCRIPTION

[0038] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

[0039] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it can be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0040] It is understandable that in the embodiments of the present application, related data such as user information is involved. When the embodiments of the present application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions.

[0041] In the following description, the terms "first\second\..." are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\..." can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0042] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0043] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.

[0044] 1) Distribution Key: A column (or a group of columns) used to determine the database partition where a specific row of data is stored. The column used as the query condition is usually selected as the distribution key, so that nodes can be pruned by the distribution key. If the distribution key is not specified when creating a table, the primary key of the data table is used as the distribution key by default. If the data table does not have a primary key, the first column is used as the distribution key by default.

[0045] 2) Join key: A key used to connect the data in two data tables. For example, for an order table and an order detail table, the order ID can be used as the join key between the two data tables to connect the order data in the two data tables.

[0046] 3) Data table: It is the basic unit for organizing data in a distributed database. A distributed database can include multiple data tables.

[0047] 4) In response to: used to indicate the conditions or states on which the executed operations depend. When the dependent conditions or states are met, one or more operations executed may be in real time or have a set delay. Unless otherwise specified, there is no restriction on the order in which the multiple operations executed are executed.

[0048] 5) Data skew: In a distributed database, some nodes process too much data due to uneven data distribution.

[0049] Data skew is a difficult problem in distributed databases. Under certain data distribution conditions, data skew will cause certain nodes in the distributed database to process hundreds or even thousands of times more data than other nodes, which will cause overall processing to be slow. In addition, there are many reasons for data skew, such as the following: the data in the data table itself is unevenly distributed and skewed; there are high-frequency values ​​corresponding to the distribution key or the connection key; data skew caused by data association after data connection or grouping; data skew caused by specific predicates, such as equal value conditions, etc. In addition, data skew will cause the following serious consequences: long tail phenomenon, such as a node not ending, slowing down the entire query; insufficient resources on a single node, while other nodes have no data processing; obvious query bottlenecks, resulting in huge resource consumption; some nodes may have insufficient content, causing the entire query to fail.

[0050] In view of this, the embodiments of the present application provide a data table processing method, device, electronic device, computer-readable storage medium and computer program product of a distributed database, which can solve the problem of data skew and thus improve data processing efficiency. The electronic device provided by the embodiments of the present application is described below. The electronic device provided by the embodiments of the present application can be implemented as a server, or can be implemented in collaboration with a terminal device and a server. The following is an example of a data table processing method for a distributed database provided by an embodiment of the present application implemented in collaboration with a terminal device and a server.

[0051] For example, see Figure 1 , Figure 1 1 is a schematic diagram of the architecture of a distributed database data table processing system 100 provided in an embodiment of the present application, in order to realize an application supporting solving the data skew problem and improving data processing efficiency, such as Figure 1 As shown, the data table processing system 100 of the distributed database includes: a server 200, a network 300, a terminal device 400, and a distributed database 500, wherein the network 300 can be a local area network or a wide area network, or a combination of the two, the terminal device 400 is a terminal device associated with a user, and a client 410 runs on the terminal device 400. The client 410 can be various types of clients, such as database applications, data table applications, browsers, etc. The distributed database 500 includes multiple nodes, such as node 510, node 520, node 530, etc.

[0052] In some embodiments, when a user needs to connect two data tables in the distributed database 500, the user can enter the identifiers corresponding to the two data tables in the human-computer interaction interface of the client 410. After receiving the data table identifier entered by the user, the client 410 can send the received data table identifier to the server 200 through the network 300, so that the server 200 can obtain the first data table (for example, data table A) and the second data table (for example, data table B) corresponding to the data table identifier from the distributed database 500 based on the received data table identifier. Then, the server 200 can also obtain the first data distribution of the first data table (for example, whether the distribution key of the first data table is a connection key, and the data corresponding to the connection key in the first data table is The server 200 may determine whether the first and second data tables in the first data table include high-frequency data, whether the first and second data tables in the second data table include high-frequency data, etc.) and the second data distribution of the second data table (for example, whether the distribution key of the second data table is a connection key, whether the data corresponding to the connection key in the second data table include high-frequency data, etc.). Subsequently, the server 200 may perform partial replication, partial redistribution, and partial unchanged operations on the full amount of data consisting of the data in the first data table and the data in the second data table based on the first data distribution and the second data distribution. Finally, the server 200 performs a connection operation on the operated data in the first data table and the operated data in the second data table to obtain a connection result, and returns the connection result to the terminal device 400 through the network 300, so that the terminal device 400 calls the human-computer interaction interface of the client 410 for presentation.

[0053] It should be noted that the data table processing method of the distributed database provided in the embodiment of the present application can also be implemented by a distributed database. For example, when connecting the first data table and the second data table, the data redistribution operation and the connection operation can be realized through the underlying capabilities of the distributed database. The embodiment of the present application does not make specific limitations on this.

[0054] In other embodiments, the embodiments of the present application can also be implemented with the aid of cloud technology. Cloud technology refers to a hosting technology that unifies a series of resources such as hardware, software, and networks within a wide area network or a local area network to achieve data calculation, storage, processing, and sharing.

[0055] Cloud technology is a general term for network technology, information technology, integration technology, management platform technology, and application technology based on the cloud computing business model. It can form a resource pool that can be used on demand and is flexible and convenient. Cloud computing technology will become an important support. The background services of the technical network system require a large amount of computing and storage resources.

[0056] For example, Figure 1The server 200 in the example may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content distribution networks (CDNs), and big data and artificial intelligence platforms. The terminal device 400 may be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, a vehicle-mounted terminal, etc., but is not limited thereto. The terminal device 400 and the server 200 may be directly or indirectly connected via wired or wireless communication, which is not limited in the embodiments of the present application.

[0057] The structure of the electronic device provided in the embodiment of the present application is further described below. Taking the electronic device as a server as an example, see Figure 2 , Figure 2 is a schematic diagram of the structure of an electronic device 600 provided in an embodiment of the present application, Figure 2 The electronic device 600 shown includes: at least one processor 610, a memory 640, and at least one network interface 620. The various components in the electronic device 600 are coupled together via a bus system 630. It is understood that the bus system 630 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 630 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, the bus system 630 is not described in detail. Figure 2 Various buses are labeled as bus system 630 .

[0058] The processor 610 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0059] The memory 640 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard drives, optical drives, etc. The memory 640 may optionally include one or more storage devices that are physically remote from the processor 610.

[0060] The memory 640 includes a volatile memory or a non-volatile memory, and may also include both volatile and non-volatile memories. The non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 640 described in the embodiments of the present application is intended to include any suitable type of memory.

[0061] In some embodiments, memory 640 is capable of storing data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as exemplarily described below.

[0062] Operating system 641, including system programs for processing various basic system services and performing hardware-related tasks, such as a framework layer, a core library layer, a driver layer, etc., for implementing various basic services and processing hardware-based tasks;

[0063] A network communication module 642, for reaching other computing devices via one or more (wired or wireless) network interfaces 620, exemplary network interfaces 620 include: Bluetooth, Wireless Compatibility Authentication (WiFi), and Universal Serial Bus (USB);

[0064] In some embodiments, the device provided in the embodiments of the present application can be implemented in software. Figure 2 The data table processing device 643 of the distributed database stored in the memory 640 is shown, which can be software in the form of programs and plug-ins, including the following software modules: an acquisition module 6431, an operation module 6432, a connection module 6433 and a determination module 6434. These modules are logical, so they can be arbitrarily combined or further split according to the functions implemented. It should be pointed out that in Figure 2 For the sake of convenience, all the above modules are shown at once, but it should not be regarded as that the data table processing device 643 of the distributed database excludes the implementation that can only include the acquisition module 6431, the operation module 6432 and the connection module 6433. The functions of each module will be explained below.

[0065] The data table processing method of the distributed database provided in the embodiment of the present application will be specifically described below in combination with the exemplary application and implementation of the server provided in the embodiment of the present application.

[0066] For example, see Figure 3 , Figure 3 is a flow chart of a data table processing method of a distributed database provided in an embodiment of the present application, which will be combined with Figure 3 The steps shown are explained.

[0067] In step 101, a first data table and a second data table to be connected in a distributed database are obtained.

[0068] Here, the data in the first data table and the data in the second data table are dispersedly stored in multiple nodes of the distributed database.

[0069] In some embodiments, taking the first data table (e.g., data table A) as an example, after selecting one or more columns from the first data table as distribution keys, the data columns can be hashed according to the distribution keys, and then the data of the column can be stored in the node corresponding to the calculated hash value. For example, you can first use CdbHash*makeCdbHash(intnumsegs, int natts, Oid*hashfuncs) to create a CdbHash structure, and then perform the following operations on each column (tuple) to calculate the hash value corresponding to the column and determine which node (segment) the data of the column should be distributed to: first perform an initialization operation, which is just to initialize the initial hash value, then call the hashDatum() function to perform different preprocessing for different types, then add the processed column value to the hash calculation, and finally map the hash value to a node.

[0070] It should be noted that, in addition to distributing the data in the first data table to multiple nodes of the distributed database by hash distribution, the data in the first data table can also be distributed to multiple nodes by random distribution. For example, the data in the first data table can be randomly distributed to different nodes of the distributed database. In this case, the same data will be stored in different nodes, and the embodiments of the present application do not specifically limit this.

[0071] In addition, it should be noted that the first data table and the second data table in the embodiment of the present application can be the connection result after connecting multiple (2 or more) data tables. For example, when it is necessary to connect data table A, data table B and data table C in sequence, data table A and data table B can be connected first to obtain the connection result (corresponding to the first data table), and then the connection result can be continued to be connected with data table C (corresponding to the second data table). The embodiment of the present application does not make specific limitations on this.

[0072] In step 102, a first data distribution of a first data table and a second data distribution of a second data table are obtained.

[0073] In some embodiments, the first data distribution of the first data table may refer to whether the distribution key of the first data table is a join key, and whether the data corresponding to the join key in the first data table includes high-frequency data; similarly, the second data distribution of the second data table may refer to whether the distribution key of the second data table is a join key, and whether the data corresponding to the join key in the second data table includes high-frequency data.

[0074] For example, taking the first data table as data table A and the second data table as data table B, the data distribution of data table A and the data distribution of data table B can be obtained respectively, for example, whether the distribution key of data table A is a join key, and whether the data corresponding to the join key in data table A includes high-frequency data, and whether the distribution key of data table B is a join key, and whether the data corresponding to the join key in data table B includes high-frequency data.

[0075] It should be noted that when connecting the first data table (e.g., data table A) and the second data table (e.g., data table B), a connection key needs to be specified, and then it can be determined whether the connection key specified by the user is the same key as the distribution key used when the data is distributed and stored. For example, if they are the same key, it means that the distribution key is the connection key; if they are not the same key, it means that the distribution key is not the connection key. In addition, the process of determining whether the data table includes high-frequency data is: determining whether the number of occurrences of the data corresponding to the connection key in the data table is greater than the number threshold. If yes, it means that the data table includes high-frequency data; if no, it means that the data table does not include high-frequency data.

[0076] For example, take data table A as the order table and data table B as the order details table. Assume that the order table uses the order number as the distribution key to store data in multiple nodes, and the order details table uses the product name as the distribution key to store data in multiple nodes. At the same time, assuming that the order table and the order details table need to be connected according to the product name (i.e., the connection key), it can be determined that the distribution key of the order table is not the connection key, and the distribution key of the order details table is the connection key. In addition, taking the order table as an example, when the connection key is the product name, it can be determined whether there is high-frequency data in the order table based on the number of occurrences of the data corresponding to the connection key in the order table (such as the purchase records corresponding to different products). For example, assuming that the number of occurrences of the purchase record of product A in the order table is greater than the number threshold (such as 5000), for example, assuming that there are 6000 purchase records of product A in order table A, it can be determined that the order table contains high-frequency data.

[0077] In step 103, based on the first data distribution and the second data distribution, operations of partially copying, partially redistributing, and partially keeping unchanged are performed on the full amount of data.

[0078] Here, the full amount of data includes data in the first data table and data in the second data table.

[0079] In some embodiments, see Figure 4 , Figure 4 is a flow chart of a data table processing method of a distributed database provided in an embodiment of the present application, such as Figure 4 As shown, Figure 3 Step 103 shown may be performed by Figure 4 Steps 1031 and 1032 shown in the figure are implemented by combining Figure 4 The steps shown are explained.

[0080] In step 1031, based on the first data distribution and the second data distribution, first data in the full data for performing a copy operation, second data in the full data for performing a redistribution operation, and third data in the full data that remains unchanged are determined.

[0081] It should be noted that the first data, second data, and third data in the embodiment of the present application do not refer to a specific piece of data, but are a general term for a certain type of data. That is, in the embodiment of the present application, the multiple pieces of data used to perform a copy operation are collectively referred to as first data, the multiple pieces of data used to perform a redistribution operation are collectively referred to as second data, and the multiple pieces of data whose distribution method remains unchanged are collectively referred to as third data.

[0082] In some embodiments, step 1031 can be implemented in the following manner: in response to the fact that the distribution keys of the first data table and the second data table are not join keys, and the data corresponding to the join keys in one of the data tables include high-frequency data, wherein the high-frequency data is data that appears more than a threshold number of times (e.g., 5000), the following processing is performed: data matching the high-frequency data in the non-target data table (e.g., data identical to the high-frequency data) is used as the first data for performing the copy operation, wherein the non-target data table is a data table that does not include the high-frequency data in the first data table and the second data table; data in the target data table other than the high-frequency data and data in the non-target data table other than the data matching the high-frequency data are used as the second data for performing the redistribution operation, wherein the target data table is a data table that includes the high-frequency data in the first data table and the second data table; and the high-frequency data is used as the third data that remains unchanged.

[0083] For example, taking the first data table as data table A and the second data table as data table B, assuming that the distribution keys of data table A and data table B are not join keys, and the data corresponding to the join keys in data table A include high-frequency data, then the data matching the high-frequency data in data table B can be used as the first data for performing the copy operation. For example, assuming that the high-frequency data in data table A is (1, 3, 5), where "1" appears 5100 times in data table A, "3" appears 5500 times in data table A, and "5" appears 5400 times in data table A, then (1, 3, 5) in data table B can be used as the data matching the high-frequency data in data table A, that is, for performing the copy operation. The first data for the row copy operation; the data in data table A except the high-frequency data and the data in data table B except the data matching the high-frequency data are used as the second data for performing the redistribution operation, that is, the data in data table A except (1, 3, 5) and the data in data table B except (1, 3, 5) are used as the second data for performing the redistribution operation; (1, 3, 5) in data table A is used as the third data with the distribution method unchanged, so that the high-frequency data is not redistributed, avoiding redistribution of a large amount of data to the same node; at the same time, since the amount of high-frequency data is large, the high-frequency data is not redistributed, which also reduces data movement.

[0084] In some other embodiments, the above-mentioned step 1031 can also be implemented in the following manner: in response to the fact that the distribution keys of the first data table and the second data table are not join keys, the data corresponding to the join key in the first data table include first high-frequency data, and the data corresponding to the join key in the second data table include second high-frequency data, wherein the first high-frequency data is data in the first data table whose number of appearances is greater than a number threshold, and the second high-frequency data is data in the second data table whose number of appearances is greater than a number threshold, and there is no overlap between the first high-frequency data and the second high-frequency data, the following processing is performed: the data in the first data table that matches the second high-frequency data and the data in the second data table that matches the first high-frequency data are used as the first data for performing the copy operation; the data in the first data table other than the first high-frequency data and the data matching the second high-frequency data and the data in the second data table other than the second high-frequency data and the data matching the first high-frequency data are used as the second data for performing the redistribution operation; the first high-frequency data and the second high-frequency data are used as the third data that remain unchanged.

[0085] For example, taking the first data table as data table A and the second data table as data table B, assuming that the distribution keys of data table A and data table B are not join keys, the data corresponding to the join key in data table A includes the first high-frequency data, the data corresponding to the join key in data table B includes the second high-frequency data, and there is no overlap between the first high-frequency data and the second high-frequency data, then the data matching the second high-frequency data in data table A and the data matching the first high-frequency data in data table B can be used as the first data for performing the copy operation; the data in data table A except the first high-frequency data and the data matching the second high-frequency data and the data in data table B except the second high-frequency data and the data matching the first high-frequency data can be used as the second data for performing the redistribution operation; the first high-frequency data and the second high-frequency data can be used as the third data with the distribution method remaining unchanged. For example, assuming that the first high-frequency data in data table A is (1, 3, 5) and the second high-frequency data in data table B is (2, 4), then (2, 4) in data table A can be used as data matching the second high-frequency data, and (1, 3, 5) in data table B can be used as data matching the first high-frequency data; the data in data table A except (1, 2, 3, 4, 5) and the data in data table B except (1, 2, 3, 4, 5) can be used as the second data for performing the redistribution operation; (1, 3, 5) in data table A and (2, 4) in data table B can be used as the third data that remains unchanged.

[0086] In some embodiments, the above-mentioned step 1031 can also be implemented in the following manner: in response to the distribution key of one of the first data table and the second data table being a join key, the distribution key of the other data table is not a join key, and the data corresponding to the join key in the non-target data table includes high-frequency data, wherein the non-target data table is a data table in the first data table and the second data table whose distribution key is not a join key, and the high-frequency data is data in the non-target data table whose number of appearances is greater than a number threshold, the following processing is performed: using the data in the target data table that matches the high-frequency data as the first data for performing the copy operation, wherein the target data table is the data table in the first data table and the second data table whose distribution key is a join key; using the data in the non-target data table other than the high-frequency data as the second data for performing the redistribution operation; using the high-frequency data and the data in the target data table other than the data that matches the high-frequency data as the third data that remains unchanged.

[0087] For example, taking the first data table as data table A and the second data table as data table B as an example, assuming that the distribution key of data table A is a connection key, the distribution key of data table B is not a connection key, and the data corresponding to the connection key in data table B includes high-frequency data, then the data matching the high-frequency data in data table A can be used as the first data for performing the copy operation; the data other than the high-frequency data in data table B can be used as the second data for performing the redistribution operation; the high-frequency data and the data other than the data matching the high-frequency data in data table A can be used as the third data with the distribution method unchanged. For example, assuming that the high-frequency data is (1, 3, 5), then (1, 3, 5) in data table A can be used as the data matching the high-frequency data; the data other than (1, 3, 5) in data table B can be used as the second data for performing the redistribution operation; and the data (1, 3, 5) in data table B and the data other than (1, 3, 5) in data table A can be used as the third data with the distribution method unchanged.

[0088] In some other embodiments, the above-mentioned step 1031 can also be implemented in the following manner: in response to the fact that the distribution keys of the first data table and the second data table are not join keys, the data corresponding to the join key in the first data table includes first high-frequency data, the data corresponding to the join key in the second data table includes second high-frequency data, and there is overlap between the first high-frequency data and the second high-frequency data, the following processing is performed: the data in the first data table and the second data table that matches the high-frequency data in the non-overlapping part is used as the first data for performing the copy operation; the high-frequency data in the overlapping part and the remaining data in the first data table and the second data table are used as the second data for performing the redistribution operation, wherein the remaining data is the data in the first data table and the second data table except the first high-frequency data, the second high-frequency data, and the data matching the high-frequency data in the non-overlapping part; the high-frequency data in the non-overlapping part is used as the third data that remains unchanged.

[0089] For example, take the first data table as data table A and the second data table as data table B. Assume that the distribution keys of data table A and data table B are not join keys, the data corresponding to the join key in data table A includes the first high-frequency data, assuming it is (1, 3, 5), and the data corresponding to the join key in data table B includes the second high-frequency data, assuming it is (2, 3, 4), and there is overlap between the first high-frequency data and the second high-frequency data, that is, data (3). Then, the data matching the high-frequency data of the non-overlapping part in data table A and data table B can be used as the first data for performing the copy operation, that is, (2, 4) in data table A and (2, 4) in data table B. (1, 5) is used as the first data for performing the copy operation; the high-frequency data of the overlapping part and the remaining data in data table A and data table B are used as the second data for performing the redistribution operation, that is, the high-frequency data (3) of the overlapping part, the data other than (1, 2, 3, 4, 5) in data table A, and the data other than (1, 2, 3, 4, 5) in data table B are used as the second data for performing the redistribution operation; the high-frequency data of the non-overlapping part is used as the third data that remains unchanged, that is, (1, 5) in data table A and (2, 4) in data table B are used as the third data whose distribution method remains unchanged.

[0090] It should be noted that the high-frequency data in the embodiments of the present application does not refer to a specific piece of data, but is a general term for a certain type of data. For example, data that appears in a data table more than a number threshold can be collectively referred to as high-frequency data. In other words, high-frequency data is a collection of data that appears more than a number threshold.

[0091] In some other embodiments, the above-mentioned step 1031 can also be implemented in the following manner: in response to the distribution key of one of the first data table and the second data table being a join key, the distribution key of the other data table is not a join key, and the data corresponding to the join key in the target data table includes high-frequency data, wherein the target data table is a data table in the first data table and the second data table whose distribution key is a join key, and the high-frequency data is data in the target data table whose number of occurrences is greater than a number threshold, the following processing is performed: the high-frequency data and the data matching the high-frequency data in the non-target data table are used as the first data for performing the copy operation, wherein the non-target data table is a data table in the first data table and the second data table whose distribution key is not a join key; the data in the non-target data table except the data matching the high-frequency data is used as the second data for performing the redistribution operation; the data in the target data table except the high-frequency data is used as the third data that remains unchanged.

[0092] For example, taking the first data table as data table A and the second data table as data table B as an example, assuming that the distribution key of data table A is a connection key, the distribution key of data table B is not a connection key, and the data corresponding to the connection key in data table A includes high-frequency data, then the high-frequency data and the data matching the high-frequency data in data table B can be used as the first data for performing the copy operation; the data in data table B other than the data matching the high-frequency data can be used as the second data for performing the redistribution operation; and the data in data table A other than the high-frequency data can be used as the third data that remains unchanged. For example, taking the high-frequency data as (1, 3, 5) as an example, then (1, 3, 5) in data table A and (1, 3, 5) in data table B can be used as the first data for performing the copy operation; the data in data table B other than (1, 3, 5) can be used as the second data for performing the redistribution operation; and the data in data table A other than (1, 3, 5) can be used as the third data whose distribution method remains unchanged.

[0093] In step 1032, a copy operation is performed on the first data, a redistribution operation is performed on the second data, and the third data remains unchanged.

[0094] In some embodiments, for the above-mentioned first data table and the second data table, the distribution keys are not join keys, and the data corresponding to the join key in one of the data tables includes high-frequency data; or, the distribution keys of the first data table and the second data table are not join keys, the data corresponding to the join key in the first data table includes the first high-frequency data, the data corresponding to the join key in the second data table includes the second high-frequency data, and there is no overlap between the first high-frequency data and the second high-frequency data; or, in the first data table and the second data table, the distribution key of one data table is a join key, the distribution key of the other data table is not a join key, and the data corresponding to the join key in a non-target data table (i.e., a data table whose distribution key is not a join key) includes high-frequency data, the above-mentioned step 1032 can be implemented in the following manner: copy the first data to each node; redistribute the second data in multiple nodes according to the join key; and keep the distribution method of the third data in multiple nodes unchanged.

[0095] For example, for the first data identified from the full data, the first data can be sent to all nodes by broadcasting, so that a complete copy of the first data is stored in each node of the distributed database; for the second data identified from the full data, the second data can be redistributed to multiple nodes according to the connection key; of course, the data in one data table can be kept unchanged, and the data in another data table can be copied to all nodes; for the third data, the distribution method of the third data in multiple nodes can be kept unchanged.

[0096] In other embodiments, for the case where the distribution keys of the first data table and the second data table are not join keys, the data corresponding to the join key in the first data table include first high-frequency data, the data corresponding to the join key in the second data table include second high-frequency data, and there is overlap between the first high-frequency data and the second high-frequency data, the above-mentioned step 1032 can be implemented in the following manner: copying the first data to each node; redistributing the high-frequency data of the overlapping part in the second data to multiple nodes so that the high-frequency data of the overlapping part is evenly distributed in multiple nodes, and redistributing the remaining data in the second data among multiple nodes according to the join key; and keeping the distribution method of the third data among multiple nodes unchanged.

[0097] For example, for the first data identified from the full amount of data, the first data can be sent to all nodes of the distributed database by broadcasting, so that a complete copy of the first data is stored in each node.

[0098] For example, for the second data identified from the full amount of data, where the second data includes the high-frequency data of the overlapping part and the remaining data (i.e., the data in the first data table and the second data table except the first high-frequency data, the second high-frequency data, and the data matching the high-frequency data of the non-overlapping part), for the high-frequency data of the overlapping part, the high-frequency data of the overlapping part can be redistributed to multiple nodes according to the set rules, so that the high-frequency data of the overlapping part is evenly distributed in the multiple nodes. For example, the high-frequency data of the overlapping part can be redistributed to multiple nodes by polling replication. For example For example, if it is assumed that the overlapping high-frequency data includes 500 data items and the multiple nodes are 10 nodes, the first data item of the 500 data items can be sent to the first node (that is, the first data item is stored in the first node), the second data item is sent to the second node (that is, the second data item is stored in the second node), and so on, the tenth data item is sent to the tenth node, and for the eleventh data item, the eleventh data item can be resent to the first node. In this way, the high-frequency data of the overlapping part can be evenly distributed in multiple nodes to avoid data skew problems.

[0099] As for the remaining data included in the second data, the remaining data can be redistributed to multiple nodes according to the connection key. For example, the connection key can be used as a classification standard to classify the remaining data, and different types of data can be stored in different nodes, that is, one node only stores one type of data. For example, taking the connection key as the name of the goods, the remaining data (such as the purchase records corresponding to multiple goods) can be first classified according to the name of the goods, and the purchase records of different types of goods can be stored in different nodes, for example, the purchase records of fresh goods can be stored in the first node, the purchase records of clothing goods can be stored in the second node, and the purchase records of electronic products can be stored in the third node, that is, one node only stores the purchase records of one type of goods.

[0100] For example, for the third data identified from the full amount of data, the distribution of the third data in the multiple nodes is kept unchanged.

[0101] In some embodiments, for the case where the distribution key of one of the first data table and the second data table is a connection key, the distribution key of the other data table is not a connection key, and the data corresponding to the connection key in the target data table includes high-frequency data, the above-mentioned step 1032 can be implemented in the following way: polling and copying the high-frequency data in the first data, and copying the data in the first data that matches the high-frequency data to each node; redistributing the second data among multiple nodes according to the connection key; and keeping the distribution method of the third data among multiple nodes unchanged.

[0102] For example, the above-mentioned polling and replication of high-frequency data in the first data can be implemented in the following manner: let n be an integer variable whose value increases from 1, the maximum value of n is N, N is the total number of data included in the high-frequency data, and iterate n to perform the following processing: determine the remainder obtained by dividing n by M, and copy the nth data to the node corresponding to the remainder in multiple nodes, where M is the total number of multiple nodes, and when the remainder is 0, copy the data corresponding to the remainder to the Mth node.

[0103] In some other embodiments, the following processing may also be performed: in response to the data volume of one of the first data table and the second data table being greater than a data volume threshold, and the data volume of the other data table being less than the data volume threshold, the data in the data table whose data volume is less than the data volume threshold is copied to each node.

[0104] For example, taking the first data table as data table A and the second data table as data table B, assuming that the data volume of data table A is greater than the data volume threshold, and the data volume of data table B is less than the data volume threshold, that is, data table A is a large table and data table B is a small table, then the data in data table B can be copied to all nodes of the distributed database, that is, a complete copy of the data in data table B is stored in each node.

[0105] In step 104, a connection operation is performed on the operated data in the first data table and the operated data in the second data table to obtain a connection result.

[0106] In some embodiments, after partially copying, partially redistributing, and partially keeping unchanged the full amount of data based on the first data distribution and the second data distribution, a connection operation can be performed on the operated data in the first data table and the operated data in the second data table to obtain a connection result. When it is necessary to connect with other data tables (such as a third data table) later, the connection result can be connected with the third data table. The specific connection process can be implemented with reference to the above, and the embodiments of the present application will not be repeated here.

[0107] The data table processing method of a distributed database provided in an embodiment of the present application performs partial replication, partial redistribution, and partial unchanged operations on the full data for the first data table and the second data table to be connected in the distributed database based on the first data distribution of the first data table and the second data distribution of the second data table. In this way, the distribution of the full data after the operation in multiple nodes is basically uniform, avoiding the data skew problem caused by copying a large amount of data to the same node when redistributing the full data in the related technology, thereby improving the data processing efficiency.

[0108] The following is an explanation of an exemplary application of an embodiment of the present application in a practical application scenario.

[0109] Data skew is a difficult problem in distributed databases. Under certain data distribution conditions, data skew can cause certain nodes in the distributed database to process hundreds or even thousands of times more data than other nodes, resulting in slow overall processing. For example, a general database test set (such as TPCDS) is tested, and the results of the data skew test are as follows: Figure 5 As shown in the figure, the left figure shows the distribution of query data skew in TPCDS, and the right figure shows the corresponding relationship between data skew and execution time. Figure 5 It can be seen that a considerable number of queries with long execution time are related to data skew. Therefore, if the problem of data skew is properly handled, query efficiency can be effectively improved.

[0110] In view of this, an embodiment of the present application provides a data table processing method for a distributed database. For two data tables that need to be connected, while keeping most of the data distribution methods unchanged, the data in the two data tables are redistributed according to different data distributions by partially copying, partially redistributing, and partially keeping unchanged (keep local), thereby solving the problem of data skew.

[0111] The following is a detailed description of the distributed database data table processing method provided in the embodiment of the present application.

[0112] In some embodiments, since the data skew situation is very complicated, the embodiments of the present application will handle the data skew situation through the following categories.

[0113] 1) The distribution keys of the two data tables to be connected (or the connection results) are not the connection keys, and the connection key of one of the data tables (or the connection results) includes the Most Common Value (MCV);

[0114] 2) The distribution keys of the two data tables to be connected (or the connection results) are not the connection keys, and the connection keys of the two data tables (or the connection results) contain high-frequency values ​​(that is, the values ​​that appear more than the threshold number of times in a column, corresponding to the high-frequency data mentioned above), and the high-frequency values ​​do not overlap;

[0115] 3) The distribution keys of the two data tables (or the connection results) to be connected are not the connection keys, and the connection keys of the two data tables (or the connection results) both contain high-frequency values, and the high-frequency values ​​overlap;

[0116] 4) The distribution key of one of the data tables (or the connection result) is the connection key, and the distribution key of the other data table (or the connection result) is not the connection key, and the data table (or the connection result) on the connection key side of the distribution key includes high-frequency values;

[0117] 5) The distribution key of one of the data tables (or the connection result) is the connection key, and the distribution key of the other data table (or the connection result) is not the connection key, and the data table (or the connection result) on the side where the distribution key is not the connection key includes high-frequency values.

[0118] The following is a detailed description of the above five situations.

[0119] In some embodiments, see Figure 6 , Figure 6 It is a schematic diagram of the principle of the data table processing method of the distributed database provided by the relevant technology, such as Figure 6As shown, for the case where the distribution keys of the data tables on both sides of the connection (or the connection results) are not connection keys, and the connection key of one side of the data table (or the connection result) contains high-frequency values, when distributing according to the hash function, the high-frequency values ​​will be distributed to the same node.

[0120] For example, see Figure 7 , Figure 7 Schematic diagram of the principle of the data table processing method of the distributed database provided in the embodiment of the present application. Figure 7 As shown, assuming that the left table includes high-frequency values, for the above situation, the improved solution provided by the embodiment of the present application includes the following parts: For the processing of high-frequency values: For the data table including the high-frequency value side, since the distribution key is not a connection key, it can be basically assumed that the high-frequency value is evenly distributed on multiple nodes, so the high-frequency value can be redistributed without redistribution, and the data matching the high-frequency value in the database on the other side is broadcast to all nodes. For example, assuming that the data table on the left keeps the high-frequency value (1, 3, 5) locally, it is necessary to copy (1, 3, 5) in the data table on the right to all nodes. This has the following advantages: First, there is no redistribution of high-frequency values, avoiding redistribution of a large amount of high-frequency data to the same node to cause data skew; second, since the amount of high-frequency value data is large, no redistribution is performed, which can reduce data movement. In addition, for the processing of non-high-frequency values, since non-high-frequency values ​​are evenly distributed, they can be redistributed according to the connection key, or one side remains unchanged and the other side copies the data to all nodes to ensure even distribution.

[0121] It should be noted that for the same value on any side, it is necessary to correspond to all the matching values ​​on the other side. When one side performs polling distribution, the data distribution can be guaranteed to be uniform. Since the data is replicated on the other side, it can be guaranteed that the same value can be matched once and only once, ensuring the correctness of the data. In addition, since it is necessary to ensure that all data with connection relationships have the opportunity to connect, for high-frequency values, one side is kept locally and the other side performs full-node replication, which can ensure that for high-frequency values, all the same values ​​in the connection column have the opportunity to connect; for other values, all values ​​are redistributed to the same node, so the same values ​​in the connection column also have the opportunity to connect. Therefore, although the connection column has replicated values, local values, and distributed values ​​at the same time, it can ensure that the same values ​​in the connection column have the opportunity to connect, thereby ensuring the correctness of the result set.

[0122] In other embodiments, see Figure 8 , Figure 8 Schematic diagram of the principle of the data table processing method of the distributed database provided in the embodiment of the present application. Figure 8As shown, for the case where the distribution keys of the data tables (or connection results) on both sides of the connection are not connection keys, the connection keys of the data tables (or connection results) on both sides contain high-frequency values, and the high-frequency values ​​do not overlap, the improvement scheme provided by the embodiment of the present application includes the following parts: For the processing of high-frequency values: For the high-frequency values ​​on both sides, since the connection key is not the distribution key and there is no overlap between the high-frequency values, the high-frequency values ​​on this side can also be kept locally, and the opposite side needs to copy the data matching the high-frequency values ​​to all nodes. For example, assuming that the high-frequency values ​​included in the data table on the left are (1, 3, 5) and the high-frequency values ​​included in the data table on the right are (2, 4), then for the data table on the left, the high-frequency values ​​(1, 3, 5) can be kept locally, and (1, 3, 5) in the data table on the right can be copied to all nodes; similarly, for the data table on the right, after keeping the high-frequency values ​​(2, 4) locally, it is necessary to copy (2, 4) in the data table on the left to all nodes. For the processing of non-high-frequency values, since non-high-frequency values ​​are evenly distributed, non-high-frequency values ​​can be redistributed to all nodes according to the connection key, which can ensure average distribution.

[0123] In some embodiments, see Fig. 9 , Fig. 9 It is a schematic diagram of the principle of the data table processing method of the distributed database provided by the relevant technology, such as Fig. 9 As shown, the distribution keys of the data tables on both sides of the connection (or the connection results) are not the connection keys. The connection keys of the data tables on both sides (or the connection results) contain high-frequency values, and the high-frequency values ​​overlap. When the high-frequency values ​​are redistributed, the data of a certain node will expand, causing the typical problem of data skew; when the high-frequency values ​​are copied, the amount of copied data will be too large; in addition, the amount of data processed by each node will also be amplified.

[0124] For example, see Fig.10 , Fig.10 Schematic diagram of the principle of the data table processing method of the distributed database provided in the embodiment of the present application. Fig.10 As shown, for the above situation, the improvement scheme provided by the embodiment of the present application includes the following parts: For the processing of high-frequency values: For high-frequency values ​​on both sides, since the connection key is not a distribution key and there is overlap, redistribution or data replication will have limitations. Therefore, the data of the overlapping part of the high-frequency values ​​is rearranged, for example, the high-frequency values ​​of the overlapping part are redistributed to different nodes according to the rules, so that the high-frequency values ​​go to different nodes for processing. In this way, when formulating the execution plan, the high-frequency values ​​are rearranged so that the data distribution is basically balanced to ensure that the amount of data processed by each node is basically average. In other words, the advantage of rearrangement is that after the data distribution is rearranged, the amount of data processed by each node is as balanced as possible, reducing the impact of data skew.

[0125] In other embodiments, see Fig.11 , Fig.11 Schematic diagram of the principle of the data table processing method of the distributed database provided in the embodiment of the present application. Fig.11 As shown, for the case where the distribution key of the data table (or connection result) on one side is the connection key, and the data table (or connection result) on the side where the connection key is not the distribution key contains high-frequency values, the improvement scheme provided by the embodiment of the present application mainly includes the following parts: For the processing of high-frequency values: the high-frequency values ​​of the data table (or connection result) on the side where the connection key is the distribution key are kept locally, and the data matching the high-frequency values ​​in the data table (or connection result) on the side where the connection key is not the distribution key need to be copied to all nodes. For the processing of non-high-frequency values, since the non-high-frequency values ​​are evenly distributed, the non-high-frequency values ​​in the data table on the side where the distribution key is the connection key can be kept locally, and the non-high-frequency values ​​in the data table on the other side can be redistributed according to the connection key.

[0126] In some embodiments, see Fig.12 , Fig.12 Schematic diagram of the principle of the data table processing method of the distributed database provided in the embodiment of the present application. Fig.12 As shown, for the case where the distribution key of the data table (or connection result) on one side is the connection key, and the connection key includes high-frequency values ​​in the data table (or connection result) on the side where the distribution key is the connection key, the improvement scheme provided by the embodiment of the present application mainly includes the following parts: For the processing of high-frequency values: for the high-frequency values ​​in the data table (or connection result) on the side where the distribution key is the connection key, polling replication is performed, that is, the first data is sent to the first node, the second data is sent to the second node, and so on, so as to ensure the average distribution of data; for the data matching the high-frequency values ​​in the data table (or connection result) on the side where the distribution key is not the connection key, it needs to be copied to all nodes. For the processing of non-high-frequency values, due to the uniform distribution, the non-high-frequency values ​​in the data table (or connection result) on the side where the distribution key is the connection key can be kept locally, and the non-high-frequency values ​​on the other side can be redistributed according to the connection key.

[0127] In other embodiments, see Fig.13 , Fig.13 Schematic diagram of the principle of the data table processing method of the distributed database provided in the embodiment of the present application. Fig.13 As shown, if the data table on one side is particularly large and the data table on the other side is particularly small, and the cost of copying the data in the small table to all nodes is not high, the data on one side of the small table can be directly copied to all nodes.

[0128] The beneficial effects of the data table processing method of the distributed database provided in the embodiment of the present application will be described below in combination with experimental data.

[0129] For example, see Fig.14 , Fig.14 is a schematic diagram for comparing the effects provided by the embodiments of the present application. Fig.14 As shown, the embodiment of the present application is tested for a typical scenario of TPCDS. For the case of data skew, the technical solution provided by the embodiment of the present application (i.e., a solution of partial replication, partial redistribution, and partial unchangedness) is improved to a certain extent compared with the technical solution provided by the related art (i.e., a solution of redistribution or replication). For example, for SQL 78, the execution time is shortened from 3666 milliseconds to 2482 milliseconds; for SQL 16, the execution time is shortened from 2416 milliseconds to 2182 milliseconds.

[0130] The following is a description of an exemplary structure of a distributed database data table processing device 643 provided in an embodiment of the present application implemented as a software module. In some embodiments, Figure 2 As shown, the software modules stored in the data table processing device 643 of the distributed database of the memory 640 may include: an acquisition module 6431 , an operation module 6432 and a connection module 6433 .

[0131] The acquisition module 6431 is used to acquire the first data table and the second data table to be connected in the distributed database, wherein the data in the first data table and the data in the second data table are stored in a plurality of nodes in a dispersed manner; the acquisition module 6431 is also used to acquire the first data distribution of the first data table and the second data distribution of the second data table; the operation module 6432 is used to perform partial replication, partial redistribution and partial unchanged operations on the full amount of data based on the first data distribution and the second data distribution, wherein the full amount of data includes the data in the first data table and the data in the second data table; the connection module 6433 is used to perform a connection operation on the operated data in the first data table and the operated data in the second data table to obtain a connection result.

[0132] In some embodiments, the data table processing device 643 of the distributed database also includes a determination module 6434, which is used to determine, based on the first data distribution and the second data distribution, first data in the full amount of data used to perform a copy operation, second data in the full amount of data used to perform a redistribution operation, and third data that remains unchanged in the full amount of data; an operation module 6432, which is used to perform a copy operation on the first data, perform a redistribution operation on the second data, and keep the third data unchanged.

[0133] In some embodiments, the determination module 6434 is also used to perform the following processing in response to the fact that the distribution keys of the first data table and the second data table are not join keys, and the data corresponding to the join keys in one of the data tables include high-frequency data, wherein the high-frequency data is data that appears more than a threshold number of times: using the data in the non-target data table that matches the high-frequency data as the first data for performing the copy operation, wherein the non-target data table is a data table that does not include the high-frequency data in the first data table and the second data table; using the data in the target data table except the high-frequency data and the data in the non-target data table except the data that matches the high-frequency data as the second data for performing the redistribution operation, wherein the target data table is a data table that includes the high-frequency data in the first data table and the second data table; using the high-frequency data as the third data that remains unchanged.

[0134] In some embodiments, the determination module 6434 is also used to perform the following processing in response to the fact that the distribution keys of the first data table and the second data table are not join keys, the data corresponding to the join key in the first data table includes first high-frequency data, and the data corresponding to the join key in the second data table includes second high-frequency data, wherein the first high-frequency data is data in the first data table whose number of appearances is greater than a number threshold, the second high-frequency data is data in the second data table whose number of appearances is greater than a number threshold, and there is no overlap between the first high-frequency data and the second high-frequency data: use the data in the first data table that matches the second high-frequency data and the data in the second data table that matches the first high-frequency data as the first data for performing a copy operation; use the data in the first data table other than the first high-frequency data and the data matching the second high-frequency data, and the data in the second data table other than the second high-frequency data and the data matching the first high-frequency data as the second data for performing a redistribution operation; use the first high-frequency data and the second high-frequency data as the third data that remain unchanged.

[0135] In some embodiments, the determination module 6434 is also used to respond to the fact that the distribution key of one of the first data table and the second data table is a join key, the distribution key of the other data table is not a join key, and the data corresponding to the join key in the non-target data table includes high-frequency data, wherein the non-target data table is a data table in the first data table and the second data table whose distribution key is not a join key, and the high-frequency data is data in the non-target data table whose number of appearances is greater than a number threshold, to perform the following processing: using the data in the target data table that matches the high-frequency data as the first data for performing the copy operation, wherein the target data table is the data table in the first data table and the second data table whose distribution key is a join key; using the data in the non-target data table other than the high-frequency data as the second data for performing the redistribution operation; using the high-frequency data and the data in the target data table other than the data that matches the high-frequency data as the third data that remains unchanged.

[0136] In some embodiments, the operation module 6432 is further used to copy the first data to each node; redistribute the second data among multiple nodes according to the connection key; and keep the distribution of the third data among multiple nodes unchanged.

[0137] In some embodiments, the determination module 6434 is also used to perform the following processing in response to the fact that the distribution keys of the first data table and the second data table are not join keys, the data corresponding to the join key in the first data table includes first high-frequency data, the data corresponding to the join key in the second data table includes second high-frequency data, and there is overlap between the first high-frequency data and the second high-frequency data: use the data in the first data table and the second data table that matches the high-frequency data in the non-overlapping part as the first data for performing the copy operation; use the high-frequency data in the overlapping part and the remaining data in the first data table and the second data table as the second data for performing the redistribution operation, wherein the remaining data is the data in the first data table and the second data table except the first high-frequency data, the second high-frequency data, and the data matching the high-frequency data in the non-overlapping part; use the high-frequency data in the non-overlapping part as the third data that remains unchanged.

[0138] In some embodiments, the operation module 6432 is also used to copy the first data to each node; redistribute the high-frequency data of the overlapping part of the second data to multiple nodes so that the high-frequency data of the overlapping part is evenly distributed in multiple nodes, and redistribute the remaining data in the second data among multiple nodes according to the connection key; and keep the distribution method of the third data in multiple nodes unchanged.

[0139] In some embodiments, the determination module 6434 is also used to respond to the fact that the distribution key of one of the first data table and the second data table is a join key, the distribution key of the other data table is not a join key, and the data corresponding to the join key in the target data table includes high-frequency data, wherein the target data table is a data table in the first data table and the second data table whose distribution key is a join key, and the high-frequency data is data in the target data table whose number of appearances is greater than a threshold number, to perform the following processing: use the high-frequency data and the data in the non-target data table that matches the high-frequency data as the first data for performing the copy operation, wherein the non-target data table is a data table in the first data table and the second data table whose distribution key is not a join key; use the data in the non-target data table except the data that matches the high-frequency data as the second data for performing the redistribution operation; use the data in the target data table except the high-frequency data as the third data that remains unchanged.

[0140] In some embodiments, the operation module 6432 is also used to perform polling replication on the high-frequency data in the first data, and to copy the data in the first data that matches the high-frequency data to each node; to redistribute the second data among multiple nodes according to the connection key; and to keep the distribution of the third data among multiple nodes unchanged.

[0141] In some embodiments, the operation module 6432 is also used to let n be an integer variable whose value increases from 1, the maximum value of n is N, N is the total number of data included in the high-frequency data, and iterate n to perform the following processing: determine the remainder obtained by dividing n by M, and copy the nth data to the node corresponding to the remainder in multiple nodes, where M is the total number of multiple nodes, and when the remainder is 0, copy the data corresponding to the remainder to the Mth node.

[0142] In some embodiments, the operation module 6432 is also used to copy the data in the data table whose data volume is less than the data volume threshold to each node in response to the data volume of one of the first data table and the second data table being greater than the data volume threshold and the data volume of the other data table being less than the data volume threshold.

[0143] It should be noted that the description of the device in the embodiment of the present application is similar to the description of the method embodiment described above, and has similar beneficial effects as the method embodiment, so it will not be repeated. Figure 3 ,or Figure 4 The present invention should be understood by referring to the description of any one of the accompanying drawings.

[0144] The embodiment of the present application provides a computer program product, which includes a computer program or a computer executable instruction, and the computer program or the computer executable instruction is stored in a computer readable storage medium. The processor of the computer device reads the computer executable instruction from the computer readable storage medium, and the processor executes the computer executable instruction, so that the computer device executes the data table processing method of the distributed database described in the embodiment of the present application.

[0145] The embodiment of the present application provides a computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, the processor will execute the data table processing method of the distributed database provided in the embodiment of the present application, for example, Figure 3 ,or Figure 4 A data table processing method for a distributed database is shown.

[0146] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface storage, optical disk, or CD-ROM; or it may be various devices including one or any combination of the above memories.

[0147] In some embodiments, executable instructions may be in the form of a program, software, software module, script or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for use in a computing environment.

[0148] As an example, the executable instructions may be deployed to be executed on one electronic device, or on multiple electronic devices located at one site, or on multiple electronic devices distributed at multiple sites and interconnected by a communication network.

[0149] The above is only an embodiment of the present application and is not intended to limit the protection scope of the present application. Any modifications, equivalent substitutions and improvements made within the spirit and scope of the present application are included in the protection scope of the present application.

Claims

1. A data table processing method for a distributed database, characterized in that: The distributed database includes a plurality of nodes; the method includes: Acquire a first data table and a second data table to be connected in the distributed database, wherein the data in the first data table and the data in the second data table are dispersedly stored in the plurality of nodes; Acquire a first data distribution of the first data table and a second data distribution of the second data table; In response to the fact that neither the distribution key of the first data table nor the second data table is a join key, and the data corresponding to the join key in one of the data tables includes high-frequency data, wherein the high-frequency data is data that appears more than a threshold number of times, the following processing is performed: using data in a non-target data table that matches the high-frequency data as first data for performing a copy operation, wherein the non-target data table is a data table that does not include the high-frequency data in the first data table and the second data table; using data other than the high-frequency data in a target data table and data other than the data matching the high-frequency data in a non-target data table as second data for performing a redistribution operation, wherein the target data table is a data table including the high-frequency data in the first data table and the second data table; Using the high frequency data as the third data that remains unchanged; Perform the copy operation on the first data, perform the redistribution operation on the second data, and keep the third data unchanged; A connection operation is performed on the data in the first data table after the operation and the data in the second data table after the operation to obtain a connection result.

2. The method according to claim 1, characterized in that The method further comprises: In response to the fact that neither the distribution key of the first data table nor the distribution key of the second data table is a join key, the data corresponding to the join key in the first data table includes first high-frequency data, and the data corresponding to the join key in the second data table includes second high-frequency data, wherein the first high-frequency data is data in the first data table that appears more than a number threshold, the second high-frequency data is data in the second data table that appears more than a number threshold, and there is no overlap between the first high-frequency data and the second high-frequency data, the following processing is performed: using the data in the first data table that matches the second high-frequency data and the data in the second data table that matches the first high-frequency data as first data for performing a copy operation; using data in the first data table other than the first high-frequency data and data matching the second high-frequency data, and data in the second data table other than the second high-frequency data and data matching the first high-frequency data as second data for performing a redistribution operation; The first high-frequency data and the second high-frequency data are used as unchanged third data.

3. The method according to claim 1, characterized in that The method further comprises: In response to the distribution key of one of the first data table and the second data table being a join key, the distribution key of the other data table being not a join key, and the data corresponding to the join key in the non-target data table including high-frequency data, wherein the non-target data table is a data table in the first data table and the second data table whose distribution key is not a join key, and the high-frequency data is data in the non-target data table whose occurrence number is greater than a number threshold, the following processing is performed: Using data in a target data table that matches the high-frequency data as first data for performing a copy operation, wherein the target data table is a data table in which the distribution key of the first data table and the second data table is a connection key; using the data in the non-target data table except the high-frequency data as second data for performing a redistribution operation; The high-frequency data and data in the target data table other than data matching the high-frequency data are used as third data that remain unchanged.

4. The method according to claim 2, characterized in that: The performing the copy operation on the first data, performing the redistribution operation on the second data, and keeping the third data unchanged, includes: Copying the first data to each of the nodes; redistributing the second data among the multiple nodes according to the connection key; The distribution mode of the third data in the plurality of nodes is kept unchanged.

5. The method according to claim 1, characterized in that The method further comprises: In response to the fact that neither the distribution key of the first data table nor the distribution key of the second data table is a join key, the data corresponding to the join key in the first data table includes first high-frequency data, the data corresponding to the join key in the second data table includes second high-frequency data, and there is overlap between the first high-frequency data and the second high-frequency data, the following processing is performed: using data in the first data table and the second data table that matches the high-frequency data in the non-overlapping portion as first data for performing a copy operation; Using the high-frequency data of the overlapping portion and the remaining data in the first data table and the second data table as second data for performing a redistribution operation, wherein the remaining data is data in the first data table and the second data table other than the first high-frequency data, the second high-frequency data, and data matching the high-frequency data of the non-overlapping portion; The high frequency data of the non-overlapping portion is regarded as the third data which remains unchanged.

6. The method according to claim 5, characterized in that The performing the copy operation on the first data, performing the redistribution operation on the second data, and keeping the third data unchanged, includes: Copying the first data to each of the nodes; redistribute the high frequency data of the overlapping part in the second data to the plurality of nodes so that the high frequency data of the overlapping part is evenly distributed in the plurality of nodes, and redistributing the remaining data in the second data among the multiple nodes according to the connection keys; The distribution mode of the third data in the plurality of nodes is kept unchanged.

7. The method according to claim 1, characterized in that The method further comprises: In response to the distribution key of one of the first data table and the second data table being a join key, the distribution key of the other data table being not a join key, and the data corresponding to the join key in the target data table includes high-frequency data, wherein the target data table is a data table in which the distribution key of the first data table and the second data table is a join key, and the high-frequency data is data in the target data table whose number of occurrences is greater than a number threshold, the following processing is performed: Using the high-frequency data and data in a non-target data table that matches the high-frequency data as first data for performing a copy operation, wherein the non-target data table is a data table in the first data table and the second data table whose distribution key is not a join key; using data in the non-target data table other than data matching the high-frequency data as second data for performing a redistribution operation; The data in the target data table except the high-frequency data is used as the third data which remains unchanged.

8. The method according to claim 7, characterized in that The performing the copy operation on the first data, performing the redistribution operation on the second data, and keeping the third data unchanged, includes: Performing polling replication on the high-frequency data in the first data, and replicating the data in the first data that matches the high-frequency data to each of the nodes; redistributing the second data among the multiple nodes according to the connection key; The distribution mode of the third data in the plurality of nodes is kept unchanged.

9. The method according to claim 8, characterized in that The polling and copying of the high-frequency data in the first data includes: Let n be an integer variable whose value increases from 1, the maximum value of n is N, N is the total number of data included in the high-frequency data, and iterate n to perform the following processing: Determine the remainder obtained by dividing n by M, and copy the nth data to the node corresponding to the remainder among the multiple nodes, where M is the total number of the multiple nodes, and when the remainder is 0, copy the data corresponding to the remainder to the Mth node.

10. The method according to any one of claims 1 to 9, characterized in that: The method further comprises: In response to the data volume of one of the first data table and the second data table being greater than a data volume threshold and the data volume of the other data table being less than the data volume threshold, the data in the data table having a data volume less than the data volume threshold is copied to each of the nodes.

11. A data table processing device for a distributed database, characterized in that: The distributed database includes a plurality of nodes; the device includes: An acquisition module, configured to acquire a first data table and a second data table to be connected in the distributed database, wherein the data in the first data table and the data in the second data table are dispersedly stored in the plurality of nodes; The acquisition module is further used to acquire a first data distribution of the first data table and a second data distribution of the second data table; a determination module, configured to, in response to the fact that neither the distribution key of the first data table nor the second data table is a join key, and the data corresponding to the join key in one of the data tables includes high-frequency data, wherein the high-frequency data is data with a number of occurrences greater than a number threshold, perform the following processing: using data in a non-target data table that matches the high-frequency data as first data for performing a copy operation, wherein the non-target data table is a data table that does not include the high-frequency data in the first data table and the second data table; using data in a target data table other than the high-frequency data and data in the non-target data table other than the data that matches the high-frequency data as second data for performing a redistribution operation, wherein the target data table is a data table that includes the high-frequency data in the first data table and the second data table; using the high-frequency data as third data that remains unchanged; an operation module, configured to perform the copy operation on the first data, perform the redistribution operation on the second data, and keep the third data unchanged; The connection module is used to perform a connection operation on the data in the first data table after the operation and the data in the second data table after the operation to obtain a connection result.

12. An electronic device, characterized in that: include: A memory for storing executable instructions; The processor is used to implement the data table processing method of the distributed database as described in any one of claims 1 to 10 when executing the executable instructions stored in the memory.

13. A computer-readable storage medium storing computer-executable instructions, characterized in that: When the computer executable instructions are executed by a processor, the data table processing method of a distributed database as described in any one of claims 1 to 10 is implemented.

14. A computer program product comprising a computer program or computer executable instructions, characterized in that When the computer program or computer executable instruction is executed by a processor, the data table processing method of a distributed database as described in any one of claims 1 to 10 is implemented.

Images