Data redistribution method and device, computer equipment, readable storage medium and program product

Abstract

The invention relates to a data redistribution method and device, computer equipment, a computer readable storage medium and a computer program product. The method is applied to a first node of a distributed database, and comprises the following steps: receiving a redistribution instruction issued by a coordination node in the distributed database; writing the target data into a sending queue in a first redistribution operator based on the redistribution instruction, and pushing the target data to a second data transceiving process of a second node based on the sending queue in the first redistribution operator and a first data transceiving process; the sending queue of the first redistribution operator and the first data transceiving process share the same memory area; the second node is a remote node; the second data transceiving process and the receiving queue of the second redistribution operator share the same memory area; the receiving queue of the second redistribution operator is used for reading the target data from the second data transceiving process. By adopting the method, the process connection number and the network connection overhead required by data redistribution can be reduced.

Description

Technical Field

[0001] The present application relates to the field of information technology, and in particular to a data redistribution method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Art

[0002] In distributed database systems, data redistribution involves reallocating data to different storage nodes to achieve balanced data distribution and optimize performance. Existing solutions primarily implement data redistribution through the RemoteSubplan operator, but this approach requires significant process and network connection overhead. Summary of the Invention

[0003] Based on this, it is necessary to provide a data redistribution method, apparatus, computer equipment, computer-readable storage medium and computer program product to address the above technical problems, so as to reduce the number of process connections and network connection overhead required for data redistribution.

[0004] In a first aspect, the present application provides a data redistribution method, applied to a first node of a distributed database, the method comprising:

[0005] Receive redistribution instructions issued by the coordination node in the distributed database;

[0006] Based on the redistribution instruction, the first redistribution operator corresponding to the first node is called, and the target data to be redistributed in the first node is written into the sending queue of the first redistribution operator, so that the target data is pushed to the second data sending and receiving process in the second node of the distributed database based on the sending queue in the first redistribution operator and the first data sending and receiving process in the first node; wherein,

[0007] The sending queue in the first redistribution operator and the first data sending and receiving process share the same memory area; and

[0008] The second node is a remote node; the second data transceiver process and the receive queue in the second redistribution operator corresponding to the second node share the same memory area; the receive queue in the second redistribution operator is used for the second node to read target data from the second data transceiver process.

[0009] In one embodiment, writing target data to be redistributed in a first node into a sending queue in a first redistribution operator, and pushing the target data to a second data sending and receiving process in a second node of a distributed database based on the sending queue in the first redistribution operator and a first data sending and receiving process in the first node, includes:

[0010] In a case where the memory space corresponding to the sending queue in the first redistribution operator is in a full state, the target data in the first node that needs to be redistributed is written into the buffer corresponding to the first redistribution operator to wait for writing into the sending queue in the first redistribution operator;

[0011] In a case where the memory space corresponding to the sending queue in the first redistribution operator switches from the full state to the not full state, the target data in the buffer corresponding to the first redistribution operator is written into the sending queue in the first redistribution operator based on the waiting order, so as to push the target data to the second data transceiving process in the second node of the distributed database based on the sending queue in the first redistribution operator and the first data transceiving process in the first node.

[0012] In one of the embodiments, writing the target data in the first node that needs to be redistributed into the sending queue in the first redistribution operator so as to push the target data to the second data transceiving process in the second node of the distributed database based on the sending queue in the first redistribution operator and the first data transceiving process in the first node further includes:

[0013] In a case where the memory space corresponding to the sending queue is in the not full state and the buffer corresponding to the first redistribution operator is empty, the target data in the first node that needs to be redistributed is written into the sending queue in the first redistribution operator, so as to push the target data to the second data transceiving process in the second node of the distributed database based on the sending queue in the first redistribution operator and the first data transceiving process in the first node.

[0014] In a second aspect, the application further provides a data redistribution method applied to a second node of a distributed database, which includes:

[0015] receiving a redistribution instruction issued by a coordination node in the distributed database;

[0016] based on the redistribution instruction, calling a preset second redistribution operator corresponding to the second node, receiving and reading target data pushed by a first data transceiving process in a first node of the distributed database based on a receiving queue in the second redistribution operator and the second data transceiving process in the second node; wherein,

[0017] the second data transceiving process and the receiving queue in the second redistribution operator share the same memory area; and

[0018] The first node is a remote node, and the first node corresponds to a first redistribution operator; the target data is data that needs to be redistributed and that is written by the first node into a sending queue in the first redistribution operator; the sending queue in the first redistribution operator and a first data transceiving process share a same memory region; and the first data transceiving process is configured to read the target data from the sending queue in the first redistribution operator and push the target data to a second data transceiving process.

[0019] In one of the embodiments, receiving and reading the target data pushed by the first data transceiving process in the first node of the distributed database includes:

[0020] receiving the target data pushed by the first data transceiving process in the first node of the distributed database, and writing the target data into a buffer corresponding to a receiving queue in a second redistribution operator;

[0021] reading a data type of the target data from the buffer corresponding to the receiving queue in the second redistribution operator;

[0022] controlling a process of receiving the target data by the second node based on the read data type of the target data.

[0023] In one of the embodiments, controlling the process of receiving the target data by the second node based on the read data type of the target data includes:

[0024] in a case where the read data type of the target data is a data row type, continuing to receive the target data;

[0025] in a case where the read data type of the target data is an end-of-file type, stopping receiving the target data and ending running of the second redistribution operator.

[0026] In a third aspect, the application further provides a data redistribution device applied to a first node of a distributed database, the device including:

[0027] an instruction receiving module configured to receive a redistribution instruction issued by a coordination node in the distributed database;

[0028] A push module is used to call the first redistribution operator corresponding to the first node based on the redistribution instruction, write the target data to be redistributed in the first node into the sending queue in the first redistribution operator, and push the target data to the second data sending and receiving process in the second node of the distributed database based on the sending queue in the first redistribution operator and the first data sending and receiving process in the first node; wherein the sending queue in the first redistribution operator and the first data sending and receiving process share the same memory area; and the second node is a remote node; the second data sending and receiving process and the receiving queue in the second redistribution operator corresponding to the second node share the same memory area; the receiving queue in the second redistribution operator is used for the second node to read the target data from the second data sending and receiving process.

[0029] In a fourth aspect, the present application further provides a data redistribution device, applied to a second node of a distributed database, the device comprising:

[0030] An instruction receiving module is used to receive redistribution instructions issued by a coordination node in a distributed database;

[0031] A target data receiving module is used to call the preset second redistribution operator corresponding to the second node based on the redistribution instruction, and receive and read the target data pushed by the first data sending and receiving process in the first node of the distributed database based on the receiving queue in the second redistribution operator and the second data sending and receiving process in the second node; wherein the second data sending and receiving process and the receiving queue in the second redistribution operator share the same memory area; and the first node is a remote node, and the first node corresponds to the first redistribution operator; the target data is the data that needs to be redistributed written by the first node to the sending queue in the first redistribution operator; the sending queue in the first redistribution operator and the first data sending and receiving process share the same memory area; the first data sending and receiving process is used to read the target data from the sending queue in the first redistribution operator and push it to the second data sending and receiving process.

[0032] In a fifth aspect, the present application further provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the method described in the first or second aspect when executing the computer program.

[0033] In a sixth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described in the first or second aspect above.

[0034] In a seventh aspect, the present application also provides a computer program product, comprising a computer program, which, when executed by a processor, implements the steps of the method described in the first or second aspect above.

[0035] In the above-mentioned data redistribution method, apparatus, computer device, computer-readable storage medium, and computer program product, when data redistribution is required, the first node sends the target data to be redistributed to the second node through the first redistribution operator to achieve data redistribution. Specifically, the first node writes the target data into the send queue of the first redistribution operator. Because the send queue of the first redistribution operator shares memory with the first data transceiver process and pushes the target data to the second node, this improves the transmission efficiency of the target data. At the same time, the first node can send the target data through the first redistribution operator, which reduces the number of session processes of the first node, thereby reducing the number of process connections and network connection overhead required for data redistribution. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the following briefly introduces the drawings required for use in the embodiments of the present application or related technical descriptions. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying any creative work.

[0037] Figure 1 A schematic diagram of a data exchange format in one embodiment;

[0038] Figure 2 Schematic diagram of the principle of an existing data redistribution solution in one embodiment;

[0039] Figure 3 1 is a flow chart of a data redistribution method according to an embodiment;

[0040] Figure 4 is another schematic flow chart of a data redistribution method according to an embodiment;

[0041] Figure 5 Schematic diagram of the specific execution process of the RDA operator in one embodiment;

[0042] Figure 6 Schematic diagram of shared memory management of RDA operator in one embodiment;

[0043] Figure 7 A schematic diagram of an overall data redistribution process in one embodiment;

[0044] Figure 8 1 is a flow chart of a sending side in a data redistribution method according to an embodiment;

[0045] Figure 9 1 is a flow chart of a receiving side in a data redistribution method according to an embodiment;

[0046] Figure 10 is a structural block diagram of a data redistribution device in one embodiment;

[0047] Figure 11 is a structural block diagram of another data redistribution device in one embodiment;

[0048] Figure 12 FIG. 1 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0049] To make the objectives, technical solutions, and advantages of this application more clearly understood, the present application is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are merely intended to illustrate this application and are not intended to limit this application. The term "multiple" herein may mean "≥2" unless otherwise specified.

[0050] Before explaining the technical solution of this application, the following terms or nouns related to the technical solution of this application are explained:

[0051] Table: In a distributed database system, a table is the basic unit of data storage, consisting of rows and columns. Each table can contain multiple fields (i.e., columns), each of which stores a specific type of data, such as integers or strings. Rows represent instances of each field, or specific data records.

[0052] Data redistribution refers to the process of redistributing data across different storage nodes in a distributed database system to achieve balanced data distribution and optimize performance. In this application, this means redistributing the data from one or more tables to each node according to the distribution key in the optimal plan when performing distributed computations on multiple tables.

[0053] CN: Coordinator, the coordinator node, is the entry point for business access. The CN only stores the system's global metadata and does not store actual business data.

[0054] DN: Datanode, data node, is the node where business data is stored.

[0055] RDA: Remote Data Access, is an operator specifically used to obtain data from remote nodes. In the technical solution of this application, RDA is a database operator used to implement data redistribution.

[0056] Forwarder: FWD, a data sending and receiving process, carries out cross-node data interaction between RDA operators.

[0057] Backend: The process is responsible for receiving and processing client requests and returning results. One client connects to one backend process.

[0058] NodeID: Each database node has a unique numeric identifier, which starts at 0 and increases in one direction. It is the subscript of the node name in the order of string sorting.

[0059] Node OID: A database node ID that is unique within a single node.

[0060] SQL, Structured Query Language, Structured Query Language.

[0061] The JOIN operation, also known as join, is a method used in SQL to retrieve data from multiple tables. It allows you to combine rows from two or more tables based on certain conditions, so that you can obtain related data from these tables.

[0062] A non-replicated table is a table in a distributed database system where the data is not fully replicated to every node. Instead, the data in this type of table is partitioned, meaning that the data is distributed across multiple nodes based on a strategy (such as hashing or ranges).

[0063] Distribution Key, also known as distribution key, refers to the column or columns used to determine how to distribute data in a table to different nodes.

[0064] Non-distribution key refers to all columns other than the distribution key column.

[0065] In database query processing and distributed computing, an operator refers to the basic unit that performs a specific operation or task.

[0066] ‌Mmap (Memory Map), a memory mapping file method, is used to map a file or other object to the address space of a process, achieving a one-to-one correspondence between the file disk address and the process virtual address space.‌

[0067] RPC (remote procedure call) allows a program to call a procedure or function in another address space (usually remote) just like calling a local function.

[0068] A redistribution key is a field or set of fields that determines how data is redistributed across nodes in a distributed system. When performing certain operations, such as JOIN, if the data involved is not on the same node, the redistribution key is used to redistribute the data so that the operation can be performed correctly on the corresponding nodes.

[0069] After the above description, the technical solution of this application is described below:

[0070] When executing SQL queries in a distributed database, the client connects to the coordinating node (CN). The CN generates the optimal query plan and distributes it to all DNs. The DNs receive the CN's query plan and execute it. Based on the execution plan, the DNs decide whether to fetch data from other nodes. After the query completes, the final or intermediate results are returned. The CN collects the results from all DNs, processes them based on the actual query, and returns them to the client.

[0071] Redistribution occurs when data needs to be exchanged between data nodes. When two non-replicated tables are joined, the distribution column key is joined with the non-distribution column key, and the redistribution process is executed. Figure 1 As shown, when all columns involved in a join are distribution columns, the query is directly sent to the corresponding DN node. After the node join is complete, the data is summarized on the CN. When a sufficiently small table is joined with a large table, if the join is with a non-distributed key of the large table, the small table is copied; otherwise, the query is sent or the small table is redistributed. During redistribution, the non-distributed columns of TBL_B are hashed again, using a hash function. The join value of the non-distributed key on each DN node is sent to all nodes, ensuring that the data of the TAB_B shard on each DN is complete. In other words, the small table is redistributed according to the join conditions so that its data distribution matches that of the large table, facilitating the join operation. This is usually achieved through a hash function, ensuring that data rows with the same join key value are assigned to the same node.

[0072] The corresponding situations can be summarized in the following table:

[0073]

[0074] For data redistribution, the existing solution is mainly implemented through the RemoteSubplan operator, such as Figure 2 In the figure, dn1 and dn2 represent the processes that execute the plan on DN1 and DN2, respectively. Their working principle is as follows:

[0075] (1) The RemoteSubplan operator sends the overall next query plan.

[0076] (2) DN will get a connection from the pool to execute the plan.

[0077] (3) DN1 does not copy all the data of DN2 tbl b (table b on DN2), but redistributes tbl b through SharedQueue (shared queue). Two processes on DN1 will bind to SharedQueue at the same time. The one that binds first becomes the producer, and the one that binds later becomes the consumer. The same is true for DN2. SharedQueue is a dynamic queue with multiple producers and consumers. During redistribution, there will be a SharedQueue on each DN node. Then the process on each node will bind to the SharedQueue on the node. The first one to bind will become the producer. After the producer is bound, a sending thread will be created, and then a buffer DataPumpBuf will be created for all consumers. The producer then puts the data into the data buffer. When the amount of data in the data buffer exceeds a certain threshold (which can be set through parameters), the sending thread will send the data, completing the data redistribution.

[0078] (4) The RemoteSubplan operator in the upper layer receives the data of the lower layer query plan.

[0079] Because the entire next query plan is only delivered when the RemoteSubplan operator is encountered during execution, if a query involves multiple redistribution levels, each DN layer will consider itself an initiator, resulting in significant multi-layer process and network connection overhead. For example, if each redistribution level is treated as a new initiation point, each redistribution operation requires reestablishing inter-process or inter-node communication connections, resulting in significant process and network connection overhead.

[0080] Let's do a simple calculation: if 200 DN nodes have 100 concurrent queries, and each query involves five data redistributions, the calculation will involve over 100,000 connections. This problem becomes even more severe when the cluster scales to thousands of nodes, and it is a common problem of MPP (Massively Parallel Processing) at large scale.

[0081] Based on the above analysis, this application proposes a data redistribution method. Since data redistribution involves different nodes, the data redistribution method provided by this application is described below from the perspective of a single-side node, such as the data sending side or the data receiving side.

[0082] In one embodiment, as shown in Figure 3 A data redistribution method is provided, and in this embodiment, the method can be applied to a first node of a distributed database, and the first node can be a node responsible for data storage in the distributed database, i.e., a data node. In this embodiment, the method includes steps S301 to S302:

[0083] Step S301: The first node receives a redistribution instruction issued by a coordination node in the distributed database.

[0084] The distributed database refers to data stored in multiple independent nodes, and each node can be located in the same place or distributed around the world.

[0085] For example, the distributed database can include a coordination node, a first node, and a second node.

[0086] In some embodiments, the distributed database can receive a data query instruction, the coordination node can generate a corresponding query plan, and issue it to the data node. A data node can determine whether to obtain data from other data nodes according to the content of the plan.

[0087] The redistribution instruction can be a specific query plan generated by the coordination node, which is a plan that needs to be redistributed. For example, the data redistribution plan corresponding to the redistribution instruction can include sending target data that needs to be redistributed from the first node to the second node.

[0088] Step S302: The first node calls a first redistribution operator corresponding to the first node based on the redistribution instruction, writes the target data that needs to be redistributed in the first node into a sending queue in the first redistribution operator, and pushes the target data to a second data transceiver process in the second node of the distributed database based on the sending queue in the first redistribution operator and the first data transceiver process in the first node; the sending queue in the first redistribution operator and the first data transceiver process share the same memory area; and the second node is a remote node; the second data transceiver process and a receiving queue in a second redistribution operator corresponding to the second node share the same memory area; and the receiving queue in the second redistribution operator is used for the second node to read the target data from the second data transceiver process.

[0089] The second node is a remote node, i.e., Remote Node, which can be another computer or server that is not on the host currently processing the request or query. In some embodiments, the first node and the second node can be connected through a network and work together to complete data storage, data processing, or service provision tasks.

[0090] In some embodiments, the number of the first redistribution operators may be one or more; similarly, the number of the second redistribution operators may be one or more.

[0091] In some embodiments, the first redistribution operator may have a corresponding label or name, such as A1, A2, etc., and the second redistribution operator may also have a corresponding label or name, such as B1, B2, etc. For example, one first redistribution operator may correspond to one second redistribution operator, such as A1 corresponds to B1, to avoid confusion.

[0092] In some embodiments, the first redistribution operator may include a send queue, which may be used to send target data in the first node to the first data transceiver process. For example, the first node may scan for target data to be redistributed in the first node based on the first redistribution operator and write the scanned target data into the send queue of the first redistribution operator.

[0093] In some embodiments, the first redistribution operator may further include a receiving queue, which may be used to obtain the second target data to be redistributed from the first data transceiver process and provide it to the first node for reading. For example, the first node may read the second target data based on the first redistribution operator.

[0094] In some embodiments, a first redistribution operator may correspond to an mmap file. The first redistribution operator creates the mmap file when communication starts and closes the mmap file when communication ends.

[0095] In some embodiments, the first data transceiving process and the second data transceiving process may be used to carry out cross-node data interaction between the first redistribution operator and the second redistribution operator.

[0096] In some embodiments, the first redistribution operator may share the same memory area with the first data transceiving process; and the second redistribution operator may share the same memory area with the second data transceiving process.

[0097] In the above technical solution, when data redistribution is required, the first node uses the first redistribution operator to send the target data to the second node to achieve data redistribution. Specifically, the first node writes the target data into the send queue of the first redistribution operator. Because the send queue of the first redistribution operator shares memory with the first data transceiver process and pushes the target data to the second node, this improves the transmission efficiency of the target data. At the same time, the first node can send the target data through the first redistribution operator, which reduces the number of session processes on the first node, thereby reducing the number of process connections and network connection overhead required for data redistribution.

[0098] In one embodiment, the aforementioned “writing target data to be redistributed in the first node into a sending queue in the first redistribution operator, and pushing the target data to a second data sending and receiving process in the second node of the distributed database based on the sending queue in the first redistribution operator and the first data sending and receiving process in the first node” may include:

[0099] When the memory space corresponding to the sending queue in the first redistribution operator is full, the target data to be redistributed in the first node is written into the buffer corresponding to the first redistribution operator to wait for being written into the sending queue in the first redistribution operator;

[0100] When the memory space corresponding to the sending queue in the first redistribution operator switches from a full state to a not full state, the target data in the buffer corresponding to the first redistribution operator is written into the sending queue in the first redistribution operator based on the waiting order, so as to push the target data to the second data sending and receiving process in the second node of the distributed database based on the sending queue in the first redistribution operator and the first data sending and receiving process in the first node.

[0101] In some embodiments, the buffer corresponding to the first redistribution operator may be a disk of the first node.

[0102] In some embodiments, the sending queue in the first redistribution operator may be a circular queue, and the corresponding memory space is limited, so the target data may be temporarily stored in a buffer corresponding to the first redistribution operator.

[0103] For example, the target data may include A, B, and C. A is in the memory space corresponding to the send queue in the first redistribution operator, causing the memory space corresponding to the send queue in the first redistribution operator to be full. B is in the buffer corresponding to the first redistribution operator. Based on this, C can be written to the buffer corresponding to the first redistribution operator and wait. The corresponding waiting order is B first, C second. Therefore, after A is sent out, the memory space corresponding to the send queue in the first redistribution operator switches from a full state to a not full state. B is written to the send queue in the first redistribution operator first, and then C is written to the send queue in the first redistribution operator.

[0104] The above technical solution takes into account the limited memory space corresponding to the send queue in the first redistribution operator. By utilizing the buffer corresponding to the first redistribution operator to temporarily store the target data, the target data is written to the send queue in the first redistribution operator based on the waiting order, depending on the state change of the memory space corresponding to the send queue in the first redistribution operator, that is, when the memory space changes from full to not full. This ensures the orderly transmission of the target data and avoids the failure of the target data transmission due to the limited memory space corresponding to the send queue in the first redistribution operator.

[0105] In one embodiment, the aforementioned “writing target data to be redistributed in the first node into a sending queue in the first redistribution operator, and pushing the target data to a second data sending and receiving process in the second node of the distributed database based on the sending queue in the first redistribution operator and the first data sending and receiving process in the first node” may further include:

[0106] When the memory space corresponding to the sending queue in the first redistribution operator of the first node is not full and the buffer corresponding to the first redistribution operator is empty, the first node writes the target data to be redistributed in the first node into the sending queue in the first redistribution operator, so as to push the target data to the second data sending and receiving process in the second node of the distributed database based on the sending queue in the first redistribution operator and the first data sending and receiving process in the first node.

[0107] In some embodiments, as described above, the memory space corresponding to the send queue in the first redistribution operator is limited. This memory space may be full, but it may also be not full. Similarly, the buffer corresponding to the first redistribution operator may have two states: containing data and not having data (i.e., empty). Therefore, the following situation may occur: the memory space corresponding to the send queue in the first redistribution operator is not full, either permanently not full or switching from full to not full, while the buffer corresponding to the first redistribution operator is empty. In this case, the first node can directly write the target data to the send queue without writing it to the buffer corresponding to the first redistribution operator.

[0108] In the above technical solution, when the memory space corresponding to the sending queue in the first redistribution operator is not full and the buffer corresponding to the first redistribution operator is empty, the first node can directly write the target data into the sending queue in the first redistribution operator without writing into the buffer corresponding to the first redistribution operator, which helps the transmission efficiency of the target data and thus improves the data redistribution efficiency.

[0109] In an exemplary embodiment, Figure 4As shown, a data redistribution method is provided, which is applied to the second node of a distributed database. The method can be applied to the second node of a distributed database, and the second node can be a node responsible for data storage in the distributed database. In this embodiment, the method includes steps S401 to S402:

[0110] Step S401: The second node receives a redistribution instruction sent by a coordinating node in a distributed database.

[0111] Step S402: The second node calls the preset second redistribution operator corresponding to the second node based on the redistribution instruction, and receives and reads the target data pushed by the first data sending and receiving process in the first node of the distributed database based on the receiving queue in the second redistribution operator and the second data sending and receiving process in the second node; wherein the second data sending and receiving process and the receiving queue in the second redistribution operator share the same memory area; and the first node is a remote node, and the first node corresponds to the first redistribution operator; the target data is the data that needs to be redistributed written by the first node to the sending queue in the first redistribution operator; the sending queue in the first redistribution operator and the first data sending and receiving process share the same memory area; the first data sending and receiving process is used to read the target data from the sending queue in the first redistribution operator and push it to the second data sending and receiving process.

[0112] For understanding of step S401 to step S402, please refer to the above description of step S101 to step S102, and the same parts will not be repeated here.

[0113] In some embodiments, the second node may receive the target data pushed by the first data transceiver process based on the second data transceiver process and the receive queue in the second redistribution operator. For example, the second node may write the target data into a buffer corresponding to the receive queue in the second redistribution operator, read the target data from the buffer, and then report it to the coordinating node.

[0114] In the above technical solution, when data redistribution is required, the second node receives the target data that needs to be redistributed sent by the first node through the second redistribution operator and sends it to the second node to achieve data redistribution. Specifically, the second node receives and reads the target data pushed by the first data transceiver process in the first node of the distributed database based on the receiving queue in the second redistribution operator and the second data transceiver process in the second node. Since the second data transceiver process and the receiving queue in the second redistribution operator share the same memory area, the second node can receive and read the target data more efficiently. At the same time, the second node can receive the target data through the second redistribution operator, which reduces the session process of the second node, thereby reducing the number of process connections and network connection overhead required for data redistribution.

[0115] In one of the embodiments, the aforementioned "receiving and reading target data pushed by the first data transceiving process in the first node of the distributed database" can include: the second node receiving the target data pushed by the first data transceiving process in the first node of the distributed database, and writing the target data into a buffer corresponding to a receiving queue in the second redistribute operator; reading the data type of the target data from the buffer corresponding to the receiving queue in the second redistribute operator; and controlling the process of the second node receiving the target data based on the read data type of the target data.

[0116] In some embodiments, the second node can determine the data type of the received target data in real time, and determine whether to end the receiving of the target data according to the change of the data type of the target data, that is, control the start and stop of the process of receiving the target data.

[0117] In some embodiments, the buffer corresponding to the receiving queue in the second redistribute operator can be a disk of the second node.

[0118] In some embodiments, the number of buffers corresponding to the receiving queue in the second redistribute operator can be multiple.

[0119] The above technical solution receives the target data and writes it into the buffer corresponding to the receiving queue in the second redistribute operator, reads the data type of the target data from the buffer, and controls the process of receiving the target data according to the data type of the target data, thereby avoiding the situation that the target data has been transmitted but the second node is still in the receiving process, and avoiding process waste.

[0120] In one of the embodiments, the aforementioned "controlling the process of the second node receiving the target data based on the read data type of the target data" can include: the second node continuing to receive the target data when the read data type of the target data is a data row type; and stopping the receiving of the target data and ending the running of the second redistribute operator when the read data type of the target data is an end-of-file symbol type.

[0121] In some embodiments, the data type of the target data can include a data row type and an end-of-file symbol type, and the end-of-file symbol type can correspond to a symbol for indicating the end of a file, and the data row type can correspond to data with actual meaning and not indicating the end of a file.

[0122] In some embodiments, the second node can control the process of receiving the target data by the corresponding buffer based on the data type of the read target data. For example, the buffers corresponding to the receiving queue in the second redistribution operator can include A, B and C. The second node can read the data type of the target data from the buffer A, and if it is a file end symbol type, it is determined that the buffer A has completed receiving the target data. When all the buffers corresponding to the receiving queue in the second redistribution operator have completed receiving the target data, the operation of the second redistribution operator is ended.

[0123] The above technical solution continues to receive the target data when the data type of the target data is a data row type, and stops receiving the target data and ends the operation of the second redistribution operator when the data type of the target data is a file end symbol type. This enables the second node to close the second redistribution operator in time, avoiding resource occupation and waste.

[0124] In an exemplary embodiment, a data redistribution method is provided, which uses an RDA operator to perform redistribution. For batch data acquisition, there are generally two kinds of pulling and pushing, and the operator triggers both pushing and pulling data processes. Pushing is cross-node, pushing local node data to remote node; pulling is reading data from local buffer. The present application selects the pushing mode, which is more efficient. At the same time, the RDA operator is used to replace the existing SharedQueue and RemoteSubplan logic to do non-distributed key management data redistribution. When RemoteSubplan processes non-distributed key association queries, the number of database connections (processes) of a single node is N (the number of DN) * J (the number of associations) * P (the number of concurrent processes). When processing complex queries, the resource consumption is too high. By introducing the RDA operator, the number of connections of a single node is reduced to N*C (a constant), thereby solving the problems of process connection number explosion and excessive network connection consumption in the existing redistribution scheme. At the same time, the method realizes the interaction of data between different nodes through shared memory and mmap method, thereby improving the transmission efficiency. Specifically:

[0125] The use scenario of the present method is when data redistribution is needed in a distributed database. For example, the present method can be applied to a distributed database or a corresponding system or platform, etc.

[0126] The RDA operator designed by the method is a database operator for remote data access. The RDA operator corresponds to the first redistribution operator and the second redistribution operator. The RDA operator life cycle includes three steps: RDAInit (RDA initialization), RDAExec (RDA execution), and RDAEnd (RDA end). A 64-bit global sequence is used as the name of the RDA operator. The sequence is created in the system initialization stage and needs to be considered for multiple node creation. As shown in Figure 5 , a possible RDA operator specific execution process is given:

[0127] Step one: RDAInit.

[0128] Initialization, resource application, internal variable initialization, etc. are completed. Mainly involve:

[0129] Shared memory file creation, according to the nodeList (node list) to create a receiving channel file. The original logic of the sending channel file is created according to the distributiorNodes (distribution node), which may not be consistent with the receiving channel. Exemplarily, the shared memory can be the shared memory between the RDA operator and the send and receive process fwd.

[0130] Redistribution algorithm locator initialization, that is, to determine which node each tuple should be sent to for processing. Hook function, used to calculate the redistribution node of the tuple.

[0131] Data buffer structure tuplestore (corresponding to the "buffer" described above), used to temporarily store data read from the lower layer but cannot be sent due to lack of space.

[0132] Register the operator to the Forwarder process, mark the receiving / sending channel, rda_id, etc.

[0133] Step two: RDAExec.

[0134] The execution logic of the RDA operator, the RDA operator needs to complete two aspects of work:

[0135] 1. Data distribution in the cluster range. Each time data is scanned, the Dest node (target node, corresponding to the first node and the second node) is calculated according to the redistribution key, the corresponding shared memory object (that is, the queue and the process that need to share memory, the RDA operator and the send and receive process fwd of each DN interact through shared memory) and the buffer are determined. First, try to write data into shared memory for RPCService process to read. If the writing fails, write to the buffer.

[0136] Exemplarily, the buffer area can correspond to a data cache structure tuplestore, the tuplestore specifying a memory size, the memory being written into memory if sufficient, and being written into a buffer area such as a disk if insufficient.

[0137] 2. The operator pulls remote data for local calculation, determines corresponding shared memory objects and buffer areas according to a Src node (source node), reads data from the shared memory and writes into the buffer area, and reads a row of data from the buffer area for calculation.

[0138] Step three: RDAEnd.

[0139] Close RDA operator resources, including notifying a local RPCService process to end shared memory data listening, cleaning up buffer areas used by the operator, closing a shared memory file, and then ending operator operation.

[0140] Each RDA operator corresponds to an mmap file, the RDA creating the file when starting communication, closing the file when communication ends, the corresponding file being managed in a ResourceOwner and being deleted together at transaction end. RDA shared memory management is as shown in Figure 6 Each RDA corresponds to a shared memory block, including a sending channel (snd channel) and a receiving channel (rcv channel) shared memory, each channel including basic channel information such as a node id (name) and a communication cyclic queue.

[0141] As shown in Figure 7 , a corresponding redistribution overall flow is given. Specifically:

[0142] A CN issues an execution plan to a DN, the DN determining whether to obtain data from other nodes according to the execution plan content, and calling an RDA operator to implement data distribution if redistribution is needed.

[0143] In the RDA operator, a sending side and a receiving side are further divided, the sending side corresponding to the first node and the first redistribution operator, and the receiving side corresponding to the second node and the second redistribution operator.

[0144] As shown in Figure 8The figure below illustrates the process on the sending side. On the sending side, the RDA operator first checks whether there is data in the tuplestore. The tuplestore is used for data caching. When data received from lower-level operators cannot be sent out in a timely manner, the data is stored in the tuplestore. If there is no data in the tuplestore, the data is directly written to the send queue. If there is data in the tuplestore, the operator then checks whether there is space in the send queue. If not, the data received from the lower-level operator is temporarily stored in the tuplestore, awaiting the next transmission. If there is space in the send queue, the data in the tuplestore is first written to the send queue, followed by the data received from the lower-level operator.

[0145] like Figure 9 The figure below shows the receiving side process. On the receiving side, data is obtained from the receive buffer. If the data type is datarow, the datarow is encapsulated into a slot and returned to the upper-layer operator. If the data type is eof (End of File), no further data is received from this receive buffer. A check is performed to see if all receive buffers have been filled. Once this is complete, the operator terminates.

[0146] The above technical solution uses the RDA operator to replace the existing SharedQueue and RemoteSubplan logic in the existing technology to perform data redistribution for non-distributed key management. The RDA operator uses a push method, which helps to improve the efficiency of data redistribution. The RDA operator triggers two processes: pushing and pulling data. While pushing local node data to remote nodes, data is read from the local receive buffer. Each DN will only have a constant level of session processes, which saves the number of process connections and network connection consumption. By adopting a push method, data redistribution is achieved, solving the problems of process connection explosion and excessive network connection consumption in existing redistribution solutions; through shared memory and mmap methods, data interaction between different nodes is achieved, improving transmission efficiency.

[0147] It should be understood that, although the various steps in the flowcharts involved in the various embodiments described above are displayed in sequence according to the instructions of the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the flowcharts involved in the various embodiments described above can include multiple steps or multiple stages, and these steps or stages are not necessarily executed and completed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of steps or stages in other steps.

[0148] Based on the same inventive concept, embodiments of the present application also provide a data redistribution device for implementing the aforementioned data redistribution method. The implementation solution provided by this device is similar to the implementation solution described in the aforementioned method. Therefore, the specific limitations in one or more embodiments of the data redistribution device provided below can be found in the above-mentioned limitations on the data redistribution method and will not be further elaborated here.

[0149] In an exemplary embodiment, Figure 10 As shown, a data redistribution device 1000 is provided, which is applied to a first node of a distributed database, including:

[0150] The instruction receiving module 1001 is used to receive the redistribution instruction issued by the coordination node in the distributed database;

[0151] The push module 1002 is used to call the first redistribution operator corresponding to the first node based on the redistribution instruction, write the target data to be redistributed in the first node into the sending queue in the first redistribution operator, and push the target data to the second data sending and receiving process in the second node of the distributed database based on the sending queue in the first redistribution operator and the first data sending and receiving process in the first node; wherein the sending queue in the first redistribution operator and the first data sending and receiving process share the same memory area; and the second node is a remote node; the second data sending and receiving process and the receiving queue in the second redistribution operator corresponding to the second node share the same memory area; the receiving queue in the second redistribution operator is used for the second node to read the target data from the second data sending and receiving process.

[0152] In one of the embodiments, the pushing module 1002 is further configured to write the target data in the first node that needs to be redistributed into a sending queue in the first redistribution operator, so as to push the target data to the second data exchange process in the second node of the distributed database based on the sending queue in the first redistribution operator and the first data exchange process in the first node, including: in the case that the memory space corresponding to the sending queue in the first redistribution operator is in a full state, writing the target data in the first node that needs to be redistributed into a buffer corresponding to the first redistribution operator, so as to wait for writing into the sending queue in the first redistribution operator; in the case that the memory space corresponding to the sending queue in the first redistribution operator switches from the full state to an empty state, writing the target data in the buffer corresponding to the first redistribution operator into the sending queue in the first redistribution operator based on the waiting order, so as to push the target data to the second data exchange process in the second node of the distributed database based on the sending queue in the first redistribution operator and the first data exchange process in the first node.

[0153] In one of the embodiments, the pushing module 1002 is further configured to write the target data in the first node that needs to be redistributed into a sending queue in the first redistribution operator, so as to push the target data to the second data exchange process in the second node of the distributed database based on the sending queue in the first redistribution operator and the first data exchange process in the first node, and further including: in the case that the memory space corresponding to the sending queue is in an empty state and the buffer corresponding to the first redistribution operator is empty, writing the target data in the first node that needs to be redistributed into the sending queue in the first redistribution operator, so as to push the target data to the second data exchange process in the second node of the distributed database based on the sending queue in the first redistribution operator and the first data exchange process in the first node.

[0154] In one of the embodiments, the pushing module 1002 is further configured to write the target data in the first node that needs to be redistributed into a sending queue in the first redistribution operator, so as to push the target data to the second data exchange process in the second node of the distributed database based on the sending queue in the first redistribution operator and the first data exchange process in the first node, and further including: in the case that the memory space corresponding to the sending queue is in an empty state and the buffer corresponding to the first redistribution operator is empty, writing the target data in the first node that needs to be redistributed into the sending queue in the first redistribution operator, so as to push the target data to the second data exchange process in the second node of the distributed database based on the sending queue in the first redistribution operator and the first data exchange process in the first node. Figure 11 In one of the embodiments, the pushing module 1002 is further configured to write the target data in the first node that needs to be redistributed into a sending queue in the first redistribution operator, so as to push the target data to the second data exchange process in the second node of the distributed database based on the sending queue in the first redistribution operator and the first data exchange process in the first node, and further including: in the case that the memory space corresponding to the sending queue is in an empty state and the buffer corresponding to the first redistribution operator is empty, writing the target data in the first node that needs to be redistributed into the sending queue in the first redistribution operator, so as to push the target data to the second data exchange process in the second node of the distributed database based on the sending queue in the first redistribution operator and the first data exchange process in the first node.

[0155] The instruction receiving module 1101 is configured to receive a redistribution instruction issued by a coordination node in the distributed database.

[0156] The target data receiving module 1102 is configured to invoke a preset second redistribution operator corresponding to the second node based on the redistribution instruction, and receive and read target data pushed by a first data transceiving process in a first node of a distributed database based on a receiving queue in the second redistribution operator and the second data transceiving process in the second node, wherein the second data transceiving process and the receiving queue in the second redistribution operator share a same memory area, and the first node is a remote node, and the first node corresponds to a first redistribution operator; the target data is data that needs to be redistributed and is written into a sending queue in the first redistribution operator by the first node; the sending queue in the first redistribution operator and the first data transceiving process share a same memory area; and the first data transceiving process is configured to read the target data from the sending queue in the first redistribution operator and push the target data to the second data transceiving process.

[0157] In one of the embodiments, the target data receiving module 1102 is further configured to receive and read the target data pushed by the first data transceiving process in the first node of the distributed database, including: receiving the target data pushed by the first data transceiving process in the first node of the distributed database and writing the target data into a buffer corresponding to the receiving queue in the second redistribution operator; reading a data type of the target data from the buffer corresponding to the receiving queue in the second redistribution operator; and controlling a process of receiving the target data by the second node based on the read data type of the target data.

[0158] In one of the embodiments, the target data receiving module 1102 is further configured to control the process of receiving the target data by the second node based on the read data type of the target data, including: in a case where the read data type of the target data is a data row type, continuing to receive the target data; and in a case where the read data type of the target data is an end-of-file type, stopping to receive the target data and ending running of the second redistribution operator.

[0159] The above modules in the data redistribution apparatus can be realized by software, hardware, or a combination thereof, in whole or in part. The above modules can be embedded in or independent of a processor in a computer device in a hardware form, or stored in a memory in a computer device in a software form, so as to be called and executed by a processor to perform operations corresponding to the above modules.

[0160] In one of the exemplary embodiments, a computer device, which can be a server, is provided, and an internal structure diagram of the computer device can be as shown in FIG. 8. Figure 12As shown in the figure. The computer device includes a processor, a memory, an Input / Output (I / O) interface and a communication interface. Among them, the processor, the memory and the input / output interface are connected through the system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capability. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operating system and the computer program in the non-volatile storage medium to run. The database of the computer device is used to store data required for executing the data redistribution method, such as target data that needs to be redistributed. The input / output interface of the computer device is used to exchange information between the processor and external devices. The communication interface of the computer device is used to communicate with external terminals through network connection. The computer program is executed by the processor to implement a data redistribution method.

[0161] Those skilled in the art can understand that, Figure 12 The structure shown in the figure is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device can include more or fewer components than those shown in the figure, or combine certain components, or have a different component arrangement.

[0162] In one exemplary embodiment, a computer device is provided, including a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps in each of the method embodiments described above.

[0163] In one embodiment, a computer readable storage medium is provided, which stores a computer program, and the computer program is executed by a processor to implement the steps in each of the method embodiments described above.

[0164] In one embodiment, a computer program product is provided, including a computer program, and the computer program is executed by a processor to implement the steps in each of the method embodiments described above.

[0165] Those skilled in the art will understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. In particular, any reference to memory, database, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The databases involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the various embodiments provided herein may be, but are not limited to, general-purpose processors, central processing units (CPUs), graphics processing units (GPUs), digital signal processors (DSPs), programmable logic devices (PLDs), quantum computing-based data processing logic devices, artificial intelligence (AI) processors, and the like.

[0166] The technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0167] The above-described embodiments are merely illustrative of several embodiments of the present application, and the description is relatively specific and detailed, but should not be understood as a limitation on the scope of the patent. It should be noted that for those skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are all within the scope of the present application. Therefore, the scope of protection of the present application should be subject to the appended claims.

Claims

1. A data redistribution method, characterized in that: Applied to a first node of a distributed database, the method comprises: Receiving a redistribution instruction issued by a coordinating node in the distributed database; Based on the redistribution instruction, a first redistribution operator corresponding to the first node is called, and the target data to be redistributed in the first node is written into a sending queue in the first redistribution operator, so that the target data is pushed to a second data sending and receiving process in a second node of the distributed database based on the sending queue in the first redistribution operator and the first data sending and receiving process in the first node; wherein, The sending queue in the first redistribution operator and the first data sending and receiving process share the same memory area; and The second node is a remote node; the second data transceiver process and the receive queue in the second redistribution operator corresponding to the second node share the same memory area; the receive queue in the second redistribution operator is used by the second node to read the target data from the second data transceiver process.

2. The method according to claim 1, characterized in that Writing the target data to be redistributed in the first node into a sending queue in the first redistribution operator, and pushing the target data to a second data sending and receiving process in a second node of the distributed database based on the sending queue in the first redistribution operator and the first data sending and receiving process in the first node, includes: When the memory space corresponding to the sending queue in the first redistribution operator is full, write the target data to be redistributed in the first node into the buffer corresponding to the first redistribution operator to wait for being written into the sending queue in the first redistribution operator; When the memory space corresponding to the sending queue in the first redistribution operator switches from a full state to a not full state, the target data in the buffer corresponding to the first redistribution operator is written into the sending queue in the first redistribution operator based on the waiting order, so as to push the target data to the second data sending and receiving process in the second node of the distributed database based on the sending queue in the first redistribution operator and the first data sending and receiving process in the first node.

3. The method according to claim 2, characterized in that The step of writing the target data to be redistributed in the first node into a sending queue in the first redistribution operator, and pushing the target data to a second data sending and receiving process in a second node of the distributed database based on the sending queue in the first redistribution operator and the first data sending and receiving process in the first node, further includes: When the memory space corresponding to the sending queue is not full and the buffer corresponding to the first redistribution operator is empty, the target data to be redistributed in the first node is written into the sending queue in the first redistribution operator, so as to push the target data to the second data sending and receiving process in the second node of the distributed database based on the sending queue in the first redistribution operator and the first data sending and receiving process in the first node.

4. A data redistribution method, characterized in that: Applied to a second node of a distributed database, the method comprises: Receiving a redistribution instruction issued by a coordinating node in the distributed database; Based on the redistribution instruction, a preset second redistribution operator corresponding to the second node is called, and based on the receiving queue in the second redistribution operator and the second data transceiver process in the second node, the target data pushed by the first data transceiver process in the first node of the distributed database is received and read; wherein, The second data transceiver process and the receiving queue in the second redistribution operator share the same memory area; and The first node is a remote node, and the first node corresponds to a first redistribution operator; the target data is the data that needs to be redistributed and is written by the first node to the sending queue in the first redistribution operator; the sending queue in the first redistribution operator and the first data sending and receiving process share the same memory area; the first data sending and receiving process is used to read the target data from the sending queue in the first redistribution operator and push it to the second data sending and receiving process.

5. The method according to claim 4, characterized in that The receiving and reading target data pushed by the first data receiving and sending process in the first node of the distributed database includes: receiving target data pushed by a first data transceiver process in a first node of the distributed database, and writing the target data into a buffer corresponding to a receiving queue in the second redistribution operator; receiving a buffer corresponding to the queue in the second redistribution operator and reading the data type of the target data; Based on the data type of the read target data, the process of the second node receiving the target data is controlled.

6. The method according to claim 5, characterized in that The controlling the process of the second node receiving the target data based on the read data type of the target data includes: When the data type of the read target data is a data row type, continue to receive the target data; In a case where the data type of the read target data is an end-of-file type, the receiving of the target data is stopped, and the operation of the second redistribution operator is terminated.

7. A data redistribution device, characterized in that: Applied to a first node of a distributed database, the apparatus comprises: An instruction receiving module, configured to receive a redistribution instruction issued by a coordination node in the distributed database; A push module is used to call the first redistribution operator corresponding to the first node based on the redistribution instruction, write the target data to be redistributed in the first node into the sending queue in the first redistribution operator, and push the target data to the second data sending and receiving process in the second node of the distributed database based on the sending queue in the first redistribution operator and the first data sending and receiving process in the first node; wherein the sending queue in the first redistribution operator and the first data sending and receiving process share the same memory area; and the second node is a remote node; the second data sending and receiving process and the receiving queue in the second redistribution operator corresponding to the second node share the same memory area; the receiving queue in the second redistribution operator is used for the second node to read the target data from the second data sending and receiving process.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images