Data redistribution methods, apparatus, computer equipment, readable storage media, and program products

Abstract

This application relates to a data redistribution method, apparatus, computer device, computer-readable storage medium, and computer program product. The method is applied to a first node of a distributed database and includes: receiving a redistribution instruction issued by a coordinating node in the distributed database; based on the redistribution instruction, writing target data into a sending queue in a first redistribution operator, and then pushing the target data to a second data transceiver process of a second node based on the sending queue in the first redistribution operator and a first data transceiver process; the sending queue of the first redistribution operator and the first data transceiver process share the same memory area; the second node is a remote node; the second data transceiver process and the receiving queue of the second redistribution operator share the same memory area; the receiving queue of the second redistribution operator is used to read target data from the second data transceiver process. This method can reduce the number of process connections and network connection overhead required for data redistribution.

Description

Technical Field

[0001] This application relates to the field of information technology, and in particular to a data redistribution method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Technology

[0002] In distributed database systems, data redistribution refers to the reallocation of data across different storage nodes to achieve balanced data distribution and optimize performance. Existing solutions primarily utilize the RemoteSubplan operator for data redistribution, but this approach incurs significant overhead from process and network connections. Summary of the Invention

[0003] Therefore, it is necessary to provide a data redistribution method, apparatus, computer device, computer-readable storage medium, and computer program product to address the aforementioned technical problems, so as to reduce the number of process connections and network connection overhead required for data redistribution.

[0004] In a first aspect, this application provides a data redistribution method applied to the first node of a distributed database, the method comprising:

[0005] Receive redistribution instructions from the coordinating node in the distributed database;

[0006] Based on the redistribution instruction, the first redistribution operator corresponding to the first node is invoked. The target data to be redistributed in the first node is written into the sending queue of the first redistribution operator. Based on the sending queue of the first redistribution operator and the first data transmission and reception process in the first node, the target data is pushed to the second data transmission and reception process in the second node of the distributed database.

[0007] The send queue and the first data transmission process in the first redistribution operator share the same memory region; and...

[0008] The second node is a remote node; the second data transmission and reception process and the receiving queue in the second redistribution operator corresponding to the second node share the same memory area; the receiving queue in the second redistribution operator is used by the second node to read target data from the second data transmission and reception process.

[0009] In one embodiment, the target data to be redistributed in the first node is written into the sending queue in the first redistribution operator, and based on the sending queue in the first redistribution operator and the first data transmission and reception process in the first node, the target data is pushed to the second data transmission and reception process in the second node of the distributed database, including:

[0010] If the memory space corresponding to the sending queue in the first redistribution operator is full, the target data that needs to be redistributed in the first node is written into the buffer corresponding to the first redistribution operator, so as to wait to be written into the sending queue in the first redistribution operator.

[0011] When the memory space corresponding to the sending queue in the first redistribution operator changes from a full state to a partially full state, the target data in the buffer corresponding to the first redistribution operator is written into the sending queue in the first redistribution operator according to the waiting order. Based on the sending queue in the first redistribution operator and the first data sending and receiving process in the first node, the target data is pushed to the second data sending and receiving process in the second node of the distributed database.

[0012] In one embodiment, the target data to be redistributed in the first node is written into the sending queue in the first redistribution operator, and based on the sending queue in the first redistribution operator and the first data transmission and reception process in the first node, the target data is pushed to the second data transmission and reception process in the second node of the distributed database, further comprising:

[0013] When the memory space corresponding to the sending queue is not full and the buffer corresponding to the first redistribution operator is empty, the target data that needs to be redistributed in the first node is written into the sending queue in the first redistribution operator. Based on the sending queue in the first redistribution operator and the first data sending and receiving process in the first node, the target data is pushed to the second data sending and receiving process in the second node of the distributed database.

[0014] Secondly, this application also provides a data redistribution method applied to the second node of a distributed database, the method comprising:

[0015] Receive redistribution instructions from the coordinating node in the distributed database;

[0016] Based on the redistribution instruction, the preset second redistribution operator corresponding to the second node is invoked. Based on the receive queue in the second redistribution operator and the second data transceiver process in the second node, the target data pushed by the first data transceiver process in the first node of the distributed database is received and read.

[0017] The second data transmission and reception process shares the same memory region with the receive queue in the second redistribution operator; and...

[0018] The first node is a remote node, and the first node corresponds to the first redistribution operator; the target data is the data that needs to be redistributed and written into the sending queue of the first redistribution operator by the first node; the sending queue in the first redistribution operator and the first data transceiver process share the same memory area; the first data transceiver process is used to read the target data from the sending queue in the first redistribution operator and push it to the second data transceiver process.

[0019] In one embodiment, receiving and reading target data pushed by a first data transmission and reception process in the first node of the distributed database includes:

[0020] Receive the target data pushed by the first data sending and receiving process in the first node of the distributed database, and write the target data into the buffer corresponding to the receiving queue in the second redistribution operator;

[0021] Read the data type of the target data from the buffer corresponding to the receiving queue in the second redistribution operator;

[0022] Based on the data type of the target data read, control the process of the second node receiving the target data.

[0023] In one embodiment, the process of the second node receiving the target data is controlled based on the data type of the target data read, including:

[0024] If the data type of the target data being read is a data row type, continue receiving target data;

[0025] If the data type of the target data being read is an end-of-file marker, stop receiving the target data and terminate the operation of the second redistribution operator.

[0026] Thirdly, this application also provides a data redistribution device applied to the first node of a distributed database, the device comprising:

[0027] The instruction receiving module is used to receive redistribution instructions issued by the coordination node in the distributed database;

[0028] The push module is used to invoke the first redistribution operator corresponding to the first node based on the redistribution instruction, write the target data to be redistributed in the first node into the sending queue of the first redistribution operator, and push the target data to the second data transceiver process in the second node of the distributed database based on the sending queue of the first redistribution operator and the first data transceiver process in the first node; wherein, the sending queue of the first redistribution operator and the first data transceiver process share the same memory area; and, the second node is a remote node; the second data transceiver process and the receiving queue of the second redistribution operator corresponding to the second node share the same memory area; the receiving queue of the second redistribution operator is used by the second node to read the target data from the second data transceiver process.

[0029] Fourthly, this application also provides a data redistribution apparatus applied to a second node of a distributed database, the apparatus comprising:

[0030] The instruction receiving module is used to receive redistribution instructions issued by the coordination node in the distributed database;

[0031] The target data receiving module is used to call a preset second redistribution operator corresponding to the second node based on a redistribution instruction. Based on the receiving queue in the second redistribution operator and the second data transceiver process in the second node, it receives and reads the target data pushed by the first data transceiver process in the first node of the distributed database. The second data transceiver process shares the same memory area as the receiving queue in the second redistribution operator. The first node is a remote node, corresponding to the first redistribution operator. The target data is the data to be redistributed that the first node writes to the sending queue in the first redistribution operator. The sending queue in the first redistribution operator and the first data transceiver process share the same memory area. The first data transceiver process reads the target data from the sending queue in the first redistribution operator and pushes it to the second data transceiver process.

[0032] Fifthly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the methods described in the first or second aspect above.

[0033] Sixthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described in the first or second aspect above.

[0034] In a seventh aspect, this application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the methods described in the first or second aspect above.

[0035] The aforementioned data redistribution method, apparatus, computer equipment, computer-readable storage medium, and computer program product, when data redistribution is required, involve a first node sending the target data to be redistributed to a second node via a first redistribution operator to achieve data redistribution. Specifically, the first node writes the target data into the transmission queue of the first redistribution operator. Since the transmission queue of the first redistribution operator shares memory with the first data transmission and reception process, and pushes the target data to the second node, this improves the transmission efficiency of the target data. Simultaneously, the first node can send the target data directly through the first redistribution operator, reducing the number of session processes required for the first node, thereby reducing the number of process connections and network connection overhead needed for data redistribution. Attached Figure Description

[0036] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0037] Figure 1 This is a schematic diagram illustrating the data exchange format in one embodiment;

[0038] Figure 2 This is a schematic diagram illustrating the principle of an existing data redistribution scheme in one embodiment;

[0039] Figure 3 This is a flowchart illustrating a data redistribution method in one embodiment;

[0040] Figure 4 This is another flowchart illustrating a data redistribution method in one embodiment;

[0041] Figure 5 This is a schematic diagram of the specific execution flow of an RDA operator in one embodiment;

[0042] Figure 6 This is a schematic diagram of shared memory management for an RDA operator in one embodiment;

[0043] Figure 7 This is a schematic diagram of the overall data redistribution process in one embodiment;

[0044] Figure 8 This is a flowchart illustrating the sending side of a data redistribution method in one embodiment;

[0045] Figure 9 This is a flowchart illustrating the receiving side of a data redistribution method in one embodiment;

[0046] Figure 10 This is a structural block diagram of a data redistribution device in one embodiment;

[0047] Figure 11 This is a structural block diagram of another data redistribution device in one embodiment;

[0048] Figure 12 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0049] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application. Unless otherwise specified, the term "more" herein can mean "≥2".

[0050] Before describing the technical solution of this application, the following explanations are provided regarding the terms or nouns related to the technical solution of this application:

[0051] Table: In a distributed database system, a "table" refers to the basic structural unit for storing data, consisting of rows and columns. Each table can contain multiple fields (columns), each storing a specific type of data, such as integers or strings. Rows represent instances of each field, that is, specific data records.

[0052] Data redistribution refers to the process in a distributed database system of redistributing data across different storage nodes to achieve balanced data distribution and optimize performance. In this application, it can be understood as redistributing the data from one or more tables to various nodes according to the distribution key in the optimal plan during distributed computing across multiple tables.

[0053] CN: Coordinator, is the entry point for business access. The CN only stores global metadata of the system and does not store actual business data.

[0054] DN: Datanode, a node that stores business data.

[0055] RDA: Remote Data Access, is an operator specifically designed to retrieve data from remote nodes. In the technical solution of this application, RDA is a database operator used to implement data redistribution.

[0056] Forwarder: also known as FWD, is the data transmission and reception process that carries out cross-node data interaction of RDA operators.

[0057] Backend: This process is responsible for receiving and processing client requests and returning results. One client connects to one backend process.

[0058] NodeID: Each database node has a unique numerical identifier. This identifier starts from 0 and increments in one direction. It is the index of the node name in the string order.

[0059] Node OID: A database node ID, unique within a single node.

[0060] SQL, Structured Query Language.

[0061] The JOIN operation, also known as join, is a method in SQL used to retrieve data from multiple tables. It allows you to combine rows from two or more tables based on certain conditions, thereby retrieving related data from those tables.

[0062] Non-replicated tables refer to tables in distributed database systems where data is not completely replicated to every node. Instead, the data in these tables is stored in partitions, meaning it is distributed across multiple nodes according to a certain strategy (such as hashing, range, etc.).

[0063] The distribution key, also known as the distribution column, refers to the column or columns used to determine how data in a table is distributed across different nodes.

[0064] A non-distribution key refers to all columns other than the column used as the distribution key.

[0065] In database query processing and distributed computing, an operator is the basic unit that performs a specific operation or task.

[0066] mmap (Memory Map) is a method for mapping files to memory. It is used to map a file or other object into the address space of a process, establishing a one-to-one correspondence between the file's disk address and the process's virtual address space.

[0067] RPC (remote procedure call) allows a program to call a procedure or function in another address space (usually remote) as if it were a local function.

[0068] A redistribution key is a field or set of fields that determines how data is redistributed among the nodes in a distributed system. When performing operations such as JOIN, if the data involved in the operation is not on the same node, the data needs to be redistributed according to the redistribution key so that these operations can be executed correctly on the appropriate nodes.

[0069] Having described the above, the technical solution of this application will now be described:

[0070] When a distributed database executes an SQL query, the client connects to the coordinating node (CN). The CN generates the optimal query plan and distributes it to all data nodes (DNs). Each DN receives and executes the query plan from the CN, deciding whether to retrieve data from other nodes based on the execution plan. After the query is complete, it returns either the final result or an intermediate result. The CN collects all the results from the DNs, processes them according to the actual query, and returns the result to the client.

[0071] Redistribution occurs when data needs to be exchanged between data nodes. When two non-replicated tables are joined, the redistribution process is executed when the distributed column key is joined with the non-distributed column key. For example... Figure 1 As shown, when all columns involved in the join are distributed columns, the query is directly sent to the corresponding DN node. After the nodes complete the join, the results are summarized on the CN. When joining a sufficiently small table with a large table, if the join is with a non-distributed key of the large table, the small table is replicated; otherwise, the query is sent or the small table is redistributed. During redistribution, the non-distributed columns of TBL_B are hashed again, i.e., a hash function is used to send the join value of the non-distributed key on each DN node to all nodes, ensuring the integrity of the TAB_B shard data on each DN. In other words, the small table is redistributed according to the join conditions so that its data distribution matches that of the large table, facilitating the join operation. This is usually achieved through a hash function, ensuring that data rows with the same join key value are assigned to the same node.

[0072] The corresponding situations can be summarized in the following table:

[0073]

[0074] For data redistribution, existing solutions mainly utilize the RemoteSubplan operator, such as... Figure 2 In the diagram, dn1 and dn2 represent the processes of the specific execution plans on DN1 and DN2, respectively, and their working principle is as follows:

[0075] (1) The RemoteSubplan operator sends the overall next query plan downwards.

[0076] (2) DN will retrieve a connection from the pool to execute the plan.

[0077] (3) DN1 does not copy all the data of DN2 tbl b (table b on DN2), but redistributes tbl b through SharedQueue. Two processes on DN1 will bind to the SharedQueue simultaneously. The one that binds first becomes the producer, and the one that binds later becomes the consumer. The same applies to DN2. SharedQueue is a producer-consumer dynamic queue. During redistribution, each DN node will have a SharedQueue. Then, the processes on each node will bind to the SharedQueue on the node. The first one to bind will become the producer. After the producer binds, it will create a sending thread and then create a buffer DataPumpBuf for each consumer. Then the producer puts the data into the data buffer. When the amount of data in the data buffer exceeds a certain threshold (which can be set by parameters), the sending thread will send the data, completing the data redistribution.

[0078] (4) The RemoteSubplan operator of the upper layer collects data from the query plan of the lower layer.

[0079] Because the overall query plan is only sent down when the RemoteSubplan operator is encountered during execution, if there are many levels of redistribution in the query, each DN at each level will consider itself an initiator, leading to a large amount of multi-level process connections and network connections. For example, when each level of redistribution is regarded as a new initiator, it means that each redistribution operation needs to re-establish communication connections between processes or nodes, which will result in a large amount of process connection and network connection overhead.

[0080] To illustrate with a simple calculation, if 200 DN nodes have 100 concurrent queries, and each query involves 5 data redistributions, the computation will require over 100,000 connections. This problem becomes even more severe when the cluster size reaches thousands of nodes, and it's a common issue in large-scale MPP (Massively Parallel Processing).

[0081] Based on the above analysis, this application proposes a data redistribution method. Since data redistribution involves different nodes, the following explanation focuses on a single node, such as the data sending side or the receiving side, to illustrate the data redistribution method provided in this application:

[0082] In one embodiment, such as Figure 3 As shown, a data redistribution method is provided. In this embodiment, this method can be applied to the first node of a distributed database. The first node can be the node responsible for data storage in the distributed database, i.e., a data node. In this embodiment, the method includes steps S301 to S302:

[0083] Step S301: The first node receives the redistribution instruction issued by the coordinating node in the distributed database.

[0084] Distributed databases refer to databases where data is stored across multiple independent nodes, which can be located in the same place or distributed around the world.

[0085] For example, a distributed database may include a coordinating node, a first node, and a second node.

[0086] In some embodiments, the distributed database can receive data query instructions, and the coordinating node can generate a corresponding query plan and distribute it to the data nodes. A data node can then decide whether to retrieve data from other data nodes based on the content of the plan.

[0087] The redistribution instruction can be a specific query plan that a coordinating node can generate, which is a plan that requires data redistribution. For example, the data redistribution plan corresponding to the redistribution instruction may include the first node sending the target data to be redistributed to the second node.

[0088] Step S302: Based on the redistribution instruction, the first node calls the first redistribution operator corresponding to the first node, writes the target data to be redistributed in the first node into the sending queue of the first redistribution operator, and pushes the target data to the second data sending and receiving process in the second node of the distributed database based on the sending queue in the first redistribution operator and the first data sending and receiving process in the first node; wherein, the sending queue in the first redistribution operator and the first data sending and receiving process share the same memory area; and, the second node is a remote node; the second data sending and receiving process and the receiving queue in the second redistribution operator corresponding to the second node share the same memory area; the receiving queue in the second redistribution operator is used by the second node to read the target data from the second data sending and receiving process.

[0089] The second node is a remote node, which can be a computer or server not on the host currently processing the request or query. In some embodiments, the first and second nodes can be connected via a network and work together to perform tasks such as data storage, data processing, or service provision.

[0090] In some embodiments, the number of first redistribution operators can be one or more; similarly, the number of second redistribution operators can be one or more.

[0091] In some embodiments, the first redistribution operator may have a corresponding label or name, such as A1, A2, etc., and the second redistribution operator may also have a corresponding label or name, such as B1, B2, etc. For example, one first redistribution operator can correspond to one second redistribution operator, such as A1 corresponding to B1, thereby avoiding confusion.

[0092] In some embodiments, the first redistribution operator may include a sending queue, which can be used to send target data from the first node to the first data transceiver process. For example, the first node may scan the target data to be redistributed in the first node based on the first redistribution operator, and write the scanned target data into the sending queue of the first redistribution operator.

[0093] In some embodiments, the first redistribution operator may further include a receive queue, which can be used to obtain the second target data to be redistributed from the first data transmission and reception process and make it available for reading by the first node. For example, the first node can read the second target data based on the first redistribution operator.

[0094] In some embodiments, a first redistribution operator can correspond to an mmap file, which is created when the first redistribution operator starts communication and closed when communication ends.

[0095] In some embodiments, the first data transceiver process and the second data transceiver process can be used to carry out cross-node data interaction between the first redistribution operator and the second redistribution operator.

[0096] In some embodiments, the first redistribution operator may share the same memory region as the first data transceiver process; the second redistribution operator may share the same memory region as the second data transceiver process.

[0097] In the above technical solution, when data redistribution is required, the first node sends the target data to be redistributed to the second node through the first redistribution operator to achieve data redistribution. Specifically, the first node writes the target data into the sending queue of the first redistribution operator. Since the sending queue of the first redistribution operator shares memory with the first data transmission and reception process, and pushes the target data to the second node, this improves the transmission efficiency of the target data. At the same time, the first node can send the target data through the first redistribution operator, which reduces the number of session processes of the first node, thereby reducing the number of process connections and network connection overhead required for data redistribution.

[0098] In one embodiment, the aforementioned "writing the target data to be redistributed in the first node into the sending queue in the first redistribution operator, and pushing the target data to the second data sending and receiving process in the second node of the distributed database based on the sending queue in the first redistribution operator and the first data sending and receiving process in the first node" may include:

[0099] If the memory space corresponding to the sending queue in the first redistribution operator is full, the target data that needs to be redistributed in the first node is written into the buffer corresponding to the first redistribution operator, so as to wait to be written into the sending queue in the first redistribution operator.

[0100] When the memory space corresponding to the sending queue in the first redistribution operator changes from a full state to a partially full state, the target data in the buffer corresponding to the first redistribution operator is written into the sending queue in the first redistribution operator according to the waiting order. Based on the sending queue in the first redistribution operator and the first data sending and receiving process in the first node, the target data is pushed to the second data sending and receiving process in the second node of the distributed database.

[0101] In some embodiments, the buffer corresponding to the first redistribution operator may be the disk of the first node.

[0102] In some embodiments, the sending queue in the first redistribution operator can be a circular queue with limited memory space, so the target data can be temporarily stored by using the buffer corresponding to the first redistribution operator.

[0103] For example, the target data may include A, B, and C. A is located in the memory space corresponding to the sending queue in the first redistribution operator, causing the memory space corresponding to the sending queue in the first redistribution operator to be in a full state. B is located in the buffer corresponding to the first redistribution operator. Based on this, C can be written into the buffer corresponding to the first redistribution operator and wait. The corresponding waiting order is B first, C second. Therefore, after A is sent, the memory space corresponding to the sending queue in the first redistribution operator changes from a full state to a partially full state. B is written into the sending queue in the first redistribution operator first, and then C is written into the sending queue in the first redistribution operator.

[0104] The above technical solution takes into account the limited memory space corresponding to the sending queue in the first redistribution operator. It utilizes a buffer corresponding to the first redistribution operator to temporarily store the target data. Based on the state change of the memory space corresponding to the sending queue in the first redistribution operator—that is, switching from a full state to a partially full state—the target data is written into the sending queue in the first redistribution operator according to the waiting order. This ensures the orderly transmission of the target data and avoids transmission failures due to the limited memory space corresponding to the sending queue in the first redistribution operator.

[0105] In one embodiment, the aforementioned "writing the target data to be redistributed in the first node into the sending queue in the first redistribution operator, and pushing the target data to the second data sending and receiving process in the second node of the distributed database based on the sending queue in the first redistribution operator and the first data sending and receiving process in the first node" may further include:

[0106] If the memory space corresponding to the sending queue in the first redistribution operator is not full and the buffer corresponding to the first redistribution operator is empty, the target data to be redistributed in the first node is written into the sending queue in the first redistribution operator. Based on the sending queue in the first redistribution operator and the first data sending and receiving process in the first node, the target data is pushed to the second data sending and receiving process in the second node of the distributed database.

[0107] In some embodiments, as mentioned above, the memory space corresponding to the sending queue in the first redistribution operator is limited. The memory space may be full, but it may also be in a partially full state. Similarly, the buffer corresponding to the first redistribution operator also has two states: with data and without data (i.e., empty). Therefore, the following situation may occur: the memory space corresponding to the sending queue in the first redistribution operator is in a partially full state, which may always be in a partially full state or switch from a full state to a partially full state. At the same time, the buffer corresponding to the first redistribution operator is empty. In this case, the first node can directly write the target data to the sending queue without writing to the buffer corresponding to the first redistribution operator.

[0108] In the above technical solution, when the memory space corresponding to the sending queue in the first redistribution operator is not full and the buffer corresponding to the first redistribution operator is empty, the first node can directly write the target data into the sending queue in the first redistribution operator without writing it into the buffer corresponding to the first redistribution operator. This helps improve the transmission efficiency of the target data and thus improves the data redistribution efficiency.

[0109] In one exemplary embodiment, such as Figure 4As shown, a data redistribution method is provided, applied to the second node of a distributed database. This method can be applied to the second node of the distributed database, which can be the node responsible for data storage. In this embodiment, the method includes steps S401 to S402:

[0110] Step S401: The second node receives the redistribution instruction issued by the coordinating node in the distributed database.

[0111] Step S402: Based on the redistribution instruction, the second node calls the preset second redistribution operator corresponding to the second node. Based on the receiving queue in the second redistribution operator and the second data transceiver process in the second node, it receives and reads the target data pushed by the first data transceiver process in the first node of the distributed database. The second data transceiver process shares the same memory area as the receiving queue in the second redistribution operator. The first node is a remote node, and the first node corresponds to the first redistribution operator. The target data is the data that needs to be redistributed and written by the first node to the sending queue in the first redistribution operator. The sending queue in the first redistribution operator and the first data transceiver process share the same memory area. The first data transceiver process is used to read the target data from the sending queue in the first redistribution operator and push it to the second data transceiver process.

[0112] For an understanding of steps S401 to S402, please refer to the aforementioned explanation of steps S101 to S102; the same parts will not be repeated here.

[0113] In some embodiments, the second node can receive target data pushed by the first data transmission process based on the second data transmission process and the receive queue in the second redistribution operator. For example, the second node can write the target data into the buffer corresponding to the receive queue in the second redistribution operator, read the target data from the buffer, and then report it to the coordinating node.

[0114] In the above technical solution, when data redistribution is required, the second node receives the target data to be redistributed from the first node via a second redistribution operator, thereby achieving data redistribution. Specifically, the second node receives and reads the target data pushed by the first data transceiver process in the first node of the distributed database, based on the receive queue in the second redistribution operator and the second data transceiver process in the second node. Since the second data transceiver process and the receive queue in the second redistribution operator share the same memory area, the second node can receive and read the target data more efficiently. Furthermore, the second node can receive the target data directly through the second redistribution operator, reducing the number of session processes required for the second node, thus reducing the number of process connections and network connection overhead needed for data redistribution.

[0115] In one embodiment, the aforementioned "receiving and reading target data pushed by the first data transmission and reception process in the first node of the distributed database" may include: the second node receiving the target data pushed by the first data transmission and reception process in the first node of the distributed database and writing the target data into the buffer corresponding to the receiving queue in the second redistribution operator; reading the data type of the target data from the buffer corresponding to the receiving queue in the second redistribution operator; and controlling the process of the second node receiving the target data based on the data type of the target data read.

[0116] In some embodiments, the second node can determine the data type of the received target data in real time, and determine whether to end the reception of target data based on the change in the data type of the target data, that is, control the start and stop of the process of receiving target data.

[0117] In some embodiments, the buffer corresponding to the receive queue in the second redistribution operator may be the disk of the second node.

[0118] In some embodiments, the number of buffers corresponding to the receive queue in the second redistribution operator can be multiple.

[0119] The above technical solution receives target data and writes it into the buffer corresponding to the receiving queue in the second redistribution operator. It then reads the data type of the target data from the buffer and controls the process of receiving the target data based on the data type of the target data. This avoids situations where the target data has been completely transmitted but the second node is still in the receiving process, thus avoiding wasted process time.

[0120] In one embodiment, the aforementioned "controlling the process of the second node receiving target data based on the data type of the target data read" may include: the second node continuing to receive target data when the data type of the target data read is a data row type; and stopping receiving target data and ending the operation of the second redistribution operator when the data type of the target data read is an end-of-file character type.

[0121] In some embodiments, the data type of the target data may include a data row type and a file end-of-file type. The file end-of-file type may correspond to a marker used to indicate the end of a file, while the data row type may correspond to data that has actual meaning and does not indicate the end of a file.

[0122] In some embodiments, the second node can control the process of receiving target data in the corresponding buffer based on the data type of the target data read. For example, the buffers corresponding to the receive queue in the second redistribution operator may include A, B, and C. The second node can read the data type of the target data from buffer A; if it is an end-of-file marker type, then it determines that buffer A has finished receiving the target data. When all buffers corresponding to the receive queue in the second redistribution operator have finished receiving the target data, the operation of the second redistribution operator ends.

[0123] The above technical solution continues to receive target data when the target data type is a data row type; and stops receiving target data and terminates the second redistribution operator when the target data type is a file end-of-file type. This allows the second node to promptly shut down the second redistribution operator, avoiding resource occupation and waste.

[0124] In an exemplary embodiment, a data redistribution method is provided, which uses RDA operators for redistribution. For batch data acquisition, there are generally two methods: pull and push. The operator triggers both push and pull processes simultaneously. Push is cross-node, pushing data from the local node to the remote node; pull reads data from the local buffer. This application chooses the push method, which is more efficient. Simultaneously, RDA operators are used to replace the existing SharedQueue and RemoteSubplan logic for non-distributed key managed data redistribution. When RemoteSubplan processes non-distributed key join queries, the number of database connections (processes) per node is N (number of DNs) * J (number of joins) * P (number of concurrent connections), resulting in excessive resource consumption when handling complex queries. By introducing RDA operators, the number of connections per node is reduced to N*C (a constant), thus solving the problems of process connection explosion and excessive network connection consumption in existing redistribution schemes. Furthermore, this method uses shared memory and mmap to enable data interaction between different nodes, thereby improving transmission efficiency. Specifically:

[0125] This method is used when data redistribution is required in a distributed database. For example, this method can be applied to distributed databases or related systems or platforms.

[0126] The RDA operator designed in this method is a database operator for remote data access, corresponding to the first and second redistribution operators mentioned above. The RDA operator lifecycle includes three steps: RDAInit (RDA initialization), RDAExec (RDA execution), and RDAEnd (RDA termination). A 64-bit global sequence is used as the name of the RDA operator. This sequence is created during the system initialization phase and needs to consider the creation of multiple nodes. For example... Figure 5 As shown, a possible specific execution process of an RDA operator is given:

[0127] Step 1: RDAInit.

[0128] Initialization involves resource allocation, internal variable initialization, etc., and mainly includes:

[0129] Shared memory files are created by creating a receive channel file based on a nodeList. The original logic for creating a send channel file was based on distributorNodes, which may differ from the receive channel file. For example, shared memory can be shared between an RDA operator and the send / receive process fwd.

[0130] The redistribution algorithm locator initialization determines which node each tuple should be sent to for processing. Hook functions are used to calculate the redistribution nodes for tuples.

[0131] The data cache structure tuplestore (corresponding to the aforementioned "buffer") is used to temporarily store data read from the lower layer but which cannot be sent due to lack of space.

[0132] Register operators to the Forwarder process, mark receive / send channels, rda_id, etc.

[0133] Step 2: RDAExec.

[0134] The execution logic of the RDA operator involves two aspects:

[0135] 1. Data distribution within the cluster: whenever data is scanned, the Dest node (target node, corresponding to the first and second nodes mentioned above) is calculated according to the redistribution key. The corresponding shared memory objects (i.e., queues and processes that need shared memory; the RDA operators and send / receive processes fwd of each DN interact through shared memory) and buffers are determined. First, the data is attempted to be written to shared memory for the RPCService process to read. If the write fails, the data is written to the buffer.

[0136] For example, the buffer can correspond to a data cache structure tuplestore, which specifies the memory size. If there is enough memory, the data is written to memory; if not, it is placed in the buffer first, such as being written to disk.

[0137] 2. The operator retrieves remote data for local computation. Based on the Src node (source node), it determines the corresponding shared memory object and buffer. It reads data from the shared memory and writes it to the buffer, and then reads a row of data from the buffer for computation.

[0138] Step 3: RDAEnd.

[0139] Shutting down RDA operator resources includes notifying the local RPCService process to end shared memory data listening, cleaning up the buffers used by the operator, closing the shared memory file, and then ending the operator's operation.

[0140] Each RDA operator corresponds to an mmap file. The file is created when RDA communication begins and closed when communication ends. The corresponding file is managed in ResourceOwner and deleted together with the transaction at the end. RDA shared memory management is as follows: Figure 6 As shown, each RDA corresponds to a shared memory block, which contains shared memory for the send channel (snd channel) and the receive channel (rcv channel). Each channel contains basic channel information, such as the node's ID (name), and a circular queue for communication.

[0141] like Figure 7 The diagram shows the overall redistribution process. Specifically:

[0142] The CN will distribute the execution plan to the DN. The DN will then decide whether to retrieve data from other nodes based on the content of the execution plan. If redistribution is required, it will call the RDA operator to distribute the data.

[0143] In the RDA operator, it is further divided into the sending side and the receiving side. The sending side corresponds to the first node and the first redistribution operator mentioned above, and the receiving side corresponds to the second node and the second redistribution operator mentioned above.

[0144] like Figure 8The diagram illustrates the sending-side process. On the sending side, the RDA operator first checks if there is data in the tuplestore. The tuplestore is a data caching tool; when data obtained from the lower-level operator cannot be sent in a timely manner, it is stored in the tuplestore. If there is no data in the tuplestore, the data is directly written to the sending queue. If there is data in the tuplestore, it checks if there is space in the sending queue. If there is no space, the data obtained from the lower-level operator is temporarily stored in the tuplestore, awaiting the next transmission. If there is space in the sending queue, the data in the tuplestore is first written to the sending queue, and then the data obtained from the lower-level operator is written to the sending queue.

[0145] like Figure 9 The diagram illustrates the process on the receiving side. On the receiving side, data is obtained from the receive buffer. If the data type is datarow, the datarow is encapsulated into a slot and returned to the upper-level operator. If the data type is eof (End of File, indicating the end of file marker or status), data will no longer be received from this receive buffer. The operator then checks whether all receive buffers have been completed, and terminates upon completion.

[0146] The above technical solution uses RDA operators to replace the existing SharedQueue and RemoteSubplan logic in current technologies for non-distributed key data redistribution. The RDA operators use a push-based approach, which helps improve the efficiency of data redistribution. The RDA operators trigger two processes: push and pull data. While pushing local node data to remote nodes, data is simultaneously read from the local receive buffer. Each DN has only a constant number of session processes, saving on process connection counts and network connection consumption. By adopting a push-based approach, data redistribution is achieved, solving the problems of excessive process connection counts and network connection consumption in existing redistribution schemes. Shared memory and mmap methods enable data interaction between different nodes, improving transmission efficiency.

[0147] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0148] Based on the same inventive concept, this application also provides a data redistribution apparatus for implementing the data redistribution method described above. The solution provided by this apparatus is similar to the implementation scheme described in the above method; therefore, the specific limitations in one or more data redistribution apparatus embodiments provided below can be found in the limitations of the data redistribution method described above, and will not be repeated here.

[0149] In one exemplary embodiment, such as Figure 10 As shown, a data redistribution device 1000 is provided, applied to the first node of a distributed database, comprising:

[0150] The instruction receiving module 1001 is used to receive redistribution instructions issued by the coordination node in the distributed database;

[0151] The push module 1002 is used to call the first redistribution operator corresponding to the first node based on the redistribution instruction, write the target data to be redistributed in the first node into the sending queue of the first redistribution operator, and push the target data to the second data receiving process in the second node of the distributed database based on the sending queue of the first redistribution operator and the first data receiving process in the first node; wherein, the sending queue of the first redistribution operator and the first data receiving process share the same memory area; and, the second node is a remote node; the second data receiving process and the receiving queue of the second redistribution operator corresponding to the second node share the same memory area; the receiving queue of the second redistribution operator is used by the second node to read the target data from the second data receiving process.

[0152] In one embodiment, the push module 1002 is further configured to write the target data to be redistributed in the first node into the sending queue in the first redistribution operator, so as to push the target data to the second data sending process in the second node of the distributed database based on the sending queue in the first redistribution operator and the first data sending and receiving process in the first node. This includes: when the memory space corresponding to the sending queue in the first redistribution operator is full, writing the target data to be redistributed in the first node into the buffer corresponding to the first redistribution operator to wait for writing into the sending queue in the first redistribution operator; when the memory space corresponding to the sending queue in the first redistribution operator changes from full to not full, writing the target data in the buffer corresponding to the first redistribution operator into the sending queue in the first redistribution operator according to the waiting order, so as to push the target data to the second data sending and receiving process in the second node of the distributed database based on the sending queue in the first redistribution operator and the first data sending and receiving process in the first node.

[0153] In one embodiment, the push module 1002 is further configured to write the target data to be redistributed in the first node into the sending queue in the first redistribution operator, so as to push the target data to the second data sending process in the second node of the distributed database based on the sending queue in the first redistribution operator and the first data sending and receiving process in the first node. The push module 1002 is further configured to write the target data to be redistributed in the first node into the sending queue in the first redistribution operator when the memory space corresponding to the sending queue is not full and the buffer corresponding to the first redistribution operator is empty, so as to push the target data to the second data sending and receiving process in the second node of the distributed database based on the sending queue in the first redistribution operator and the first data sending and receiving process in the first node.

[0154] In one exemplary embodiment, such as Figure 11 As shown, a data redistribution device 1100 is also provided, applied to the second node of a distributed database, comprising:

[0155] The instruction receiving module 1101 is used to receive redistribution instructions issued by the coordination node in the distributed database;

[0156] The target data receiving module 1102 is used to call a preset second redistribution operator corresponding to the second node based on a redistribution instruction, and to receive and read target data pushed by the first data transceiver process in the first node of the distributed database based on the receiving queue in the second redistribution operator and the second data transceiver process in the second node; wherein, the second data transceiver process and the receiving queue in the second redistribution operator share the same memory area; and, the first node is a remote node, and the first node corresponds to the first redistribution operator; the target data is the data that needs to be redistributed written by the first node to the sending queue in the first redistribution operator; the sending queue in the first redistribution operator and the first data transceiver process share the same memory area; the first data transceiver process is used to read the target data from the sending queue in the first redistribution operator and push it to the second data transceiver process.

[0157] In one embodiment, the target data receiving module 1102 is further configured to receive and read target data pushed by the first data sending and receiving process in the first node of the distributed database, including: receiving the target data pushed by the first data sending and receiving process in the first node of the distributed database, and writing the target data into the buffer corresponding to the receiving queue in the second redistribution operator; reading the data type of the target data from the buffer corresponding to the receiving queue in the second redistribution operator; and controlling the process of the second node receiving the target data based on the data type of the target data read.

[0158] In one embodiment, the target data receiving module 1102 is further configured to control the process of the second node receiving target data based on the data type of the target data read, including: continuing to receive target data when the data type of the target data read is a data row type; and stopping receiving target data and ending the operation of the second redistribution operator when the data type of the target data read is a file end character type.

[0159] Each module in the aforementioned data redistribution device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0160] In one exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 12As shown, this computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operating system and computer programs in the non-volatile storage media to run. The database stores data required for implementing a data redistribution method, such as the target data to be redistributed. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network connection. When the computer program is executed by the processor, it implements a data redistribution method.

[0161] Those skilled in the art will understand that Figure 12 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0162] In one exemplary embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0163] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps in the above method embodiments.

[0164] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0165] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0166] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0167] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A data redistribution method, characterized in that, The method, applied to the first node of a distributed database, includes: Receive redistribution instructions issued by the coordination node in the distributed database; Based on the redistribution instruction, the first redistribution operator corresponding to the first node is invoked, and the target data to be redistributed in the first node is written into the sending queue in the first redistribution operator. Then, based on the sending queue in the first redistribution operator and the first data transmission / reception process in the first node, the target data is pushed to the second data transmission / reception process in the second node of the distributed database. The sending queue in the first redistribution operator and the first data transceiver process share the same memory region; and, The second node is a remote node; the second data transceiver process and the receive queue in the second redistribution operator corresponding to the second node share the same memory area; the receive queue in the second redistribution operator is used by the second node to read the target data from the second data transceiver process.

2. The method according to claim 1, characterized in that, The step of writing the target data to be redistributed in the first node into the sending queue in the first redistribution operator, and then, based on the sending queue in the first redistribution operator and the first data sending and receiving process in the first node, pushing the target data to the second data sending and receiving process in the second node of the distributed database, includes: When the memory space corresponding to the sending queue in the first redistribution operator is full, the target data that needs to be redistributed in the first node is written into the buffer corresponding to the first redistribution operator, so as to wait to be written into the sending queue in the first redistribution operator. When the memory space corresponding to the sending queue in the first redistribution operator changes from a full state to a partially full state, the target data in the buffer corresponding to the first redistribution operator is written into the sending queue in the first redistribution operator according to the waiting order, so as to push the target data to the second data sending process in the second node of the distributed database based on the sending queue in the first redistribution operator and the first data sending and receiving process in the first node.

3. The method according to claim 2, characterized in that, The step of writing the target data to be redistributed in the first node into the sending queue in the first redistribution operator, and then pushing the target data to the second data sending and receiving process in the second node of the distributed database based on the sending queue in the first redistribution operator and the first data sending and receiving process in the first node, further includes: When the memory space corresponding to the sending queue is not full and the buffer corresponding to the first redistribution operator is empty, the target data to be redistributed in the first node is written into the sending queue in the first redistribution operator. Based on the sending queue in the first redistribution operator and the first data sending and receiving process in the first node, the target data is pushed to the second data sending and receiving process in the second node of the distributed database.

4. A data redistribution method, characterized in that, The method, applied to the second node of a distributed database, includes: Receive redistribution instructions issued by the coordination node in the distributed database; Based on the redistribution instruction, a preset second redistribution operator corresponding to the second node is invoked. Based on the receive queue in the second redistribution operator and the second data transceiver process in the second node, the target data pushed by the first data transceiver process in the first node of the distributed database is received and read. The second data transmission and reception process shares the same memory region with the receive queue in the second redistribution operator; and, The first node is a remote node, and the first node corresponds to the first redistribution operator; the target data is the data that needs to be redistributed and written into the sending queue of the first redistribution operator by the first node; the sending queue in the first redistribution operator and the first data transceiver process share the same memory area; the first data transceiver process is used to read the target data from the sending queue in the first redistribution operator and push it to the second data transceiver process.

5. The method according to claim 4, characterized in that, The process of receiving and reading the target data pushed by the first data sending and receiving process in the first node of the distributed database includes: Receive target data pushed by the first data sending and receiving process in the first node of the distributed database, and write the target data into the buffer corresponding to the receiving queue in the second redistribution operator; Read the data type of the target data from the buffer corresponding to the receiving queue in the second redistribution operator; Based on the data type of the target data read, the process of the second node receiving the target data is controlled.

6. The method according to claim 5, characterized in that, The process of controlling the second node to receive the target data based on the data type of the read target data includes: If the data type of the target data read is a data row type, continue to receive the target data; If the data type of the target data read is an end-of-file marker, stop receiving the target data and terminate the operation of the second redistribution operator.

7. A data redistribution device, characterized in that, The device, applied to the first node of a distributed database, includes: The instruction receiving module is used to receive redistribution instructions issued by the coordination node in the distributed database; The push module is used to invoke the first redistribution operator corresponding to the first node based on the redistribution instruction, write the target data to be redistributed in the first node into the sending queue of the first redistribution operator, and push the target data to the second data sending / receiving process in the second node of the distributed database based on the sending queue of the first redistribution operator and the first data sending / receiving process in the first node; wherein the sending queue of the first redistribution operator and the first data sending / receiving process share the same memory area; and the second node is a remote node; the second data sending / receiving process and the receiving queue of the second redistribution operator corresponding to the second node share the same memory area; the receiving queue of the second redistribution operator is used by the second node to read the target data from the second data sending / receiving process.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images