Data processing method and device, computer equipment, readable storage medium and product

Abstract

The invention relates to a data processing method and device, computer equipment, a readable storage medium and a product, and relates to the technical field of big data. The method comprises the following steps: in response to a data processing request, creating a data processing task containing a data redistribution operator through a coordination node, and issuing the data processing task to each data node; if each data node executes the task node where the data redistribution operator is located, creating a data sending process; executing a sub-processing task corresponding to the data redistribution operator through a data sending process, and sending a corresponding data processing result to a data transfer process; transmitting a data processing result to other data nodes through a data transfer process; and converting the process currently used for receiving the data processing task into a data receiving process, and receiving data processing results from other data nodes through the data receiving process and the data transfer process. By adopting the method, efficient data redistribution can be realized.

Description

Technical Field

[0001] This application relates to the field of big data technology, and in particular to a data processing method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Technology

[0002] In distributed database systems, data redistribution is the core operation for achieving cross-node collaborative data computation. Traditional solutions use data redistribution operators to achieve data redistribution. During the execution of these operators, each data node must act as the initiator to establish network connections with all other nodes and start the corresponding processes.

[0003] However, as data processing complexity increases, multi-distribution operations lead to a surge in the number of processes, requiring individual nodes to create and manage massive amounts of processes. This results in a sharp depletion of system resources, limiting stability and scalability, and consequently impacting the efficiency of data redistribution. Therefore, achieving efficient data redistribution has become a pressing issue that needs to be addressed. Summary of the Invention

[0004] Therefore, it is necessary to provide a data processing method, apparatus, computer equipment, computer-readable storage medium, and computer program product that can achieve efficient data redistribution in response to the above-mentioned technical problems.

[0005] In a first aspect, this application provides a data processing method applied to a distributed data system, the distributed data system including a coordinating node and multiple data nodes, the method comprising:

[0006] In response to data processing requests, a data processing task containing a data redistribution operator is created through the coordinating node, and the data processing task is distributed to each data node.

[0007] For each data node, if the execution of a data processing task reaches the task node where the data redistribution operator is located, a data sending process is created.

[0008] The data sending process executes the sub-processing tasks corresponding to the data redistribution operator and sends the data processing results corresponding to the sub-processing tasks to the data relay process of the data node. The sub-processing tasks belong to the data processing tasks, and the data relay process is connected to the data sending process.

[0009] The data processing results are transmitted to other data nodes through a data relay process;

[0010] The process currently used for receiving and processing data tasks is converted into a data receiving process, which is connected to the data relay process.

[0011] The data receiving process and the data relay process receive data processing results from other data nodes.

[0012] In one embodiment, a data processing task containing a data redistribution operator is created via a coordinating node, including:

[0013] A task to be processed is created by a coordinating node to match the data processing request, wherein the task to be processed contains at least one initial redistribution operator;

[0014] Locate the level at which each initial redistribution operator belongs;

[0015] Filter out the initial redistribution operators where the number of levels exceeds the level threshold;

[0016] Replace the initial redistribution operator with a number of levels exceeding the level threshold with a data redistribution operator, and update the task to be processed to obtain a data processing task containing the data redistribution operator.

[0017] In one embodiment, creating a pending task that matches a data processing request via a coordinating node includes:

[0018] An initial processing task matching the data processing request is created by a coordinating node;

[0019] Perform execution strategy matching on the initial processing task to obtain at least one task execution strategy that matches the initial processing task;

[0020] Cost analysis is performed on each task execution strategy to obtain the task execution cost of each strategy.

[0021] Based on the task execution strategy that satisfies the task execution cost condition, the initial processing task is updated to obtain the task to be processed.

[0022] In one embodiment, the data processing method further includes:

[0023] Obtain the operator identifier of the data redistribution operator;

[0024] A shared memory region corresponding to a data node is created based on the operator identifier. The shared memory region refers to the memory region where data is shared among the data sending process, the data receiving process, and the data relay process.

[0025] In one embodiment, the shared memory region includes a send buffer region corresponding to the target data node, where the target data node is at least one of the other data nodes used to receive the data processing results of the sub-processing task; the data relay process that sends the data processing results corresponding to the sub-processing task to the data node includes:

[0026] If there is remaining capacity in the send buffer area corresponding to the target data node, the data processing results of the sub-processing task will be sent to the send buffer area.

[0027] The data processing results are transmitted to other data nodes through a data relay process, including:

[0028] The data relay process reads the data processing results corresponding to the sub-processing tasks from the sending buffer area and transmits the data processing results corresponding to the sub-processing tasks to the target data node.

[0029] In one embodiment, the shared memory region further includes a receive buffer region corresponding to the target data node; and receives data processing results from other data nodes through a data receiving process and a data relay process, including:

[0030] The data processing results from other data nodes are stored in the receiving buffer area through the data relay process;

[0031] The data receiving process reads the data processing results from other data nodes within the receiving buffer area.

[0032] Secondly, this application also provides a data processing apparatus for use in a distributed data system based on the data processing method of the first aspect. The distributed data system includes a coordinating node and multiple data nodes. The apparatus includes:

[0033] The task creation and distribution module is used to respond to data processing requests, create data processing tasks containing data redistribution operators through the coordination node, and distribute the data processing tasks to each data node.

[0034] The data sending process creation module is used to create a data sending process for each data node when the execution of data processing tasks reaches the task node where the data redistribution operator is located.

[0035] The data relay process module is used to execute the sub-processing tasks corresponding to the data redistribution operator through the data sending process, and send the data processing results corresponding to the sub-processing tasks to the data relay process of the data node. The sub-processing tasks belong to the data processing tasks, and the data relay process is connected to the data sending process.

[0036] The data sending module is used to transmit the data processing results to other data nodes through the data relay process;

[0037] The data receiving process creation module is used to convert the process currently used for receiving and processing data into a data receiving process, wherein the data receiving process is connected to the data relay process;

[0038] The data receiving module is used to receive data processing results from other data nodes through the data receiving process and the data relay process.

[0039] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0040] In response to data processing requests, a data processing task containing a data redistribution operator is created through the coordinating node, and the data processing task is distributed to each data node.

[0041] For each data node, if the execution of a data processing task reaches the task node where the data redistribution operator is located, a data sending process is created.

[0042] The data sending process executes the sub-processing tasks corresponding to the data redistribution operator and sends the data processing results corresponding to the sub-processing tasks to the data relay process of the data node. The sub-processing tasks belong to the data processing tasks, and the data relay process is connected to the data sending process.

[0043] The data processing results are transmitted to other data nodes through a data relay process;

[0044] The process currently used for receiving and processing data tasks is converted into a data receiving process, which is connected to the data relay process.

[0045] The data receiving process and the data relay process receive data processing results from other data nodes.

[0046] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the following steps:

[0047] In response to data processing requests, a data processing task containing a data redistribution operator is created through the coordinating node, and the data processing task is distributed to each data node.

[0048] For each data node, if the execution of a data processing task reaches the task node where the data redistribution operator is located, a data sending process is created.

[0049] The data sending process executes the sub-processing tasks corresponding to the data redistribution operator and sends the data processing results corresponding to the sub-processing tasks to the data relay process of the data node. The sub-processing tasks belong to the data processing tasks, and the data relay process is connected to the data sending process.

[0050] The data processing results are transmitted to other data nodes through a data relay process;

[0051] The process currently used for receiving and processing data tasks is converted into a data receiving process, which is connected to the data relay process.

[0052] The data receiving process and the data relay process receive data processing results from other data nodes.

[0053] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, performs the following steps:

[0054] In response to data processing requests, a data processing task containing a data redistribution operator is created through the coordinating node, and the data processing task is distributed to each data node.

[0055] For each data node, if the execution of a data processing task reaches the task node where the data redistribution operator is located, a data sending process is created.

[0056] The data sending process executes the sub-processing tasks corresponding to the data redistribution operator and sends the data processing results corresponding to the sub-processing tasks to the data relay process of the data node. The sub-processing tasks belong to the data processing tasks, and the data relay process is connected to the data sending process.

[0057] The data processing results are transmitted to other data nodes through a data relay process;

[0058] The process currently used for receiving and processing data tasks is converted into a data receiving process, which is connected to the data relay process.

[0059] The data receiving process and the data relay process receive data processing results from other data nodes.

[0060] The aforementioned data processing method, apparatus, computer equipment, computer-readable storage medium, and computer program product are applied to a distributed data system, which includes a coordinating node and multiple data nodes. In response to a data processing request, a data processing task containing a data redistribution operator is first created through the coordinating node and distributed to each data node. For each data node, during the execution of the data processing task, if the execution reaches the task node containing the data redistribution operator, a data sending process is created. This data sending process executes the sub-processing task corresponding to the data redistribution operator and sends the data processing result corresponding to the sub-processing task to the data relay process of the data node. The sub-processing task is part of the data processing task, and the data relay process is connected to the data sending process, thus allowing the data processing result to be transmitted to other data nodes. The process currently used to receive data processing tasks is automatically converted into a data receiving process, which is also connected to the data relay process. Through the data receiving process and the data relay process, data nodes can receive data processing results from other data nodes. Therefore, this solution splits the data redistribution operator into corresponding data sending and receiving processes, forming a fixed process collaboration architecture. This reduces the number of processes required to perform redistribution operations on a single node from a multiple related to cluster size and the number of operators to a constant related only to the number of operators, remaining constant with increasing cluster size. This fundamentally solves the problem of a dramatic increase in processes and connections in large-scale clusters. Simultaneously, by unifying network communication through a data relay process, network overhead is significantly reduced. Ultimately, this solution ensures efficient data redistribution while maintaining controllability and stability of system resources. Attached Figure Description

[0061] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0062] Figure 1 This is a diagram illustrating the application environment of a data processing method in one embodiment.

[0063] Figure 2 This is a flowchart illustrating a data processing method in one embodiment;

[0064] Figure 3 This is a schematic diagram of the data processing task generation process in one embodiment;

[0065] Figure 4This is a schematic diagram of the data processing task generation process in another embodiment;

[0066] Figure 5 This is a schematic diagram of the shared memory creation process in one embodiment;

[0067] Figure 6 This is a flowchart illustrating a data processing method in a specific embodiment;

[0068] Figure 7 This is a flowchart illustrating the RDA operator replacement process in a specific embodiment.

[0069] Figure 8 This is a schematic diagram of the RDA operator initialization process in a specific embodiment;

[0070] Figure 9 This is a flowchart illustrating the execution of an RDA operator in a specific embodiment;

[0071] Figure 10 This is a flowchart illustrating the process of ending the execution of an RDA operator in a specific embodiment.

[0072] Figure 11 This is a schematic diagram illustrating the transmission of RDA operators between nodes in a specific embodiment;

[0073] Figure 12 This is a structural block diagram of a data processing device in one embodiment;

[0074] Figure 13 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0075] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0076] It should be noted that the terms "first," "second," etc., used in this application can be used to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish the first element from the second element. The terms "comprising" and "having," and any variations thereof, used in this application, are intended to cover non-exclusive inclusion. The term "multiple" used in this application refers to two or more. The term "and / or" used in this application refers to one of the embodiments, or any combination of multiple embodiments.

[0077] In distributed database systems, data redistribution is the core operation for achieving cross-node collaborative data computation. It is responsible for redistributing or balancing data among different nodes. Therefore, the data redistribution operator is a key operator in the distributed data processing system.

[0078] In traditional solutions, when handling data redistribution, such as multi-table joins, the coordinating node introduces a dedicated redistribution operator responsible for cross-node data exchange within the generated data processing task. A typical implementation is the Remotesubplan operator. The Remotesubplan operator employs a fully interconnected architecture during execution. Each data node, upon encountering this operator, acts as an independent initiator, establishing a dedicated network connection with all other data nodes in the cluster and launching corresponding processes to handle data transmission and reception tasks. This means that a single redistribution operator results in each data node generating a number of processes proportional to the cluster size. Specifically, with N data nodes, each data node's processes include one session process handling the execution plan issued by the coordinating node, and N data processes handling data consumption requests from other data nodes and the node itself. Therefore, for one Remotesubplan operator, a single data node needs to maintain N+1 processes; for M Remotesubplan operators, a single data node needs to maintain M*(N+1) processes. If the system also enables parallel execution within nodes, assuming a parallelism degree of K, then the number of processes that a single data node needs to maintain can reach the order of M*(N+1)*K. This rapid expansion of process size consumes a large amount of system resources, becoming a key bottleneck restricting the scalability and stability of distributed databases, and consequently affecting the efficiency of data redistribution. Therefore, how to achieve efficient data redistribution has become an urgent problem to be solved.

[0079] To address the aforementioned problems, embodiments of this application provide a data processing method that can be applied to, for example... Figure 1 The application environment shown is as follows. In this application environment, terminal 102 communicates with distributed data system 104 via a network. Terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, etc. Distributed data system 104 is a server cluster or distributed system composed of multiple physical servers, including a coordinator node (CN) and multiple data node nodes (DN). The coordinator node is responsible for coordinating and managing the entire cluster. Its main functions include receiving user data processing requests, generating optimal data processing tasks, and distributing tasks to multiple data nodes for execution. Data nodes are responsible for storing data and indexes and are the physical nodes that actually perform data processing.

[0080] For example, in response to a data processing request initiated by a user through terminal 102, distributed data system 104 first creates a data processing task containing a data redistribution operator through a coordinating node in distributed data system 104, and distributes the data processing task to each data node in distributed data system 104; for each data node, when executing the data processing task, if the execution reaches the task node where the data redistribution operator is located, a data sending process is created; the data sending process executes the sub-processing task corresponding to the data redistribution operator, and sends the data processing result corresponding to the sub-processing task to the data relay process of the data node, wherein the sub-processing task belongs to the data processing task, and the data relay process is connected to the data sending process; the data relay process transmits the data processing result to other data nodes; the process currently used to receive data processing tasks is converted into a data receiving process, wherein the data receiving process is connected to the data relay process; the data receiving process and the data relay process receive data processing results from other data nodes.

[0081] It should be understood that the data processing request initiated through terminal 102 is merely an exemplary method for triggering a data processing task. In other embodiments of this application, the data processing request may also be automatically generated internally by the system, such as by a scheduled task, event-driven mechanism, or other application programming interface (API) calls. The scope of protection of this application is not limited to the specific source of the data processing request.

[0082] In one exemplary embodiment, such as Figure 2 As shown, a data processing method is provided, which can be applied to... Figure 1 Taking the distributed data system 104 in the example, the following steps are included:

[0083] In step S202, in response to the data processing request, a data processing task containing a data redistribution operator is created through the coordinating node, and the data processing task is distributed to each data node.

[0084] In this context, a data processing request refers to a request that requires processing by the distributed data system, such as a data query request involving multiple table joins. The coordinating node is the control node of the distributed data system, responsible for receiving user requests, parsing requests, generating data processing tasks, and coordinating the execution of tasks by various data nodes. The coordinating node does not store data; it is only responsible for task optimization and scheduling. Data nodes are the physical nodes in the distributed data system that store data and execute data processing tasks. Each data node is responsible for reading and writing local data and executing operators. Data redistribution operators are used in distributed data processing tasks to redistribute data among data nodes. In this embodiment, the data redistribution operator is specifically the RDA (Remote Data Access) operator, replacing the original Remotesubplan operator. The core logic of the RDA operator is to realize data interaction between nodes through a fixed number of dedicated processes combined with shared memory communication and a unified network forwarding process. That is, each RDA operator creates only one data sending process, which is converted from the execution master process to the data receiving process. The fundamental difference between the RDA operator and the Remotesubplan operator is that the Remotesubplan operator needs to create a number of processes on each data node that are proportional to the cluster size, i.e., the number of data nodes and the parallelism within the node, causing the number of processes to increase dramatically as the cluster expands. In contrast, the RDA operator, through process separation and shared memory communication mechanisms, controls the number of processes on a single node to a constant level of "number of RDA operators + 1", thereby solving the problem of process expansion under large-scale clusters and significantly improving the efficiency and scalability of distributed data processing. A data processing task refers to a distributed data processing task or plan generated by a coordinating node, such as a distributed data query task. It may contain a series of execution operators such as scanning, joining, and redistribution, which are used to execute in parallel on various data nodes.

[0085] For example, a user submits a data processing request, such as an SQL query, to a distributed data system. Upon receiving the request, the coordinating node in the distributed data system parses its syntax to obtain the corresponding syntax tree. Based on the syntax tree, it finds the optimal data processing path and generates a data processing task containing that path. The coordinating node also inserts at least one data redistribution operator (RDA operator) into this data processing task according to preset rules. Finally, the data processing task containing the RDA operator is distributed by the coordinating node to each data node via the network.

[0086] Step S204: For each data node, if the execution of the data processing task reaches the task node where the data redistribution operator is located, a data sending process is created.

[0087] In this context, a task node refers to a specific operator node in a data processing task, such as a scan node, a join node, or a redistribution operator node. The data sending process, also known as the RDA sender process, is created by the data node when executing the data redistribution operator (RDA operator). It is specifically responsible for executing the sharding task corresponding to the RDA operator and sending the data.

[0088] For example, when any data node receives a data processing task from the coordinating node, its executor can start a backend process to execute the task. The executor is a software module within the data node responsible for specifically executing the data processing task, while the backend process is an independent operation process. The backend process can use a volcano model to call operator execution functions layer by layer. The volcano model is a task execution model, also known as an iterator model. In this model, each operator implements an execution function. Upper-level operators repeatedly call the execution functions of lower-level operators to obtain result tuples row by row, thereby realizing pipelined data processing. When the execution reaches the task node where the data redistribution operator resides, i.e., the RDA operator node, the executor calls the corresponding initialization function (RDA Init function). This function is responsible for completing the preliminary preparation work for the RDA operator, including: generating a globally unique identifier for the current RDA operator, such as a RAD ID, used as an identifier for cross-node and cross-process communication; allocating a shared memory region and a shared memory file. The shared memory region is a memory area shared by the data sending process, data receiving process, and data relay process. The shared memory file is a file named after the RDA operator identifier, serving as a handle to the shared memory region. Related processes can access the same shared memory region by mapping this file. Finally, the RDA operator identifier is sent to a fixed data relay process on this node, also known as the Rorwarder process, through the local Socket mechanism. The local Socket mechanism is a mechanism for inter-process communication within the data node. After completing the above operations, the executor then creates a new operation process on this node as a data sending process. This process is specifically responsible for executing the processing tasks under the current RDA operator and sending data.

[0089] In some embodiments, after the RDA sender process is successfully created, the backend process immediately performs task slicing, that is, from the complete data processing task, the current RDA operator node and its entire left subtree, which is the sub-processing task that the operator is responsible for, are identified and segmented to form an independent execution task and sent to the RDA sender process.

[0090] Step S206: The data sending process executes the sub-processing task corresponding to the data redistribution operator and sends the data processing result corresponding to the sub-processing task to the data relay process of the data node. The sub-processing task belongs to the data processing task, and the data relay process is connected to the data sending process.

[0091] In this context, a sub-processing task refers to the sub-plan corresponding to the data redistribution operator, i.e., the left subtree of the current RDA operator, representing the data computation part that needs to be redistributed, such as scanning, filtering, and aggregation. The data processing result is a tuple generated after the sub-processing task is executed. The data relay process, or Rorwarder process, is responsible for data forwarding between nodes. Each data node has a global Rorwarder process that handles network communication for all RDA operators.

[0092] For example, the core responsibility of the RDA sender process is to execute the left subtree of the RDA operator, that is, the local data processing tasks handled by the RDA operator, such as data scanning and filtering, and to obtain the data processing results, i.e., tuples, layer by layer through the volcano model. For each tuple obtained, the RDA sender process recalculates which target data node the tuple should be sent to based on a preset data distribution method, such as a hash value based on the join key. Subsequently, the process writes the tuple into a shared memory area uniquely identified by the RDAID. This shared memory area is logically divided into a send buffer area and a receive buffer area, and the process places the tuple into the send buffer area. At the same time, the Rorwarder process on the data node efficiently senses the data sending status through a coordination mechanism with the RDA sender process, such as event notification. Once the Rorwarder process detects that there is data in the send buffer area, it can read this data and accurately send it to the corresponding target data node.

[0093] Step S208: The data processing results are transmitted to other data nodes through a data relay process.

[0094] Other data nodes refer to the nodes in the data node cluster of the distributed data system that are not the current data node.

[0095] For example, the data relay process, i.e., the Rorwarder process, continuously monitors the send buffer area in the shared memory region, or receives event notifications from the data sending process, i.e., the RDA sender process. When data processing results, i.e., tuples, exist in the send buffer area, the Rorwarder process sends these tuples to the Rorwarder process of the target data node through a pre-established long-lived socket connection. After receiving the tuples, the Rorwarder process of the target data node places them into the receive buffer area of ​​the corresponding shared memory region of the target data node according to their unique RDA ID. Subsequently, the data receiving process of the target data node retrieves the tuples from the receive buffer area and passes them up to the upper-level operators in the data processing plan for further processing, thereby completing the entire data redistribution process between nodes.

[0096] Step S210: Convert the process currently used for receiving data processing tasks into a data receiving process, wherein the data receiving process is connected to the data relay process.

[0097] The process currently used to receive data processing tasks—that is, the backend process that initially receives data processing tasks for the data node—automatically transforms into a data receiving process, also known as the RDA sender process, after completing the initialization function call and creating the data sending process. The core of this role transformation lies in the redistribution of responsibilities. The original process transfers the sub-processing tasks corresponding to the RDA operators to the newly created RDA sender process, freeing itself to become a data receiving process, specifically responsible for receiving and transmitting data upwards.

[0098] For example, after the backend process successfully calls the initialization function and creates a dedicated data sending process, it understands the role transition and becomes the data receiving process, i.e., the RDA recver process. This transition involves several key steps to complete the change in its responsibilities: First, it clears the sub-processing tasks. Since the sub-processing tasks, i.e., the left subtree of the current RDA operator, have been completely handed over to the RDA sender process for execution, the RDA recver process clears its internal left subtree references. Second, it establishes a receive channel. The RDA recver process maps a shared memory file to access the receive buffer area in the shared memory region, thereby receiving data from other nodes. Finally, it registers the process. The RDA sender process registers with the Rorwarder process on its local node via a local socket to establish a connection with the Rorwarder process. This allows the Rorwarder process to write the data received with the corresponding RAD ID into the receive buffer area monitored by the RDA sender process, ensuring accurate data reception. After completing the above steps, the RDA recver process enters the working state, continuously monitoring the receive buffer area, waiting for and consuming data from other nodes. In this way, by changing the process role, the system does not need to create additional operation processes for data reception, which significantly reduces the number of processes.

[0099] Step S212: Receive data processing results from other data nodes through the data receiving process and the data relay process.

[0100] For example, the Rorwarder process receives data from other data nodes and places the data into the receive buffer area of ​​the shared memory region of the local node according to the RDAID. The RDA recver process continuously monitors the receive buffer area to read tuples from other data nodes and pass them to the upper-layer operators for consumption.

[0101] In some embodiments, when a subprocessing task completes or the RAD operator needs to be terminated, an ordered resource cleanup and exit process can be executed by calling the operator termination function (RDA End function). This process proceeds according to the process role: For the RDA sender process, it first notifies the Rorwarder process to refresh and send all remaining tuples in the shared memory region to ensure no data loss. Then, it terminates the execution of its internal left subtree and finally clears the shared memory, temporary files, and other resources used by the task execution, and releases variables. For the RDA recver process, it first notifies the Rorwarder process to discard any tuples still in transmission but no longer needed to quickly release network and memory resources. Then, it sends a stop flag, such as a "close" flag, to the RDA sender process belonging to the same RDA operator via inter-process communication, collaboratively triggering the other's termination process. Finally, the RDA recver process performs its own resource cleanup. This bidirectional collaborative termination mechanism not only ensures data integrity but also avoids resource leaks, guaranteeing system stability in high-concurrency scenarios.

[0102] In this embodiment, in response to a data processing request, the distributed data system first creates a data processing task containing a data redistribution operator through its coordinating node and distributes the data processing task to each data node in the system. For each data node, during the execution of the data processing task, if the execution reaches the task node containing the data redistribution operator, a data sending process is created. This data sending process executes the sub-processing task corresponding to the data redistribution operator and sends the data processing result corresponding to the sub-processing task to the data relay process of the data node. The sub-processing task is part of the data processing task, and the data relay process is connected to the data sending process, thus allowing the data processing result to be transmitted to other data nodes. The process currently used to receive data processing tasks is automatically converted into a data receiving process, which is also connected to the data relay process. Through the data receiving process and the data relay process, data nodes can receive data processing results from other data nodes. Therefore, this embodiment constructs a fixed process collaboration architecture by splitting the data redistribution operator into corresponding data sending and receiving processes. This reduces the number of processes required to perform redistribution operations on a single node from a multiple related to cluster size and the number of operators to a constant related only to the number of operators, without increasing with cluster size. This fundamentally solves the problem of a dramatic increase in the number of processes and connections in large-scale clusters. Simultaneously, by unifying network communication through a data relay process, network overhead is significantly reduced. Ultimately, this embodiment ensures efficient data redistribution while also achieving controllability and stability of system resources.

[0103] In one exemplary embodiment, such as Figure 3 As shown, a data processing task containing a data redistribution operator is created through a coordinating node, including:

[0104] Step S302: Create a task to be processed that matches the data processing request through the coordination node, wherein the task to be processed contains at least one initial redistribution operator.

[0105] Among them, the pending tasks are those that have not undergone redistribution operator replacement. They are data processing tasks obtained after logical optimization and strategy selection based on the initial processing tasks. After the redistribution operator replacement is completed, the pending tasks can form the final data processing tasks and be sent to the data nodes for execution. The initial redistribution operator refers to the operator in the pending tasks used to achieve data redistribution between nodes, such as the Remotesubplan operator.

[0106] For example, when a coordinating node receives a user's data processing request, it first performs lexical and syntactic analysis on the request. Lexical analysis breaks down the request text, such as SQL text, into multiple lexical units, such as keywords, identifiers, and operators. Syntactic analysis checks whether the permutations and combinations of these lexical units conform to the database syntax rules, thereby constructing a structured syntax tree that reflects the surface syntactic structure of the data processing request. Next, semantic analysis is performed on the syntax tree. This stage verifies the semantic correctness of each element in the syntax tree, such as verifying the existence and accessibility of table or column names, checking data type matching, and confirming the validity of function call parameters, among other things. After semantic analysis, the node generates a query logic tree, which is a data structure reflecting the internal logic of the syntax tree and clearly expresses the core logical operations required to execute the data processing request. The query logic tree then enters the optimizer in the coordinating node. The optimizer is the core component of the coordinating node responsible for generating efficient execution tasks. It performs rule-based logical optimization on the query logic tree, which can be understood as a series of optimization processes based on relational algebraic equivalence transformation rules, such as predicate pushdown, subquery expansion, and expression simplification, aiming to generate a logically equivalent but structurally superior query tree. Finally, the optimizer finds the optimal data processing path based on the optimized query tree and generates a complete plan containing the execution steps of that path; this plan is the task to be processed. When generating this task, the optimizer inserts an initial redistribution operator, the Remotesubplan operator, at all locations where cross-node data exchange is required to complete operations such as joins or aggregations, to ensure that data is correctly distributed to each node according to computational requirements, such as the join key. It is understandable that the coordinating node, as the initiator and planner of distributed task execution, needs to first generate a logically correct execution path. Inserting the initial redistribution operator is the basic guarantee for realizing distributed parallel computing. At the same time, it also provides a clear operation object and optimization space for replacing it with a better-performing data redistribution operator, i.e., the RDA operator, under specific conditions.

[0107] Step S304: Locate the level of each initial redistribution operator.

[0108] In this context, "level" refers to the path length or depth traversed from the root node to a specific operator node in the query tree structure corresponding to the task to be processed.

[0109] For example, after generating the query tree corresponding to the tasks to be processed, the optimizer of the coordinating node traverses the query tree, typically using a depth-first search algorithm. During the traversal, the algorithm records the depth from the root node to each operator node. When it reaches an initial redistribution operator, it records its current depth value as the operator's level. By accurately locating the depth of each initial redistribution operator in the query tree, crucial decision-making information can be provided for the next step of the level-based filtering strategy.

[0110] Step S306: Filter out the initial redistribution operators whose number of levels exceeds the level threshold.

[0111] The level threshold is a preset level, usually an integer value such as 1. The specific value can be determined according to the actual situation. The level threshold is used to delineate the boundaries of the initial redistribution operators that need to be replaced.

[0112] For example, after obtaining the hierarchical information of all initial redistribution operators, the optimizer iterates through these operators and compares the level of each operator with a preset level threshold, such as 1. All initial redistribution operators with a level greater than this threshold (e.g., a level number greater than 1) are selected as targets for replacement, while initial redistribution operators with a level not greater than this threshold (e.g., a level number less than or equal to 1) are not replaced. This embodiment aims to accurately identify deeper redistribution operators that could cause a multiplication of process numbers as optimization targets. Based on this, the stability of the basic communication framework is maintained by retaining initial redistribution operators at specified levels, such as the first level.

[0113] Step S308: Replace the initial redistribution operator with a number of levels exceeding the level threshold with a data redistribution operator, and update the task to be processed to obtain a data processing task containing the data redistribution operator.

[0114] For example, for initial redistribution operators whose number of levels exceeds the level threshold, the optimizer will replace them with data redistribution operators, i.e., RDA operators. After all initial redistribution operators that meet the conditions have been replaced with RDA operators, the optimizer will update the structure of the entire query tree, thereby generating the final optimized data processing task.

[0115] In this embodiment, a dynamic replacement strategy based on hierarchical thresholds is introduced, enabling the optimizer of the coordinating node to intelligently identify and replace deep initial redistribution operators in the tasks to be processed. These operators typically have a significant impact on the number of processes, while retaining necessary level operators to maintain the stability of the execution framework. This reduces the number of processes and further achieves precise control over the optimization process, avoiding the performance risks or architectural shocks that might result from replacing all operators.

[0116] In one exemplary embodiment, such as Figure 4 As shown, a task to be processed is created by the coordinating node to match the data processing request, including:

[0117] Step S402: Create an initial processing task that matches the data processing request through the coordinating node.

[0118] The initial processing task is a task that has only completed logical optimization. It belongs to the intermediate state of data processing tasks. The initial processing task clarifies the core operations required to execute the data processing request and the logical order between these operations, but the execution method has not yet been determined, such as which connection algorithm to use and how the data is transmitted.

[0119] For example, when the coordinating node receives a user's data processing request, it can perform lexical analysis, syntax analysis, and semantic analysis on the request to obtain the corresponding query logic tree. The coordinating node's optimizer can then perform rule-based logical optimization on this query logic tree to obtain a logically equivalent but structurally superior query tree, which serves as the tree structure for the initial processing task.

[0120] Step S404: Perform execution strategy matching on the initial processing task to obtain at least one task execution strategy that matches the initial processing task.

[0121] The task execution strategy is a specific execution plan designed to implement the logical operations described in the initial processing task. It may include the selection of operator algorithms, execution order, data flow mode, etc.

[0122] For example, after generating the query tree, i.e., the initial processing task, the optimizer can enumerate multiple equivalent execution schemes as candidate task execution strategies based on its logical structure. For instance, for a join operation, the optimizer can match strategies using different algorithms such as hash join, merge join, or nested loops. For the order of multi-table joins, it will also generate different permutations and combinations.

[0123] Step S406: Perform cost analysis on each task execution strategy to obtain the task execution cost of each strategy.

[0124] The task execution cost refers to the quantified value of the resources required for the task execution strategy during execution. It is usually a value that integrates indicators such as CPU (Central Processing Unit) overhead, disk I / O overhead, and network transmission overhead. The specific indicators used to quantify the task execution cost can be determined according to the actual situation, and this embodiment does not impose any restrictions on this.

[0125] For example, the optimizer, for each candidate task execution strategy, utilizes its built-in cost model and combines statistical information collected by the distributed data system, such as table size, data distribution, and index selectivity, to quantitatively estimate the resource consumption that the strategy might generate during execution. The cost model comprehensively considers factors such as the computational complexity of each operator, the expected amount of data to be transferred, and the cost of accessing storage devices, ultimately calculating a cost value representing the total overhead of each strategy. By objectively comparing the efficiency of different execution strategies through quantified costs, the optimization process is transformed from experience-based judgment to data-driven decision-making, ensuring the selection of efficient task execution strategies and thus achieving efficient data redistribution.

[0126] Step S408: Based on the task execution strategy that satisfies the task execution cost condition, update the initial processing task to obtain the task to be processed.

[0127] The task execution cost condition is a criterion used to select the optimal task execution strategy. For example, it can be the strategy with the lowest execution cost among all candidate execution strategies. The specific condition can be determined according to the actual situation, and this embodiment does not impose any restrictions on it.

[0128] For example, the optimizer compares the task execution costs of all candidate execution strategies to select the optimal strategy that satisfies the task execution cost condition, i.e., the strategy with the lowest task execution cost. Subsequently, the optimizer updates and specifies the initial processing task based on the details of the optimal strategy to obtain the task to be processed.

[0129] In this embodiment, by introducing an execution strategy matching and cost analysis mechanism, the coordination node can dynamically select the solution with the lowest cost from multiple feasible candidate execution strategies. This not only directly improves the execution performance of a single task, but also provides a high-quality task to be processed as input for subsequent deep optimizations such as operator replacement, forming a multi-level progressive optimization system and further improving the overall performance of the solution.

[0130] In one exemplary embodiment, such as Figure 5 As shown, the data processing method also includes:

[0131] Step S502: Obtain the operator identifier of the data redistribution operator.

[0132] Among them, the operator identifier is a globally unique identifier for the data redistribution operator, providing a unique addressing basis for subsequent inter-process communication, shared memory mapping and network data routing, ensuring that data can be accurately directed and transmitted in complex multi-process and multi-node interaction processes.

[0133] For example, when the executor of a data node begins initializing an RDA operator, it first assigns a globally unique identifier, such as an RDAID, to the operator by calling a system-level identifier generator. The operator identifier can be a random long string or a unique value generated based on information such as a timestamp or node ID to ensure its uniqueness throughout the entire cluster environment.

[0134] Step S504: Create a shared memory region corresponding to the data node based on the operator identifier. The shared memory region refers to the memory region where data is shared among the data sending process, the data receiving process, and the data relay process.

[0135] In this context, a shared memory region is a physical memory region that allows multiple processes to access it. In this embodiment, the shared memory region refers to the memory region where data is shared among data sending processes, data receiving processes, and data relay processes.

[0136] For example, after successfully obtaining the operator identifier, the executor uses that identifier to request the creation of a shared memory region from the system. Specifically, the system creates a shared memory file named after the operator identifier and maps it to the address spaces of the data sending process, the data receiving process, and the data relay process, so that these three processes can directly access the same physical memory.

[0137] In this embodiment, by creating a globally unique operator identifier for each data redistribution operator and establishing a dedicated shared memory region based on this identifier, a unified and efficient data exchange hub is built for the three key processes of data sending, receiving, and relaying. This not only eliminates unnecessary data copying overhead between processes and significantly improves communication efficiency, but also ensures the isolation and precise management of system resources when multiple operators are executed concurrently through the uniqueness of the identifier, thereby guaranteeing the stable implementation of high-performance remote data access.

[0138] In an exemplary embodiment, the shared memory region includes a send buffer region corresponding to the target data node, where the target data node is at least one of the other data nodes used to receive the data processing results of the sub-processing task. The data relay process for sending the data processing results corresponding to the sub-processing task to the data node includes: sending the data processing results of the sub-processing task to the send buffer region if there is remaining capacity in the send buffer region corresponding to the target data node; and transmitting the data processing results to other data nodes through the data relay process includes: reading the data processing results corresponding to the sub-processing task from the send buffer region through the data relay process and transmitting the data processing results corresponding to the sub-processing task to the target data node.

[0139] In this context, the target data node refers to other data nodes that need to receive data during the data redistribution process. The sending buffer area is used to temporarily store the data processing results to be sent. Depending on the different target data nodes, the sending buffer area can be divided into different point units, with each target data node corresponding to one point unit, used to temporarily store the data processing results sent from the current data node to the corresponding target data node.

[0140] For example, after the data sending process completes its sub-processing task and obtains the corresponding data processing result, it first checks whether there is remaining capacity in the sending buffer area of ​​the corresponding target data node in the shared memory region. If the buffer is not full and there is remaining capacity, the data processing result is directly written to the buffer area. If the buffer is full and there is no remaining capacity, the data processing result is temporarily stored in a local temporary file, and written to the buffer when there is available capacity. At the same time, the data relay process is notified to send the data processing result already existing in the sending buffer area. The data relay process continuously monitors the status of the sending buffer area of ​​each data redistribution operator. When it detects that there is a data processing result in the buffer corresponding to a target data node, it reads these data processing results and accurately sends them to the data relay process of the target data node through a pre-established Socket long connection. In this way, the data sending process focuses on task execution, while the data relay process focuses on network transmission. The two cooperate asynchronously through shared memory, which not only avoids the data sending process being blocked due to network latency, but also enables the data relay process to perform batch scheduling and sending of data from multiple buffers, optimizing network resource utilization and significantly improving the efficiency of cross-node data redistribution.

[0141] In an exemplary embodiment, the shared memory region further includes a receiving buffer region corresponding to the target data node; receiving data processing results from other data nodes through a data receiving process and a data relay process includes: storing the data processing results from other data nodes into the receiving buffer region through the data relay process; and reading the data processing results from other data nodes from the receiving buffer region through the data receiving process.

[0142] The receiving buffer area is used to temporarily store the data processing results to be received. Similar to the sending buffer area, the receiving buffer area can also be divided into different point units according to different target data nodes. Each target data node corresponds to a point unit, which is used to temporarily store the data processing results transmitted from the corresponding target data node.

[0143] For example, the data relay process receives data from other data nodes and places it into the receive buffer area of ​​the shared memory region of the local node according to the RDAID. The data receiving process continuously monitors the receive buffer area to read tuples from other data nodes and pass them to the upper-layer operators for consumption. In this way, by using shared memory, the communication overhead between the data receiving process and the data relay process is significantly reduced, thereby improving the overall efficiency of data redistribution.

[0144] Figure 6 A flowchart illustrating a data processing method in a specific embodiment is shown. The distributed data system includes a coordinating node (CN) and multiple data nodes (DNs) working together. The data redistribution process is reconstructed by introducing an RDA operator to address the problem of a dramatic increase in the number of processes caused by the existing Remotesubplan operator when the cluster size expands.

[0145] Specifically, when the CN receives an SQL query request submitted by a user, it first performs lexical and syntactic analysis to generate a syntax tree. Then, semantic analysis is performed to verify the correctness of tables, columns, and other elements in the syntax tree, generating a query logic tree, which is the tree structure of the initial processing task. After the query logic tree enters the CN's optimizer, it undergoes rule-based logical optimization and distributed cost optimization based on statistical information and cost models, generating a task to be processed containing at least one Remotesubplan operator. Finally, by performing RDA operator replacement on this task, the data processing task, i.e., the distributed query task, is obtained and distributed to each DN node. Figure 7 The diagram illustrates the RDA operator replacement process. The CN (Client-Controller) recursively traverses the query tree corresponding to the task to be processed. When an operator is found, it first checks if it is a Remotesubplan operator. If so, it further checks if the level of the Remotesubplan operator is greater than 1. If it is, it replaces the Remotesubplan operator with a high-performance RDA operator. If the currently traversed operator is not a Remotesubplan operator, it checks if the query tree has been traversed completely. If not, it returns to the recursive traversal of the query tree; if so, it generates a distributed query task. If the currently traversed operator is a Remotesubplan operator, but its level is less than or equal to 1, it similarly checks if the query tree has been traversed completely. This process is repeated until the query tree is completely traversed, resulting in a distributed query task.

[0146] Figure 8The diagram illustrates the RDA operator initialization process. Taking a DN node as an example, its executor starts a backend process to receive data processing tasks issued by the CN. When executing an RDA operator, the executor calls the RDA operator initialization function, such as RDA Init. This function is responsible for completing the preliminary work of the RDA operator, including: initializing the resources and related variables required by the RDA operator, and generating a globally unique RAD ID as the operator identifier; creating and mapping a shared memory region based on the RAD ID, which includes a send buffer region and a receive buffer region; registering the RAD ID with the local node's data relay process (Rorwarder process) through the local socket; subsequently creating a new data sending process (RDA sender process) and sending the RDA operator's task slice and task execution commands to this process; the current process then becomes the data receiving process (RDA receiver process).

[0147] Figure 9 The diagram illustrates the execution flow of an RDA operator. RDA operator execution is achieved by calling RDA operator execution functions such as RDAExec. During RDA operator execution, the RDA sender process executes its assigned left subtree (i.e., sub-processing task), retrieving tuples row by row using the volcano model. It then recalculates the target DN to which the tuple should be sent based on the data distribution method and writes the tuple into the send buffer area corresponding to the target DN in the shared memory region. When the buffer reaches full capacity, the RDA sender process notifies the Rorwarder process to transmit the tuples in the buffer to the Rorwarder process of the target DN. Other DNs can also send their generated tuples to the receive buffer area in shared memory through the Rorwarder process. The RDA receiver process of the current node retrieves the tuples from the receive buffer area and passes them to the upper-layer operator for consumption.

[0148] Figure 10The diagram illustrates the execution flow of an RDA operator. After the query is completed, the RDA operator termination process begins, which involves calling the operator termination function (RDA End function). This process proceeds according to the process role: For the RDA sender process, it first notifies the Rorwarder process to refresh and send all remaining tuples in the shared memory region. Then, it terminates the execution of its internal left subtree and finally clears the shared memory, temporary files, and other resources used by the task execution, releasing variables. For the RDA recver process, it first notifies the Rorwarder process to discard any tuples still in transmission but no longer needed. Next, it sends a stop flag to the local RDA sender process via inter-process communication, coordinating to trigger the termination process. Finally, the RDA recver process performs its own resource cleanup.

[0149] Figure 11 This diagram illustrates the data transfer between RDA operators. The RDA sender process executes sub-processing tasks and retrieves tuples, writing them to a shared-memory send buffer uniquely identified by a RAD ID. If the buffer is full, the tuples are temporarily stored in a local temporary file. The Rorwarder process monitors the send buffer and immediately sends the tuples to other DNs via a Socket connection as soon as data arrives. Similarly, the Rorwarder process receives tuples from other DNs via Socket connections and places them into the corresponding shared-memory receive buffer based on their RAD IDs. Finally, the RDA sender process retrieves the tuples from the receive buffer and passes them upwards, thus completing the cross-node data redistribution.

[0150] This embodiment replaces the Remotesubplan operator with an RDA operator and splits the RDA operator into RDA Asender and RDA Recver processes, forming a fixed process collaboration architecture. This reduces the number of processes required to perform redistribution operations on a single node from a multiple related to cluster size and the number of operators to a constant related only to the number of operators, not increasing with cluster size. This fundamentally solves the problem of a dramatic increase in processes and connections in large-scale clusters. Simultaneously, by unifying network communication through the Recver process, network overhead is significantly reduced. Ultimately, this embodiment ensures efficient data redistribution while maintaining controllability and stability of system resources.

[0151] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps. It is understood that the steps in different embodiments can be freely combined as needed, and all non-contradictory solutions formed by such combinations are within the scope of protection of this application.

[0152] Based on the same inventive concept, this application also provides a data processing apparatus for implementing the data processing method described above. The solution provided by this apparatus is similar to the implementation scheme described in the above method; therefore, the specific limitations in one or more data processing apparatus embodiments provided below can be found in the limitations of the data processing method described above, and will not be repeated here.

[0153] In one exemplary embodiment, such as Figure 12 As shown, a data processing apparatus is provided for a distributed data system applied to a data processing method. The distributed data system includes a coordinating node and multiple data nodes, including:

[0154] The task creation and distribution module 1202 is used to respond to data processing requests, create data processing tasks containing data redistribution operators through the coordination node, and distribute the data processing tasks to each data node.

[0155] The data sending process creation module 1204 is used to create a data sending process for each data node when the data processing task is executed on the data node and the task node where the data redistribution operator is located is reached.

[0156] The data relay process module 1206 is used to execute the sub-processing tasks corresponding to the data redistribution operator through the data sending process, and send the data processing results corresponding to the sub-processing tasks to the data relay process of the data node. The sub-processing tasks belong to the data processing tasks, and the data relay process is connected to the data sending process.

[0157] The data sending module 1208 is used to transmit the data processing results to other data nodes through the data relay process;

[0158] The data receiving process creation module 1210 is used to convert the process currently used for receiving data processing tasks into a data receiving process, wherein the data receiving process is connected to the data relay process;

[0159] The data receiving module 1212 is used to receive data processing results from other data nodes through the data receiving process and the data relay process.

[0160] In one embodiment, the task creation and distribution module 1202 is further configured to:

[0161] A task to be processed is created by a coordinating node to match the data processing request, wherein the task to be processed contains at least one initial redistribution operator;

[0162] Locate the level at which each initial redistribution operator belongs;

[0163] Filter out the initial redistribution operators where the number of levels exceeds the level threshold;

[0164] Replace the initial redistribution operator with a number of levels exceeding the level threshold with a data redistribution operator, and update the task to be processed to obtain a data processing task containing the data redistribution operator.

[0165] In one embodiment, the task creation and distribution module 1202 is further configured to:

[0166] An initial processing task matching the data processing request is created by a coordinating node;

[0167] Perform execution strategy matching on the initial processing task to obtain at least one task execution strategy that matches the initial processing task;

[0168] Cost analysis is performed on each task execution strategy to obtain the task execution cost of each strategy.

[0169] Based on the task execution strategy that satisfies the task execution cost condition, the initial processing task is updated to obtain the task to be processed.

[0170] In one embodiment, the data processing apparatus is further configured to:

[0171] Obtain the operator identifier of the data redistribution operator;

[0172] A shared memory region corresponding to a data node is created based on the operator identifier. The shared memory region refers to the memory region where data is shared among the data sending process, the data receiving process, and the data relay process.

[0173] In one embodiment, the shared memory region includes a send buffer region corresponding to the target data node, and the target data node is at least one of the other data nodes used to receive the data processing results of the sub-processing task; the data relay process module 1206 is further configured to: send the data processing results of the sub-processing task to the send buffer region when there is remaining capacity in the send buffer region corresponding to the target data node; the data sending module 1208 is further configured to: read the data processing results corresponding to the sub-processing task from the send buffer region through the data relay process, and transmit the data processing results corresponding to the sub-processing task to the target data node.

[0174] In one embodiment, the shared memory region further includes a receive buffer region corresponding to the target data node; the data receiving module 1212 is also used for:

[0175] The data processing results from other data nodes are stored in the receiving buffer area through the data relay process;

[0176] The data receiving process reads the data processing results from other data nodes within the receiving buffer area.

[0177] Each module in the aforementioned data processing device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0178] In one exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 13 As shown, this computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores data for data processing. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network connection. When the computer program is executed by the processor, it implements a data processing method.

[0179] Those skilled in the art will understand that Figure 13The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0180] In one exemplary embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0181] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps in the above method embodiments.

[0182] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0183] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0184] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0185] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0186] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A data processing method, characterized by, The method is applied to a distributed data system comprising a coordination node and a plurality of data nodes, and comprises: in response to a data processing request, creating, by the coordination node, a data processing task comprising a data redistribution operator, and issuing the data processing task to each data node; for each data node, during execution of the data processing task by the data node, if execution reaches a task node where the data redistribution operator is located, creating a data sending process; executing, by the data sending process, a sub-processing task corresponding to the data redistribution operator, and sending a data processing result corresponding to the sub-processing task to a data transfer process of the data node, wherein the sub-processing task belongs to the data processing task, and the data transfer process is connected to the data sending process; transmitting, by the data transfer process, the data processing result to other data nodes; converting a process currently used to receive the data processing task into a data receiving process, wherein the data receiving process is connected to the data transfer process; receiving, by the data receiving process and the data transfer process, data processing results from the other data nodes.

2. The method of claim 1, wherein, The method further comprises: creating, by the coordination node, a to-be-processed task matching the data processing request, wherein the to-be-processed task comprises at least one initial redistribution operator; locating a level of each initial redistribution operator; filtering out initial redistribution operators whose level numbers exceed a level threshold; replacing the initial redistribution operators whose level numbers exceed the level threshold with a data redistribution operator, and updating the to-be-processed task to obtain a data processing task comprising the data redistribution operator.

3. The method of claim 2, wherein, The method further comprises: creating, by the coordination node, an initial processing task matching the data processing request; performing execution strategy matching on the initial processing task to obtain at least one task execution strategy matching the initial processing task; performing cost analysis on each task execution strategy to obtain a task execution cost of each task execution strategy; based on a task execution strategy satisfying a task execution cost condition, updating the initial processing task to obtain a to-be-processed task.

4. The method of claim 1, wherein, The method further comprises: obtaining an operator identifier of the data redistribution operator; based on the operator identifier, creating a shared memory area corresponding to the data node, wherein the shared memory area is a memory area for data sharing between the data sending process, the data receiving process, and the data transfer process.

5. The method of claim 4, wherein, The shared memory area comprises a sending buffer area corresponding to a target data node, wherein the target data node is at least one data node in the other data nodes used to receive the data processing result of the sub-processing task. The method further comprises: sending, by the data sending process, a data processing result corresponding to the sub-processing task to a data transfer process of the data node, wherein the data transfer process is connected to the data sending process. In a case where there is remaining capacity in a sending buffer area corresponding to the target data node, the data processing result of the sub-processing task is sent to the sending buffer area; The transmitting, by the data transfer process, of the data processing result to other data nodes comprises: The data transfer process reads the data processing result corresponding to the sub-processing task from the sending buffer area and transmits the data processing result corresponding to the sub-processing task to the target data node.

6. The method of claim 4, wherein, The shared memory area further comprises a receiving buffer area corresponding to the target data node; The receiving, by the data receiving process and the data transfer process, of the data processing result from the other data nodes comprises: The data transfer process stores the data processing result from the other data nodes to the receiving buffer area; The data receiving process reads the data processing result of the other data nodes from the receiving buffer area.

7. A data processing apparatus, characterized by, The distributed data system applied to the data processing method of claim 1 comprises a coordination node and a plurality of data nodes, and the device comprises: A task creation and distribution module is configured to create a data processing task containing a data redistribution operator by the coordination node in response to a data processing request, and distribute the data processing task to each data node; A data sending process creation module is configured to create a data sending process for each data node during execution of the data processing task by the data node, if the execution reaches a task node where the data redistribution operator is located. A data transfer process module is configured to execute a sub-processing task corresponding to the data redistribution operator by the data sending process, and send the data processing result corresponding to the sub-processing task to a data transfer process of the data node, wherein the sub-processing task belongs to the data processing task, and the data transfer process is connected to the data sending process. A data sending module is configured to transmit the data processing result to other data nodes by the data transfer process. A data receiving process creation module is configured to convert a process currently used for receiving the data processing task into a data receiving process, wherein the data receiving process is connected to the data transfer process. A data receiving module is configured to receive the data processing result from the other data nodes by the data receiving process and the data transfer process.

8. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that, The processor executes the computer program to implement the steps of the method of any one of claims 1 to 6.

9. A computer readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method of any one of claims 1 to 6.

10. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method of any one of claims 1 to 6.

Images