Distributed database data processing methods, devices, servers, and storage media

Abstract

The data processing method, apparatus, server, and storage medium for a distributed database provided in this invention include: receiving a data processing request sent by a client, the processing request including a data processing template; after sending a processing response to the processing request to the client, receiving an execution request sent by the client, the execution request including a template identifier and template parameters corresponding to the data processing template; invoking a data query processing thread and a data insertion processing thread according to the template identifier and template parameters; streamwise acquiring a preset number of target data rows from query data obtained from a first database node based on the data query processing thread; and sending the target data to a second database node based on the data insertion processing thread, so that the second database node performs a data insertion operation on the target data until the query data insertion is completed. This application can realize simultaneous data query and target data insertion operations, shortening data processing time and improving data processing efficiency.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a data processing method, apparatus, server and storage medium for a distributed database. Background Technology

[0002] Compute nodes can be servers of a database management system, used to manage and control the database nodes (DNs) in the database management system. Typically, compute nodes use Structured Query Language (SQL) to store data and query, update, and manage database nodes. Communication between the database management system and clients is based on data transmission protocols.

[0003] Prepare Statement (hereinafter referred to as PS protocol) is a data transmission protocol. Currently, the data processing scheme for executing Insert-Select statements under PS protocol is to first perform a data query to obtain the data to be inserted. After all the data queries are completed, the data insertion protocol packet is generated and the data insertion is performed on the relevant database nodes. This data processing method essentially executes the data query and data insertion processes serially. Especially for distributed databases with many database nodes, this data processing method will consume a lot of time and reduce data processing efficiency. Summary of the Invention

[0004] In view of this, embodiments of the present invention provide a data processing method, apparatus, server, and storage medium for a distributed database, which enables data query and data insertion processes to be parallelized to a certain extent, greatly shortening data processing time and improving data processing efficiency.

[0005] In a first aspect, embodiments of the present invention provide a data processing method for a distributed database, wherein the method is applied to a computing node, and the computing node is communicatively connected to a client and a database node; the method includes:

[0006] Receive data processing requests sent by clients, including data processing templates;

[0007] After sending a processing response to the client in response to the processing request, the system receives an execution request for data processing from the client. The execution request includes the template identifier and template parameters corresponding to the data processing template.

[0008] The data query processing thread and the data insertion processing thread are retrieved based on the template identifier and template parameters;

[0009] The target data, with a preset number of rows, is streamed from the query data obtained by the data query processing thread from the first database node corresponding to the data processing template; wherein, the amount of target data is less than the amount of query data.

[0010] The data insertion processing thread sends the target data to the second database node corresponding to the data processing template, so that the second database node performs a data insertion operation on the target data until the data insertion is completed.

[0011] In one possible implementation, after receiving a data processing request sent by the client, the processing request including a data processing template, the method further includes:

[0012] Obtain information from the first and second data tables from the data processing template;

[0013] Based on the data processing template, the information from the first data table, and the information from the second data table, a processing response is generated for the processing request, and the processing response is sent to the client.

[0014] In one possible implementation, a processing response to the processing request is generated based on the data processing template, first data table information, and second data table information, including:

[0015] Generate template identifiers corresponding to the data processing templates, and split the data processing templates into data query templates and data insertion templates;

[0016] The first data processing request to generate a data query template and the second data processing request to insert data into a template;

[0017] Send the first data processing request to the first database node corresponding to the first data table information, and send the second data processing request to the second database node corresponding to the second data table information;

[0018] Receive a first data processing response from a first database node in response to a first data processing request and a second data processing response from a second database node in response to a second data processing request;

[0019] A processing response is generated to form a first data processing response and a second data processing response; wherein the processing response includes a template identifier.

[0020] In one possible implementation, a Prepare request that requests a Prepared Statement is processed, and an Execute request that requests a Prepared Statement is executed.

[0021] In one possible implementation, the data query processing thread and the data insertion processing thread are invoked based on the template identifier and template parameters, including:

[0022] Query the data processing template corresponding to the template identifier;

[0023] Constructing SOL text based on template parameters and data processing templates;

[0024] Data retrieval and data insertion processing threads based on SQL text.

[0025] In one possible implementation, the first data table information includes first database identification information and first data table identification information;

[0026] Based on the data query processing thread, a preset number of target data are streamed from the query data obtained from the first database node corresponding to the data processing template, including:

[0027] A query execution request for SQL text is generated and sent to the first database node corresponding to the first database identifier information, so that the first database node can query the query data from the first data table corresponding to the first data table identifier information based on the SQL text;

[0028] Receive the query execution response sent by the first database node in response to the query execution request;

[0029] A streaming data acquisition request is sent to the first database node, so that the first database node can stream a preset number of target data from the query data and send the target data to the data storage; wherein, the data storage is pre-configured on the computing node;

[0030] Retrieve the stored target data from the data storage device.

[0031] In one possible implementation, the second data table information includes second database identification information and second data table identification information;

[0032] The data insertion processing thread sends the target data to the second database node corresponding to the data processing template, so that the second database node performs a data insertion operation on the target data, including:

[0033] The target data is assembled according to the protocol format and SQL text of the second database node corresponding to the second database identifier information to obtain the execution protocol packet;

[0034] The execution protocol packet is sent to the second database node, so that the second database node can parse the execution protocol packet to obtain the assembled target data and insert the assembled target data into the second data table corresponding to the second data table identifier information.

[0035] In one possible implementation, the method further includes:

[0036] Send an execution response to the client in response to the execution request.

[0037] In one possible implementation, after sending an execution response to the client in response to the execution request, the method further includes:

[0038] Receive a close request sent by the client;

[0039] Clear the data processing template, template identifier, and template parameters;

[0040] Send a close request to the first database node and the second database node, and receive a first close response generated by the first database node in response to the close request and a second close response generated by the second database node in response to the close request;

[0041] Generate a first close response and a second close response;

[0042] Send a close response to the client.

[0043] In a second aspect, embodiments of the present invention provide a data processing apparatus for a distributed database, wherein the apparatus is applied to a computing node, the computing node being communicatively connected to a client and a database node; the apparatus includes:

[0044] The first receiving module is used to receive data processing requests sent by the client, and the processing requests include data processing templates.

[0045] The second receiving module is used to receive the data processing execution request sent by the client after sending the processing response to the processing request to the client. The execution request includes the template identifier and template parameters corresponding to the data processing template.

[0046] The retrieval module is used to retrieve the data query processing thread and the data insertion processing thread based on the template identifier and template parameters.

[0047] The acquisition module is used to stream target data of a preset number of rows from the query data obtained by the data query processing thread from the first database node corresponding to the data processing template; wherein the amount of target data is less than the amount of query data.

[0048] The sending module is used to send the target data to the second database node corresponding to the data processing template based on the data insertion processing thread, so that the second database node can perform data insertion operation on the target data until the data insertion is completed.

[0049] Thirdly, embodiments of the present invention provide a server, comprising: a processor and a memory, wherein the processor is used to execute a data processing program stored in the memory to implement the above-described data processing method for a distributed database.

[0050] Fourthly, embodiments of the present invention provide a storage medium, wherein the storage medium stores one or more programs, which can be executed by one or more processors to implement the above-described data processing method for a distributed database.

[0051] The data processing method, apparatus, server, and storage medium for a distributed database provided in this invention include: receiving a data processing request sent by a client, the processing request including a data processing template; after sending a processing response to the processing request to the client, receiving an execution request sent by the client, the execution request including a template identifier and template parameters corresponding to the data processing template; invoking a data query processing thread and a data insertion processing thread according to the template identifier and template parameters; stream-acquiring a preset number of target data rows from the query data obtained by the data query processing thread from the first database node corresponding to the first data table information; and sending the target data to the second database node corresponding to the second data table information based on the data insertion processing thread, so that the second database node performs a data insertion operation on the target data until the query data insertion is completed. In the data processing process, this application can stream-acquire target data with a data volume smaller than the query data volume, enabling simultaneous data query and target data insertion operations without waiting for all query data to be acquired before performing the data insertion operation. This achieves parallel processing of data query and data insertion to a certain extent, significantly shortening data processing time and improving data processing efficiency. Attached Figure Description

[0052] Figure 1 This is a schematic diagram illustrating an application scenario of a data processing method for a distributed database provided in an embodiment of the present invention.

[0053] Figure 2 A flowchart illustrating a data processing method for a distributed database provided in an embodiment of the present invention;

[0054] Figure 3 A flowchart illustrating another data processing method for a distributed database provided in an embodiment of the present invention;

[0055] Figure 4 A flowchart illustrating another data processing method for a distributed database provided in an embodiment of the present invention;

[0056] Figure 5 This is a schematic diagram of the structure of a data processing device for a distributed database provided in an embodiment of the present invention;

[0057] Figure 6 This is a schematic diagram of the structure of a server provided in an embodiment of the present invention. Detailed Implementation

[0058] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0059] To facilitate understanding of the embodiments of the present invention, further explanations and descriptions will be provided below with reference to the accompanying drawings and specific embodiments. These embodiments do not constitute a limitation on the embodiments of the present invention.

[0060] In this embodiment, the data processing method of the above-mentioned distributed database can be provided by, for example: Figure 1 The computing node 101 shown is implemented as follows: Figure 1 As shown, the computing node 101 is communicatively connected to multiple clients 102 and multiple database nodes 103. Figure 1 Only three clients 102 and three database nodes 103 are shown in the diagram. The specific number of clients 102 and database nodes 103 can be set according to actual needs and is not limited here. The computing node 101 communicates with the clients 102 and database nodes 103 via a network. In some embodiments, the network can be any type of wired or wireless network, or a combination thereof. For example, the network can include cable networks, wired networks, fiber optic networks, telecommunications networks, intranets, the Internet, local area networks (LANs), wide area networks (WANs), wireless local area networks (WLANs), metropolitan area networks (MANs), public switched telephone networks (PSTNs), Bluetooth networks, ZigBee networks, near field communication (NFC) networks, or similar networks, etc., and is not limited here.

[0061] Client 102 refers to the server carrying client software (application). Database nodes 103 are the individual database servers that constitute the distributed database, responsible for all data access and processing. All requests for data lookup, addition, deletion, and update are handled by the database nodes. Compute nodes 101 can intercept requests sent from client 102 to database nodes 103. Compute nodes 101 process the intercepted requests and then send them back to database nodes 103. For example, if client 102 requests a table of products listed alphabetically, client 102 submits the request to compute node 101, which processes the request and sends it to database node 103. Database node 103 processes the request, filtering, discarding, and sorting data as needed, and then sends the results back to client 102 through compute node 101.

[0062] In this embodiment, a data processing method for a distributed database can be executed by computing node 101, such as... Figure 2 As shown, the method may include the following steps:

[0063] Step 201: Receive a data processing request sent by the client. The processing request includes a data processing template.

[0064] Step 202: After sending a processing response to the processing request to the client, receive an execution request for data processing sent by the client. The execution request includes the template identifier and template parameters corresponding to the data processing template.

[0065] In some embodiments, the above processing request is a Prepare request of a Prepared Statement, and the execution request is an Execute request of a Prepared Statement. That is, the communication between the client 102 and the database management system composed of the computing node 101 and the database node 103 is based on the PS protocol.

[0066] Under the PS protocol, data transmission between the client and the database management system is divided into two phases: Prepare and Execute. In the Prepare phase, the client sends a processing request (Prepare request) to the compute node, which includes a data processing template. For example, the data processing template can be obtained by replacing template parameters with placeholders; the template parameters are the parameters in the data processing template that have been replaced by placeholders. For instance, if a data processing statement is "insert into DB1.t1 select col1 from DB1.t2 where col1=3", then the data processing template is "insert into DB1.t1 select col1 from DB1.t2 where col1=?"; the template parameter is "3". After receiving the processing request, the compute node parses it to obtain the SQL text without template parameters, saves the parsing result, processes the SQL text without template parameters, and then sends a new processing request, including the processed SQL text without template parameters, to the corresponding database node. Once the database node has received all responses to the new processing request, it generates a processing response and returns it to the client.

[0067] After receiving the processing response, the client enters the Execute phase. The client only sends an Execute request to the compute node, containing binary template parameters replaced with placeholders and a template identifier (StmtId) for the data processing template. The compute node can directly obtain the data processing template corresponding to the StmtId and execute all data processing requests identical to the template based on the template and its parameters. The PS protocol allows for one Prepare and multiple Execute operations. Under the PS protocol, the client and compute node only need to transmit the template identifier (StmtId) representing the statement and the template parameters to complete the corresponding data processing. This effectively reduces the CPU (Central Processing Unit) overhead on the database side, saves network bandwidth, reduces the impact of network jitter on the database management system, and improves data processing efficiency.

[0068] Step 203: Retrieve the data query processing thread and the data insertion processing thread based on the template identifier and template parameters;

[0069] A processing thread refers to a model with a series of thread segments, each performing an action. In this embodiment, the data query processing thread and data insertion processing thread include a series of thread segments. For example, there is a thread segment corresponding to the process of streaming the target data of a preset number of rows from the query data obtained from the first database node corresponding to the data processing template, and then the thread segment corresponding to the process of sending the target data to the second database node corresponding to the data processing template. In general, the purpose of calling the data query processing thread and the data insertion processing thread is to perform a series of operations of streaming the target data from the query data and inserting it. This operation is repetitive and continuous, and can perform real-time insertion processing on the target data obtained in real time, thereby realizing the simultaneous data query operation and the target data insertion operation.

[0070] Step 204: Based on the data query processing thread, the target data of the preset number of rows is obtained in a streaming manner from the query data obtained from the first database node corresponding to the data processing template;

[0071] The target data volume is less than the query data volume. The first database node is one or more database nodes that need to be queried. The number of first database nodes is related to the specific content of the data processing template and is not limited here. If the data processing template is "insert into DB1.t1 select col1from DB1.t2 where col1=?", then there is only one first database node, which is the database node corresponding to the database name DB1. If the data processing template is "insert into DB1.t1,DB3.t1 select col1 fromDB1.t2,col2 from DB2.t3where col1=?,col2=?", then there are two first database nodes, which are the database nodes corresponding to the database names DB1 and DB2. These will not be elaborated on here.

[0072] In this embodiment, the data query processing thread can retrieve only a preset number of target data rows from the query data each time to perform the data insertion operation in step 205. This eliminates the need to collect all the query data obtained from the first database node at once before performing the data insertion operation. This greatly improves the concurrency of data processing during the Execute phase, thereby speeding up the response during the Execute phase.

[0073] Step 205: Based on the data insertion processing thread, the target data is sent to the second database node corresponding to the data processing template, so that the second database node performs a data insertion operation on the target data until the data insertion is completed.

[0074] The second database node is one or more database nodes that need to be inserted. The number of second database nodes depends on the specific content of the data processing template and is not limited here. Continuing the previous example, if the data processing template is "insert into DB1.t1 select col1 from DB1.t2 where col1=?", then there is only one second database node, which is the database node corresponding to the database name DB1. If the data processing template is "insert into DB1.t1,DB3.t1 select col1from DB1.t2,col2 from DB2.t3where col1=?,col2=?", then there are two second database nodes, which are the database nodes corresponding to the database names DB1 and DB3. The details will not be elaborated here.

[0075] In practical use, the second database node and the first database node can be the same database node or different database nodes. This depends on the data processing template. For example, if the data processing template is "insert into DB1.t1 select col1 from DB1.t2 where col1=?", then both the first and second database nodes are database nodes with the database name DB1. If the data processing template is "insert into DB2.t1 select col1 from DB1.t2 where col1=?", then the first database node is the database node with the database name DB1, and the second database node is the database node with the database name DB2.

[0076] After the data insertion processing thread sends the target data to the second database node corresponding to the data processing template, the second database node can perform data insertion operations on the target data in real time to insert the target data into the data table stored under the second database node.

[0077] The distributed database data processing method provided in this invention includes: receiving a data processing request sent by a client, the processing request including a data processing template; after sending a processing response to the processing request to the client, receiving an execution request sent by the client, the execution request including a template identifier and template parameters corresponding to the data processing template; invoking a data query processing thread and a data insertion processing thread according to the template identifier and template parameters; stream-acquiring a preset number of target data rows from the query data obtained by the data query processing thread from the first database node corresponding to the first data table information; and sending the target data to the second database node corresponding to the second data table information based on the data insertion processing thread, so that the second database node performs a data insertion operation on the target data until the query data insertion is completed. In the data processing process, this application can stream-acquire target data with a data volume smaller than the query data volume, enabling simultaneous data query and target data insertion operations without waiting for all query data to be acquired before performing the data insertion operation. This achieves parallel processing of data query and data insertion to a certain extent, significantly shortening data processing time and improving data processing efficiency.

[0078] See Figure 3 This is a flowchart illustrating an embodiment of a data processing method for a distributed database provided by the present invention. Figure 3 The process shown above Figure 2 This is implemented based on the process shown.

[0079] like Figure 3 As shown, the process may include the following steps:

[0080] Step 301: Receive a data processing request sent by the client. The processing request includes a data processing template.

[0081] Step 302: Obtain the information of the first data table and the second data table from the data processing template;

[0082] Typically, a data processing template includes information about a first data table and information about a second data table. The first data table information specifies which database node and table the data query is performed in, and the second data table information specifies which database node and table the data insertion is performed in. Both the first and second data table information include the database name and the table name. The database names in the first and second data table information can be the same or different, but the table names are generally different.

[0083] Continuing from the previous example, if the data processing template is "insert into DB1.t1 select col1 from DB1.t2 where col1=?", then the first data table information obtained is DB1.t2, which is used to indicate that data is queried in table t2 of database name DB1, and the second data table information obtained is DB1.t2, which is used to indicate that data is inserted in table t1 of database name DB1.

[0084] Step 303: Generate a processing response for the processing request based on the data processing template, the first data table information, and the second data table information, and send the processing response to the client;

[0085] The specific process of generating and processing the response can be achieved through steps A1 to A5:

[0086] Step A1: Generate template identifiers corresponding to the data processing templates, and split the data processing templates into data query templates and data insertion templates;

[0087] Upon receiving a Prepare request from the client, the compute node parses the Prepare request according to the PS protocol, extracts the encapsulated data processing template from the parsing result, and stores the data processing template in at least one memory of the compute node. The compute node generates a template identifier for the data processing template and establishes a mapping between the template identifier and the data processing template. In this way, when the compute node receives a template identifier again, it can find the data processing template corresponding to the template identifier from at least one memory.

[0088] The compute node needs to generate a possible execution plan based on the data processing template parsing results and the actual data distribution. Typically, the compute node transforms the processing request and then sends the new data processing request to the corresponding database node. At this time, the data processing template sent by the compute node to the database node is different from the original data processing template sent by the client. In this embodiment, the data processing template is split into a data query template and a data insertion template. The data query template is used to perform data queries, and the data insertion template is used to perform data insertion. Through steps A2 and A3, the first data processing request of the generated data query template is sent to the first database node, and the second data processing request of the generated data insertion template is sent to the second database node.

[0089] Step A2: Generate the first data processing request for the data query template and the second data processing request for the data insertion template;

[0090] Step A3: Send the first data processing request to the first database node corresponding to the first data table information, and send the second data processing request to the second database node corresponding to the second data table information;

[0091] Step A4: Receive the first data processing response sent by the first database node in response to the first data processing request and the second data processing response sent by the second database node in response to the second data processing request;

[0092] The first data processing response is the response returned to the computing node in response to the first data processing request. This response can be either a response that is correct or a response that has a problem. Similarly, the second data processing response can be either a response that is correct or a response that has a problem in response to the second data processing request.

[0093] A response indicating no issues means that the first and second database nodes are functioning correctly, i.e., they can process data normally. A response indicating problems means that the first and second database nodes are not functioning correctly, i.e., they cannot process data normally.

[0094] Step A5 is executed only if both the first and second data processing responses are error-free; otherwise, step A5 is not executed, i.e., the Execute process is not executed.

[0095] Step A5: Generate a processing response for the first data processing response and the second data processing response; wherein, the processing response includes a template identifier.

[0096] Step 304: Receive the data processing execution request sent by the client. The execution request includes the template identifier and template parameters corresponding to the data processing template.

[0097] Step 305: Retrieve the data query processing thread and the data insertion processing thread based on the template identifier and template parameters;

[0098] The specific process of retrieving the processing thread is as follows: query the data processing template corresponding to the template identifier; construct the SQL text based on the template parameters and the data processing template; and retrieve the data query processing thread and the data insertion processing thread based on the SQL text.

[0099] When a compute node receives an execution request, since the execution request includes a template identifier, the data processing template stored in the memory can be found through this template identifier during the Prepare stage. It is then combined with the template data to form a complete SQL text. Based on the execution plan of the data processing corresponding to the SQL text, the corresponding data query processing thread and data insertion processing thread are invoked to enable parallel execution of data query and data insertion processing.

[0100] Step 306: Based on the data query processing thread, a preset number of target data rows are streamed from the query data obtained from the first database node corresponding to the data processing template; wherein, the amount of target data is less than the amount of query data.

[0101] In practical use, the aforementioned first data table information includes first database identifier information and first data table identifier information. The first database identifier information is used to uniquely identify the first database node, and the first data table identifier information is used to uniquely identify the data table under the first database node. In other words, the first data table information indicates which database node and which data table the data query is performed on. The first database identifier information can be represented by numbers or characters, preferably the database node name; the first data table identifier information can also be represented by numbers or characters, preferably the data table name.

[0102] Specifically, the process of obtaining a preset number of target data rows based on the data query processing thread can be implemented by steps B1 to B4:

[0103] Step B1: Generate a query execution request for the SQL text and send the query execution request to the first database node corresponding to the first database identifier information, so that the first database node can query the query data from the first data table corresponding to the first data table identifier information based on the SQL text;

[0104] The compute node packages the SQL text into a request packet, i.e., a query execution request, in a protocol format that the first database node can recognize. The first database node then sends the query execution request to the first database node corresponding to the first database identifier information. Upon receiving the query execution request, the first database node parses the query execution request according to the PS protocol, extracts the SQL text from the parsing result, obtains the first data table identifier information from the SQL text, and then queries the first data table corresponding to the first data table identifier information. The first data table is then queried according to the SQL text to obtain the queried data.

[0105] Step B2: Receive the query execution response sent by the first database node in response to the query execution request;

[0106] After completing the data query, the first database node will generate a query execution response for the query execution request and send it to the compute node. The compute node can then check the query execution response to determine if there is a problem with the data queried by the first database node. If the query execution response indicates a problem, it means that the first database node's data query based on the SQL text is incorrect, and there is no need to proceed to step B3. If the query execution response indicates no problem, it means that the first database node's data query based on the SQL text is correct, and step B3 can proceed.

[0107] Step B3: Send a streaming data acquisition request to the first database node so that the first database node can stream the target data of a preset number of rows from the query data and send the target data to the data storage; wherein, the data storage is pre-configured on the computing node;

[0108] The preset number of rows is a parameter pre-configured on the compute node. It is usually less than the number of rows in the data table to be queried. In actual use, there may be cases where the number of rows in the data table is less than the preset number of rows. In this case, the compute node can retrieve all the queried data at once and then perform data insertion processing. This is the same as the existing data query and data insertion process, which will not be elaborated here. If the preset number of rows is less than the number of rows in the data table, then the preset number of rows of data will be retrieved.

[0109] Step B4: Retrieve the stored target data from the data storage device.

[0110] In this embodiment, to further improve execution speed, the computing node can call the streaming fetch interface to directly retrieve target data from the data storage.

[0111] Step 307: Based on the data insertion processing thread, the target data is sent to the second database node corresponding to the data processing template, so that the second database node performs a data insertion operation on the target data until the data insertion is completed.

[0112] In practical use, the aforementioned second data table information includes second database identifier information and second data table identifier information. The second database identifier information is used to uniquely identify the second database node, and the first data table identifier information is used to uniquely identify the data table under the second database node. In other words, the second data table information indicates which database node and which data table the data insertion process is performed on. The second database identifier information can be represented by numbers or characters, preferably the database node name; the second data table identifier information can also be represented by numbers or characters, preferably the data table name.

[0113] Specifically, the process of sending the target data to the second database node corresponding to the data processing template based on the data insertion processing thread, so that the second data node can perform data insertion, can be implemented by steps C1 to C2:

[0114] Step C1: Assemble the target data according to the protocol format and SQL text of the second database node corresponding to the second database identification information to obtain the execution protocol packet;

[0115] The target data acquired by the compute node is unrecognizable by the second database node. Therefore, directly sending the target data to the second database node will not enable data insertion. In this embodiment, the target data needs to be assembled to generate an execution protocol packet that the second database node can recognize. To improve concurrency, the data insertion processing thread will hand over the assembly part to the execution thread pool in the compute node for processing.

[0116] In actual packet assembly, if data insertion is required across multiple second database nodes, the target data must be pre-divided into multiple sub-target data based on the actual amount of data to be inserted into each second database node. Then, the sub-target data is assembled according to the protocol format and SQL text of the corresponding second database node to obtain an execution protocol packet that the corresponding second database node can recognize. If data insertion is required within a single second database node, it is not necessary to divide the target data into multiple sub-target data. The target data can be directly assembled according to the protocol format and SQL text of the corresponding second database node to obtain an execution protocol packet that the corresponding second database node can recognize.

[0117] Step C2 involves sending the execution protocol packet to the second database node, enabling the second database node to parse the packet to obtain the assembled target data and insert it into the second data table corresponding to the second data table identifier information. In practical use, to facilitate informing the client that the data processing is complete, the second database node needs to perform the data insertion operation on all queried data before the compute node sends an execution response to the client in response to the execution request.

[0118] Typically, after the client receives the execution response, it receives the close request sent by the client; clears the data processing template, template identifier, and template parameters; sends close requests to the first database node and the second database node, and receives the first close response generated by the first database node in response to the close request and the second close response generated by the second database node in response to the close request; and generates a close response that combines the first and second close responses.

[0119] Send a close response to the client.

[0120] The above-described closed-loop process indicates that the entire data processing is complete, and the computing node can then execute the same data processing as the data processing template again.

[0121] In summary, to facilitate understanding of the process described above, Figure 4 A flowchart illustrating another data processing method for distributed databases is shown, simplifying the process described above as follows:

[0122] Step 401: The client sends a data processing request to the computing node;

[0123] Step 402: The computing node processes the data processing model, including the data processing, and generates a new processing request after processing, which is then sent to the database node.

[0124] Step 403: The database node generates a new processing response for the new processing request and sends it to the compute node;

[0125] Step 404: The compute node generates a processing response for the new processing response and sends it to the client;

[0126] Step 405: The client sends a data processing execution request to the compute node;

[0127] Step 406: The compute node invokes the data processing thread according to the execution request, generates a query execution request for the SOL text, and sends the query execution request to the database node;

[0128] Step 407: The database node generates a query execution response for the query execution request and sends it to the compute node;

[0129] Step 408: The compute node sends a streaming data retrieval request to the database node;

[0130] Step 409: The database node streams the target data of a preset number of rows from the query data obtained from the SOL text query and sends the target data to the computing node.

[0131] Step 410: The computing node obtains the target data, assembles the target data to generate an execution protocol packet, and sends it to the database node;

[0132] Step 411: After the database node has completed the data insertion process for all the assembled target data, it sends a data insertion processing response to the computing node.

[0133] Step 412: The compute node sends an execution response to the client in response to the execution request;

[0134] Step 413: The client sends a shutdown request to the compute node;

[0135] Step 414: After the compute node clears the data during the data processing, it sends a shutdown request to the database node;

[0136] Step 415: The database node sends a shutdown response to the compute node in response to the shutdown request;

[0137] Step 416: The compute node sends a close response to the client.

[0138] The distributed database data processing method provided in this application has no impact on front-end business and does not require any modification to SQL statements for use. For processing Insert-Select data issued by the PS protocol, data query and data insertion can be performed simultaneously during the Execute phase, which greatly improves data processing efficiency, enables fast response to the client, and enhances the user experience of the client.

[0139] Corresponding to the above method embodiments, this embodiment of the invention provides a data processing device for a distributed database, wherein the device is applied to a computing node, and the computing node is communicatively connected to the client and the database node; Figure 5 A schematic diagram of the structure of a data processing device for a distributed database is shown, such as... Figure 5 As shown, the device includes: The device includes:

[0140] The first receiving module 51 is used to receive a data processing request sent by the client, the processing request including a data processing template;

[0141] The second receiving module 52 is used to receive a data processing execution request sent by the client after sending a processing response to the processing request to the client. The execution request includes a template identifier and template parameters corresponding to the data processing template.

[0142] The retrieval module 53 is used to retrieve the data query processing thread and the data insertion processing thread based on the template identifier and template parameters.

[0143] The acquisition module 54 is used to stream target data of a preset number of rows from the query data obtained by the data query processing thread from the first database node corresponding to the data processing template; wherein the amount of target data is less than the amount of query data.

[0144] The sending module 55 is used to send the target data to the second database node corresponding to the data processing template based on the data insertion processing thread, so that the second database node performs a data insertion operation on the target data until the data insertion is completed.

[0145] The distributed database data processing apparatus provided in this invention includes: receiving a data processing request sent by a client, the processing request including a data processing template; after sending a processing response to the processing request to the client, receiving an execution request sent by the client, the execution request including a template identifier and template parameters corresponding to the data processing template; invoking a data query processing thread and a data insertion processing thread according to the template identifier and template parameters; stream-acquiring a preset number of target data rows from query data obtained from a first database node corresponding to a first data table based on the data query processing thread; and sending the target data to a second database node corresponding to a second data table based on the data insertion processing thread, so that the second database node performs a data insertion operation on the target data until the query data insertion is completed. In the data processing process, this application can stream-acquire target data with a data volume smaller than the query data volume, enabling simultaneous data query and target data insertion operations without waiting for all query data to be acquired before performing the data insertion operation. This achieves parallel processing of data query and data insertion to a certain extent, significantly shortening data processing time and improving data processing efficiency.

[0146] Figure 6 This is a schematic diagram of the structure of a server provided in an embodiment of the present invention. Figure 5 The server 500 shown includes at least one processor 501, memory 502, at least one network interface 504, and other user interfaces 503. The various components in the server 500 are coupled together via a bus system 505. It is understood that the bus system 505 is used to implement communication between these components. In addition to a data bus, the bus system 505 also includes a power bus, a control bus, and a status signal bus. However, for clarity, ... Figure 6 The general designated all buses as Bus System 505.

[0147] The user interface 503 may include a display, keyboard, or clicking device (e.g., mouse, trackball, touchpad, or touchscreen).

[0148] It is understood that the memory 502 in the embodiments of the present invention can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), Synchronous Dynamic Random Access Memory (SDRAM), Double Data Rate Synchronous Dynamic Random Access Memory (DDRSDRAM), Enhanced Synchronous Dynamic Random Access Memory (ESDRAM), Synchlink Dynamic Random Access Memory (SLDRAM), and Direct Rambus RAM (DRRAM). The memory 502 described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0149] In some implementations, memory 502 stores elements, executable units or data structures, or subsets thereof, or extended sets thereof: operating system 5021 and application program 5022.

[0150] The operating system 5021 includes various system programs, such as the framework layer, core library layer, and driver layer, used to implement various basic business functions and handle hardware-based tasks. The application program 5022 includes various applications, such as a media player and a browser, used to implement various application functions. The program implementing the method of this embodiment can be included in the application program 5022.

[0151] In this embodiment of the invention, the processor 501 executes the method steps provided in each method embodiment by calling the program or instructions stored in the memory 502, specifically the program or instructions stored in the application program 5022.

[0152] The methods disclosed in the above embodiments of the present invention can be applied to or implemented by processor 501. Processor 501 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the integrated logic circuit of the hardware in processor 501 or by instructions in the form of software. The processor 501 may be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of the present invention. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of the present invention can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software units in the decoding processor. The software units may be located in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in memory 502. Processor 501 reads the information in memory 502 and, in conjunction with its hardware, completes the steps of the above method.

[0153] It is understood that the embodiments described herein can be implemented in hardware, software, firmware, middleware, microcode, or a combination thereof. For hardware implementation, the processing unit can be implemented in one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), general-purpose processors, controllers, microcontrollers, microprocessors, other electronic units for performing the functions described herein, or combinations thereof.

[0154] For software implementation, the techniques described herein can be implemented by units that perform the functions described herein. The software code can be stored in memory and executed by a processor. The memory can be implemented in the processor or external to the processor.

[0155] The server provided in this embodiment can be as follows: Figure 6The server shown can perform, for example Figure 2-4 All steps of data processing methods in distributed databases, thereby achieving... Figure 2-4 For details on the technical effects of the distributed database data processing method shown, please refer to [link / reference]. Figure 2-4 The relevant descriptions are presented concisely and will not be elaborated upon here.

[0156] This invention also provides a storage medium (computer-readable storage medium). This storage medium stores one or more programs. The storage medium may include volatile memory, such as random access memory; the memory may also include non-volatile memory, such as read-only memory, flash memory, hard disk, or solid-state drive; the memory may also include combinations of the above types of memory.

[0157] When one or more programs in the storage medium can be executed by one or more processors to implement the data processing method of the distributed database described above.

[0158] The processor is used to execute data processing programs stored in the memory to implement the steps of a distributed database data processing method.

[0159] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0160] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented in hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0161] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A data processing method of a distributed database, characterized by, The method is applied to a computing node, which communicates with a client and a database node; the method includes: Receive a data processing request sent by the client, the processing request including a data processing template; Obtain the first data table information and the second data table information from the data processing template; Based on the data processing template, the first data table information, and the second data table information, a processing response is generated for the processing request, and the processing response is sent to the client. This includes: generating a template identifier corresponding to the data processing template; splitting the data processing template into a data query template and a data insertion template; generating a first data processing request for the data query template and a second data processing request for the data insertion template; sending the first data processing request to a first database node corresponding to the first data table information, and sending the second data processing request to a second database node corresponding to the second data table information; receiving a first data processing response from the first database node for the first data processing request and a second data processing response from the second database node for the second data processing request; and generating a processing response that includes the template identifier. After sending a processing response to the processing request to the client, the system receives an execution request for data processing sent by the client. The execution request includes a template identifier and template parameters corresponding to the data processing template. The data query processing thread and the data insertion processing thread are retrieved based on the template identifier and the template parameters; The target data with a preset number of rows is streamed from the query data obtained by the data query processing thread from the first database node corresponding to the data processing template; wherein, the data volume of the target data is less than the data volume of the query data; The data insertion processing thread sends the target data to the second database node corresponding to the data processing template, so that the second database node performs a data insertion operation on the target data until the queried data insertion is completed.

2. The method of claim 1, wherein, The processing request is a Prepare request for a Prepared Statement, and the execution request is an Execute request for a Prepared Statement.

3. The method according to claim 1, characterized in that, The process of retrieving the data query processing thread and the data insertion processing thread based on the template identifier and the template parameters includes: Query the data processing template corresponding to the template identifier; Construct SQL text based on the template parameters and the data processing template; The SQL text is used to retrieve data query processing threads and data insertion processing threads.

4. The method according to claim 3, characterized in that, The first data table information includes first database identification information and first data table identification information; The step of streaming a preset number of target data from the query data obtained by the data query processing thread from the first database node corresponding to the data processing template includes: Generate a query execution request for the SQL text and send the query execution request to the first database node corresponding to the first database identifier information, so that the first database node can query the query data from the first data table corresponding to the first data table identifier information based on the SQL text; Receive the query execution response sent by the first database node in response to the query execution request; A streaming data acquisition request is sent to the first database node, so that the first database node can stream a preset number of target data from the query data and send the target data to the data storage; wherein, the data storage is pre-configured on the computing node; Retrieve the stored target data from the data storage.

5. The method according to claim 3, characterized in that, The second data table information includes second database identification information and second data table identification information; The step of sending the target data to the second database node corresponding to the data processing template based on the data insertion processing thread, so that the second database node performs a data insertion operation on the target data, includes: The target data is assembled according to the protocol format of the second database node corresponding to the second database identifier information and the SQL text to obtain the execution protocol packet; The execution protocol packet is sent to the second database node, so that the second database node parses the execution protocol packet to obtain the assembled target data and inserts the assembled target data into the second data table corresponding to the second data table identifier information.

6. The method according to claim 1, characterized in that, The method further includes: Send an execution response to the client in response to the execution request.

7. The method according to claim 6, characterized in that, After sending an execution response to the client in response to the execution request, the method further includes: Receive the close request sent by the client; Clear the data processing template, the template identifier, and the template parameters; Send the shutdown request to the first database node and the second database node, and receive the first shutdown response generated by the first database node in response to the shutdown request and the second shutdown response generated by the second database node in response to the shutdown request; Generate a shutdown response that includes both the first shutdown response and the second shutdown response; Send the close response to the client.

8. A data processing device for a distributed database, characterized in that, The device is applied to a computing node, which is communicatively connected to a client and a database node; the device includes: The first receiving module is configured to receive a data processing request sent by the client, the processing request including a data processing template. A response sending module is configured to obtain first data table information and second data table information from the data processing template; generate a processing response for the processing request based on the data processing template, the first data table information, and the second data table information, and send the processing response to the client, including: generating a template identifier corresponding to the data processing template, and splitting the data processing template into a data query template and a data insertion template; generating a first data processing request for the data query template and a second data processing request for the data insertion template; sending the first data processing request to a first database node corresponding to the first data table information, and sending the second data processing request to a second database node corresponding to the second data table information; receiving a first data processing response for the first data processing request sent by the first database node and a second data processing response for the second data processing request sent by the second database node; and generating a processing response for the first data processing response and the second data processing response; wherein, the processing response includes the template identifier; The second receiving module is configured to receive a data processing execution request sent by the client after sending a processing response to the processing request to the client. The execution request includes a template identifier and template parameters corresponding to the data processing template. The retrieval module is used to retrieve the data query processing thread and the data insertion processing thread based on the template identifier and the template parameters; The acquisition module is used to stream target data of a preset number of rows from the query data obtained by the data query processing thread from the first database node corresponding to the data processing template; wherein the amount of target data is less than the amount of query data; The sending module is used to send the target data to the second database node corresponding to the data processing template based on the data insertion processing thread, so that the second database node performs a data insertion operation on the target data until the query data insertion is completed.

9. A server, characterized in that, include: A processor and a memory, the processor being configured to execute a data processing program stored in the memory to implement the data processing method for a distributed database as described in any one of claims 1 to 7.

10. A storage medium, characterized in that, The storage medium stores one or more programs, which can be executed by one or more processors to implement the data processing method of the distributed database according to any one of claims 1 to 7.

Images