Database preprocessing method, device, electronic device and storage medium

Abstract

The present invention discloses a database preprocessing method, device, electronic device and storage medium. The method is applied to any node in the computing cluster of the database, and the method comprises: when the preprocessing statement corresponding to the preprocessing request cannot be found in the local preprocessing cache, querying from the metadata preprocessing cache of the metadata storage node; if the metadata preprocessing cache includes the target metadata preprocessing cache corresponding to the preprocessing statement, synchronizing the target metadata preprocessing cache to the local, interacting with the storage cluster to generate the storage preprocessing cache corresponding to the preprocessing statement, and transmitting the preprocessing information in the target metadata preprocessing cache to the client; if the metadata preprocessing cache does not include the target metadata preprocessing cache, generating a new local preprocessing cache and storage preprocessing cache corresponding to the preprocessing statement, and transmitting the preprocessing information in the new local preprocessing cache to the client. The reliability of preprocessing in the database is improved.

Description

Technical Field

[0001] Embodiments of the present invention relate to the field of database technology, and in particular to a database preprocessing method, device, electronic device, and storage medium. Background Art

[0002] Databases use a preprocessing protocol to preprocess Structured Query Language (SQL). A client sends SQL to the database for preprocessing. Upon successful preprocessing, the database returns a corresponding identifier. This process is called preprocessing (also known as "prepare"). After preprocessing is complete, the identifier is used to execute the corresponding SQL statement (also known as "execute").

[0003] The database node that receives client preprocessing requests stores the SQL sent by the client during preprocessing, along with the corresponding preprocessing identifier, abstract syntax tree, and execution plan. If a master / slave switch occurs in the database, the information stored during preprocessing may be lost, resulting in lower reliability for preprocessing in the database. Summary of the Invention

[0004] The present invention provides a database preprocessing method, device, electronic device and storage medium, which can improve the reliability of database preprocessing in abnormal situations.

[0005] In a first aspect, an embodiment of the present invention provides a database preprocessing method, which is applied to any node in a computing cluster of the database, wherein the database further includes a storage cluster and a metadata storage node, and the method includes:

[0006] If no prepared statement corresponding to the preprocessing request can be found in the local preprocessing cache, searching from the metadata preprocessing cache of the metadata storage node;

[0007] If the metadata preprocessing cache includes a target metadata preprocessing cache corresponding to the prepared statement, synchronizing the target metadata preprocessing cache to the local computer, interacting with the storage cluster to generate a storage preprocessing cache corresponding to the prepared statement, and transmitting the preprocessing information in the target metadata preprocessing cache to the client.

[0008] If the metadata preprocessing cache does not include the target metadata preprocessing cache, a new local preprocessing cache and a storage preprocessing cache corresponding to the prepared statement are generated, and preprocessing information in the new local preprocessing cache is transmitted to the client.

[0009] In a second aspect, an embodiment of the present invention provides a database preprocessing device, which is configured at any node in a computing cluster of the database, wherein the database further includes a storage cluster and a metadata storage node, and the device includes:

[0010] A query module, configured to query from the metadata preprocessing cache of the metadata storage node if no prepared statement corresponding to the preprocessing request is found in the local preprocessing cache;

[0011] A first preprocessing module is configured to, if the metadata preprocessing cache includes a target metadata preprocessing cache corresponding to the prepared statement, synchronize the target metadata preprocessing cache to a local device, interact with the storage cluster to generate a storage preprocessing cache corresponding to the prepared statement, and transmit preprocessing information in the target metadata preprocessing cache to the client;

[0012] The second preprocessing module is used to generate a new local preprocessing cache and a storage preprocessing cache corresponding to the prepared statement if the metadata preprocessing cache does not include the target metadata preprocessing cache, and transmit the preprocessing information in the new local preprocessing cache to the client.

[0013] In a third aspect, an embodiment of the present invention provides an electronic device, including:

[0014] at least one processor; and

[0015] a memory communicatively connected to the at least one processor; wherein,

[0016] The memory stores a computer program executable by the at least one processor. The computer program is executed by the at least one processor to enable the at least one processor to perform the method according to the first aspect.

[0017] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in the first aspect.

[0018] The technical solution of the embodiment of the present invention is that when the preprocessing statement corresponding to the preprocessing request cannot be found in the local preprocessing cache, the metadata preprocessing cache is used to query and synchronize the target metadata preprocessing cache to the local. If the metadata preprocessing cache cannot be found, a new local preprocessing cache is generated. When synchronizing the target metadata preprocessing cache or generating a new local preprocessing cache, a corresponding storage preprocessing cache must be generated and the preprocessing information must be transmitted to the client. In the event that the information stored in the local preprocessing cache or the storage preprocessing cache is lost due to the master-slave node switching in the database, the client can still complete the preprocessing without being aware of it through the preprocessing request, thereby improving the reliability of preprocessing in the database under abnormal conditions.

[0019] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present invention, nor is it intended to limit the scope of the present invention. Other features of the present invention will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0021] Figure 1 is a flowchart of a database preprocessing method provided according to the first embodiment of the present invention;

[0022] Figure 2 This is a system architecture diagram of a database provided according to the first embodiment of the present invention;

[0023] Figure 3 This is a system architecture diagram of a database provided according to the second embodiment of the present invention;

[0024] Figure 4 1 is a schematic structural diagram of a database preprocessing device provided according to a third embodiment of the present invention;

[0025] Figure 5 It is a schematic structural diagram of an electronic device implementing an embodiment of the present invention. DETAILED DESCRIPTION

[0026] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0027] It should be noted that the terms "first," "second," and the like in the present invention are used to distinguish similar objects and are not necessarily used to describe a specific order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having," as well as any variations thereof, are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or apparatus comprising a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to these processes, methods, products, or apparatuses.

[0028] Example 1

[0029] Figure 1 This is a flowchart of a database preprocessing method provided according to the first embodiment of the present invention. This embodiment is applicable to the case of performing preprocessing in a database. The method can be executed by a database preprocessing device, which can be implemented in the form of software and / or hardware and integrated into an electronic device. Furthermore, the electronic device includes but is not limited to: computers, laptops, servers, etc. Figure 1 As shown, the method includes S110-S130.

[0030] An embodiment of the present invention provides a database preprocessing method, which is applied to any node in a computing cluster of the database. The database also includes a storage cluster and a metadata storage node. Figure 2 is a system architecture diagram of a database provided according to the first embodiment of the present invention, such as Figure 2 As shown, the database includes a computing cluster, a storage cluster and a metadata storage node that are respectively connected to the computing cluster. The database can provide data services for one or more clients.

[0031] The client can be the party that initiates a preprocessing request to the database system based on a preprocessing protocol. A preprocessing request can be understood as a request to preprocess an SQL statement. A preprocessing protocol, such as the Prepare Statement protocol, can be used to preprocess SQL statements within the database. The Prepare Statement protocol is a binary protocol that can preprocess SQL statements with parameter placeholders. Parameter placeholders, such as "?", primarily reserve space in the SQL statement to be filled with specific values during execution.

[0032] A compute cluster in a database can be responsible for providing Prepare Statement protocol services. A compute cluster can include multiple compute nodes, each of which can provide Prepare Statement protocol services. The compute cluster operates in an active-active state, and client preprocessing requests are evenly distributed to a compute node within the cluster. Each compute node maintains its own preprocessing cache, which is a cache related to preprocessing.

[0033] A storage cluster in a database is a cluster responsible for providing data storage services to a compute cluster. A storage cluster can include multiple storage nodes. Data required for storage on any compute node can be split and stored across multiple storage nodes. Storage nodes can provide Prepare Statement protocol services for their own data shards to compute nodes.

[0034] The metadata storage node can be a highly available metadata storage server responsible for maintaining the global pre-processing cache.

[0035] Combine Figure 2 The architecture shown in the figure takes any computing node in the computing cluster, also called the current computing node, as the execution subject. Figure 1 The method flow shown is described as follows:

[0036] S110 : When no prepared statement corresponding to the preprocessing request is found in the local preprocessing cache, a query is made from the metadata preprocessing cache of the metadata storage node.

[0037] One or more local pre-processing caches can be maintained in the current compute node. A local pre-processing cache is a cache stored locally on the current compute node, obtained by performing pre-processing based on a client's pre-processing request. This cache is also known as a server prepare cache.

[0038] Any local preprocessing cache may store the SQL statement corresponding to the preprocessing, the preprocessing identifier determined based on the SQL statement, the abstract syntax tree, meta information, the execution plan, and the executor. The preprocessing identifier may be a globally unique identifier corresponding to a preprocessing request from the client. The abstract syntax tree is a tree-like data structure that is an abstract representation of the grammatical structure of the SQL statement. The meta information may be information indicating information corresponding to the SQL statement, such as information indicating the columns of the SQL statement in the database table, the types of parameter placeholders contained in the SQL statement, etc. The execution plan may be a physical plan generated based on the abstract syntax tree of the SQL statement. The executor may be an execution operator constructed by the execution plan.

[0039] One or more metadata preprocessing caches, also known as MetaPrepare Caches, can be maintained in a metadata storage node. A metadata preprocessing cache can be considered a global preprocessing cache. Any compute node can interact with a metadata storage node to synchronize the SQL statements, preprocessing flags, abstract syntax trees, and metadata in any local preprocessing cache of that compute node to the metadata storage node and serve as a metadata preprocessing cache in the metadata storage node. Any compute node can also interact with a metadata storage node to synchronize any metadata preprocessing cache in the metadata storage node to its local location.

[0040] In the current computing node, a hash table can be used to maintain the correspondence between each local preprocessing cache and the SQL statements included in the local preprocessing cache. When the current computing node receives a preprocessing request transmitted by a client, it can query the hash table to see whether each local preprocessing cache of the current computing node includes the prepared statement corresponding to the received preprocessing request. If not, it can query the metadata preprocessing cache of the metadata storage node. The preprocessing request transmitted by the client can be a prepared statement with parameter placeholders, that is, an SQL statement that needs to be preprocessed.

[0041] S120. If the metadata preprocessing cache includes the target metadata preprocessing cache corresponding to the prepared statement, synchronize the target metadata preprocessing cache to the local, interact with the storage cluster to generate a storage preprocessing cache corresponding to the prepared statement, and transmit the preprocessing information in the target metadata preprocessing cache to the client.

[0042] In the metadata storage node, a hash table can be used to maintain the correspondence between each metadata preprocessing cache and the SQL statements included in the metadata preprocessing cache. The current compute node can interact with the metadata storage node to query whether each metadata preprocessing cache of the metadata storage node includes the prepared statement corresponding to the received preprocessing request. If so, the metadata preprocessing cache including the prepared statement is used as the target metadata preprocessing cache and synchronized to the local computer.

[0043] In one embodiment, the target metadata preprocessing cache includes at least: the prepared statement, the preprocessing identifier corresponding to the prepared statement, the abstract syntax tree, and the meta-information. When the target metadata preprocessing cache is synchronized to the local computer, the execution plan and executor corresponding to the prepared statement can be determined based on the abstract syntax tree in the target metadata preprocessing cache. The execution plan can indicate multiple split statements obtained by splitting the prepared statement, and the storage node in the storage cluster that processes each split statement. The split statement can be an SQL statement obtained by splitting the prepared statement, also known as Sub SQL.

[0044] The current compute node can interact with the storage cluster to generate a store prepare cache (also known as a store prepare cache) corresponding to prepared statements. Specifically, each split statement generated based on the prepared statement can interact with the storage node in the storage cluster that processes the split statement to generate a corresponding store prepare cache. The store prepare cache can be the store prepare cache of the storage node in the current compute node that processes the split statement.

[0045] In one embodiment, the preprocessing cache stores at least: split statements obtained by splitting the prepared statement, split identifiers corresponding to the split statements, and split meta-information. The split identifier may be an identifier uniquely corresponding to the split statement; and the split meta-information may be information indicating information corresponding to the split statement, which is not limited herein.

[0046] In one embodiment, there are multiple split statements, and each split statement generates a corresponding storage pre-processing cache. The generation process of the storage pre-processing cache includes:

[0047] Initiate a storage preprocessing request based on the target split statement to a target storage node in the storage cluster that processes the target split statement;

[0048] Obtaining a target split identifier and target split meta information corresponding to the target split statement determined by the target storage node;

[0049] A storage preprocessing cache corresponding to the target split statement is generated, where the storage preprocessing cache includes the target split statement, the target split identifier, and the target split meta information.

[0050] The target split statement may be any split statement obtained by splitting the prepared statement. The target storage node may be a storage node in the storage cluster that processes the target split statement. The storage preprocessing request may be understood as a request to preprocess the split statement. The target split identifier may be the split identifier corresponding to the target split statement. The target split meta information may be the split meta information corresponding to the target split statement.

[0051] The current computing node can transmit the target split statement to the target storage node, that is, initiate a storage preprocessing request to the target storage node; the target storage node determines the target split identifier and target split meta-information based on the target split statement, and returns the determined content to the current computing node; the current computing node generates a storage preprocessing cache corresponding to the target split statement based on the target split statement, the target split identifier and the target split meta-information.

[0052] Optionally, the target storage node may also include a storage preprocessing cache corresponding to the target split statement, which may include: the target split statement, the target split identifier, the target split meta-information, the execution plan corresponding to the target split statement, and the executor corresponding to the target split statement.

[0053] Optionally, after generating a storage preprocessing cache corresponding to the target split statement, a correspondence between the target split identifier and the preprocessing identifier in the storage preprocessing cache may be established.

[0054] In this step, the current computing node may transmit preprocessing information in the target metadata preprocessing cache to the client. The preprocessing information may be information that needs to be returned to the client after preprocessing the prepared statement based on the preprocessing request. In one embodiment, the preprocessing information includes: a preprocessing identifier and meta-information in the target metadata preprocessing cache.

[0055] S130: If the metadata preprocessing cache does not include the target metadata preprocessing cache, generate a new local preprocessing cache and a storage preprocessing cache corresponding to the prepared statement, and transmit the preprocessing information in the new local preprocessing cache to the client.

[0056] If the metadata preprocessing cache does not include the target metadata preprocessing cache, the prepared statement is preprocessed in the current compute node. The preprocessing process is as follows:

[0057] In the current computing node, a prepared statement is parsed to generate an abstract syntax tree and meta-information corresponding to the prepared statement; a preprocessing identifier corresponding to the prepared statement is generated; and a new local preprocessing cache corresponding to the prepared statement is generated, where the new local preprocessing cache includes at least the prepared statement, the preprocessing identifier corresponding to the prepared statement, the abstract syntax tree, and the meta-information. The new local preprocessing cache may be a local preprocessing cache newly created in the current computing node for the prepared statement.

[0058] Generate a storage preprocessing cache corresponding to the prepared statement. The process of generating the storage preprocessing cache in this step is basically the same as the process of generating the storage preprocessing cache described in S120 above, and will not be repeated here.

[0059] The preprocessing information in the newly created local preprocessing cache is transmitted to the client. In one embodiment, the preprocessing information includes: a preprocessing identifier and meta information in the newly created local preprocessing cache.

[0060] Optionally, after a new local preprocessing cache corresponding to the prepared statement is generated in the current computing node, the new local preprocessing cache may be synchronously updated to the metadata storage node as a new metadata preprocessing cache in the metadata storage node.

[0061] It should be noted that the embodiment of the present invention does not limit the execution order of S120 and S130, and can be determined based on whether the metadata pre-processing cache includes the target metadata pre-processing cache.

[0062] The technical solution of the embodiment of the present invention is that when the preprocessing statement corresponding to the preprocessing request cannot be found in the local preprocessing cache, the metadata preprocessing cache is used to query and synchronize the target metadata preprocessing cache to the local. If the metadata preprocessing cache cannot be found, a new local preprocessing cache is generated. When synchronizing the target metadata preprocessing cache or generating a new local preprocessing cache, a corresponding storage preprocessing cache must be generated and the preprocessing information must be transmitted to the client. In the event that the information stored in the local preprocessing cache or the storage preprocessing cache is lost due to the master-slave node switching in the database, the client can still complete the preprocessing without being aware of it through the preprocessing request, thereby improving the reliability of preprocessing in the database under abnormal conditions.

[0063] In one embodiment, the method further comprises:

[0064] If the local preprocessing cache includes a target local preprocessing cache corresponding to the prepared statement, the preprocessing information in the target local preprocessing cache is transmitted to the client.

[0065] The target local preprocessing cache may be a local preprocessing cache that includes prepared statements. The target local preprocessing cache may include: prepared statements, corresponding preprocessing identifiers, abstract syntax trees, meta-information, execution plans, and executors. The preprocessing information in the target local preprocessing cache may include the preprocessing identifiers and meta-information in the target local preprocessing cache.

[0066] The above is the process of preprocessing performed by the current computing node based on the preprocessing request transmitted by the client. The following describes the process of executing the execution request transmitted by the current computing node based on the client:

[0067] In one embodiment, the method further comprises:

[0068] If the to-be-executed preprocessing identifier corresponding to the execution request is not found in the local preprocessing cache, the execution statement corresponding to the execution request is used as the preprocessing statement, and the execution is jumped to the operation queried from the metadata preprocessing cache of the metadata storage node.

[0069] The execution request can be understood as a request to execute an SQL statement. The SQL statement to be executed is the execution statement, which can be a statement preprocessed in the database cluster. The pending preprocessing identifier can be the preprocessing identifier returned to the client during the preprocessing of the execution statement. The execution request can include the pending preprocessing identifier.

[0070] In the current computing node, the correspondence between each local preprocessing cache and the preprocessing identifier included in the local preprocessing cache can be maintained through a hash table. In the case that the current computing node receives an execution request transmitted by the client, it can be queried in the hash table whether each local preprocessing cache of the current computing node includes the preprocessing identifier to be executed. If not, it indicates that there is no local preprocessing cache corresponding to the execution statement in the current computing node, and the execution statement corresponding to the execution request is used as the preprocessing statement, and the execution is jumped to the operation queried from the metadata preprocessing cache of the metadata storage node, that is, the execution statement is preprocessed in the current computing node, and the execution statement is executed after the preprocessing is completed.

[0071] In one embodiment, the method further comprises:

[0072] If a pending preprocessing identifier corresponding to the execution request is found in the local preprocessing cache, the local preprocessing cache corresponding to the pending preprocessing identifier is determined as the pending local preprocessing cache;

[0073] Updating the execution parameters corresponding to the execution request to the executor in the local pre-processing cache to be executed;

[0074] The executor queries whether there is a to-be-executed storage preprocessing cache corresponding to the to-be-executed preprocessing identifier. If so, it interacts with the to-be-executed storage node corresponding to the to-be-executed storage preprocessing cache to request execution; otherwise, it generates the to-be-executed storage preprocessing cache, and interacts with the to-be-executed storage node corresponding to the to-be-executed storage preprocessing cache to request execution.

[0075] If the local preprocessing cache of the current computing node includes a to-be-executed preprocessing identifier, the local preprocessing cache including the to-be-executed preprocessing identifier is determined as the to-be-executed local preprocessing cache.

[0076] Checks whether there is an executor cache in the local preprocessing cache for the statement to be executed. If so, updates the execution parameters in the execution request to the executor and starts the executor. If not, generates a corresponding execution plan and executor based on the abstract syntax tree in the local preprocessing cache for the statement to be executed, updates the execution parameters to the executor, and starts the executor. Execution parameters can be parameters required for executing the statement, such as the actual parameters corresponding to parameter placeholders in the statement. Execution parameters can be determined based on metadata returned to the client during statement preprocessing.

[0077] The to-be-executed storage preprocessing cache may be a storage preprocessing cache generated during statement preprocessing. During statement preprocessing, a correspondence between the to-be-executed preprocessing identifier and the split identifier in the to-be-executed storage preprocessing cache may be established.

[0078] The executor can query the storage preprocessing cache of the current computing node based on the preprocessing identifier to be executed to see whether there is a split identifier corresponding to the preprocessing identifier to be executed. If so, the storage preprocessing cache including the split identifier will be determined as the storage preprocessing cache to be executed; if not, based on the execution plan in the local preprocessing cache to be executed, according to the aforementioned generation process involving the storage preprocessing cache, a storage preprocessing cache to be executed corresponding to the execution statement is generated.

[0079] Upon determining the pending storage preprocessing cache, the executor may transmit the split statements in the pending storage preprocessing cache and some of the execution parameters to the pending storage node, which then executes the split statements in the pending storage preprocessing cache. The pending storage node may be a storage node in the storage cluster that processes the split statements in the pending storage preprocessing cache. Some of the execution parameters may be parameters required for executing the split statements in the pending storage preprocessing cache, and may be determined based on the split metadata in the pending storage preprocessing cache.

[0080] In one embodiment, the method further comprises:

[0081] Based on the batch acquisition request transmitted by the client corresponding to the execution request, the execution results are acquired in batches from the storage node to be executed, and the execution results are returned to the client corresponding to the execution request.

[0082] A batch retrieval request, also known as a Fetch request, is a client requesting the database to return the execution results of the request in batches. If the current compute node determines that the client corresponding to the execution request has Fetch enabled, it initiates streaming processing and returns a message confirming the successful initiation of streaming processing to the client. The client then transmits the batch retrieval request to the current compute node. If Fetch is not enabled, the execution results are returned to the client all at once.

[0083] When the current computing node receives the batch acquisition request transmitted by the client corresponding to the execution request, it obtains the execution results in batches from the storage node to be executed and returns the execution results to the client corresponding to the execution request.

[0084] The above technical solution is for the process of executing the execution request transmitted by the client by the current computing node. When the local preprocessing cache or the storage preprocessing cache of the execution statement corresponding to the execution request is lost, it can jump to the preprocessing process of the execution statement for preprocessing, and continue execution after the preprocessing is completed. The client can still complete the execution without being aware of the execution request, thereby improving the reliability of execution in the database under abnormal circumstances.

[0085] Example 2

[0086] The embodiments of the present invention are illustrative descriptions of the above embodiments. Figure 3 This is a system architecture diagram of a database provided according to the second embodiment of the present invention. Figure 3 The architecture shown in can be considered as the interaction between the current computing node, the metadata storage node, and the storage node corresponding to the current computing node in the above embodiment.

[0087] like Figure 3 As shown, in the current computing node, a local preprocessing cache can be maintained, which can include: the SQL statement corresponding to the preprocessing, that is, the preprocessing statement, the preprocessing identifier determined based on the SQL statement, the abstract syntax tree, meta information, the execution plan and the executor.

[0088] In the metadata storage node, a metadata preprocessing cache may be maintained, which may include: a SQL statement corresponding to the preprocessing, a preprocessing identifier determined based on the SQL statement, an abstract syntax tree, and meta information.

[0089] In the current computing node, multiple storage preprocessing caches corresponding to the local preprocessing cache can be maintained. Each storage preprocessing cache may include: split statements obtained by splitting the corresponding SQL statement, split identifiers corresponding to the split statements, and split meta information.

[0090] Each storage preprocessing cache in the current computing node may correspond to a storage node, and the storage node may also include a storage preprocessing cache. The storage preprocessing cache may include: a split identifier, an execution plan, and an executor corresponding to the split statement.

[0091] based on Figure 3 The architecture shown in FIG. 1 illustrates the process of preprocessing the current computing node based on the preprocessing request transmitted by the client, and the process of executing the current computing node based on the execution request transmitted by the client:

[0092] Preprocessing process:

[0093] 1) receiving a preprocessing request transmitted by a client, wherein the preprocessing request includes a prepared statement;

[0094] 2) Querying whether each local preprocessing cache includes a prepared statement; if not, jump to 3); if yes, determine the local preprocessing cache including the prepared statement as the target local preprocessing cache, transmit the preprocessing identifier and meta information in the target local preprocessing cache to the client, and end;

[0095] 3) Querying whether each metadata preprocessing cache of the metadata storage node includes a prepared statement; if not, jump to 4); if yes, determine the metadata preprocessing cache including the prepared statement as the target metadata preprocessing cache, synchronize the target metadata preprocessing cache to the local computer, jump to 6), and transmit the preprocessing identifier and meta information in the target metadata preprocessing cache to the client, ending;

[0096] 4) Parse the prepared statement and generate the abstract syntax tree and meta-information corresponding to the prepared statement. If successful, jump to 5). If unsuccessful, clear the cache and end.

[0097] 5) Generate a preprocessing identifier corresponding to the prepared statement; generate a new local preprocessing cache corresponding to the prepared statement, the new local preprocessing cache including at least: the prepared statement, the preprocessing identifier corresponding to the prepared statement, the abstract syntax tree, and meta-information; transmit the preprocessing identifier and meta-information in the new local preprocessing cache to the client; synchronously update the new local preprocessing cache to the metadata storage node as a new metadata preprocessing cache in the metadata storage node;

[0098] 6) Transmit any split statement obtained by splitting the prepared statement, i.e., the target split statement, to the target storage node, i.e., initiate a storage preprocessing request to the target storage node. The target storage node determines the target split identifier and target split meta-information based on the target split statement and returns the determined content to the current compute node. The current compute node generates a storage preprocessing cache corresponding to the target split statement based on the target split statement, the target split identifier, and the target split meta-information. A correspondence between the target split identifier and the preprocessing identifier is established. If failure occurs, the cache is cleared, and the process ends.

[0099] Execution process:

[0100] 1) Receive an execution request transmitted by a client, the execution request including a pre-processing identifier to be executed and execution parameters;

[0101] 2) Check whether each local preprocessing cache contains a preprocessing identifier to be executed. If yes, jump to 3); if no, use the execution statement corresponding to the execution request as the preprocessing statement and jump to preprocessing process 3);

[0102] 3) Determine the local preprocessing cache including the pending preprocessing identifier as the pending local preprocessing cache, query whether there is an executor cache in the pending local preprocessing cache, and if so, jump to 4); if not, generate a corresponding execution plan and executor based on the abstract syntax tree in the pending local preprocessing cache and store them in the pending local preprocessing cache;

[0103] 4) Update the execution parameters to the executor in the local pre-processing cache to be executed and start the executor;

[0104] 5) The executor queries the storage preprocessing cache of the current computing node based on the preprocessing identifier to be executed to see if there is a split identifier corresponding to the preprocessing identifier to be executed. If so, the storage preprocessing cache including the split identifier is determined as the storage preprocessing cache to be executed, and jumps to 6); if not, jumps to preprocessing process 6) to generate the storage preprocessing cache to be executed. If successful, jumps to 6), otherwise, the request ends;

[0105] 6) The executor transmits the split statements in the storage preprocessing cache to be executed and some parameters in the execution parameters to the storage node to be executed, and the storage node to be executed implements the execution of the split statements in the storage preprocessing cache to be executed; if it is determined that the client corresponding to the execution request has enabled Fetch, streaming processing is started and a message indicating that streaming processing has been successfully started is returned to the client; based on the batch acquisition request transmitted by the client, the execution results are obtained in batches from the storage node to be executed, and the execution results are returned to the client.

[0106] The technical solution of the embodiment of the present invention has high reliability in preprocessing or executing SQL statements in the database under abnormal circumstances. Specifically:

[0107] 1) If the client cannot connect to the compute node it previously connected to and connects to a new compute node: if the client transmits a preprocessing request, it will be processed normally through the above preprocessing process; if the client transmits an execution request, it will be processed normally through the above execution process. If the new compute node does not have a local preprocessing cache for the execution statement corresponding to the execution request, it will be re-preprocessed and then executed, which will not affect the execution flow.

[0108] 2) When the storage node cannot be connected: If the client transmits a preprocessing request, it can be processed normally through the above preprocessing process; if the client transmits an execution request, when the above execution process is executed to the execution process 5), because the split identifier corresponding to the preprocessing identifier to be executed does not exist in the storage preprocessing cache of the current computing node, it jumps to the preprocessing process 6) to generate the storage preprocessing cache to be executed, and continues to execute after success. The above process is not perceived by the client.

[0109] The technical solution of the embodiment of the present invention can reduce memory overhead in the following aspects: the preprocessing caches involved can all be cleanable caches, which can be automatically restored when needed after being cleaned; and the ability to support streaming computing can be avoided by reading the entire execution result into the memory of the computing node at one time.

[0110] The technical solution of the embodiment of the present invention manages global preprocessing through the metadata preprocessing cache of the metadata storage node, so that prepared statements repeated across connections or computing nodes will only be precompiled once, thereby achieving execution performance optimization.

[0111] Example 3

[0112] Figure 4 This is a structural diagram of a database preprocessing device provided according to Example 3 of the present invention. This embodiment is applicable to situations where preprocessing is performed in a database. The device is configured at any node in the computing cluster of the database. The database also includes a storage cluster and a metadata storage node.

[0113] like Figure 4 As shown, the device includes:

[0114] A query module 41 is configured to query the metadata preprocessing cache of the metadata storage node if no prepared statement corresponding to the preprocessing request is found in the local preprocessing cache;

[0115] A first preprocessing module 42 is configured to, if the metadata preprocessing cache includes a target metadata preprocessing cache corresponding to the prepared statement, synchronize the target metadata preprocessing cache to a local device, interact with the storage cluster to generate a storage preprocessing cache corresponding to the prepared statement, and transmit preprocessing information in the target metadata preprocessing cache to the client;

[0116] The second preprocessing module 43 is used to generate a new local preprocessing cache and a storage preprocessing cache corresponding to the prepared statement if the metadata preprocessing cache does not include the target metadata preprocessing cache, and transmit the preprocessing information in the new local preprocessing cache to the client.

[0117] The technical solution provided by this embodiment is that, when the query module cannot find the preprocessing statement corresponding to the preprocessing request in the local preprocessing cache, it queries from the metadata preprocessing cache of the metadata storage node; if the metadata preprocessing cache includes the target metadata preprocessing cache corresponding to the preprocessing statement, the first preprocessing module synchronizes the target metadata preprocessing cache to the local, interacts with the storage cluster to generate the storage preprocessing cache corresponding to the preprocessing statement, and transmits the preprocessing information in the target metadata preprocessing cache to the client; if the metadata preprocessing cache does not include the target metadata preprocessing cache, the second preprocessing module generates a new local preprocessing cache and storage preprocessing cache corresponding to the preprocessing statement, and transmits the preprocessing information in the new local preprocessing cache to the client. In the case where the information stored in the local preprocessing cache or the storage preprocessing cache is lost due to the master-slave node switching in the database, the client can still complete the preprocessing without noticing through the preprocessing request, thereby improving the reliability of preprocessing in the database under abnormal circumstances.

[0118] Furthermore, the target metadata preprocessing cache or the newly created local preprocessing cache includes at least: the preprocessing statement, the preprocessing identifier, the abstract syntax tree and meta information corresponding to the preprocessing statement;

[0119] The preprocessing information at least includes: the preprocessing identifier and the meta information;

[0120] The storage preprocessing cache at least includes: split statements obtained by splitting the preprocessing statement, split identifiers and split meta-information corresponding to the split statements.

[0121] Furthermore, there are multiple split statements, and each split statement generates a corresponding storage preprocessing cache. The generation process of the storage preprocessing cache includes:

[0122] Initiate a storage preprocessing request based on the target split statement to a target storage node in the storage cluster that processes the target split statement;

[0123] Obtaining a target split identifier and target split meta information corresponding to the target split statement determined by the target storage node;

[0124] A storage preprocessing cache corresponding to the target split statement is generated, where the storage preprocessing cache includes the target split statement, the target split identifier, and the target split meta information.

[0125] Furthermore, the device also includes a third pre-processing module, which is used to:

[0126] If the local preprocessing cache includes a target local preprocessing cache corresponding to the prepared statement, the preprocessing information in the target local preprocessing cache is transmitted to the client.

[0127] Furthermore, the device further includes a first execution module, configured to:

[0128] If the to-be-executed preprocessing identifier corresponding to the execution request is not found in the local preprocessing cache, the execution statement corresponding to the execution request is used as the preprocessing statement, and the execution is jumped to the operation queried from the metadata preprocessing cache of the metadata storage node.

[0129] Furthermore, the device further includes a second execution module, configured to:

[0130] If a pending preprocessing identifier corresponding to the execution request is found in the local preprocessing cache, the local preprocessing cache corresponding to the pending preprocessing identifier is determined as the pending local preprocessing cache;

[0131] Updating the execution parameters corresponding to the execution request to the executor in the local pre-processing cache to be executed;

[0132] The executor queries whether there is a to-be-executed storage preprocessing cache corresponding to the to-be-executed preprocessing identifier. If so, it interacts with the to-be-executed storage node corresponding to the to-be-executed storage preprocessing cache to request execution; otherwise, it generates the to-be-executed storage preprocessing cache, and interacts with the to-be-executed storage node corresponding to the to-be-executed storage preprocessing cache to request execution.

[0133] Furthermore, the device also includes an execution result returning module, which is used to:

[0134] Based on the batch acquisition request transmitted by the client corresponding to the execution request, the execution results are acquired in batches from the storage node to be executed, and the execution results are returned to the client corresponding to the execution request.

[0135] The database preprocessing device provided in the embodiment of the present invention can execute the database preprocessing method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0136] Example 4

[0137] Figure 5 is a schematic diagram of the structure of an electronic device that implements an embodiment of the present invention. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or claimed herein.

[0138] like Figure 5 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., which is communicatively connected to the at least one processor 11. The memory stores a computer program that can be executed by the at least one processor. The processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 into the random access memory (RAM) 13. Various programs and data required for the operation of the electronic device 10 can also be stored in the RAM 13. The processor 11, ROM 12, and RAM 13 are connected to each other via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0139] Multiple components in the electronic device 10 are connected to the I / O interface 15, including an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a magnetic disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0140] The processor 11 can be any general-purpose and / or specialized processing component with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 executes the various methods and processes described above, such as the database preprocessing method.

[0141] In some embodiments, the database preprocessing method can be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as the storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the database preprocessing method described above can be performed. Alternatively, in other embodiments, the processor 11 can be configured to execute the database preprocessing method in any other appropriate manner (for example, by means of firmware).

[0142] Various embodiments of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0143] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when the computer program is executed by the processor, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The computer program may be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0144] In the context of the present invention, computer-readable storage media can be tangible media that can contain or store a computer program for use with an instruction execution system, device or equipment or used in combination with an instruction execution system, device or equipment. Computer-readable storage media can include but are not limited to electronic, magnetic, optical, electromagnetic, infrared or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, computer-readable storage media can be machine-readable signal media. More specific examples of machine-readable storage media can include electrical connections based on one or more lines, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0145] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0146] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0147] A computing system may include clients and servers. The clients and servers are typically remote from each other and typically interact via a communication network. This client-server relationship arises through computer programs running on the respective computers, creating a client-server relationship. The server may be a cloud server, also known as a cloud computing server or cloud host. This server is a hosting product within the cloud computing service ecosystem that addresses the management difficulties and limited scalability of traditional physical hosting and VPS services.

[0148] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in the present invention can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solution of the present invention can be achieved. This is not limited herein.

[0149] The above specific embodiments do not limit the scope of protection of the present invention. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention are intended to be included within the scope of protection of the present invention.

Claims

1. A database preprocessing method, characterized in that: Applied to any node in a computing cluster of a database, the database further comprising a storage cluster and a metadata storage node, the method comprising: If the prepared statement corresponding to the preprocessing request cannot be found in the local preprocessing cache, the metadata preprocessing cache of the metadata storage node is searched; the metadata preprocessing cache is a global preprocessing cache; If the metadata preprocessing cache includes a target metadata preprocessing cache corresponding to the prepared statement, synchronizing the target metadata preprocessing cache to the local computer, interacting with the storage cluster to generate a storage preprocessing cache corresponding to the prepared statement, and transmitting the preprocessing information in the target metadata preprocessing cache to the client. If the metadata preprocessing cache does not include the target metadata preprocessing cache, generating a new local preprocessing cache and a storage preprocessing cache corresponding to the prepared statement, and transmitting the preprocessing information in the new local preprocessing cache to the client; The storage preprocessing cache includes at least: split statements obtained by splitting the preprocessing statement, split identifiers and split meta information corresponding to the split statements; the split statements are used to interact with the storage nodes in the storage cluster that process the split statements to generate the split identifiers and split meta information corresponding to the split statements; Among them, any local preprocessing cache in the computing node stores the SQL statement corresponding to the preprocessing, the preprocessing identifier, abstract syntax tree, meta information, execution plan and executor determined based on the SQL statement; any metadata preprocessing cache in the metadata storage node stores the SQL statement corresponding to the preprocessing, preprocessing identifier, abstract syntax tree and meta information; There are multiple split statements, and each split statement generates a corresponding storage preprocessing cache. The generation process of the storage preprocessing cache includes: Initiating a storage preprocessing request based on the target split statement to a target storage node in the storage cluster that processes the target split statement; wherein the storage preprocessing request is a request to preprocess the target split statement; Obtaining a target split identifier and target split meta information corresponding to the target split statement determined by the target storage node; A storage preprocessing cache corresponding to the target split statement is generated, where the storage preprocessing cache includes the target split statement, the target split identifier, and the target split meta information.

2. The method according to claim 1, characterized in that The preprocessing information at least includes: the preprocessing identifier and the meta information.

3. The method according to claim 1, characterized in that Also includes: If the local preprocessing cache includes a target local preprocessing cache corresponding to the prepared statement, the preprocessing information in the target local preprocessing cache is transmitted to the client.

4. The method according to claim 1, wherein Also includes: If the to-be-executed preprocessing identifier corresponding to the execution request is not found in the local preprocessing cache, the execution statement corresponding to the execution request is used as the preprocessing statement, and the execution is jumped to the operation queried from the metadata preprocessing cache of the metadata storage node.

5. The method according to claim 1, wherein Also includes: If a pending preprocessing identifier corresponding to the execution request is found in the local preprocessing cache, the local preprocessing cache corresponding to the pending preprocessing identifier is determined as the pending local preprocessing cache; Updating the execution parameters corresponding to the execution request to the executor in the local pre-processing cache to be executed; The executor queries whether there is a to-be-executed storage preprocessing cache corresponding to the to-be-executed preprocessing identifier. If so, it interacts with the to-be-executed storage node corresponding to the to-be-executed storage preprocessing cache to request execution; otherwise, it generates the to-be-executed storage preprocessing cache, and interacts with the to-be-executed storage node corresponding to the to-be-executed storage preprocessing cache to request execution.

6. The method according to claim 5, characterized in that Also includes: Based on the batch acquisition request transmitted by the client corresponding to the execution request, the execution results are acquired in batches from the storage node to be executed, and the execution results are returned to the client corresponding to the execution request.

7. A database preprocessing device, characterized in that: Any node in a computing cluster of a database, the database further comprising a storage cluster and a metadata storage node, the device comprising: A query module, configured to query from the metadata preprocessing cache of the metadata storage node if no prepared statement corresponding to the preprocessing request can be found in the local preprocessing cache; the metadata preprocessing cache is a global preprocessing cache; A first preprocessing module is configured to, if the metadata preprocessing cache includes a target metadata preprocessing cache corresponding to the prepared statement, synchronize the target metadata preprocessing cache to a local device, interact with the storage cluster to generate a storage preprocessing cache corresponding to the prepared statement, and transmit preprocessing information in the target metadata preprocessing cache to the client; A second preprocessing module is configured to generate a new local preprocessing cache and a storage preprocessing cache corresponding to the prepared statement if the metadata preprocessing cache does not include the target metadata preprocessing cache, and transmit the preprocessing information in the new local preprocessing cache to the client; The storage preprocessing cache includes at least: split statements obtained by splitting the preprocessing statement, split identifiers and split meta information corresponding to the split statements; the split statements are used to interact with the storage nodes in the storage cluster that process the split statements to generate the split identifiers and split meta information corresponding to the split statements; Among them, any local preprocessing cache in the computing node stores the SQL statement corresponding to the preprocessing, the preprocessing identifier, abstract syntax tree, meta information, execution plan and executor determined based on the SQL statement; any metadata preprocessing cache in the metadata storage node stores the SQL statement corresponding to the preprocessing, preprocessing identifier, abstract syntax tree and meta information; There are multiple split statements, and each split statement generates a corresponding storage preprocessing cache. The generation process of the storage preprocessing cache includes: Initiating a storage preprocessing request based on the target split statement to a target storage node in the storage cluster that processes the target split statement; wherein the storage preprocessing request is a request to preprocess the target split statement; Obtaining a target split identifier and target split meta information corresponding to the target split statement determined by the target storage node; A storage preprocessing cache corresponding to the target split statement is generated, where the storage preprocessing cache includes the target split statement, the target split identifier, and the target split meta information.

8. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor. The computer program is executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images