Data query methods and devices, distributed storage systems, computing equipment

Abstract

This specification provides a data query method and apparatus, a distributed storage system, and a computing device. The method is applied to a master control node in a distributed storage system, which includes the master control node and multiple storage nodes. The master control node is communicatively connected to the multiple storage nodes, and each storage node stores a global data table. The method includes: receiving a data query statement; querying a memory table identifier based on the data query statement, wherein the memory table is obtained by loading the global data table into the memory of the master control node; generating target query information pointing to the memory table based on the data query statement and the memory table identifier; and obtaining a query result from the memory table based on the target query information. This method can improve data query efficiency.

Description

Technical Field

[0001] This specification relates to the field of database technology, and in particular to a data query method. This specification also relates to a data query device, a distributed storage system, a computing device, and a computer-readable storage medium. Background Technology

[0002] With the development of information technology, distributed storage systems have become increasingly popular due to their high reliability.

[0003] A distributed storage system comprises a master control node and multiple connected storage nodes. The master control node manages and coordinates the work of each storage node. Each node in the distributed storage system can be a database server. For example, different types of data within the same business system can be stored separately on these multiple storage nodes. When implementing a specific business function, the master control node and the various storage nodes can communicate and collaborate to process the data corresponding to that function. Data that all business functions in the business system depend on can be stored in a global table across all storage nodes, and the data in this global table across all storage nodes must be completely consistent.

[0004] The client can send a data query statement to the master control node, which then controls the corresponding storage nodes to perform the data query and return the results. However, this data query process is inefficient. Summary of the Invention

[0005] In view of this, this specification provides a data query method that can improve data query efficiency. This specification also relates to a data query device, a distributed storage system, a computing device, and a computer-readable storage medium.

[0006] According to one aspect of the embodiments of this specification, a data query method is provided, applied to a master control node in a distributed storage system, the distributed storage system including the master control node and multiple storage nodes, the master control node being communicatively connected to the multiple storage nodes, and the storage nodes storing a global data table; the method includes:

[0007] Receive data query statements;

[0008] The memory table identifier is queried based on the data query statement, wherein the memory table is obtained by loading the global data table into the memory of the main control node;

[0009] Based on the data query statement and the memory table identifier, generate target query information pointing to the memory table;

[0010] Based on the target query information, the query results are obtained from the memory table.

[0011] Optionally, generating target query information pointing to the memory table based on the data query statement and the memory table identifier includes:

[0012] The identifier of the global data table in the initial query information is changed to the identifier of the memory table to obtain the target query information; wherein, the initial query information is query information pointing to the global data table generated based on the data query statement.

[0013] Optionally, the method further includes:

[0014] In response to a load instruction for the global data table, the global data table is loaded into the memory to obtain the memory table;

[0015] Set the identifier of the global data table to indicate the memory table;

[0016] Accordingly, the data query statement includes the identifier of the global data table; querying the memory table identifier based on the data query statement includes:

[0017] Based on the identifier of the global data table included in the data query statement, query the identifier of the memory table.

[0018] Optionally, the method further includes:

[0019] In response to a data update instruction for the global data table, the memory table is updated.

[0020] Based on the data update of the memory table, the global data table in each of the storage nodes is updated using asynchronous replication.

[0021] Optionally, updating the global data table in each of the storage nodes using asynchronous replication based on data updates to the memory table includes:

[0022] During the process of updating the data in the memory table, a redo file is generated, wherein the redo file is used to record the change information of the data in the memory table;

[0023] Based on the redo file, the global data table in each of the storage nodes is updated using asynchronous replication.

[0024] Optionally, before updating the data in the memory table, the method further includes:

[0025] Create a synchronization record table corresponding to the memory table; wherein, the synchronization record table is used to record the data update operations performed on the memory table, and the update status of the global data table of each storage node for each data update operation;

[0026] The step of updating the global data table in each storage node using asynchronous replication based on data updates in the memory table includes:

[0027] Based on the synchronous record table, the global data table in each of the storage nodes is updated using asynchronous replication.

[0028] Optionally, updating the global data table in each of the storage nodes using asynchronous replication based on the synchronous record table includes:

[0029] Based on the synchronization record table, a target storage node is determined that has not updated the global data table for the target operation, wherein the target operation is any data update operation recorded in the synchronization record table;

[0030] For the target operation, control the target storage node to update the global data table using asynchronous replication;

[0031] Modify the update status of the global data table of the target storage node in the synchronization record table for the target operation.

[0032] Optionally, the method further includes:

[0033] In response to an unload instruction for the memory table, data processing on the memory table is prohibited; wherein, the data processing includes at least one of data updating and data querying;

[0034] Once the global data tables in each of the storage nodes are updated to be consistent with the memory table, the memory table is cleaned up.

[0035] Optionally, the method further includes:

[0036] In response to the unload command for the memory table, the synchronization record table is cleaned up when the global data tables in each of the storage nodes are updated to be consistent with the memory table.

[0037] Optionally, the distributed storage system further includes a backup control node of the master control node, and the method further includes:

[0038] The memory table is sent to the backup control node so that the backup control node creates a backup table of the memory table in memory.

[0039] According to another aspect of the embodiments of this specification, a data query device is provided, applied to a master control node in a distributed storage system, the distributed storage system including the master control node and multiple storage nodes, the master control node being communicatively connected to the multiple storage nodes, and the storage nodes storing a global data table; the data query device includes:

[0040] The receiving module is used to receive data query statements;

[0041] The first query module is used to query the memory table identifier based on the data query statement, wherein the memory table is obtained by loading the global data table into the memory of the main control node;

[0042] The information generation module is used to generate target query information pointing to the memory table based on the data query statement and the memory table identifier;

[0043] The second query module is used to retrieve query results from the memory table based on the target query information.

[0044] According to another aspect of the embodiments of this specification, a computing device is provided, the computing device comprising: a processor and a memory, the memory storing program instructions, the processor being configured to execute the program instructions to perform the steps in the method described above.

[0045] According to another aspect of the embodiments of this specification, a computer-readable storage medium is provided, wherein program instructions are stored therein, which, when executed by a processor, are used to implement the steps in the method described above.

[0046] According to another aspect of the embodiments of this specification, a distributed storage system is provided, the distributed storage system including a master control node and a plurality of storage nodes connected in communication, the master control node including the data query device or the computing device described above.

[0047] In one embodiment of this specification, the master control node of the distributed storage system can load a global data table into memory to obtain a memory table. Based on the received data query statement, the master control node can query the memory table identifier and generate target query information pointing to the memory table based on the data query statement and the memory table identifier. Then, based on this target query information, it retrieves the query result from the memory table. In this way, the master control node can directly obtain the query result of the data query statement without interacting with the storage node, and data querying in memory is faster, thus improving data query efficiency. Attached Figure Description

[0048] Figure 1This is a schematic diagram of a distributed storage system provided in one embodiment of this specification;

[0049] Figure 2 This is a flowchart of a data query method provided in one embodiment of this specification;

[0050] Figure 3 This is a flowchart of another data query method provided in one embodiment of this specification;

[0051] Figure 4 This is a schematic diagram of a SQL statement processing flow provided in one embodiment of this specification;

[0052] Figure 5 This is a schematic diagram of the structure of a data query device provided in one embodiment of this specification;

[0053] Figure 6 This is a structural block diagram of a computing device provided in one embodiment of this specification. Detailed Implementation

[0054] Many specific details are set forth in the following description to provide a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar extensions without departing from the spirit of this specification. Therefore, this specification is not limited to the specific implementations disclosed below.

[0055] The terminology used in one or more embodiments of this specification is for the purpose of describing particular embodiments only and is not intended to limit the scope of one or more embodiments of this specification. The singular forms “a,” “described,” and “the” as used in one or more embodiments of this specification and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items. The term “at least one” as used in one or more embodiments of this specification means “one or more,” and “a plurality of” means “two or more.” The term “comprising” is an open-ended description and should be understood as “including but not limiting,” and may include other content in addition to what has been described.

[0056] It should be understood that although the terms “first,” “second,” etc., may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, “first” may also be referred to as “second” without departing from the scope of one or more embodiments of this specification, and similarly, “second” may also be referred to as “first.” Depending on the context, the word “if” as used herein may be interpreted as “when,” “in response to a determination,” or “when…”.

[0057] Database systems can include centralized database systems and distributed database systems. In a centralized database system, data is stored on a single computer or server. In this specification, a distributed database system is referred to as a distributed storage system. Figure 1 This is a schematic diagram of a distributed storage system provided in one embodiment of this specification. Figure 1 As shown, the distributed storage system 10 includes a master control node 101 and multiple storage nodes 102. The master control node 101 is communicatively connected to the multiple storage nodes 102, and each node can be a database server.

[0058] Storage nodes 102 possess both storage and computing capabilities. These multiple storage nodes 102 can be located in different physical locations, and data is stored on the disks of these storage nodes 102. The master control node 101 is the control center of the entire distributed storage system, responsible for coordinating and managing the entire system, including core functions such as data query optimization, task scheduling, and data distribution. The master control node 101 can control the collaboration of each storage node 102 to achieve data management and processing within the distributed storage system 10. The master control node 101 can also store metadata within the distributed storage system, including data storage location, data fragmentation information, and information about the physical storage blocks on which the data is stored.

[0059] The embodiments in this specification do not limit the number of storage nodes 102 in the distributed storage system 10. Figure 1 Only three storage nodes 102 are shown in the illustration. Figure 1 The direction of the directed line segments between nodes can represent the flow of data or instructions. Optionally, please refer to... Figure 1 The distributed storage system 10 may further include one or more backup control nodes 103 of the master control node 101, which are communicatively connected to the master control node 101. The backup control node 103 can back up the data stored in the master control node 101 to ensure data reliability. Figure 1 Taking a distributed storage system 10 including a backup control node 103 as an example.

[0060] In the distributed storage system 10, data can be sharded and distributed to different storage nodes 102 according to different needs and rules. For example... Figure 1 As shown, data in a business system is sharded according to different business types to obtain data table 1, data table 2, and data table 3. These three data tables can be stored in three storage nodes 102 respectively. For example, in a financial business system, financial data can be sharded according to the location of the stock exchange and stored in different storage nodes 102.

[0061] Optionally, storage node 102 can store data from various business systems. For example, in addition to financial data, storage node 102 can also store other data such as logistics data and e-commerce data. Figure 1 The data table n in the table can refer to the data table obtained after the other data is sharded. Optionally, redundant data replication can also be performed between the various storage nodes 102 to improve data availability and fault tolerance.

[0062] The distributed storage system 10 may also contain a global data table. This global data table includes data that all business processes in the business system depend on. It can be stored on each of the storage nodes 102 where the data in the business system resides, and the data in the global data table stored on each storage node 102 is completely consistent. Each storage node 102 can directly access this global data table locally when executing business processes, ensuring high business execution efficiency.

[0063] like Figure 1 As shown, data table A is a global data table, and it is stored in each storage node 102. For example, in a financial business system, the execution of each business operation depends on national macroeconomic adjustment policies and stipulated financial parameters, which can be stored in a global data table. Each business system can have one or more global data tables, and different global data tables can store different types of data that each business operation depends on. Global data tables are typically modified infrequently.

[0064] The distributed storage system 10 can also be expanded, such as by adding new storage nodes 102 during use. After adding a new storage node 102, the global data table can be copied from the original storage node 102 to the new storage node 102.

[0065] Distributed storage system 10 can connect to a client. Specifically, the master control node 101 in distributed storage system 10 connects to the client, and the user can send a data processing request to the master control node 101 through the client to request data processing on the data stored in storage node 102. This data processing request can include Structured Query Language (SQL) statements, and the data processing can include at least one of data querying and data updating. Currently, the master control node 101 can, based on the received data query request, control the corresponding storage node 102 to perform data read queries on its disk and return the query results, which the master control node 101 then feeds back to the client. This data query process performed by distributed storage system 10 based on the data query request sent by the client is inefficient.

[0066] This specification provides a data query method. Based on this method, the master control node in a distributed storage system can efficiently perform data queries. This specification also relates to a data query device, a distributed storage system, a computing device, and a computer-readable storage medium. This distributed storage system can, as... Figure 1 As shown.

[0067] Figure 2 This is a flowchart illustrating a data query method provided in one embodiment of this specification. This method can be applied to... Figure 1 The master control node 101 in the distributed storage system 10 shown. For example... Figure 2 As shown, the method may include:

[0068] Step 201: Receive the data query statement.

[0069] In the embodiments described in this specification, the data query statement can be an SQL statement. The data query statement can be sent by a client. For example, a user can edit a data query statement based on the client, and the client can generate a data query request based on the data query statement and send the data query request to the main control node. The data query request can be packaged from the data query statement and other information. After receiving the data query request, the main control node can parse the data query request to obtain the data query statement.

[0070] In this embodiment, the data query request is used to request a data query on only one data table. Optionally, the data query request can also request different data queries on multiple data tables simultaneously, or the data query request can also request the main control node to perform other operations; this embodiment does not limit the scope of the request.

[0071] Step 203: Query the memory table identifier based on the data query statement. The memory table is obtained by loading the global data table into the memory of the main control node.

[0072] In the embodiments described in this specification, the master control node can load a global data table into memory to obtain a memory table. Upon receiving a data query statement targeting the global data table, the master control node can query the memory table identifier based on the data query statement, and execute subsequent steps if the memory table identifier is found. Specifically, the master control node can determine the target data table (i.e., the data table to be queried) based on the data query statement. For example, the data query statement can carry the identifier of the data table to be queried, or the master control node can determine the data table to be queried based on the information to be queried.

[0073] For example, if the memory table exists in the master control node's memory before receiving the data query statement, the master control node can retrieve the memory table identifier based on the data query statement. If the memory table does not exist in the master control node's memory before receiving the data query statement, the master control node cannot retrieve the memory table identifier in step 203, and the master control node can choose not to execute subsequent steps but instead execute other steps. These other steps may include controlling the corresponding storage node to execute the data query statement.

[0074] Optionally, the master control node may also query the memory table identifier for all data query statements. If the data query statement is not targeting the global data table, the master control node will not be able to find the memory table identifier in step 203, and therefore may skip the subsequent steps and instead execute other steps. These other steps may include controlling the corresponding storage node to execute the data query statement.

[0075] Step 205: Generate target query information pointing to the memory table based on the data query statement and the memory table identifier.

[0076] For example, this target query information can indicate the operations that the master control node needs to perform in order to obtain the query results corresponding to the data query statement.

[0077] Step 207: Based on the target query information, retrieve the query results from the memory table.

[0078] For example, the master control node can operate based on the instructions of the target query information to execute a data query statement and retrieve the query results from the in-memory table. The master control node can also return the obtained query results to the client.

[0079] In summary, in the data query method provided in the embodiments of this specification, the master control node of the distributed storage system can load a global data table into memory to obtain a memory table. Based on the received data query statement, the master control node can query the memory table identifier and generate target query information pointing to the memory table based on the data query statement and the memory table identifier. Then, based on this target query information, it retrieves the query result from the memory table. In this way, the master control node can directly obtain the query result of the data query statement without interacting with the storage node, and the data query speed in memory is relatively fast, thus improving data query efficiency.

[0080] Figure 3 This is a flowchart of another data query method provided in one embodiment of this specification, which can be applied to... Figure 1 The master control node 101 in the distributed storage system 10 shown. For example... Figure 3 As shown, the method may include:

[0081] Step 301: In response to the load instruction for the global data table, load the global data table into memory to obtain the memory table.

[0082] Users can send load commands to the master control node from the client for global data tables in the distributed storage system. The master control node can respond to these commands by loading the global data table from the storage nodes into its memory, thus obtaining the corresponding in-memory table. Optionally, the master control node can also generate these load commands itself. If, by default, all global data tables in the distributed storage system are loaded into the master control node's memory, the master control node can generate load commands for each global data table.

[0083] For example, the master control node can partition storage areas in memory; then, it retrieves the structure of the global data table from the storage nodes and partitions the storage areas based on this structure; subsequently, it reads data from the global data table and loads it into the partitioned storage areas, and builds an index on the data in those storage areas. This completes the loading of the global data table, resulting in an in-memory table.

[0084] A distributed storage system's storage nodes can store one or more global data tables. When multiple global data tables are stored in a storage node, a load command sent by the client can instruct the loading of at least one of these global data tables, such as by carrying an identifier for that global data table. If the load command instructs the loading of multiple global data tables, the master control node can obtain the memory tables corresponding to each of the multiple global data tables. This specification's embodiments only illustrate loading a single global data table; the global data table mentioned below refers to the loaded global data table.

[0085] Optionally, before loading the global data table, the master control node can lock the global data table in each storage node, prohibiting operations on that global data table. This prevents changes to the global data table in the storage nodes during the loading process, ensuring that the master control node loads the latest records from the global data table. Optionally, after the master control node obtains the memory table, it can unlock the global data table in the storage nodes.

[0086] In the embodiments described in this specification, the global data table loaded into the memory of the main control node can meet the condition of a small data volume, such as less than a set data volume threshold. This can avoid the memory table occupying too much storage space of the main control node, and avoid affecting the original working efficiency of the main control node due to the storage of the memory table. Optionally, the global data table can also meet the condition of having a certain update requirement, but the update frequency is not too high, so as to avoid affecting the storage logic of the main control node.

[0087] Step 303: Create the synchronization record table corresponding to the memory table.

[0088] The synchronization record table records data update operations performed on the in-memory table, as well as the update status of the global data table on each storage node for each data update operation. This data update operation can be recorded using a corresponding transaction identifier. For ease of description, the update of the global data table on the storage node will be referred to as the storage node update, and the update status of the global data table on the storage node will be referred to as the storage node update status.

[0089] The in-memory table must maintain consistency with the global data table in the storage nodes. This synchronization record table tracks the data synchronization status between the in-memory table and the global data table in the storage nodes. If data is updated in the in-memory table, the global data table must also be updated accordingly. When performing a data update operation on the in-memory table, the information about this operation can be recorded in the synchronization record table, along with whether each storage node has updated the global data table accordingly based on the update operation (e.g., indicated by the update status).

[0090] In one alternative approach, the synchronization log table can record all data update operations performed on the in-memory table, as well as the update status of all storage nodes for each data update operation. This update status includes both updated and unupdated states.

[0091] In another alternative approach, the synchronization record table can also record target data update operations for the memory table, as well as the update status of the global data table on all storage nodes for each target update operation. Here, a target data update operation refers to an operation where some storage nodes have not yet performed the corresponding data update. When all storage nodes have performed the corresponding data update for a certain data update operation, the information for that data update operation, along with the update status information for each storage node for that data update operation, can be deleted from the synchronization record table.

[0092] In another alternative approach, the synchronization record table can record target data update operations for the in-memory table, as well as target storage nodes that have not completed the corresponding data update for each target update operation. When a storage node's global data table performs a corresponding data update for a certain data update operation, the information for that storage node's data update operation can be deleted from the synchronization record table. For example, the global data table is stored in a first storage node, a second storage node, and a third storage node. A first data update operation and a second data update operation are sequentially performed on the in-memory table of this global data table. The first and second storage nodes complete the first data update operation on the global data table, while all three storage nodes fail to complete the second data update operation. In this case, the synchronization record table can record that for the first data update operation, the storage node that failed to complete the update includes the third storage node; and for the second data update operation, the storage nodes that failed to complete the update include the first, second, and third storage nodes.

[0093] Step 305: Set the identifier of the global data table to indicate the memory table.

[0094] After loading the global data table into memory to obtain the in-memory table, the master control node can also set the identifier of the global data table to indicate this in-memory table, so that the processing flow of subsequent SQL statements targeting the global data table can be performed on this in-memory table. In this mode, the global data table in each storage node is equivalent to a storage copy.

[0095] For example, the master control node can bind the identifier of a global data table to the identifier of a memory table. Subsequently, based on the identifier of the global data table, the identifier of the memory table can be determined, allowing the memory table to be located and processed. SQL statements include Data Manipulation Language (DML) statements and Data Definition Language (DDL) statements. DML statements are used to perform CRUD operations on data in the database, while DDL statements are used to define or change the structure and data types of data tables.

[0096] Step 307: Send the memory table to the backup control node so that the backup control node can create a backup table of the memory table in memory.

[0097] After obtaining the memory table, the master control node can send the memory table to the backup control node, so that the backup control node can store the received memory table as a backup, thereby achieving backup of the memory table and ensuring high data storage security and high reliability of the memory table.

[0098] For example, the backup control node can create a backup table of the memory table in memory; that is, it stores the received memory table in memory to obtain the backup table. If the memory table in the primary control node suffers data loss or other problems, or if the primary control node fails, the backup control node can take over the work of the primary control node. The backup control node can perform the same operations on the backup table based on the operations that the primary control node needs to perform on the memory table.

[0099] As another example, the backup control node can also be used solely for data backup, and the backup table can be stored on the backup control node's disk instead of in memory. If the in-memory table in the primary control node experiences data loss or other problems, the primary control node can retrieve the backup table from the backup control node to restore the in-memory table.

[0100] The following example illustrates the process of the main control node executing SQL statements based on a memory table, using the scenario of updating and querying data in a global data table as an example.

[0101] Step 309: In response to the data update instruction for the global data table, update the data in the memory table.

[0102] Users can send data update commands for global data tables to the master control node via the client. These commands can include SQL statements instructing data updates. The master control node can respond to these commands by first updating the corresponding in-memory table. Data updates can include adding, deleting, and modifying data. They can also include data import updates and data structure changes.

[0103] For example, the master control node can first parse the SQL statement included in the data update instruction to determine the data table targeted by the SQL statement. If the data table targeted by the SQL statement includes a global data table, the master control node then updates the data in the memory table based on the SQL statement. Optionally, the master control node can also update the data in the memory table based on the SQL statement if it is determined that the global data table targeted by the SQL statement corresponds to a memory table. Optionally, if the master control node determines that the global data table does not correspond to a memory table, it can directly control each storage node to update the data in the global data table based on the SQL statement.

[0104] Figure 4 This is a schematic diagram illustrating the processing flow of an SQL statement according to an embodiment of this specification. For example... Figure 4 As shown, the client can send SQL statements to the master control node, which can then process the SQL statement sequentially using a parser, analyzer, rewriter, planner, and executor. After obtaining the processing result, the master control node can then send the result back to the client.

[0105] The parser can parse plain text SQL statements to generate a parse tree. A parse tree is a tree-like structure used to represent grammatical structures. In a parse tree, the nodes represent symbols or terminal symbols of grammatical rules, while the edges represent the process of rule application.

[0106] The analyzer can perform semantic analysis on the parse tree to generate initial query information, such as a query tree. This query tree can reflect information such as the command type on which its generation was based, the data tables used in the query, and the connections between the data tables. For example, the initial query information may include the identifier of a global data table.

[0107] The rewriter can transform initial query information using rules stored in the rule system to generate target query information. For example, the rewriter can transform the query tree generated by the analyzer to generate a new query tree, providing support for rules and views. If the target in the query tree may define transformation rules, the rewriter can rewrite the query tree based on those rules. In the embodiments of this specification, the identifier of the global data table can indicate a memory table. The rewriter can change the identifier of the global data table in the initial query information to a memory table identifier so that the SQL statement points to the memory table. For example, if the initial query information points to the disk of storage node a, indicating the retrieval of data with indices 1 to 10 in the global data table; then the target query information can point to the memory of the main control node, indicating the retrieval of data with indices 1 to 10 in the memory table.

[0108] The planner can determine the most efficient plan tree based on the query tree. Alternatively, it can select the optimal plan tree based on the SQL statement and a cost estimation model, minimizing the execution cost. Since the SQL statement only specifies the desired query result and not the specific query route, it can be executed using different schemes. Although the execution results of each scheme may be the same, their execution efficiency can differ. The planner can then select the scheme with the lowest cost as the plan tree.

[0109] The executor can access data tables and indexes to execute SQL statements in the execution order indicated by the plan tree and obtain the execution results of the SQL statements.

[0110] In the embodiments described in this specification, the backup table in the standby control node must be consistent with the memory table in the primary control node. When the memory table in the primary control node is updated, the backup table in the standby control node also needs to be updated accordingly. If the primary control node has multiple standby control nodes, each of these standby control nodes must store a backup table of the memory table, and when the memory table in the primary control node is updated, each of these standby control nodes must also update its backup table accordingly. The update process of the backup table in the standby control node can refer to the update process of the memory table in the primary control node, and will not be described in detail in the embodiments described in this specification.

[0111] Step 311: Based on the update of the memory table, update the global data table in each storage node using asynchronous replication.

[0112] After updating the in-memory table, the master control node can use asynchronous replication to update the global data table on each storage node. For example, the master control node can update the global data table on each storage node during a relatively quiet period in the distributed storage system, thus avoiding any impact on other operations of the distributed storage system caused by updating the global data table.

[0113] In the embodiments described in this specification, during the process of updating data in the memory table, the master control node can generate a redo file, which records the changes to the data in the memory table. Based on this redo file, the master control node can update the global data table in each storage node using asynchronous replication. During the process of updating the global data table in each storage node, the master control node can also update the synchronization record table based on the update process. Each time a storage node completes its update, the master control node can modify the update status of that storage node in the synchronization record table.

[0114] The master control node can determine the target storage node that has not updated the global data table in response to a target operation, based on the synchronization record table. The target operation is any data update operation recorded in the synchronization record table, specifically the aforementioned target data update operation. Furthermore, the master control node can control the target storage node to update the global data table asynchronously in response to the target operation. After the target storage node updates the global data table, the master control node can modify the update status of the target storage node's global data table in the synchronization record table for the target operation.

[0115] In the embodiments of this specification, if the redo file and synchronization record table contains multiple data update operations that the storage nodes have not completed updating, then each storage node needs to update the global data table in sequence according to the order of these multiple data update operations.

[0116] In the embodiments described in this specification, all SQL statements for the global data table can be processed based on the in-memory table. After updating the in-memory table, the master control node can directly process subsequent SQL statements for the global data table based on the updated in-memory table. The global data table is updated using asynchronous replication; this update process does not affect the processing of SQL statements for the global data table.

[0117] In related technologies, after receiving an update instruction for the global data table, the master control node directly instructs all storage nodes to update the global data table based on the instruction. Only after all storage nodes have completed the update can data processing operations on the global data table be resumed. Since the global data table is typically stored across all storage nodes in a distributed storage system, the time required for all storage nodes to complete the data update is relatively long. Therefore, the global data table becomes unavailable for an extended period due to the data update, resulting in low data update efficiency and reduced efficiency for other data processing operations. Furthermore, as the distributed storage system is built and needs to be expanded, the number of storage nodes will increase, further extending the data update time and reducing data processing efficiency.

[0118] In the embodiments described in this specification, updating the in-memory table requires, at most, updating the in-memory table on the backup control node. Since the total number of primary and backup control nodes in a distributed storage system is relatively small (generally not exceeding 3), the number of operation nodes targeted by the update transaction is small, requiring a maximum of 3 data tables to be updated. This update process is relatively short. The in-memory table is only unavailable during the update process, thus preventing the processing of SQL statements targeting the global data table. The shorter processing time facilitates faster release of database resources, reduces the impact of data updates on other data processing operations, lowers the cost of data updates, and improves the data processing efficiency of the distributed storage system.

[0119] The following example illustrates the processing flow of the main control node based on an SQL statement that is subsequently received by the main control node to query the global data table.

[0120] Step 313: Receive a data query statement for the global data table. The data query statement includes the identifier of the global data table.

[0121] For example, a user can send a data query statement targeting a global data table to the master control node via a client, and the master control node can receive the data query statement. This data query statement is an SQL statement and may include an identifier for the global data table to indicate that the query statement is used to query data from the global data table.

[0122] Step 313 can be referred to the relevant description in step 201, and will not be repeated here in the embodiments of this specification.

[0123] Step 315: Based on the data query statement, generate initial query information pointing to the global data table.

[0124] After receiving a data query statement, the master control node can execute a processing flow for the SQL statement. This processing flow can be found in the documentation. Figure 4 As described above. For example, in step 315, the main control node can use a parser and analyzer to parse and analyze the data query statement to generate the initial query information.

[0125] Step 317: Based on the identifier of the global data table included in the data query statement, query the identifier of the memory table.

[0126] After receiving a data query statement for a global data table, the master control node can query the memory table identifier based on the identifier of the global data table. Optionally, the query for the memory table identifier can be performed during the analysis of the data query statement by the analyzer, or after the analyzer has obtained the initial query information from the data query statement.

[0127] When the global data table is loaded into the memory of the master control node, the master control node binds the identifier of the global data table to the memory table. This allows the master control node to query the memory table identifier based on the global data table identifier. If the global data table is not loaded into the memory of the master control node, the master control node cannot query the memory table identifier. Therefore, the master control node can skip subsequent steps and directly control the storage node to execute the data query statement.

[0128] Step 319: Change the identifier of the global data table in the initial query information to the identifier of the memory table to obtain the target query information.

[0129] The master control node can use the rewriter to execute step 319 to change the identifier of the global data table in the initial query information to the identifier of the memory table, thereby obtaining the target query information. This target query information can be the new query tree obtained after rewriting. Step 319 can refer to the above description. Figure 4 The details regarding the rewriter described in the previous instructions will not be repeated in the embodiments of this manual. Afterwards, the main control node can operate on the memory table in memory based on this data query statement.

[0130] Step 321: Based on the target query information, retrieve the query results from the memory table.

[0131] The master control node can use the planner and executor to execute step 321 to process the data query statement and retrieve the query results from the memory table. Step 319 can refer to the above... Figure 4 The descriptions of the planner and executor in the previous section will not be repeated in the embodiments of this manual.

[0132] Step 323: In response to the unload instruction for the memory table, prohibit data processing on the memory table.

[0133] Data processing includes at least one of data updating and data querying.

[0134] Users can send unload commands for memory tables to the master control node via the client, instructing the master control node to unload the memory table. The master control node can also analyze the memory table to determine if it has received an unload command, and then unload the memory table when certain unload conditions are met.

[0135] For example, the unloading condition could include the global data table's data size exceeding a data size threshold. This data size threshold could be 1 gigabyte (GB), 2 gigabytes, or other values. If a global data table needs to be updated, and the updated global data table's data size would become excessive, the master control node can unload the memory table corresponding to that global data table.

[0136] When unloading a memory table, the master control node can first perform a queue lock on the memory table, prohibiting its use and thus preventing data processing on the memory table.

[0137] Step 325: After all global data tables in each storage node have been updated to match the memory tables, clean up the memory tables.

[0138] After locking the memory table, the master control node can also check whether the global data tables on each storage node are consistent with the memory table. For example, it checks whether the global data tables on each storage node have completed the corresponding data updates for data update operations on the memory table. After ensuring that the global data tables on each storage node are updated to be consistent with the memory table, the memory table is then cleaned up, along with other information related to the memory table (such as synchronization record tables), thus unloading the memory table.

[0139] The master control node can check the synchronization record table to determine whether the global data tables on each storage node have been updated. If a storage node has not completed the data update for a certain data update operation, the master control node will first perform the corresponding data update for that data update operation. Afterward, the memory table will be marked as unavailable, and then the memory table will be cleaned up, along with the synchronization record table.

[0140] In this embodiment, the global data table is loaded into the memory of the main control node to obtain a memory table, thereby changing operations on the global data table to operations on the memory table. The global data table is stored on the disk of the storage node. Reading data from the disk introduces factors such as disk seek and rotational latency, resulting in lower disk read / write speeds compared to higher memory read / write speeds. Therefore, in this embodiment, the main control node's operations on the memory table are more efficient, and the interaction with the storage node is reduced, improving the processing efficiency of SQL statements on the global data table, such as improving data query efficiency.

[0141] In the embodiments described in this specification, users can trigger the main control node to load the global data table and unload the memory table through commands, ensuring that the loading and unloading actions are controllable, avoiding the memory table from residing in memory for a long time, and preventing additional burden on the distributed storage system.

[0142] Furthermore, in related technologies, during the process of updating the global data table on each storage node, each storage node needs to update independently based on the data update statement, or the global data table needs to be copied to another node after one storage node has completed the update. If an update error occurs on a certain storage node, the consistency of the global data table across all storage nodes cannot be guaranteed. Moreover, the network instability between storage nodes is relatively high, resulting in poor replication efficiency and stability, which leads to low overall update efficiency.

[0143] In the embodiments described in this specification, each storage node updates based on the update of the memory table in the master control node, which makes it easier to ensure data consistency among storage nodes. Furthermore, the network connection between the storage nodes and the master control node is relatively stable, ensuring high update efficiency and stability. Moreover, for each data update statement, the memory table is updated first, and then each storage node updates based on the memory table. This avoids the impact on data consistency caused by different storage nodes executing different statements, and it also facilitates concurrency control of storage nodes.

[0144] In summary, in the data query method provided in the embodiments of this specification, the master control node of the distributed storage system can load a global data table into memory to obtain a memory table. Based on the received data query statement, the master control node can query the memory table identifier and generate target query information pointing to the memory table based on the data query statement and the memory table identifier. Then, based on this target query information, it retrieves the query result from the memory table. In this way, the master control node can directly obtain the query result of the data query statement without interacting with the storage node, and the data query speed in memory is relatively fast, thus improving data query efficiency.

[0145] Corresponding to the above method embodiments, this specification also provides an embodiment of a data query device. Figure 5 This is a schematic diagram of a data query device provided in one embodiment of this specification. This data query device can be applied to... Figure 1 The master control node 101 in the distributed storage system 10 shown. For example... Figure 5 As shown, the data query device 50 includes:

[0146] The receiving module 501 is used to receive data query statements.

[0147] The first query module 502 is used to query the memory table identifier based on the data query statement. The memory table is obtained by loading the global data table into the memory of the main control node.

[0148] The information generation module 503 is used to generate target query information pointing to the memory table based on the data query statement and the memory table identifier.

[0149] The second query module 504 is used to retrieve query results from the memory table based on the target query information.

[0150] In summary, in the data query device provided in the embodiments of this specification, the main control node of the distributed storage system can load a global data table into memory to obtain a memory table. Based on the received data query statement, the main control node can query the memory table identifier and generate target query information pointing to the memory table based on the data query statement and the memory table identifier. Then, based on this target query information, it retrieves the query result from the memory table. In this way, the main control node can directly obtain the query result of the data query statement without interacting with the storage node, and the data query speed in memory is relatively fast, thus improving data query efficiency.

[0151] Optionally, the information generation module 503 is also used for:

[0152] The identifier of the global data table in the initial query information is changed to the identifier of the memory table to obtain the target query information; the initial query information is the query information pointing to the global data table generated based on the data query statement.

[0153] Optionally, the data query device 50 also includes:

[0154] The loading module is used to load the global data table into memory in response to a load command for the global data table, thus obtaining the in-memory table;

[0155] The configuration module is used to set the identifier of the global data table to indicate the in-memory table;

[0156] Accordingly, the data query statement includes the identifier of the global data table; the first query module 502 is used to query the identifier of the memory table based on the identifier of the global data table included in the data query statement.

[0157] Optionally, the data query device 50 also includes:

[0158] The first update module is used to update the data in the memory table in response to data update instructions for the global data table;

[0159] The second update module is used to update the global data table in each storage node using asynchronous replication based on the data update of the memory table.

[0160] Optionally, the second update module is used for:

[0161] During the process of updating data in the memory table, a redo file is generated. The redo file is used to record the change information of the data in the memory table.

[0162] Based on redo files, the global data table in each storage node is updated using asynchronous replication.

[0163] Optionally, the data query device 50 also includes:

[0164] The first creation module is used to create a synchronization record table corresponding to the memory table before updating the data in the memory table; the synchronization record table is used to record the data update operations performed on the memory table, as well as the update status of the global data table of each storage node for each data update operation.

[0165] The second update module is used to update the global data table in each storage node based on the synchronous record table and using asynchronous replication.

[0166] Optionally, the second update module is used to: determine the target storage node that has not updated the global data table for the target operation based on the synchronization record table, wherein the target operation is any data update operation recorded in the synchronization record table;

[0167] For the target operation, control the target storage node to update the global data table using asynchronous replication;

[0168] Modify the update status of the target storage node's global data table in the synchronization record table for the target operation.

[0169] Optionally, the data query device 50 also includes:

[0170] A disable module is used to disable data processing on a memory table in response to an unload command for the memory table; wherein data processing includes at least one of data updating and data querying;

[0171] The first cleanup module is used to clean up the memory table after the global data tables on each storage node have been updated to be consistent with the memory table.

[0172] Optionally, the data query device 50 also includes:

[0173] The second cleanup module is used to clean up the synchronization record table in response to the unload command for the memory table, provided that the global data tables in each storage node have been updated to be consistent with the memory table.

[0174] Optionally, the data query device 50 also includes:

[0175] The second creation module is used to send the memory table to the backup control node so that the backup control node can create a backup table of the memory table in memory.

[0176] In summary, in the data query device provided in the embodiments of this specification, the main control node of the distributed storage system can load a global data table into memory to obtain a memory table. Based on the received data query statement, the main control node can query the memory table identifier and generate target query information pointing to the memory table based on the data query statement and the memory table identifier. Then, based on this target query information, it retrieves the query result from the memory table. In this way, the main control node can directly obtain the query result of the data query statement without interacting with the storage node, and the data query speed in memory is relatively fast, thus improving data query efficiency.

[0177] The above is an illustrative scheme of a data query device according to this embodiment. It should be noted that the technical solution of this data query device and the technical solution of the data query method described above belong to the same concept. For details not described in detail in the technical solution of the data query device, please refer to the description of the technical solution of the data query method described above.

[0178] Figure 6 This is a structural block diagram of a computing device provided in one embodiment of this specification. This computing device can be applied to... Figure 1 The distributed storage system 10 shown is a master control node 101. The components of the computing device 400 include, but are not limited to, a memory 410 and a processor 420. The processor 420 is connected to the memory 410 via a bus 430, and a database 450 is used to store data. The memory 410 stores program instructions, which the processor 420 executes to perform... Figure 2 and Figure 3 The data query method described above.

[0179] The computing device 400 also includes an access device 440, which enables the computing device 400 to communicate via one or more networks 460. Examples of these networks include Public Switched Telephone Network (PSTN), Local Area Network (LAN), Wide Area Network (WAN), Personal Area Network (PAN), or combinations of communication networks such as the Internet. The access device 440 may include one or more of any type of wired or wireless network interface (e.g., Network Interface Controller (NIC)), such as an IEEE 802.11 Wireless Local Area Networks (WLAN) wireless interface, a Wi-MAX (Worldwide Interoperability for Microwave Access) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, a Near Field Communication (NFC) interface, and so on.

[0180] In one embodiment of this specification, the aforementioned components of the computing device 400 and Figure 6 Other components, not shown, can also be connected to each other, for example, via a bus. It should be understood that... Figure 6 The block diagram of the computing device shown is for illustrative purposes only and is not intended to limit the scope of this application. Those skilled in the art can add or replace other components as needed.

[0181] The computing device 400 can be any type of stationary or mobile computing device, including mobile computers or mobile computing devices (e.g., tablet computers, personal digital assistants, laptop computers, notebook computers, netbooks, etc.), mobile phones (e.g., smartphones), wearable computing devices (e.g., smartwatches, smart glasses, etc.) or other types of mobile devices, or stationary computing devices such as desktop computers or PCs. The computing device 400 can also be a mobile or stationary server.

[0182] The above is an illustrative scheme of a computing device according to this embodiment. It should be noted that the technical solution of this computing device and the technical solution of the data query method described above belong to the same concept. For details not described in detail in the technical solution of the computing device, please refer to the description of the technical solution of the data query method described above.

[0183] This specification also provides a computer-readable storage medium storing computer instructions that, when executed by a processor, are used to implement... Figure 2 and Figure 3 The data query method is described above. The computer instructions include computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable storage medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc.

[0184] The above is an illustrative scheme of a computer-readable storage medium according to this embodiment. It should be noted that the technical solution of this storage medium and the technical solution of the data query method described above belong to the same concept. For details not described in detail in the technical solution of the storage medium, please refer to the description of the technical solution of the data query method described above.

[0185] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0186] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0187] In the above embodiments, the descriptions of each embodiment have different focuses. Parts not described in detail in a particular embodiment can be found in the relevant descriptions of other embodiments. For example, regarding... Figure 2 Introduction and Figure 3 The introductions can be used for mutual reference.

[0188] The preferred embodiments disclosed above are merely illustrative of this application. The optional embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the content of this application. These embodiments are selected and specifically described in this application to better explain the principles and practical applications of this application, thereby enabling those skilled in the art to better understand and utilize this application. This application is limited only by the claims and their full scope and equivalents.

Claims

1. A data query method, characterized by, A master control node applied to a distributed storage system, the distributed storage system comprising the master control node and a plurality of storage nodes, the master control node being in communication connection with the plurality of storage nodes, and the storage nodes storing a global data table; the method comprising: receiving a data query statement; querying a memory table identifier based on the data query statement, wherein a memory table is obtained by loading the global data table in a memory of the master control node, and an identifier of the global data table is associated with the memory table identifier; generating target query information pointing to the memory table according to the data query statement and the memory table identifier, comprising: generating initial query information by parsing and analyzing the data query statement; and changing the identifier of the global data table in the initial query information to the memory table identifier to obtain the target query information; querying a query result from the memory table based on the target query information.

2. The method of claim 1, wherein, The generating target query information pointing to the memory table according to the data query statement and the memory table identifier comprises: changing the identifier of the global data table in the initial query information to the memory table identifier to obtain the target query information; wherein the initial query information is query information pointing to the global data table generated based on the data query statement.

3. The method of claim 1, wherein, The method further comprises: loading the global data table into the memory to obtain the memory table in response to a loading instruction for the global data table; setting the identifier of the global data table as indicating the memory table; Correspondingly, the data query statement comprises the identifier of the global data table; and the querying the memory table identifier based on the data query statement comprises: querying the memory table identifier based on the identifier of the global data table included in the data query statement.

4. The method according to any one of claims 1 to 3, characterized in that, The method further comprises: performing data update on the memory table in response to a data update instruction for the global data table; updating the global data table in each of the storage nodes in an asynchronous replication manner based on the data update on the memory table.

5. The method of claim 4, wherein, The updating the global data table in each of the storage nodes in an asynchronous replication manner based on the data update on the memory table comprises: generating a redo file in the process of performing data update on the memory table, wherein the redo file is used to record change information of data in the memory table; and updating the global data table in each of the storage nodes in an asynchronous replication manner based on the redo file.

6. The method of claim 4, wherein, Before performing data update on the memory table, the method further comprises: creating a synchronization record table corresponding to the memory table, wherein the synchronization record table is used to record data update operations performed on the memory table, and update states of the global data table of each storage node for each data update operation; The updating the global data table in each of the storage nodes in an asynchronous replication manner based on the data update on the memory table comprises: updating the global data table in each of the storage nodes in an asynchronous replication manner based on the synchronization record table.

7. The method of claim 6, wherein, updating, based on the synchronization record table, the global data table in each storage node in an asynchronous replication manner, comprising: determining, based on the synchronization record table, a target storage node that fails to update the global data table for a target operation, wherein the target operation is any data update operation recorded in the synchronization record table; controlling the target storage node to update the global data table for the target operation in an asynchronous replication manner; modifying, in the synchronization record table, an update state of the global data table of the target storage node for the target operation.

8. The method of claim 4, wherein, The method further comprises: in response to an unloading instruction for the memory table, prohibiting data processing on the memory table; wherein the data processing comprises at least one of data update and data query; in the case that the global data table in each storage node is updated to be consistent with the memory table, cleaning up the memory table.

9. The method according to claim 6 or 7, characterized in that, The method further comprises: in response to an unloading instruction for the memory table, in the case that the global data table in each storage node is updated to be consistent with the memory table, cleaning up the synchronization record table.

10. The method according to any one of claims 1 to 3, characterized in that, The distributed storage system further comprises a backup control node of the master control node, and the method further comprises: sending the memory table to the backup control node, so that the backup control node creates a backup table of the memory table in memory.

11. A data query apparatus, characterized by comprising: The data query device is applied to a master control node in a distributed storage system, the distributed storage system comprising the master control node and a plurality of storage nodes, the master control node being in communication connection with the plurality of storage nodes, and the storage nodes storing global data tables; the data query device comprises: a receiving module configured to receive a data query statement; a first query module configured to query a memory table identifier based on the data query statement, wherein a memory table is obtained by loading the global data tables in memory of the master control node, and an identifier of the global data tables is associated with the memory table identifier; an information generation module configured to generate target query information pointing to the memory table according to the data query statement and the memory table identifier, comprising: generating initial query information by parsing and analyzing the data query statement; and changing the identifier of the global data tables in the initial query information to the memory table identifier to obtain the target query information; a second query module configured to query a query result from the memory table based on the target query information.

12. A computing device, comprising: The computing device comprises a processor and a memory, the memory storing program instructions, and the processor is configured to execute the program instructions to implement steps in the method of any one of claims 1 to 10.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores program instructions, and the program instructions, when executed by a processor, are configured to implement steps in the method of any one of claims 1 to 10.

14. A distributed storage system, characterized by, The distributed storage system comprises a master control node and a plurality of storage nodes in communication connection, the master control node comprising the data query device of claim 11 or the computing device of claim 12.

15. A computer program product, characterised in that, Computer program comprising computer instructions which, when executed by a processor, implement the steps of the method according to any one of claims 1 to 10.

Images