Data processing method and corresponding device

Abstract

The invention provides a data processing method which is applied to a distributed database, base tables of the distributed database are stored in different data nodes DN in a distributed mode, a coordination node CN can generate an XC table return plan of a global secondary index GSI according to non-batch query of the base tables, and the CN executes the table return plan. According to the method, a table return plan is executed once through two remote queries, and target data are obtained from a DN where the target data to be queried is located. In this way, the data query speed is increased, and the performance of the distributed database is also improved.

Description

Technical Field

[0001] This application relates to the field of database technology, specifically to a data processing method and corresponding apparatus. Background Technology

[0002] A distributed database typically consists of one or more coordinator nodes (CNs) and multiple data nodes (DNs). The CN can schedule tasks to the DNs, and the DNs can execute the tasks scheduled by the CN, thereby performing operations such as data queries. Data in a distributed database is usually stored in the form of base tables. Data belonging to the same base table can be stored as shards on different DNs; the global secondary index (GSI) corresponding to the base table can also be stored as shards on different DNs.

[0003] When executing a query, the Data Node (DN) can use GSI to retrieve data from the base table; this process is also called a table lookup. The table lookup process differs depending on the scenario, and can be understood as different table lookup plans. Currently, table lookup plans for queries either execute within a single DN, preventing cross-DN execution and making them unsuitable for distributed databases using GSI; or they use streaming plans that execute interactively between multiple DNs, suitable for batch queries but not for non-batch queries. This is because the streaming plan requires stream thread initialization before query execution, which consumes time and resources. In batch queries, with large amounts of data, the time and resources spent on stream thread initialization are a small percentage of the total time and resources consumed by the batch query. However, in non-batch query scenarios, because the amount of data queried is small, the time and resources spent on stream thread initialization constitute a large percentage of the total time and resources consumed by the non-batch query, thus reducing database performance.

[0004] Therefore, there is an urgent need for a table lookup plan for query scenarios involving distributed storage base tables across data nodes, in order to implement the corresponding query process. Summary of the Invention

[0005] This application provides a data processing method for improving the speed and performance of non-batch queries in a distributed database. This application also provides corresponding apparatus, computer-readable storage media, and computer program products.

[0006] This application provides a data processing method applicable to a coordinator node (CN) of a distributed database, wherein the distributed database further includes multiple data nodes (DNs). The method includes: generating a first lookup plan based on a query statement; wherein the query statement includes indication information of a first base table, and the first lookup plan includes a first query request and a second query request; wherein the first query request instructs a first data node to scan the global secondary index corresponding to the first base table, and the first data node is the data node where the global secondary index corresponding to the first base table is located; the second query request instructs a second data node to query target data, and the second data node is the data node where the target data is located; and executing the first lookup plan to obtain the target data.

[0007] In one possible implementation, the method further includes receiving a query statement, the query statement including indication information of a first base table; and returning the target data after obtaining the target data.

[0008] In this application, the query statement can be a structured query language (SQL) statement entered by the user. This query statement can be sent by the client. After the CN obtains the target data, it will return the target data to the client. This target data is the query result corresponding to the query statement.

[0009] In this application, the indication information of the first base table may be the identifier or name of the first base table or other information that can indicate the first base table.

[0010] In this application, the first base table may include multiple columns and multiple rows; each of the multiple rows is used to store data, and the columns may describe the attributes of the data in that column. To speed up queries, an index may be created on one or more columns of the base table; the indexed column may be called the index key or the index column.

[0011] In this application, the first data node and the second data node can be the same data node or different data nodes.

[0012] In this application, the global secondary index (GSI) corresponds to the base table. "Global" refers to its relation to the distributed storage base table. To distinguish it from local secondary indexes (auxiliary secondary indexes created for a local table (part of the base table) stored on a specific data node), an auxiliary secondary index created for the entire base table is called a global secondary index. "Secondary" refers to its relation to the index key (also called the index column) in the base table. The index key in the base table can be understood as a primary index, and auxiliary indexes created for other non-index columns can be called secondary indexes.

[0013] In this application, "table lookup" refers to the process of querying a non-indexed column that cannot be directly retrieved using the index key in the base table. Querying each data node individually would incur significant overhead. To improve query speed, a global secondary index can be used to query the data node containing the target data. This process is called "table lookup".

[0014] In this application, the first table lookup plan refers to the plan of the execution process of querying target data in the first base table, using GSI to query information related to the storage location of the target data, and querying the target data based on the information related to the storage location of the target data. The content related to this execution process (such as code) can be called the "first table lookup plan".

[0015] In this application, the global secondary index corresponding to the first base table may include the index key in the first base table, as well as the logical values ​​and row information of the target data in the data nodes. Of course, the global secondary index may also include other information, which is not limited here.

[0016] In the first aspect mentioned above, in the table lookup scenario based on a global secondary index, CN executes a table lookup plan with only two query requests, without needing to start the stream thread required for interaction between data nodes. This saves time in initializing and starting the stream thread, improves the speed of data query, and also improves the performance of the distributed database.

[0017] In one possible implementation, the query statement also includes query conditions; generating a first table lookup plan based on the query statement includes: generating a first table lookup plan if the query conditions indicate that the query statement is a non-batch query.

[0018] In this application, non-batch query refers to querying data with a relatively limited number of rows, such as one row or a few rows.

[0019] In this possible implementation, when the query condition indicates that the query statement is a non-batch query, a first table lookup plan is generated. For non-batch data, only two query requests are needed to execute a single query on the non-batch data, instead of frequently sending query requests for different data within the batch data. This improves the speed of data retrieval for non-batch queries and also enhances the performance of the distributed database.

[0020] In one possible implementation, the above steps: executing a first lookup plan to obtain target data include: sending a first query request to a first data node according to the first lookup plan; receiving the logical value of the data node where the target data is located and the row information of the target data, wherein the logical value is used to determine that the target data is located in a second data node; sending a second query request to the second data node according to the first lookup plan, wherein the second query request contains the row information of the target data; and receiving the target data from the second data node.

[0021] In this application, the logical value of the data node where the target data is located is used to indicate the data node where the target data is located. The logical value may include a hash value representing the data node where the data is located, or a range value representing the data node where a certain range of data is located.

[0022] In this application, target data refers to the data of the row corresponding to the row information, which can be one row of data or multiple rows of data.

[0023] In this application, the coordinating node can obtain the logical value of the data node containing the target data through a first query request, and then determine the second data node containing the target data based on this logical value. In this way, the coordinating node can directly query data from the second data node, without needing to query each of the multiple data nodes storing the first base table individually, thereby improving the speed of data querying. The coordinating node transmits the row information of the target data to the second data node through a second query request, so that the second data node can directly use the row information to determine the target data, thereby improving the speed of querying the target data.

[0024] In one possible implementation, the method further includes generating a second table lookup plan for the batch query if the query condition indicates that the query statement is a batch query.

[0025] In this possible implementation, if the query conditions indicate that the query statement is a batch query, a second table lookup plan can be generated for the batch query. This can improve the speed of batch queries.

[0026] A second aspect of this application provides a distributed database, including a coordinating node and multiple data nodes;

[0027] The coordinating node is used to receive query statements, which include indication information of the first base table. The data of the first base table is distributed and stored on different data nodes.

[0028] The coordinating node is used to generate a first table lookup plan based on the query statement; the first table lookup plan includes a first query request and a second query request; the first query request is used to instruct the first data node to scan the global secondary index corresponding to the first base table, and the first data node is the data node where the global secondary index corresponding to the first base table is located; the second query request is used to instruct the second data node to query the target data, and the second data node is the data node where the target data is located;

[0029] The coordinating node is used to execute the first table lookup plan to obtain the target data;

[0030] The coordinating node is used to return the target data.

[0031] In one possible implementation, the coordinating node is specifically used to generate the first table lookup plan if the query statement is determined to be a non-batch query based on the query conditions when the query statement also includes query conditions.

[0032] In one possible implementation, the coordinating node is used to: send a first query request to the first data node according to the first table lookup plan;

[0033] The first data node is used to scan the global secondary index corresponding to the first base table according to the first query request in order to obtain the logical value of the data node where the target data is located and the row information of the target data;

[0034] The coordinating node is also used to receive the logical value of the data node where the target data is located and the row information of the target data, and determine that the target data is located in the second data node based on the logical value;

[0035] The coordinating node is also used to send a second query request to the second data node according to the first table lookup plan. The second query request contains row information of the target data.

[0036] The second data node is used to query the target data based on the row information of the target data.

[0037] The coordinating node is also used to receive target data from the second data node.

[0038] In one possible implementation, the coordinating node is also used to generate a second table-back plan for the batch query when the query conditions indicate that the query statement is a batch query.

[0039] A third aspect of this application provides a data processing apparatus, comprising:

[0040] The processing unit is configured to generate a first table lookup plan based on a query statement; wherein the query statement includes indication information of a first base table, and the first table lookup plan includes a first query request and a second query request; wherein the first query request is configured to instruct a first data node to scan the global secondary index corresponding to the first base table, and the first data node is the data node where the global secondary index corresponding to the first base table is located; the second query request is configured to instruct a second data node to query the target data, and the second data node is the data node where the target data is located;

[0041] The processing unit is also used to execute the first table lookup plan to obtain the target data.

[0042] In one possible implementation, the device further includes:

[0043] The transceiver unit is used to receive query statements, which include indication information from the first base table; and to return the target data after the target data is retrieved.

[0044] In one possible implementation, the processing unit is specifically used to generate a first table lookup plan if the query statement is determined to be a non-batch query based on the query conditions when the query statement also includes query conditions.

[0045] In one possible implementation, the transceiver unit is also used for:

[0046] According to the first table lookup plan, send the first query request to the first data node;

[0047] Receive the logical value of the data node where the target data is located and the row information of the target data. The logical value is used to determine that the target data is located in the second data node.

[0048] According to the first table return plan, a second query request is sent to the second data node. The second query request contains row information of the target data.

[0049] Receive target data from the second data node.

[0050] In one possible implementation, the processing unit is further configured to generate a second table lookup plan for the batch query if the query condition indicates that the query statement is a batch query.

[0051] A fourth aspect of this application provides a data processing apparatus. The apparatus may include at least one processor, a memory, and a communication interface. The processor is coupled to the memory and the communication interface. The memory is used to store instructions, the processor is used to execute the instructions, and the communication interface is used to communicate with other devices under the control of the processor. When executed by the processor, the instructions cause the processor to perform the methods of the first aspect or any possible implementation thereof.

[0052] The fifth aspect of this application provides a computer-readable storage medium storing one or more computer-executable instructions, wherein when the computer-executable instructions are executed by a processor, the processor performs a method as described in the first aspect or any possible implementation thereof.

[0053] The sixth aspect of this application provides a computer program product that stores one or more computer execution instructions, wherein when the computer execution instructions are executed by a processor, the processor executes the method described in the first aspect or any possible implementation thereof.

[0054] A seventh aspect of this application provides a chip system including a processor for supporting a computer device in implementing the functions described in the first aspect or any possible implementation thereof. In one possible design, the chip system may further include a memory for storing necessary program instructions and data. This chip system may be composed of chips or may include chips and other discrete devices.

[0055] The technical effects of the second aspect, the third aspect, any possible implementation of the second aspect or the third aspect, and the fourth to seventh aspects can be found in the first aspect or the technical effects of different possible implementations of the first aspect, and will not be repeated here. Attached Figure Description

[0056] Figure 1A This is a schematic diagram of the architecture of a database system;

[0057] Figure 1B This is a schematic diagram of the architecture of a distributed database system provided in an embodiment of this application;

[0058] Figure 1C This is another schematic diagram of the architecture of the distributed database system provided in the embodiments of this application;

[0059] Figure 1D This is another schematic diagram of the architecture of the distributed database system provided in the embodiments of this application;

[0060] Figure 2 This is another schematic diagram of the architecture of the distributed database system provided in the embodiments of this application;

[0061] Figure 3 This is a schematic diagram of an embodiment of the data processing method provided in this application;

[0062] Figure 4 This is a schematic diagram of another embodiment of the data processing method provided in this application;

[0063] Figure 5 This is a schematic diagram of a scenario of the data processing method provided in an embodiment of this application;

[0064] Figure 6 This is another scenario illustration of the data processing method provided in the embodiments of this application;

[0065] Figure 7 This is another scenario illustration of the data processing method provided in the embodiments of this application;

[0066] Figure 8 This is another scenario illustration of the data processing method provided in the embodiments of this application;

[0067] Figure 9 This is another scenario illustration of the data processing method provided in the embodiments of this application;

[0068] Figure 10A This is an example schematic diagram of a non-batch query back-table plan provided in the embodiments of this application;

[0069] Figure 10B This is an example schematic diagram of a back-table plan for batch query provided in an embodiment of this application;

[0070] Figure 11 This is a schematic diagram of the structure of the data processing apparatus provided in an embodiment of this application;

[0071] Figure 12 This is another structural schematic diagram of the data processing apparatus provided in the embodiments of this application;

[0072] Figure 13 This is another schematic diagram of the cluster database system provided in the embodiments of this application. Detailed Implementation

[0073] The embodiments of this application are described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. As those skilled in the art will understand, with the development of technology and the emergence of new scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0074] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in a sequence other than that illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0075] This application provides a data processing method to improve the speed and performance of non-batch queries in a distributed database. This application also provides corresponding apparatus, computer-readable storage media, and computer program products. These will be described in detail below.

[0076] The method provided in this application embodiment can be applied to a database system. Figure 1A This illustrates a typical logical architecture of a database system, based on Figure 1A Database system 100 includes database 110 and database management system (DBMS) 130.

[0077] Database 110 is an organized collection of data stored in data storage 120; that is, a related collection of data organized, stored, and used according to a specific data model. Depending on the data model used to organize the data, the data can be divided into various types, such as relational data, graph data, and time series data. Relational data is data modeled using a relational model, usually represented as a table, where rows represent a set of related values ​​for an object or entity. Graph data, simply called a "graph," is used to represent relationships between objects or entities, such as social relationships. Time series data is a series of data columns recorded and indexed in chronological order, used to describe the state changes of an object over time.

[0078] The database management system 130 is the core of the database system, serving as system software for organizing, storing, and maintaining data. Clients 200 can access the database 110 through the database management system 130, and database administrators also use the database management system for database maintenance. The database management system 130 provides various functions for clients 200 to create, modify, and query the database; clients 200 can be applications or user devices. The functions provided by the database management system 130 may include, but are not limited to, the following: (1) Data definition function: The database management system 130 provides a data definition language (DDL) to define the structure of the database 110. The DDL is used to describe the database framework and can be stored in the data dictionary; (2) Data access function: The database management system 130 provides a data manipulation language (DML) to implement basic access operations on the database 110, such as retrieval, insertion, modification and deletion; (3) Database operation management function: The database management system 130 provides data control function to effectively control and manage the operation of the database 110 to ensure that the data is correct and valid; (4) Database creation and maintenance function: including loading the initial data of the database, database dumping, recovery, reorganization, system performance monitoring and analysis, etc.; (5) Database transmission: The database management system provides data transmission to realize communication between the client and the database management system, which is usually coordinated with the operating system.

[0079] Data storage 120 includes, but is not limited to, solid-state drives (SSDs), disk arrays, cloud storage, or other types of non-transitory computer-readable storage media. Those skilled in the art will understand that a database system may include more than […]. Figure 1A The components shown are fewer or more components, or include those with Figure 1A The components shown are different components. Figure 1A Only components more relevant to the implementation disclosed in the embodiments of the present invention are shown.

[0080] The database system provided in this application embodiment can be a distributed database system (DDBS). DDBS is a logically unified database composed of multiple physically dispersed database units connected by a computer network. Each connected database unit is called a site or node. The following describes the process in conjunction with... Figure 1B and Figure 1C Introducing DDBS.

[0081] Figure 1BThis diagram illustrates a distributed database system employing a shared-nothing architecture. The system includes one or more coordinator nodes (CNs) and multiple data nodes (DNs). Each DN can store a global secondary index (GSI) and base tables. If a query can be accelerated by using a GSI lookup plan, the query process can be completed using this plan. The GSI lookup process is as follows: After receiving the query, the CN determines an execution plan based on the query and then queries the GSI according to the plan. From the GSI, it determines the logical value and row information of the data node containing the target data. The logical value identifies the DN containing the target data, and the row information indicates the row containing the target data. Thus, the target data can be retrieved through the lookup process.

[0082] It should be noted that a distributed database system may also include one or more GTMs (such as...). Figure 1B The system comprises a first GTM and a second GTM. The first GTM acts as the master GTM, while the second GTM backs up the data of the first GTM and takes over its operation when the first GTM fails, ensuring the high reliability of DDBS. CNs and DNs communicate via a network channel. Each DN has its own dedicated hardware resources (such as data storage), operating system, and database. CNs, DNs, the first GTM, or the second GTM communicate via a network channel. In one embodiment, this network channel can consist of network devices such as switches, routers, and gateways. CNs, DNs, and GTMs jointly implement the functions of the database management system, providing clients with services such as database querying (retrieval), insertion, modification, and deletion. In one embodiment, a database management system is deployed on each CN, DN, and GTM. In this distributed database system, data is allocated to various DNs according to the database model and application characteristics. Query tasks are divided into several parts by the CN and executed in parallel on multiple DNs, coordinating computation to provide database services as a whole. All communication functions can be implemented on a high-bandwidth network interconnection system.

[0083] In a distributed database system, the CN, DN, first GTM, or second GTM can be a physical machine, such as a database server, or a virtual machine (VM) or container running on abstract hardware resources. In one embodiment, the CN, DN, first GTM, or second GTM is a VM or container, and the network channel is a virtual switching network, which includes virtual switches. The database management system deployed in the CN, DN, first GTM, or second GTM is a DBMS instance, which can be a process or a thread. These DBMSs work together to complete the functions of the database relational system. In another embodiment, the CN, DN, first GTM, or second GTM is a physical machine, and the network channel includes one or more switches, which can be a storage area network (SAN) switch, an Ethernet switch, a fiber optic switch, or other physical switching equipment.

[0084] Figure 1C This is a schematic diagram of a distributed database system using a shared-storage architecture. Figure 1C In the distributed database system shown, a shared data storage device stores data that can be shared by multiple Data Nodes (DNs). DNs can perform read and write operations on the data in the data storage device via a network channel. The shared data storage device can be a shared disk array. The CNs, DNs, and the first or second GTM communicate via a network channel, which can be found in the above description. Figure 1B Please refer to the relevant introductions for understanding. In this system, data can be stored in a shared data storage device. Query tasks are divided into several parts by the CN (Network Controller), executed in parallel on multiple DNs (Data Networks), and compute collaboratively, providing database services as a whole. All communication functions can be implemented on a high-bandwidth interconnected network. (Similar to...) Figure 1B Similar to the shared-nothing architecture of the distributed database system described, CN, DN, first GTM, or second GTM can be either physical machines or virtual machines.

[0085] In all embodiments of this application, the data storage of the database system includes, but is not limited to, solid-state drives (SSDs), disk arrays, or other types of non-transient computer-readable media. Figure 1A-1C Although a database is not shown, it should be understood that a database is stored in a data storage device. Those skilled in the art will understand that a database system may include more than [a certain number of data storage devices]. Figure 1A-1C The components shown are fewer or more components, or include those with Figure 1A-1C The components shown are different components. Figure 1A-1C Only components more relevant to the implementation disclosed in the embodiments of this application are shown. However, those skilled in the art will understand that a distributed database system can contain any number of CNs and DNs. The database management system functions of each CN and DN can be implemented by appropriate combinations of software, hardware, and / or firmware running on each CN and DN.

[0086] The above Figure 1B and Figure 1C The described distributed database system includes multiple Data Domains (DNs) and multiple Data Centers (CNs). DNs are responsible for executing specific execution plans. Aside from the different data they store, DNs are architecturally identical and support various data distribution methods. CNs can be elected from DNs using an election algorithm or can adopt a different architecture. Depending on the specific deployment of the distributed database, there can be one or more CNs. Database modules can be deployed equally across CNs and DNs, and their roles within the distributed database can be configured through configuration files.

[0087] For details regarding the distribution of base tables and global secondary indexes on DNs, and the relationship between CNs and DNs, please refer to [link to relevant documentation]. Figure 1D To understand.

[0088] like Figure 1D As shown, base table 1 can be distributed across data node 1 and data node 2. For example, the first part of base table 1's data is stored in data node 1, and the second part of base table 1's data is stored in data node 2. Global secondary index 1 can also be distributed across data node 1 and data node 2. For example, the first part of global secondary index 1's data is stored in data node 1, and the second part of global secondary index 1's data is stored in data node 2. It should be noted that... Figure 1D The diagram only illustrates two data nodes distributively storing base table 1 and global secondary index 1. In reality, multiple data nodes can distribute base table 1 and global secondary index 1. Furthermore, the global secondary index 1 corresponding to base table 1 and the data in base table 1 can be distributed and stored on different data nodes, or they can be distributed and stored on the same data node.

[0089] After receiving a query statement, CN can generate an execution plan for the query statement, that is, use the global secondary index 1 to perform a table lookup on base table 1, thereby determining the target data corresponding to the query statement.

[0090] For ease of understanding, the technical terms involved in the embodiments of this application are briefly introduced below:

[0091] 1. Base Table: Also known as a table, a table is an object used to store data in a database. It is a structured collection of data and forms the foundation of the entire database system. A table is a database object containing all the data in the database, and it is typically defined as a collection of columns. Rows in a base table can be understood as data, and columns describe the attributes of that data. To speed up queries, indexes can be created on one or more columns of the base table. The column on which the index is created is called the index key or index column. Furthermore, a base table can be distributed across different data nodes, and the correspondence between different parts of the base table and the data nodes can be represented by a distribution key.

[0092] 2. Index: refers to a sorted data structure in a database management system to assist in quickly querying and updating data in database tables.

[0093] 3. Primary Key: A base table often has one or more columns whose values ​​uniquely identify each row in the table. This column or columns is called the table's primary key, which enforces entity integrity within the table. A primary key is created when a table is created or modified by defining a primary key constraint. A table can only have one primary key constraint, and columns in the primary key constraint cannot accept null (NULL) values. Because primary key constraints ensure unique data, they are often used to define identity columns.

[0094] 4. NULL value: This is a special marker used in Structured Query Language. It is an identifier for unknown or missing attributes in the database and is used to indicate uncertain values ​​in the database.

[0095] 5. Distribution Key: In a distributed database, a combination of one or more columns is used to determine the data node containing each row of data in the base table.

[0096] 6. Global Secondary Index (GSI): A GSI can correspond to a base table, and can be understood as an index table. "Global" refers to its relation to the distributed storage base table. To distinguish it from local secondary indexes (auxiliary secondary indexes created for a partial table (a part of the base table) stored on a specific data node), an auxiliary secondary index created for the entire base table is called a global secondary index. "Secondary" refers to its relation to the index key (also called the index column) in the base table. The index key in the base table can be understood as a primary index, and auxiliary indexes created for other non-index columns are called secondary indexes. In a distributed database system, data in a base table can be distributed across multiple domains (DNs). To facilitate searching for data on non-distributed keys in the base table, the DN containing the data to be queried can be determined first using GSI, and then the corresponding data can be retrieved from that DN. This improves query efficiency compared to querying each DN individually.

[0097] 7. Table Lookup: For an SQL query, if the tables involved in the query have indexes, an index scan plan is generally generated first. Furthermore, if the columns involved in the query include non-indexed columns, a table lookup plan will be generated. This table lookup process first scans the index to obtain the primary key or row pointers of the data, and then scans the primary key index or base table based on the primary key or row pointers to obtain the complete data.

[0098] 8. GSI table lookup involves first querying the GSI index table to obtain the logical value of the DN (Data Node) containing the data in the base table to be queried, as well as the row pointer. Then, the DN is determined based on the logical value, and the row data is retrieved from the corresponding DN based on the row pointer.

[0099] 9. Execution plan in a distributed database: This refers to the plan by which the Node Controller (CN) interacts with the Data Node (DN) to complete the data query after receiving the query statement. Common execution plans in distributed databases fall into three categories: distributed pushdown plans, distributed cross-data node streaming plans, and distributed XC plans (distributed XC plans).

[0100] 10. Distributed Direct Pushdown Plan: This plan involves the coordinating node directly sending the query statement to the data nodes, which then execute the query and return the results. For table lookup scenarios, the direct pushdown plan is suitable for scenarios where the lookup only occurs within the data nodes. However, since global secondary indexes need to be accessed across nodes, the distributed direct pushdown plan cannot be used in query scenarios involving global secondary indexes.

[0101] 11. Distributed Cross-Data Node Streaming Plan: This plan involves data interaction between data nodes, with all computations performed on the data nodes as much as possible. For table lookup scenarios, distributed cross-data node streaming plans are suitable for batch query table lookup scenarios but not for non-batch query table lookup scenarios. This is because the streaming plan requires stream thread initialization before starting the query, which consumes time and resources. In batch queries, the amount of data to be queried is large, so the time and resources consumed by stream thread initialization account for a small percentage of the total time and resources consumed by the batch query. However, in non-batch query scenarios, because the amount of data queried is small, the time and resources consumed by stream thread initialization account for a large percentage of the total time and resources consumed by the non-batch query. For example, if stream thread initialization takes 1 millisecond (ms), and the actual execution time of a batch query is usually over 1000 ms, then the proportion of stream thread initialization in the total time is 1 / (1000+1) ≈ 0.1%. Conversely, if the actual execution time of a non-batch query is very small, such as 4 ms, then the proportion of stream thread initialization in the total time is 1 / (4+1) = 20%. It is evident that using streaming plans to execute non-batch queries incurs significant overhead and reduces database performance.

[0102] 12. Distributed XC Plan: This involves the coordinating node interacting with data nodes via RemoteQuery, suitable for non-batch data query scenarios. For table lookup scenarios, the distributed XC plan is suitable for non-batch query table lookup scenarios, but currently, the distributed XC plan does not support global secondary index table lookup.

[0103] The following section, based on the above discussion of distributed database systems, combines... Figure 2 This paper describes the data processing process of the distributed database system provided in the embodiments of this application.

[0104] like Figure 2 As shown in the embodiment of this application, the distributed database system includes: a coordinating node and multiple data nodes, wherein the multiple data nodes include a first data node and a second data node.

[0105] The coordinating node receives query statements, which include indication information about the first base table. The data in the first base table is distributed and stored across different data nodes. This indication information can be the identifier or name of the first base table, or other information that identifies it. The first base table can include multiple columns and multiple rows; each row stores data, and the columns describe the attributes of the data in that column. To speed up queries, indexes can be created on one or more columns of the base table. These indexed columns are called index keys or indexed columns.

[0106] The first data node stores the global secondary index corresponding to the first base table, and this global secondary index can be an index table of the global secondary index.

[0107] The second data node stores a portion of the data from the first base table. Other data from the first base table can also be stored on the first data node or other data nodes.

[0108] After receiving a query statement, the coordinating node can generate a first table lookup plan based on the query statement; the first table lookup plan includes a first query request and a second query request.

[0109] The coordinating node can send a first query request to the first data node. This first query request can be RemoteQuery1.

[0110] The first data node will scan the global secondary index corresponding to the first base table according to the first query request to obtain the logical value of the data node where the target data is located and the row information of the target data. The logical value may include a hash value representing the data node where the target data is located, or a range value representing the data range of the target data in the data nodes.

[0111] After receiving the logical value of the target data node and the row information of the target data from the first data node, the coordinating node can determine the second data node containing the target data to be queried based on the logical value, and then send a second query request to the second data node, which contains the row information. This second query request can be RemoteQuery2.

[0112] The second data node can determine the target data based on the row information and then return the target data to the coordinating node. The target data refers to the data in the row corresponding to the row information; this target data can be one row or multiple rows.

[0113] After receiving the target data, the coordinating node can return the target data to the client.

[0114] The above combination Figure 2 The distributed database system described herein, in a table lookup scenario based on a global secondary index, executes a table lookup plan with only two query requests. This eliminates the need to start the streaming threads required for interaction between data nodes, saving time on initializing and starting the streaming threads, thus improving the speed of data queries and the performance of the distributed database.

[0115] based on Figure 2 The distributed database system described herein, and the data processing method provided in this application embodiment, for the coordination node of the distributed database, can: generate a first table lookup plan based on a query statement; wherein the query statement includes indication information of a first base table, and the first table lookup plan includes a first query request and a second query request; wherein the first query request is used to instruct a first data node to scan the global secondary index corresponding to the first base table, and the first data node is the data node where the global secondary index corresponding to the first base table is located; the second query request is used to instruct a second data node to query target data, and the second data node is the data node where the target data is located; and execute the first table lookup plan to obtain the target data.

[0116] In one possible embodiment, it can be combined with Figure 3 Understand the data processing methods provided in the embodiments of this application.

[0117] like Figure 3 As shown, the data processing method provided in this application embodiment includes:

[0118] S301. The coordinating node receives a query statement, which includes indication information from the first base table.

[0119] The query can come from the client.

[0120] S302. The coordinating node generates the first table lookup plan based on the query statement.

[0121] The first table lookup plan includes a first query request and a second query request. The first query request instructs the first data node to scan the global secondary index corresponding to the first base table, and the first data node is the data node where the global secondary index corresponding to the first base table is located. The second query request instructs the second data node to query the target data, and the second data node is the data node where the target data is located.

[0122] Optionally, the query statement may also include query conditions, and S302 may include: if the query conditions indicate that the query statement is a non-batch query, then generate a first table lookup plan.

[0123] In this application, non-batch query refers to querying data with a relatively limited number of rows, such as one row or a few rows.

[0124] S303. The coordinating node sends a first query request to the first data node. Correspondingly, the first data node receives the first query request.

[0125] The first query request may include indication information of the global secondary index or indication information of the first base table, so that the first data node can determine the global secondary index to be scanned based on the indication information of the global secondary index or the indication information of the first base table.

[0126] S304. The first data node scans the global secondary index corresponding to the first base table according to the first query request to obtain the logical value of the data node where the target data is located and the row information of the target data.

[0127] The form of the first base table can be understood by referring to Table 0 below.

[0128] Table 0: First Base Table

[0129] Attribute 1 (School) Attribute 2 (Name) Attribute 3 (Age) No.1 Middle School Name 1 17 Second Middle School Name 2 18 Third Middle School Name 3 18 No.1 Middle School Name 4 16

[0130] Taking a table where the first base table is an athlete table as an example, attribute 1 can indicate the athlete's school, attribute 2 can indicate the athlete's name, and attribute 3 can indicate the athlete's age. An index can be created on any column in this first base table; that column then becomes the index column. If an index is created on the name, then attribute 2 becomes the index column. This attribute 2 column can also be called the index key, where different names have different key values.

[0131] An example of a global secondary index corresponding to the first base table can be understood by referring to Table 1 below.

[0132] Table 1: Global Secondary Indexes of the First Base Table

[0133]

[0134] The process of the first data node obtaining the logical value of the data node where the target data is located and the row information of the target data can be understood by referring to Table 1 above. For example, if the index key is required to be the name 2, then the logical value of the data node where the name 2 is located can be obtained as the logical value 2, and the row information is (1,3).

[0135] S305. The first data node sends the logical value of the data node containing the target data and the row information of the target data to the coordinating node. Correspondingly, the coordinating node receives the logical value of the data node containing the target data and the row information of the target data.

[0136] S306. The coordinating node determines that the target data is located in the second data node based on the logical value of the data node where the target data is located.

[0137] S307. The coordinating node sends a second query request to the second data node. Correspondingly, the second data node receives the second query request.

[0138] The second query request contains row information for the target data.

[0139] S308. The second data node queries the target data based on the row information of the target data.

[0140] S309. The second data node returns the target data to the coordinating node. Correspondingly, the coordinating node receives the target data.

[0141] S310. The coordinating node returns the target data.

[0142] The data processing method provided in this application embodiment enables the coordinating node to execute a table lookup plan with only two query requests, thereby improving the speed of data query and the performance of the distributed database.

[0143] The above describes the data processing procedure in a global secondary index lookup scenario. In reality, after receiving an SQL statement, the coordinating node evaluates the statement and determines the appropriate execution plan based on the evaluation result. This process can be found in [reference needed]. Figure 4 To understand.

[0144] like Figure 4 As shown, the data processing procedure includes:

[0145] S401. The coordinating node receives SQL statements.

[0146] S402. The coordinating node performs a fastpath lightweight assessment on the SQL statement. If the assessment result meets the conditions for generating a global secondary index lookup plan, then execute S403; otherwise, execute S406.

[0147] In this application, the global secondary index table return plan can also be referred to as the XC table return plan of the global secondary index.

[0148] The process of determining the lightweight nature of FastPath in this application can be found in [reference needed]. Figure 5 To understand.

[0149] like Figure 5 As shown, the process of determining the lightweight nature of FastPath can include:

[0150] S501. The coordinating node determines whether the SQL statement is a query statement.

[0151] The coordinating node can determine whether an SQL statement is a query statement by its statement type. If the statement type is a query, then the SQL statement is a query statement; if the statement type is a non-query type, such as a delete statement, then the SQL statement is a non-query statement.

[0152] S502. The coordinating node determines whether the query statement contains GSI information.

[0153] The coordinating node can determine whether a query can be executed using a GSI by identifying the GSI identifier in the query statement.

[0154] S503. The coordinating node can also determine whether a query is a non-batch query.

[0155] The coordinating node can determine whether a query is a batch query or a non-batch query by the query conditions in the query statement.

[0156] In this application, the query conditions will indicate the number of rows of the target data to be queried or the value of the index key, etc., which can indicate conditions for non-batch queries.

[0157] S403. The coordinating node generates a global secondary index lookup plan based on the SQL statement.

[0158] S404. Coordinate the execution plan of the nodes.

[0159] The S404 can be the coordinating node executing a global secondary index lookup plan.

[0160] S405. The coordinating node obtains the query results.

[0161] S406. The coordinating node performs a distributed pushdown plan assessment on the SQL statement. If the conditions for generating a distributed pushdown plan are met, proceed to S407; otherwise, proceed to S408.

[0162] S407. The coordinating node generates a distributed pushdown plan based on the SQL statement.

[0163] S408. The coordinating node performs a distributed streaming plan evaluation on the SQL statement. If the conditions for generating a distributed streaming plan are met, proceed to S409; otherwise, proceed to S410.

[0164] S409. The coordinating node generates a distributed streaming plan based on the SQL statement.

[0165] S410. The coordinating node generates a distributed XC plan based on the SQL statement.

[0166] The coordinating node can execute S404 after S407, S409, and S410, but the plans executed after each step are different. After S407, a distributed pushdown plan is executed; after S409, a distributed streaming plan is executed; and after S410, a distributed XC plan is executed.

[0167] For an understanding of the process of generating a global secondary index lookup plan based on the SQL statement and executing the global secondary index lookup plan, please refer to the following scenario examples.

[0168] 1. Non-batch query scenario one;

[0169] SQL statement 1 is: SELECT * FROM t1 WHERE c2=1;

[0170] The coordinating node parses the SQL statement 1. The query "SELECT * FROM t1" indicates that this is a query statement aiming to retrieve target data from the base table t1. Here, t1 can be an indicator of the base table t1.

[0171] The base table t1 can be understood by referring to Table 2.

[0172] Table 2: Base Table t1

[0173] c1 c2 c3 4 1 1 3 5 4 7 2 6 6 8 8

[0174] The coordinating node can determine the data node where base table t1 is stored based on base table t1. If it is determined that base table t1 is distributed across multiple data nodes, then the query statement can use a global secondary index.

[0175] If column c2 in base table t1 is used as the index column, then the index table corresponding to the global secondary index idx_gsi_t1 of base table t1 can be understood by referring to Table 3 below.

[0176] Table 3: Global secondary indexes idx and gsi of base table t1; index table of t1

[0177] c2 xc_node_hash ctid xmin xmax 1 Hash(4) (1,1) 100 0 5 Hash(3) (1,3) 101 0 2 Hash(7) (2,1) 150 0 8 Hash(6) (2,2) 190 0

[0178] As shown in Table 3, xc_node_hash represents the hash value of the data node where each row of data in base table t1 is located; ctid represents the row pointer, which indicates the page where the data is located and the offset value on that page. xmin represents the insertion flag, xmax represents the deletion flag, and xmax = 0 indicates that the data has not been deleted.

[0179] In SQL statement 1, "WHERE c2=1" can be used as a query condition, indicating that the query should retrieve data where the index key is c2=1. Furthermore, the fact that c2=1 confirms that this is a non-batch query.

[0180] After the coordinating node parses SQL statement 1, it can determine that a global secondary index lookup plan needs to be generated for SQL statement 1. The global secondary index lookup plan for this scenario 1 can be:

[0181]

[0182] The lookup plan for this global secondary index is represented by 8 rows. This includes two remote queries. The first remote query indicates that the global secondary index `idx_gsi_t1` of base table `t1` is located on `datanodes1`. The second remote query indicates that the target data with `c2=1` is retrieved based on `xc_node_hash` and `ctid`.

[0183] After the coordinating node generates the global secondary index lookup plan for SQL statement 1, it executes the global secondary index lookup plan for SQL statement 1.

[0184] The execution process can be found in [reference]. Figure 6 To understand. For example Figure 6 As shown, the second row (3,5,4) and the third row (7,2,6) in base table t1 are stored on DN1, and the first row (4,1,1) and the fourth row (6,8,8) in base table t1 are stored on DN2. The first row (1,hash(4),(1,1),100,0) and the second row (5,hash(3),(1,3),101,0) in the global secondary index idx_gsi_t1 are stored on DN1, and the third row (2,hash(7),(2,1),150,0) and the fourth row (8,hash(6),(2,2),190,0) in the global secondary index idx_gsi_t1 are stored on DN2.

[0185] Since the query condition (also known as the predicate condition) of SQL statement 1 is c2=1, it can be determined that the data with c2=1 in idx_gsi_t1 is located in DN1. From the global secondary index table return plan listed in the above 8 lines, it can also be determined that the first Remote query needs to be sent to DN1, that is, Remote query1, which can contain c2=1.

[0186] Based on the Remote query 1 and c2 = 1, DN1 can determine that the xc_node_hash corresponding to c2 = 1 is hash(4) and the ctid is (1,1). Then DN1 can return hash(4) and (1,1) to the coordinating node. Among them, hash(4) can indicate the data node where the target data corresponding to c2 = 1 is located. In this scenario, based on hash(4), it is determined that the target data corresponding to c2 = 1 is stored in DN2.

[0187] The coordinating node sends Remote query2 to DN2, which contains ctid(1,1).

[0188] After receiving Remote query2, DN2 can determine that the corresponding data is stored on page1 based on ctid(1,1), and the offset on page1 is 1. For example, if the data corresponding to offset 1 on page1 is (4,1,1), then the target data can be determined to be (4,1,1).

[0189] DN2 returns the target data (4,1,1) to the coordinating node, and CN can further return the target data (4,1,1) to the client.

[0190] 2. Non-batch query scenario two;

[0191] SQL statement 2 is: SELECT * FROM t1 WHERE c2>1 limit 1;

[0192] The coordinating node parses SQL statement 2. The query "SELECT * FROM t1" indicates that SQL statement 2 is a query statement that seeks target data from base table t1. Here, t1 represents the information indicating the base table t1.

[0193] The base table t1 can be understood by referring to Table 2 above.

[0194] The coordinating node can determine the data node where base table t1 is stored based on base table t1. If it is determined that base table t1 is distributed across multiple data nodes, then the query statement can use a global secondary index.

[0195] The index table corresponding to the global secondary index idx_gsi_t1 corresponding to the base table t1 can be understood by referring to Table 3 above.

[0196] In SQL statement 2, "WHERE c2>1 LIMIT 1" can be a query condition (also called a predicate condition), indicating that the query should retrieve data where the index key is c2>1, and LIMIT 1 means retrieving the first data where c2>1. Furthermore, the presence of c2>1 LIMIT 1 confirms that this is a non-batch query.

[0197] After the coordinating node parses SQL statement 2, it can determine that a global secondary index lookup plan needs to be generated for SQL statement 2. The global secondary index lookup plan for this scenario 2 can be:

[0198]

[0199]

[0200] The table lookup plan for this global secondary index is represented by 10 rows. This includes two remote queries. The first remote query indicates that the global secondary index `idx_gsi_t1` of base table `t1` is located on `Alldatanodes`. The second remote query indicates that the target data with c2 > 1 is retrieved based on `xc_node_hash` and `ctid`.

[0201] After the coordinating node generates the global secondary index lookup plan for SQL statement 2, it executes the global secondary index lookup plan for SQL statement 2.

[0202] The execution process can be found in [reference]. Figure 7 To understand. For example Figure 7 As shown, the second row (3,5,4) and the third row (7,2,6) in base table t1 are stored on DN1, and the first row (4,1,1) and the fourth row (6,8,8) in base table t1 are stored on DN2. The first row (1,hash(4),(1,1),100,0) and the second row (5,hash(3),(1,3),101,0) in the global secondary index idx_gsi_t1 are stored on DN1, and the third row (2,hash(7),(2,1),150,0) and the fourth row (8,hash(6),(2,2),190,0) in the global secondary index idx_gsi_t1 are stored on DN2.

[0203] Since the query condition of SQL statement 2 is c2>1, it is necessary to query each data node that stores idx_gsi_t1. We can first query DN1. From the global secondary index table return plan listed in the above 10 rows, we can also determine that we need to send the first remote query to DN1, that is, remote query1. This remote query1 can contain c2>1limit1.

[0204] Based on the Remote query 1 and the limit 1, DN1 can determine that the first value of c2>1 is c2=5, the corresponding xc_node_hash is hash(3), and the ctid is (1,3). Since limit 1 restricts the data to the first c2>1, there is no need to send a query request to DN2. DN1 can then return hash(3) and (1,3) to the coordinating node. Among them, hash(3) can indicate the data node where the target data corresponding to c2=5 is distributed. In this scenario, hash(3) determines that the target data corresponding to c2=5 is stored in DN1.

[0205] The coordinating node sends Remote query2 to DN1, which contains ctid(1,3).

[0206] After receiving Remote query2, DN1 can determine that the corresponding data is stored on page1 based on ctid(1,3), and the offset on page1 is 3. For example, if the data corresponding to offset 3 on page1 is (3,5,4), then the target data can be determined to be (3,5,4).

[0207] DN1 returns the target data (3,5,4) to the coordinating node, and CN can further return the target data (3,5,4) to the client.

[0208] 3. Non-batch query scenario three;

[0209] SQL statement 3 is: SELECT * FROM t2 WHERE c2=400;

[0210] The coordinating node parses SQL statement 3. The query "SELECT * FROM t2" indicates that SQL statement 3 is a query statement that seeks target data from base table t2. Here, t2 represents the information indicating the base table t2.

[0211] The base table t2 can be understood by referring to Table 4.

[0212] Table 4: Base Table t2

[0213] c1 c2 c3 100 300 100 150 400 400 270 100 600 400 200 800

[0214] The coordinating node can determine the data nodes where base table t2 is stored based on base table t2. If it is determined that base table t2 is distributed across multiple data nodes, then the query statement can use a global secondary index.

[0215] If column c2 in base table t2 is used as the index column, the index table corresponding to the global secondary index idx_gsi_t2 of base table t2 can be understood by referring to Table 5 below.

[0216] Table 5: Global secondary indexes idx and gsi of base table t2; index table of t1

[0217]

[0218]

[0219] As shown in Table 5, xc_node_rang represents the range of data nodes containing each row of data in base table t2; ctid represents the row pointer, which indicates the page containing the data and its offset within that page; xmin represents the insertion flag; xmax represents the deletion flag; xmax = 0 indicates that the data has not been deleted.

[0220] In SQL statement 3, "WHERE c2=400" can be used as a query condition, indicating that the query should retrieve data where the index key is c2=400. Furthermore, the fact that c2=400 confirms that this is not a batch query.

[0221] After the coordinating node parses SQL statement 3, it can determine that a global secondary index lookup plan needs to be generated for SQL statement 3. The global secondary index lookup plan for scenario 3 can be:

[0222]

[0223] The lookup plan for this global secondary index is represented by 8 rows. This includes two remote queries. The first remote query indicates that the global secondary index `idx_gsi_t1` of base table `t2` is located on `datanodes2`. The second remote query indicates that the target data with `c2=400` is retrieved based on `xc_node_rang` and `ctid`.

[0224] After the coordinating node generates the global secondary index lookup plan for SQL statement 3, it executes the global secondary index lookup plan for SQL statement 3.

[0225] The execution process can be found in [reference]. Figure 8To understand. For example Figure 8 As shown, the first row (100,300,100) and the second row (150,400,400) of the base table t2 are stored on DN1, and the third row (270,100,600) and the fourth row (400,200,800) of the base table t2 are stored on DN2. The third row (100, rang(270), (2,1), 150,0) and the fourth row (200, rang(400), (2,2), 190,0) of the global secondary index idx_gsi_t2 are stored on DN1, and the first row (300, rang(100), (1,1), 100,0) and the second row (400, rang(150), (1,3), 101,0) of the global secondary index idx_gsi_t2 are stored on DN2.

[0226] Since the query condition (also known as the predicate condition) of SQL statement 3 is c2=400, it can be determined that the data with c2=400 in idx_gsi_t2 is located in DN2. From the global secondary index table return plan listed in the above 8 lines, it can also be determined that the first remote query, namely remote query1, needs to be sent to DN2. This remote query1 can contain c2=400.

[0227] Based on Remote query 1 and c2 = 400, DN2 can determine that the xc_node_rang corresponding to c2 = 400 is rang(150) and the ctid is (1,3). DN2 can then return rang(150) and (1,3) to the coordinating node. Here, 1 and 3 indicate the data nodes where the target data corresponding to c2 = 400 is distributed. In this scenario, based on rang(150), it is determined that the target data corresponding to c2 = 400 is stored in DN1.

[0228] The coordinating node sends Remote query2 to DN1, which contains ctid(1,3).

[0229] After receiving Remote query2, DN1 can determine that the corresponding data is stored on page1 based on ctid(1,3), and the offset on page1 is 3. For example, if the data corresponding to offset 3 on page1 is (150,400,400), then the target data can be determined to be (150,400,400).

[0230] DN1 returns the target data (150,400,400) to the coordinating node, and CN can further return the target data (150,400,400) to the client.

[0231] 4. Non-batch query scenario four;

[0232] SQL statement 4 is: SELECT * FROM t2 WHERE c2>200 LIMIT 1;

[0233] The coordinating node parses SQL statement 4. The query "SELECT * FROM t2" indicates that SQL statement 3 is a query statement that seeks target data from base table t2. Here, t2 represents the information indicating the base table t2.

[0234] The base table t2 can be understood by referring to Table 4 above.

[0235] The coordinating node can determine the data nodes where base table t2 is stored based on base table t2. If it is determined that base table t2 is distributed across multiple data nodes, then the query statement can use a global secondary index.

[0236] The index table corresponding to the global secondary index idx_gsi_t2 corresponding to the base table t2 can be understood by referring to Table 5 above.

[0237] In SQL statement 4, "WHERE c2>200limit 1" can be used as a query condition, indicating that the first data with the index key c2>200 should be retrieved. Furthermore, the fact that c2>200limit 1 indicates that this query is not a batch query.

[0238] After the coordinating node parses SQL statement 4, it can determine that a global secondary index lookup plan needs to be generated for SQL statement 4. The global secondary index lookup plan for scenario 3 can be:

[0239]

[0240] The table lookup plan for this global secondary index is represented by 10 rows. This includes two remote queries. The first remote query indicates that the global secondary index `idx_gsi_t1` of base table `t2` is located on `datanodes2`. The second remote query indicates that the target data with `c2>200` is retrieved based on `xc_node_rang` and `ctid`.

[0241] After the coordinating node generates the global secondary index lookup plan for SQL statement 4, it executes the global secondary index lookup plan for SQL statement 4.

[0242] The execution process can be found in [reference]. Figure 9 To understand. For example Figure 9As shown, the first row (100,300,100) and the second row (150,400,400) of the base table t2 are stored on DN1, and the third row (270,100,600) and the fourth row (400,200,800) of the base table t2 are stored on DN2. The third row (100, rang(270), (2,1), 150,0) and the fourth row (200, rang(400), (2,2), 190,0) of the global secondary index idx_gsi_t2 are stored on DN1, and the first row (300, rang(100), (1,1), 100,0) and the second row (400, rang(150), (1,3), 101,0) of the global secondary index idx_gsi_t2 are stored on DN2.

[0243] Since the query condition (also known as the predicate condition) of SQL statement 4 is c2>200, it can be determined that the data in idx_gsi_t2 with c2>200 is located in DN2. From the global secondary index table return plan listed in the above 10 rows, it can also be determined that the first remote query, namely remote query1, needs to be sent to DN2. This remote query1 can contain c2>200.

[0244] Based on the Remote query1 and the condition c2>200, DN2 can determine that the first data point of c2>200 is c2=300, and the xc_node_rang corresponding to c2=300 is rang(100), with ctid (1,1). DN2 can then return rang(100) and (1,1) to the coordinating node. Here, 1,1 indicates the data nodes where the target data corresponding to c2=300 is distributed. In this scenario, based on rang(100), it is determined that the target data corresponding to c2=300 is stored in DN1.

[0245] The coordinating node sends Remote query2 to DN1, which contains ctid(1,1).

[0246] After receiving Remote query2, DN1 can determine that the corresponding data is stored on page1 based on ctid(1,1), and the offset on page1 is 1. For example, if the data corresponding to offset 1 on page1 is (100,300,100), then the target data can be determined to be (100,300,100).

[0247] DN1 returns the target data (100,300,100) to the coordinating node, and CN can further return the target data ((100,300,100)) to the client.

[0248] The above scenarios illustrate the process of non-batch queries. As can be seen from the above scenarios, the solution provided in this application embodiment can quickly and accurately generate a global secondary index table return plan through fastpath, and under the distributed architecture, it supports fast table return for non-batch queries of the global secondary index through two layers of RemoteQuery.

[0249] In this embodiment of the application, a first table lookup plan can be generated for non-batch queries, such as a global secondary index table lookup plan. For batch queries, a second table lookup plan can be generated. The following describes the process in conjunction with... Figure 10A and Figure 10B Compare the plans for non-batch query scenarios and batch query scenarios.

[0250] like Figure 10A The image shows the first lookup plan for the non-batch query statement "explain verbose select * from t1 where c2 = 1;", which uses two layers of RemoteQuery to perform the lookup query.

[0251] like Figure 10B The image shows the second table lookup plan for the batch query statement "explain verbose select * from t1 where c2>1;", starting from... Figure 10B It can be seen that the second table return plan is a distributed streaming table return plan.

[0252] Thus, the solution provided in this application can generate a global second-level index lookup plan for non-batch queries. This plan can be executed with just two query requests. By executing the global second-level index lookup plan, it is unnecessary to start the streaming threads required for interaction between data nodes, saving time on initializing and starting these threads, thereby improving data query speed and the performance of the distributed database. For batch queries, other plans can be generated, further improving the speed of batch queries, thus enabling the distributed database to meet diverse query scenarios.

[0253] The above embodiments describe methods for data processing. The data processing apparatus 1100 provided in this application is described below with reference to the accompanying drawings.

[0254] like Figure 11 As shown, the data processing device 1100 provided in this application embodiment can be the coordination node described above. The data processing device 1100 includes:

[0255] Processing unit 1101 is configured to generate a first table lookup plan based on a query statement; wherein the query statement includes indication information of a first base table, and the first table lookup plan includes a first query request and a second query request; wherein the first query request is configured to instruct a first data node to scan the global secondary index corresponding to the first base table, and the first data node is the data node where the global secondary index corresponding to the first base table is located; the second query request is configured to instruct a second data node to query target data, and the second data node is the data node where the target data is located;

[0256] The processing unit is also used to execute the first table lookup plan to obtain the target data.

[0257] Optionally, the data processing apparatus further includes: a transceiver unit 1102, configured to receive a query statement, the query statement including indication information of a first base table; and to return the target data after the target data is retrieved.

[0258] The solution provided in this application embodiment, in a table lookup scenario based on a global secondary index, allows the data processing device to execute a table lookup plan with just two query requests. This eliminates the need to start the streaming thread required for interaction between data nodes, saving time on initializing and starting the streaming thread, thus improving the speed of data querying and the performance of the distributed database.

[0259] Optionally, the processing unit 1101 is specifically used to generate a first table lookup plan if the query statement is determined to be a non-batch query based on the query conditions when the query statement also includes query conditions.

[0260] Optionally, the transceiver unit 1102 is further configured to send a first query request to the first data node according to the first table lookup plan; and receive the logical value of the data node where the target data is located and the row information of the target data, wherein the logical value is used to determine that the target data is located in the second data node.

[0261] The transceiver unit 1102, according to the first table return plan, sends a second query request to the second data node, the second query request containing row information of the target data; and receives the target data from the second data node.

[0262] Optionally, the processing unit 1101 is further configured to generate a second table lookup plan for the batch query if the query condition indicates that the query statement is a batch query.

[0263] The above-mentioned details regarding the transceiver unit 1102 and the processing unit 1101 of the data processing device 1100 can be understood by referring to the relevant content of the coordination node in the aforementioned method embodiment section, and will not be repeated here.

[0264] Figure 12The diagram shown illustrates a possible logical structure of the data processing device 1200 described in the embodiments of this application. The data processing device 1200 includes a processor 1201, a communication interface 1202, a memory 1203, and a bus 1204. The processor 1201, communication interface 1202, and memory 1203 are interconnected via the bus 1204. In the embodiments of this application, the processor 1201 is used to control and manage the operation of the data processing device 1200; for example, the processor 1201 is used to execute... Figures 2 to 9 The steps related to processing include, for example, the processing steps performed by the read node or target node, such as S302 and S306. The communication interface 1202 is used to support communication between the data processing device 1200 and other data processing units. For example, the communication interface 1202 can perform the receiving or sending steps described in the above method embodiments. Figure 11 The transceiver unit 1102 in the database system 120 has the following functions. The memory 1203 is used to store the program code and data of the database system 120.

[0265] The processor 1201 can be a central processing unit, a general-purpose processor, a digital signal processor, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. The processor can also be a combination that implements computational functions, such as a combination of one or more microprocessors, a combination of a digital signal processor and a microprocessor, etc. The bus 1204 can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 11 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0266] See Figure 13 This application also provides a distributed database system, including: a hardware layer 1307 and a virtual machine monitor (VMM) 1301 running on the hardware layer 1307, and multiple virtual machines 1302. A virtual machine can serve as a read node or a write node in the distributed database system. Optionally, a virtual machine can also be designated as a coordinating node.

[0267] Specifically, Virtual Machine 1302 is a virtual computer simulated on public hardware resources using virtual machine software. Operating systems and applications can be installed on the virtual machine, and it can also access network resources. For applications running in the virtual machine, it's as if they are working on a real computer.

[0268] Hardware Layer 1307: The hardware platform running in the virtualized environment, which can be abstracted from the hardware resources of one or more physical hosts. The hardware layer may include various hardware components, such as a processor 1304 (e.g., CPU) and memory 1305, as well as a network interface card 1303 (e.g., RDMA network card), high-speed / low-speed input / output (I / O) devices, and other devices with specific processing functions.

[0269] Virtual machine 1302, based on the VMM and the hardware resources provided by hardware layer 1307, runs executable programs to achieve the above. Figures 2 to 9 In the relevant embodiments, some or all of the functions of the coordination node are described. For the sake of brevity, these will not be elaborated further here.

[0270] Furthermore, the distributed database system may also include a host: acting as a management layer to manage and allocate hardware resources; presenting a virtual hardware platform for virtual machines; and implementing virtual machine scheduling and isolation. The host may be a Virtual Machine Monitor (VMM); or it may be a combination of a VMM and a privileged virtual machine. The virtual hardware platform provides various hardware resources to the virtual machines running on it, such as virtual processors (e.g., VCPUs), virtual memory, virtual disks, virtual network cards, etc. The virtual disk may correspond to a file or a logical block device on the host. Virtual machines run on the virtual hardware platform prepared for them by the host, and one or more virtual machines run on the host. The VCPU of virtual machine 1302 executes an executable program stored in its corresponding virtual memory to implement or perform the method steps described in the above-described method embodiments of the present invention. For example, to implement the above... Figures 2 to 9 In related embodiments, some or all of the functions of the coordination node are used.

[0271] In another embodiment of this application, a computer-readable storage medium is also provided, which stores computer-executable instructions. When at least one processor of the device executes the computer-executable instructions, the device performs the aforementioned... Figures 2 to 9 The data processing methods described in some embodiments.

[0272] In another embodiment of this application, a computer program product is also provided, comprising computer-executable instructions stored in a computer-readable storage medium; at least one processor of the device can read the computer-executable instructions from the computer-readable storage medium, and the at least one processor executes the computer-executable instructions to cause the device to perform the above-described actions. Figures 2 to 9 The data processing methods described in some embodiments.

[0273] In another embodiment of this application, a chip system is also provided, the chip system including a processor for supporting the coordination node to implement the above. Figures 2 to 9 The data processing method described in some embodiments. In one possible design, the chip system may further include a memory for coordinating necessary program instructions and data for the nodes. The chip system may consist of chips or may include chips and other discrete devices.

[0274] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the embodiments of this application.

[0275] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0276] In the embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual couplings, direct couplings, or communication connections may be through some interfaces; indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms.

[0277] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0278] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0279] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of this application, essentially, or the parts that contribute to the prior art, or parts of the technical solutions, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

Claims

1. A method of data processing, characterized by, The method comprises: generating a first return table plan according to a query statement, wherein the query statement comprises indication information of a first base table, and the first return table plan comprises a first query request and a second query request; the first query request is used to instruct a first data node to scan a global secondary index corresponding to the first base table, and the first data node is a data node where the global secondary index corresponding to the first base table is located; the second query request is used to instruct a second data node to query target data, and the second data node is a data node where the target data is located; executing the first return table plan to obtain the target data.

2. The method of claim 1, wherein, The method further comprises: receiving the query statement; returning the target data.

3. The method according to claim 1 or 2, characterized in that, The query statement further comprises a query condition; and the generating the first return table plan according to the query statement comprises: if the query condition indicates that the query statement is a non-batch query, generating the first return table plan.

4. The method according to any one of claims 1 to 3, characterized in that, The executing the first return table plan to obtain the target data comprises: sending, according to the first return table plan, the first query request to the first data node; receiving a logical value of a data node where the target data is located and row information of the target data, the logical value being used to determine that the target data is located in the second data node; sending, according to the first return table plan, the second query request to the second data node, the second query request containing the row information of the target data; receiving the target data from the second data node.

5. The method of claim 2, wherein, The method further comprises: if the query condition indicates that the query statement is a batch query, generating a second return table plan for batch query.

6. A data processing apparatus, characterized by comprise: a processing unit, configured to generate a first return table plan according to a query statement, wherein the query statement comprises indication information of a first base table, and the first return table plan comprises a first query request and a second query request; the first query request is used to instruct a first data node to scan a global secondary index corresponding to the first base table, and the first data node is a data node where the global secondary index corresponding to the first base table is located; the second query request is used to instruct a second data node to query target data, and the second data node is a data node where the target data is located; the processing unit is further configured to execute the first return table plan to obtain the target data.

7. The apparatus of claim 6, wherein, The apparatus further comprises: a transceiving unit, configured to receive the query statement and return the target data.

8. The apparatus of claim 6 or 7, wherein: the processing unit is specifically configured to, when the query statement further comprises a query condition, if the query condition indicates that the query statement is a non-batch query, generate the first return table plan.

9. The device of any of claims 6-8, wherein, The apparatus further comprises: the transceiving unit is configured to: send, according to the first return table plan, the first query request to the first data node; receive a logical value of a data node where the target data is located and row information of the target data, the logical value being used to determine that the target data is located in the second data node; According to the first table scan plan, a second query request is sent to the second data node, the second query request containing row information of the target data; The target data is received from the second data node.

10. The apparatus of claim 8, wherein, The processing unit is further configured to generate a second table scan plan for batch query if the query condition indicates that the query statement is a batch query.

11. A data processing apparatus, characterized by A computer readable storage medium storing a computer program; The processing unit is further configured to generate a second table scan plan for batch query if the query condition indicates that the query statement is a batch query.

12. A computer readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processing unit to implement the method of any one of claims 1-5.

13. A computer program product, characterised in that, The computer program product comprises computer program code which, when executed on a computer device, causes the computer device to perform the method of any one of claims 1-5.

14. A distributed database, characterized by The computer program product comprises computer program code which, when executed on a computer device, causes the computer device to perform the method of any one of claims 1-5. The computer program product comprises computer program code which, when executed on a computer device, causes the computer device to perform the method of any one of claims 1-5. The computer program product comprises computer program code which, when executed on a computer device, causes the computer device to perform the method of any one of claims 1-5.

Images