Data query method, apparatus, device, system, medium, and product

Abstract

The present disclosure provides a data query method, device, equipment, system, medium and product, relates to the technical field of computers, and can improve the efficiency of data query. The data query method comprises the following steps: receiving a first query sub-request in a query request; the first query sub-request comprises cursor position information of a second query sub-request in the query request last time and a first shard identifier; the first shard identifier is used for identifying a storage shard that has been queried by the second query sub-request; according to the cursor position information of the second query sub-request and the first shard identifier, a broadcast request is broadcasted to each storage node in a distributed database; a response result sent by each storage node is received, and a query sub-result of the first query sub-request is generated according to the response result sent by each storage node; and the query sub-result of the first query sub-request is used for generating a query result of the query request.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, specifically to a data query method, apparatus, device, system, medium, and product. Background Technology

[0002] Distributed relational database management systems (NewSQL) typically support local secondary indexes and provide query interfaces. Users can use these interfaces to filter data based on query conditions (similar to the WHERE clause in SQL). However, when there is a large amount of data that meets the query conditions, a single request may not be able to retrieve all the data. Therefore, multiple requests are usually sent to the database to retrieve the data that meets the query conditions in stages.

[0003] When retrieving data that meets the query criteria in multiple steps, it is usually necessary to determine the starting position and the amount of data to be retrieved in this request. A common technique is to use the method of "offset M, limit number of rows N (OFFSET M LIMIT N)". That is, first find the Mth data that meets the query criteria (i.e., the starting position of this query) using OFFSET M, and then retrieve N data starting from the Mth data (i.e., the amount of data to be retrieved in this query).

[0004] As shown above, when querying data, general technology requires first retrieving M+N data entries, then discarding the first M entries, and finally returning the remaining N entries to the user. As the number of queries increases, M becomes increasingly larger, leading to a decrease in data query efficiency. Summary of the Invention

[0005] This disclosure provides a data query method, apparatus, device, system, medium, and product that can improve the efficiency of data query.

[0006] In a first aspect, this disclosure provides a data query method, comprising: receiving a first query sub-request from a query request; the first query sub-request including cursor position information of a previous second query sub-request in the query request and a first fragment identifier; the first fragment identifier being used to identify the storage fragment queried by the second query sub-request; broadcasting a broadcast request to each storage node in a distributed database based on the cursor position information of the second query sub-request and the first fragment identifier; the broadcast request being used to request each storage node to query data based on the cursor position information of the second query sub-request and the first fragment identifier; receiving response results sent by each storage node, and generating a query sub-result of the first query sub-request based on the response results sent by each storage node; the query sub-result of the first query sub-request being used to generate the query result of the query request.

[0007] Secondly, this disclosure also provides a data query method, comprising: receiving a broadcast request; the broadcast request being a query node's response to a first query sub-request in a query request, broadcast based on the cursor position information of the previous second query sub-request in the query request and a first fragment identifier; the first fragment identifier being used to identify the storage fragment queried by the second query sub-request; determining the response result of the first query sub-request based on the cursor position information of the second query sub-request and the first fragment identifier; sending the response result to the query node, so that the query node generates a query sub-result of the first query sub-request based on the response results sent by each storage node in the distributed database; the query sub-result of the first query sub-request being used to generate the query result of the query request.

[0008] Thirdly, this disclosure also provides a data query apparatus, comprising: a communication unit and a processing unit; the communication unit is configured to receive a first query sub-request in a query request; the first query sub-request includes cursor position information of the previous second query sub-request in the query request and a first fragment identifier; the first fragment identifier is used to identify the storage fragment queried by the second query sub-request; the communication unit is further configured to broadcast a broadcast request to each storage node in the distributed database based on the cursor position information of the second query sub-request and the first fragment identifier; the broadcast request is used to request each storage node to query data based on the cursor position information of the second query sub-request and the first fragment identifier; the communication unit is further configured to receive response results sent by each storage node; the processing unit is configured to generate a query sub-result of the first query sub-request based on the response results sent by each storage node; the query sub-result of the first query sub-request is used to generate the query result of the query request.

[0009] Fourthly, this disclosure also provides a data query apparatus, comprising: a communication unit and a processing unit; the communication unit is configured to receive a broadcast request; the broadcast request is a response by a query node to a first query sub-request in a query request, broadcast based on the cursor position information of the previous second query sub-request in the query request and a first fragment identifier; the first fragment identifier is used to identify the storage fragment queried by the second query sub-request; the processing unit is configured to determine the response result of the first query sub-request based on the cursor position information of the second query sub-request and the first fragment identifier; the communication unit is further configured to send the response result to the query node, so that the query node generates a query sub-result of the first query sub-request based on the response results sent by each storage node in the distributed database; the query sub-result of the first query sub-request is used to generate the query result of the query request.

[0010] Fifthly, this disclosure also provides an electronic device comprising: a processor and a memory configured to store processor-executable instructions; wherein the processor is configured to execute instructions to implement the data query method of the first aspect or the data query method of the second aspect described above.

[0011] Sixthly, this disclosure also provides a non-volatile storage medium storing a computer program, which, when read and executed, implements the data query method of the first aspect or the data query method of the second aspect described above.

[0012] In a seventh aspect, this disclosure also provides a computer program product, including instructions that, when executed on an electronic device, cause the electronic device to implement the data query method of the first aspect or the data query method of the second aspect described above.

[0013] Eighthly, this disclosure also provides a data query system, including a query node and a storage node; the query node is used to implement the data query method of the first aspect described above; the storage node is used to implement the data query method of the second aspect described above.

[0014] In the data query method and data query mechanism provided in this disclosure, the query node can receive a first query sub-request from the query request. Since the first query sub-request includes the cursor position information of the previous second query sub-request and a first shard identifier (used to identify the storage shard queried by the second query sub-request), the query node can broadcast a broadcast request to each storage node in the distributed database based on the cursor position information and the first shard identifier of the second query sub-request. The broadcast request is used to request each storage node to query data based on the cursor position information and the first shard identifier of the second query sub-request. Subsequently, the query node can receive the response results sent by each storage node and generate a query sub-result of the first query sub-request based on the response results sent by each storage node, so as to facilitate the subsequent generation of the query result of the query request.

[0015] As can be seen from the above, after receiving the first query sub-request, the query node does not need to query the data that has already been queried before, but can directly query the data required for this query (i.e. the query sub-result of the first query sub-request) based on the cursor position information of the second query sub-request and the first fragment identifier, since the first query sub-request includes the cursor position information of the previous second query sub-request in the query request. This improves the efficiency of data query.

[0016] Secondly, in a distributed database, since each storage node may store the data required by the first query sub-request, the query node can broadcast the request to each storage node in the distributed database based on the cursor position information of the second query sub-request and the first shard identifier, thereby obtaining the data (i.e., the response result) required by the first query sub-request from each storage node. Subsequently, the query node can generate the query sub-result of the first query sub-request based on the response results sent by each storage node, ensuring the integrity and accuracy of the query sub-result obtained from each storage node of the distributed database.

[0017] Furthermore, since this application does not require users to set the offset, it can automatically perform data queries in scenarios where users are unaware of the process (i.e., scenarios where there is no need to visualize the internal implementation of data queries), reducing user operations and enriching the user experience. Attached Figure Description

[0018] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0019] Figure 1 This is a schematic diagram of a data query system provided in an embodiment of the present disclosure;

[0020] Figure 2 A flowchart illustrating a data query method provided in an embodiment of this disclosure;

[0021] Figure 3 A flowchart illustrating yet another data query method provided in this disclosure embodiment;

[0022] Figure 4 A flowchart illustrating yet another data query method provided in this disclosure embodiment;

[0023] Figure 5 Example diagram of another data query method provided in the embodiments of this disclosure;

[0024] Figure 6 A flowchart illustrating yet another data query method provided in this disclosure embodiment;

[0025] Figure 7 A flowchart illustrating yet another data query method provided in this disclosure embodiment;

[0026] Figure 8 A flowchart illustrating yet another data query method provided in this disclosure embodiment;

[0027] Figure 9 A flowchart illustrating yet another data query method provided in this disclosure embodiment;

[0028] Figure 10 A flowchart illustrating yet another data query method provided in this disclosure embodiment;

[0029] Figure 11 This is a schematic diagram of the structure of a data query device provided in an embodiment of the present disclosure;

[0030] Figure 12 A schematic diagram of the structure of another data query device provided in this disclosure embodiment;

[0031] Figure 13 This is a schematic block diagram of an electronic device provided in an embodiment of the present disclosure. Detailed Implementation

[0032] The technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0033] It is understood that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0034] It should also be understood that the term "comprising" indicates the presence of the described feature, whole, step, operation, element and / or component, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements and / or components.

[0035] As described in the background section, general techniques for querying data require first retrieving M+N data entries, then discarding the first M entries, and finally returning the remaining N entries to the user. As the number of queries increases, M becomes increasingly large, leading to decreasing efficiency in data retrieval.

[0036] To address the aforementioned issues, this application provides a data query method, comprising: a query node receiving a first query sub-request from a query request. Since the first query sub-request includes cursor position information of the previous second query sub-request and a first shard identifier (used to identify the storage shard queried by the second query sub-request), the query node can broadcast a broadcast request to each storage node in the distributed database based on the cursor position information and the first shard identifier of the second query sub-request. The broadcast request requests each storage node to query data based on the cursor position information and the first shard identifier of the second query sub-request. Subsequently, the query node can receive response results from each storage node and generate a query sub-result of the first query sub-request based on the response results, facilitating the subsequent generation of the query result of the query request.

[0037] As can be seen from the above, after receiving the first query sub-request, the query node does not need to query the data that has already been queried before, but can directly query the data required for this query (i.e. the query sub-result of the first query sub-request) based on the cursor position information of the second query sub-request and the first fragment identifier, since the first query sub-request includes the cursor position information of the previous second query sub-request in the query request. This improves the efficiency of data query.

[0038] Secondly, in a distributed database, since each storage node may store the data required by the first query sub-request, the query node can broadcast the request to each storage node in the distributed database based on the cursor position information of the second query sub-request and the first shard identifier, thereby obtaining the data (i.e., the response result) required by the first query sub-request from each storage node. Subsequently, the query node can generate the query sub-result of the first query sub-request based on the response results sent by each storage node, ensuring the integrity and accuracy of the query sub-result obtained from each storage node of the distributed database.

[0039] Furthermore, since this application does not require users to set the offset, it can automatically perform data queries in scenarios where users are unaware of the process (i.e., scenarios where there is no need to visualize the internal implementation of data queries), reducing user operations and enriching the user experience.

[0040] The data query method provided in this application embodiment can be applied to... Figure 1 The data query system shown. Figure 1 This is a schematic diagram of the structural scenario of a data query system provided in an embodiment of the present disclosure. The data query system may include: at least one request terminal 101 ( Figure 1 (Taking one request client 101 as an example for illustration), query node 102 and distributed database 103.

[0041] The physical device of the requesting end 101 can be a terminal, a server, or other types of devices, and this application embodiment does not limit this.

[0042] In this embodiment of the application, the requesting end 101 can receive the data query request input by the user and generate a query request based on the data query request to request the data corresponding to the data query request from the distributed database 103.

[0043] In one possible implementation, the requesting end 101 can send a query request to the querying node 102 via a software development kit (SDK).

[0044] The SDK can be pre-configured with the amount of data requested for each query. If the query result does not retrieve complete data in a single query, the SDK can send multiple query requests to query node 102 (e.g., the first query sub-request and the second query sub-request in this embodiment).

[0045] The query node 102 can be a management node deployed on the distributed database 103, or it can be a proxy node deployed independently of the distributed database 103. This application embodiment does not limit this.

[0046] The distributed database 103 (also known as Storage) deploys multiple storage nodes (also known as Storaged or Node), for example... Figure 1 Storage node 1, storage node 2 and storage node 3 are shown.

[0047] Each storage node comprises multiple shards (also known as partitions, etc.), for example Figure 1 The diagram shows P1, P4, and P7 in storage node 1, P2, P5, and P8 in storage node 2, and P3, P6, and P9 in storage node 3.

[0048] Each shard stores multiple rows of data (List) using a key-value pair. <row>A data row consists of values ​​from multiple columns, each representing a specific attribute or field. For example, in a table storing student information, a row might contain values ​​for fields such as student ID, name, age, and gender. The student ID, name, age, and gender are the indexes to these values.

[0049] In one possible implementation, the distributed database 103 can be a database that supports local secondary indexes, meaning that the indexes and the data in each storage node are co-distributed, i.e., the index that satisfies the query request may exist on each storage node. Therefore, in this embodiment of the application, when obtaining query results, the query node needs to broadcast the query request to each storage node.

[0050] Understandable Figure 1 The data query system provided in this application embodiment is only one example in this application embodiment. The data query system and corresponding scenarios described in this application embodiment are for the purpose of more clearly illustrating the technical solutions provided in this application embodiment and do not constitute a limitation on the technical solutions provided in this application embodiment. As those skilled in the art will know, with the emergence of new business scenarios, the technical solutions provided in this application embodiment are also applicable to similar technical problems.

[0051] Figure 2 A flowchart illustrating a data query method provided in an embodiment of this application is shown. Figure 2 As shown, the data query method includes: S201-S204.

[0052] S201, The query node receives the first query sub-request in the query request.

[0053] Specifically, when the query request sent by the requesting end requires a large amount of data, in order to ensure that the requesting end can quickly obtain the required data, the SDK can send multiple query sub-requests, retrieve the required data in batches, and return it to the requesting end in the form of a data stream. In this case, the query node can retrieve multiple query sub-requests sequentially.

[0054] Since the query node can determine which data in the database was retrieved after each query sub-result, it can return information about the breakpoint data retrieved in each sub-result to the requesting end when returning the query sub-results. Thus, when the requesting end sends the first query sub-result (the current query sub-result) to the query node, it can send information about the breakpoint data of the second query sub-result (the previous query sub-result), including the cursor position information and the first fragment identifier.

[0055] In this scenario, the query node can receive the first query sub-request from the query request. This first query sub-request includes the cursor position information of the previous second query sub-request in the query request and a first shard identifier. The first shard identifier identifies the storage shard queried by the second query sub-request.

[0056] In some embodiments, the cursor position information includes: a third fragment identifier and the primary key of the data row of the breakpoint data; the third fragment identifier is used to identify the storage fragment to which the breakpoint data belongs.

[0057] For example, in combination Figure 1 Assuming the query result of the second query sub-request is that the first 40 data items were obtained from P4 of storage node 4, and all data items in P1 of storage node 1, P2 of storage node 2, and P3 of storage node 3 have been queried, then the cursor position information includes: the shard identifier of P4 (i.e., the third shard identifier), the primary key of the data row of the 40th data item in P4 (i.e., the primary key of the data row of the breakpoint data), and the first shard identifier includes: the shard identifier of P2, the shard identifier of P3, and the shard identifier of P4.

[0058] It should be noted that when the requesting end sends the first query sub-request to the query node, the first query sub-request does not include the cursor position information of the previous query sub-request and the fragment identifier of the queried storage fragment, or the content of the relevant fields of the cursor position information of the previous query sub-request and the fragment identifier of the queried storage fragment is empty.

[0059] In addition, each query sub-request received by the query node also includes the amount of data required for that query sub-request and index information.

[0060] The index information is used to instruct storage nodes to query the data corresponding to that index, which is the descriptive information of the data required by the requesting end. For example, in a distributed database storing student information, the database may contain values ​​for fields such as student ID, name, age, and gender. When the requesting end wants to retrieve the student IDs of all students, each query sub-request carries the index information for the "student ID" index.

[0061] The amount of data required by this query sub-request indicates the number of data items the storage node needs to retrieve corresponding to the index information. Using the example above, assuming the distributed database stores information about 50,000 students, the SDK can send 100 query sub-requests, each retrieving the student IDs of 500 students.

[0062] In one possible implementation, the requesting client can send query sub-requests (e.g., a first query sub-request and a second query sub-request) to the query node via the SDK. The SDK can provide a dedicated scanning interface, allowing the SDK to maintain cursor position information internally (also known as a cursor), or the user can maintain the cursor and pass it into the SDK. In this way, when the requesting client sends a first query sub-request to the query node via the SDK, the SDK will add the cursor position information of the previous second query sub-request, which it maintains internally, to the first query sub-request.

[0063] In this way, after passing the cursor into the SDK, the requesting end can use the iterator provided by the SDK to continuously obtain the next query sub-request, which improves the automation and efficiency of data query.

[0064] It should be understood that when the cursor is maintained by the user and passed into the SDK, the user can change the cursor position information as needed (e.g., change the starting position of the data query in the first query sub-request), which improves the flexibility of data query.

[0065] Of course, whether the cursor position information is maintained internally by the SDK or maintained by the user and passed to the SDK, the data structure for the requesting end to send the first query sub-request to the query node through the SDK is as follows:

[0066]

[0067] Optionally, the data structure of the first query sub-request described above can also be referred to as the outer cursor (Scan Cursor) of the first query sub-request.

[0068] S202. The query node broadcasts the broadcast request to each storage node in the distributed database based on the cursor position information of the second query sub-request and the first shard identifier.

[0069] The broadcast request is used to request each storage node to query data based on the cursor position information of the second query sub-request and the first fragment identifier.

[0070] Specifically, since the embodiments of this application can be applied to distributed databases, and distributed databases can be databases that support local secondary indexes, meaning that the indexes and the data in each storage node are uniformly distributed, the index that satisfies the first query sub-request may exist on each storage node. Therefore, after receiving each query sub-request, the query node needs to broadcast the request to each storage node in the distributed database to request data to be queried from each storage node.

[0071] Since the broadcast request is broadcast to all storage nodes and instructs them to query data, the querying node needs to construct the broadcast request based on the cursor position information of the second query sub-request and the first shard identifier. This broadcast request may include: the table ID to be queried by the first query sub-request, the primary key of the data row in the breakpoint data, the amount of data required by the first query sub-request, index information, and a list of shard IDs for the storage shards to be queried on each storage node.

[0072] Among them, the primary key of the breakpoint data row is extracted from the cursor position information of the second query sub-request, the amount of data required by the first query sub-request and the index information are extracted from the information carried by the first query sub-request, and the list of shard IDs of the storage shards to be queried on each storage node is determined according to the first shard identifier.

[0073] It should be noted that since breakpoint data typically belongs to one or a subset of storage nodes' shards, and broadcast requests are broadcast to all storage nodes, the primary key field of the breakpoint data's data row will have a value for the storage node to which the breakpoint data belongs. Conversely, for storage nodes to which the breakpoint data does not belong, the primary key field of the breakpoint data's data row will be empty.

[0074] Secondly, for the storage node to which the breakpoint data belongs, the query needs to start from the breakpoint data. Therefore, in this broadcast request, for the storage node to which the breakpoint data belongs, the first ID in the list of fragment IDs of the storage fragment to be queried is the fragment ID of the storage fragment to which the breakpoint data belongs. This ensures that the storage node to which the breakpoint data belongs can start the query from the breakpoint data row of the breakpoint storage fragment to which the breakpoint data belongs. Correspondingly, for storage nodes that do not belong to which the breakpoint data belongs, the list of fragment IDs of the storage fragment to be queried can be sorted according to a preset order (e.g., the size order of the fragment identifiers).

[0075] After the broadcast request is constructed, the query node can broadcast the broadcast request to each storage node in the distributed database.

[0076] For example, the broadcast request data structure constructed and broadcast by the query node is as follows:

[0077]

[0078] Optionally, the primary key of the data row in the breakpoint data of the above broadcast request can also be called an internal cursor.

[0079] S203. The query node receives the response results sent by each storage node.

[0080] Specifically, after the query node broadcasts a request to each storage node in the distributed database, each storage node can respond to the broadcast request by querying its stored data for data that satisfies the first query sub-request (i.e., data corresponding to the index information in the first query sub-request) and returning a response result to the query node. Correspondingly, the query node can receive the response results sent by each storage node.

[0081] S204. The query node generates the query sub-result of the first query sub-request based on the response results sent by each storage node.

[0082] The query sub-result of the first query sub-request is used to generate the query result of the query request.

[0083] As shown above, since the index satisfying the first query sub-request may exist on all storage nodes, each storage node can send a response containing the amount of data required by the first query sub-request to the query node. In this case, the query node may receive more data than required by the first query sub-request.

[0084] For example, suppose the distributed database has 5 storage nodes. In the broadcast request broadcast by the query node to each storage node, if the first query sub-request requires 100 data items, then the query node may receive a maximum of 500 data items sent by the 5 storage nodes.

[0085] In this scenario, since the query node only needs to return the amount of data required for the first query sub-request (e.g., 100 data points in the example above), it needs to aggregate and trim the data in the responses sent by each storage node to obtain the amount of data required for the first query sub-request. Subsequently, the query node can generate the query sub-results for the first query sub-request based on the aggregated and trimmed data.

[0086] As can be seen from the above, after receiving the first query sub-request, the query node does not need to query the data that has already been queried before, but can directly query the data required for this query (i.e. the query sub-result of the first query sub-request) based on the cursor position information of the second query sub-request and the first fragment identifier, since the first query sub-request includes the cursor position information of the previous second query sub-request in the query request. This improves the efficiency of data query.

[0087] Secondly, in a distributed database, since each storage node may store the data required by the first query sub-request, the query node can broadcast the request to each storage node in the distributed database based on the cursor position information of the second query sub-request and the first shard identifier, thereby obtaining the data (i.e., the response result) required by the first query sub-request from each storage node. Subsequently, the query node can generate the query sub-result of the first query sub-request based on the response results sent by each storage node, ensuring the integrity and accuracy of the query sub-result obtained from each storage node of the distributed database.

[0088] In some embodiments, after the query node generates the query sub-results of the first query sub-request, it can also generate the cursor position information of the first query sub-request and send the query sub-results and cursor position information of the first query sub-request to the requesting end, so that the requesting end can aggregate the query sub-results of each sub-request to determine the query result of the query request. In this case, combined with Figure 2 ,like Figure 3 As shown, the data query method provided in this application embodiment also includes: S301-S302.

[0089] S301. The query node generates the cursor position information of the first query sub-request based on the query sub-result of the first query sub-request.

[0090] Specifically, in order to quickly locate the breakpoint of the first query sub-request when the next query sub-request is received, and then obtain the query sub-result of the next query sub-request, the query node can generate the cursor position information of the first query sub-request based on the query sub-result of the first query sub-request after determining the query sub-result of the first query sub-request.

[0091] In some embodiments, when a query node generates cursor position information for a first query sub-request based on the query sub-result of the first query sub-request, it can determine the position information of the breakpoint data of the query sub-result of the first query sub-request as the cursor position information for the first query sub-request.

[0092] The cursor position information can include the primary key of the data row of the breakpoint data in the query sub-result of the first query sub-request, and the storage shard to which the breakpoint data belongs. In this way, the cursor position information of the first query sub-request allows for quick location of the breakpoint data when the next query sub-request arrives, improving the efficiency of data retrieval.

[0093] For example, assuming the breakpoint data of the query sub-result of the first query sub-request is the 40th row of data of P1 in storage node 1, the query node can encode the primary key of the data row of the 40th row and the fragment identifier of P1 to obtain the cursor position information of the first query sub-request.

[0094] Optionally, the query node can also store the fragment identifiers of the storage fragments that have been queried but whose queries have been completed in the query sub-results of the first query sub-request, i.e., update the finished_parts field. Alternatively, the query node can add the fragment identifiers of the storage fragments that have been queried but whose queries have been completed to the cursor position information of the first query sub-request. Or, the query node can send the fragment identifiers of the storage fragments that have been queried but whose queries have been completed to the requesting end separately. Subsequently, when the requesting end sends the next query sub-request to the query node, it can add the cursor position information of the first query sub-request to the next query sub-request. In this way, when the query node retrieves the query sub-results of the next query sub-request, it can exclude the storage fragments that have been queried but whose queries have been completed, avoiding duplicate queries on these storage fragments.

[0095] S302. The query node sends the query sub-result of the first query sub-request and the cursor position information of the first query sub-request to the requesting end corresponding to the query request, so that the requesting end can determine the query result of the query request based on the cursor position information and query sub-results corresponding to each of the multiple query sub-requests, including the first query sub-request and the second query sub-request.

[0096] Specifically, after determining the query sub-result and cursor position information of the first query sub-request, the query node can send the query sub-result and cursor position information of the first query sub-request to the requesting end corresponding to the query request. Then, the requesting end can send the next query sub-request carrying the cursor position information of the first query sub-request to the query node. This process is repeated until the requesting end can obtain the query sub-results of all query sub-requests (i.e., the query sub-results corresponding to each of the multiple query sub-requests, including the first and second query sub-requests), thereby determining the query result of the query request.

[0097] Subsequently, the requesting end can output each query sub-result sequentially in the form of a data stream according to the order in which the query sub-results were received; or it can aggregate all the query sub-results after obtaining them and output the aggregated complete query result. This application embodiment does not limit this.

[0098] As can be seen from the above, after the query node generates the query sub-result of the first query sub-request, it can also generate the cursor position information of the first query sub-request, so that the requesting end can perform the next query sub-request based on the cursor position information of the first query sub-request and repeat the above steps to obtain the complete query result corresponding to the query request.

[0099] In some embodiments, when a query node aggregates and prunes data from the response results sent by each storage node, it can do so based on the order in which the response results belong to the storage shards. In this case, combined with Figure 3 ,like Figure 4 As shown, in S204 above, the method for the query node to generate the query sub-result of the first query sub-request based on the response results sent by each storage node specifically includes: S401-S402.

[0100] S401. The query node determines the order of the storage fragments to which the response results sent by each storage node belong, based on the second fragment identifier sent by each storage node.

[0101] The second fragment identifier is the storage fragment included in the response result, used to identify the to which the response result belongs.

[0102] Specifically, in a distributed database, each storage node can pre-set a unique identifier for each storage shard during the sharding process.

[0103] Optionally, the unique identifier mentioned above can be a fixed-length hash value calculated using a hash function based on the key attributes of the object stored in the storage shard (such as shard ID, filename, etc.). Subsequently, the hash value can be mapped to a specific range of storage nodes based on the number of storage nodes, thereby determining the shard identifier.

[0104] Optionally, the unique identifier mentioned above can also be marked sequentially according to the creation order of each storage fragment.

[0105] After receiving the response results sent by each storage node, since the data in the response results can belong to different storage shards, the query node can determine the order of the storage shards to which the response results sent by each storage node belong based on the second shard identifier sent by each storage node.

[0106] S402. The query node retrieves a preset number of data from the response results sent by each storage node in the order of the storage shards to which the response results sent by each storage node belong, and determines the preset number of data as the query sub-result of the first query sub-request.

[0107] The preset quantity is the amount of data required for the first query sub-request.

[0108] Correspondingly, query nodes can discard data beyond a preset quantity.

[0109] For example, in combination Figure 1 ,like Figure 5 As shown, assuming the first query sub-request requires 100 data items (i.e., the preset quantity (Limit) = 100), storage nodes 1, 2, and 3 can each send a response of 100 data items to the query node. The 100 data items from storage node 1 include: 60 data items from P1 (i.e., ... Figure 5 P1 => List <row>:60) and 40 data points of P4 (i.e. Figure 5 P4 => List <row>:40). The 100 data items in storage node 2 include: 50 data items from P2 (i.e. Figure 5 P2 => List <row>:50) and 50 data points of P5 (i.e. Figure 5 P5 => List <row>:50). The 100 data items in storage node 3 include: 40 data items in P3 (i.e. Figure 5 P3 => List <row>:40) and 60 data points of P6 (i.e. Figure 5 P6 => List <row>:60).

[0110] Next, the query node can determine the order of the storage fragments P1, P2, P3, P4, P5, and P6 based on the second fragment identifiers sent by each storage node, namely the fragment identifiers P1, P2, P3, P4, P5, and P6. Assuming P1, P2, P3, P4, P5, and P6 are the identifiers of the aforementioned six storage fragments, then the order of the storage fragments to which the response results sent by storage nodes 1-3 belong is P1, P2, P3, P4, P5, and P6.

[0111] Since the query node needs to retrieve 100 data items, and the response result sent by storage node 1 includes 60 data items from P1, and the response result sent by storage node 2 includes 50 data items from P2, according to the order of the storage shards to which the response results belong, the query node can retain the first 40 data items from the 60 data items from P1 and the 50 data items from P2, and determine these 100 data items as the query sub-result of the first query sub-request.

[0112] Correspondingly, the query node can discard the last 10 of the 50 data points in P2, as well as the data in P3, P4, P5, and P6.

[0113] Furthermore, the query node can also use the 40th data in P2 as the breakpoint data, and determine the position information of the breakpoint data (including the fragment identifier of P2 and the primary key of the data row of the 40th data in P2) as the cursor position information of the first query sub-request.

[0114] As shown above, when the query node aggregates and prunes the data in the response results sent by each storage node, it can do so according to the order of the storage shards to which the response results sent by each storage node belong, thus avoiding the omission of unqueried storage shards during data query.

[0115] In some embodiments, when aggregating and pruning data from the response results sent by each storage node, the query node can also perform aggregation and pruning based on a load balancing strategy to avoid uneven load distribution among the storage nodes. In this case, combined with Figure 3 ,like Figure 6 As shown, in S204 above, the method for the query node to generate the query sub-result of the first query sub-request based on the response results sent by each storage node specifically includes: S601.

[0116] S601. The query node, based on the load balancing strategy, obtains data from the response results sent by each storage node to obtain the query sub-result of the first query sub-request.

[0117] In some embodiments, the load balancing strategy may be that the query node obtains data from the response results sent by each storage node on an average basis to obtain the query sub-result of the first query sub-request.

[0118] Continuing with the example above, assuming the first query sub-request needs to retrieve 100 data items, the query node can retrieve an average of 33 data items from P1 in storage node 1, 33 data items from P2 in storage node 2, and 34 data items from P3 in storage node 3, thereby ensuring load balancing across the three storage nodes.

[0119] In some other embodiments, the load balancing strategy may be that the query node obtains the load ratio of each storage node and retrieves data from the response results sent by each storage node according to the load ratio of each storage node to obtain the query sub-result of the first query sub-request.

[0120] Continuing with the example above, suppose the first query sub-request needs to retrieve 100 data items. The load ratio of storage node 1 is 20%, the load ratio of storage node 2 is 30%, and the load ratio of storage node 3 is 50%. Then, the query node can retrieve 50 data items from P1 in storage node 1, 30 data items from P2 in storage node 2, and 20 data items from P3 in storage node 3, thus ensuring load balancing among the three storage nodes.

[0121] Of course, the query node can also obtain data from the response results sent by each storage node based on other load balancing strategies, and this application embodiment does not limit this.

[0122] In some embodiments, the data query method provided in this application can also be applied to Figure 1 Each storage node in the system. Figure 7 This illustration shows a flowchart of yet another data query method provided in an embodiment of this application. Figure 7 As shown, the data query method includes: S701-S703.

[0123] S701, the storage node receives a broadcast request.

[0124] The broadcast request is a response from the query node to the first query sub-request in the query request, broadcast based on the cursor position information of the previous second query sub-request in the query request and the first shard identifier. The first shard identifier is used to identify the storage shard that has been queried by the second query sub-request.

[0125] For a detailed description of S701, please refer to the relevant description of S202 above, which will not be repeated here.

[0126] S702. The storage node determines the response result of the first query sub-request based on the cursor position information of the second query sub-request and the first fragment identifier.

[0127] Upon receiving a broadcast request, since the broadcast request is constructed based on the cursor position information of the previous second query sub-request and the first fragment identifier in the query request, the broadcast request can carry the cursor position information of the second query sub-request and the first fragment identifier. In this way, the storage node can quickly query the data based on the cursor position information of the second query sub-request and the first fragment identifier, thereby determining the response result of the first query sub-request.

[0128] S703, The storage node sends a response result to the query node, so that the query node generates the query sub-result of the first query sub-request based on the response results sent by each storage node in the distributed database.

[0129] The query sub-result of the first query sub-request is used to generate the query result of the query request.

[0130] For a detailed description of S703, please refer to the relevant descriptions of S203-S204 above, which will not be repeated here.

[0131] In some embodiments, combined with Figure 7 ,like Figure 8 As shown, in S702 above, the method by which the storage node determines the response result of the first query sub-request based on the cursor position information of the second query sub-request and the first fragment identifier specifically includes:

[0132] S801, The storage node determines the storage shard to be queried based on the first shard identifier.

[0133] Specifically, since the broadcast request carries the first fragment identifier, in order to avoid repeated queries on the storage fragment corresponding to the first fragment identifier, the storage node can determine the storage fragment to be queried based on the first fragment identifier.

[0134] Optionally, the storage node stores the fragment identifiers of all storage fragments. Upon receiving a broadcast request, the storage node can identify the storage fragments other than the one corresponding to the first fragment identifier as the storage fragment to be queried.

[0135] S802. The storage node queries a preset number of data from the storage shard to be queried based on the cursor position information of the second query sub-request, and determines the preset number of data as the response result of the first query sub-request.

[0136] The preset quantity is the amount of data required for the first query sub-request.

[0137] Specifically, since the broadcast request carries the cursor position information of the second query sub-request, in order to quickly and accurately retrieve data from the data row corresponding to the cursor position information of the second query sub-request, the storage node can first determine the starting position of the data query based on the cursor position information of the second query sub-request, and then retrieve the preset number of data. However, because different storage nodes may have different breakpoint positions in the data query of the second query sub-request, or some storage nodes may not have queried data during the second query sub-request, the method for querying the preset number of data in the storage shard to be queried will differ for different storage nodes.

[0138] In this case, combination Figure 8 ,like Figure 9 As shown, the method by which a storage node queries a preset number of data from the storage shard to be queried based on the cursor position information of the second query sub-request specifically includes:

[0139] S901, if the cursor position information of the second query sub-request is empty, the storage node queries a preset number of data from the data storage start position of the storage fragment to be queried.

[0140] For example, suppose the query result of the second query sub-request includes data stored by storage node 1, but does not include data stored by storage node 2. In this case, the cursor position information for storage node 2 in the broadcast request constructed by the query node will be empty. After receiving the broadcast request, storage node 2 can query a preset number of data from the starting position of the data storage of the storage shard to be queried.

[0141] In other words, if the storage node does not query data during the second query sub-request, the broadcast request it receives does not carry the breakpoint row primary key. Thus, the storage node can query a predetermined number of data points from the starting position of the data storage of the storage shard to be queried (i.e., the starting position of the list of shard IDs to be scanned (owned_part_ids)).

[0142] S902, if the cursor position information of the second query sub-request includes the primary key of the data row of the breakpoint data and the third fragment identifier, the storage node determines the query start position of the first query sub-request based on the primary key of the data row of the breakpoint data and the third fragment identifier.

[0143] The third fragment identifier is used to identify the storage fragment to which the breakpoint data belongs.

[0144] For example, assuming the query result of the second query sub-request includes data stored in storage node 1, then the cursor position information of storage node 1 in the broadcast request constructed by the query node is the primary key of the data row of the breakpoint data and the third shard identifier in the query result of the second query sub-request. For instance, if the breakpoint data is the 40th row of data in P1, then the primary key of the data row of the breakpoint data is the primary key of the data row of the 40th row of data in P1, and the third shard identifier is the shard identifier of P1.

[0145] After receiving the broadcast request, storage node 1 can determine the 41st row of data in P1 as the starting position of the first query sub-request.

[0146] S903. The storage node queries a preset number of data from the storage shard to be queried based on the query start position of the first query sub-request.

[0147] Referring to the example above, assuming the first query sub-request requires 100 data items, the storage node can start querying from the 41st row of data in P1, retrieving 100 data items. It should be understood that if the amount of data after the 41st row of data in P1 is less than 100 data items, the storage node will query the remaining data from the next storage shard.

[0148] In other words, if the storage node has already queried some data in the second query sub-request, the broadcast request it receives will carry the breakpoint row primary key. This allows the storage node to directly query a preset number of data points from the location of the breakpoint data corresponding to the breakpoint row primary key, improving data query efficiency.

[0149] In one possible approach, storage nodes can iterate through storage shards within themselves, querying based on indexes and returning to the table.

[0150] Specifically, the storage node can first determine the starting position of the current query based on the primary key of the breakpoint row (i.e., the internal cursor (scan_cursor)) in the cursor position information of the second query sub-request. Then, combined with the index information (i.e., index descriptions (indices)) in the cursor position information of the second query sub-request, it reads the values ​​of the index columns of that data row. For example, if the data row has 10 columns, but the index information in the cursor position information of the second query sub-request corresponds to the data column age+city, then the storage node reads the values ​​of the age and city columns of that data row.

[0151] It should be noted that since storage nodes typically read data based on binary encoding, they need to encode the information carried in the broadcast request to obtain the index key, for example: IndexKey(PartID+TableID+IndexVals+PrimaryKeys).

[0152] Among them, PartID is used to represent the shard ID of the storage shard to be queried, TableID is used to represent the table ID to be queried by the first query sub-request, IndexVals is used to represent the index information of the data required by the first query sub-request, and PrimaryKeys is used to represent the primary key of the data row of the breakpoint data.

[0153] By combining these different elements, a storage node can create a unique index key for quickly locating and retrieving data.

[0154] For example, when the storage node is a database based on the RocksDB storage engine, RocksDB iterators are typically deployed within RocksDB. RocksDB is a high-performance embedded key-value store engine. Iterators in RocksDB provide the ability to traverse key-value pairs in the database. Storage nodes can leverage the LSM-Tree-based seek capability of RocksDB iterators (the ability of iterators to quickly locate a specific key) to move the data query point to the IndexKey, thus enabling data retrieval from the last breakpoint.

[0155] As can be seen from the above, the LSM-Tree-based seek mechanism of this application has a logarithmic time complexity, which is a significant improvement over the constant-time position movement of MySQL.

[0156] Furthermore, this embodiment of the application can also push down the logic of mapping breakpoint data rows to the index to the storage node. That is, when the query node generates cursor position information, it is not necessary to obtain the primary key of the breakpoint data row again through the storage node. Instead, the index information of the breakpoint data is added to the cursor position information. Subsequently, after receiving the cursor position information, the storage node can determine the primary key of the breakpoint data row based on the index information, and then directly obtain the data required by the query sub-request. This reduces the data interaction between the query node and the storage node, thereby reducing the overhead of the query node's Remote Procedure Call (RPC).

[0157] The above mainly describes the data query method provided in this application from the perspectives of client-side query nodes and storage nodes. The following section details the data query method provided in this application from the perspective of node interaction, such as... Figure 10 As shown, this application also provides a data query method, including:

[0158] S1001, The requesting end calls the scanning interface provided by the SDK to send the first query sub-request to the query node.

[0159] The first query sub-request includes the cursor position information of the previous second query sub-request in the query request and the first fragment identifier. The first fragment identifier is used to identify the storage fragment that the second query sub-request has queried.

[0160] S1002. The query node broadcasts the broadcast request to each storage node in the distributed database based on the cursor position information of the second query sub-request and the first shard identifier.

[0161] The broadcast request is used to request each storage node to query data based on the cursor position information and the first fragment identifier of the second query sub-request. Each storage node can be... Figure 10 The storage node shown is 1... storage node N.

[0162] S1003. The storage node determines the response result of the first query sub-request based on the cursor position information of the second query sub-request and the first fragment identifier.

[0163] S1004. The storage node sends the response result to the query node.

[0164] S1005. The query node generates the query sub-result of the first query sub-request and the cursor position information of the first query sub-request based on the response results sent by each storage node.

[0165] S1006. The query node sends the query sub-result of the first query sub-request and the cursor position information of the first query sub-request to the requesting end corresponding to the query request.

[0166] Repeat the above steps until all query sub-requests of the SDK's iterator have been queried.

[0167] For example, such as Figure 5 As shown, assuming the requesting end needs to retrieve 1000 data items, the requesting end can call the SDK to send 10 query sub-requests to the query node. Each query sub-request can retrieve 100 data items (that is, retrieve the required data in batches and return it to the requesting end in the form of a data stream).

[0168] After receiving the first query sub-request, the query node can broadcast the request to each storage node in the distributed database to request the data required by the requesting end.

[0169] Understandably, since this is the first query sub-request, meaning the query node hasn't previously retrieved the required data, the first query sub-request doesn't include the cursor position information from the previous query sub-request or the fragment identifier of the queried storage fragment. Alternatively, the fields related to the cursor position information and fragment identifier of the queried storage fragment from the previous query sub-request may be empty. In this case, each storage node can query 100 data points (i.e., the response result) starting from the first data point that satisfies the query request and return them to the query node.

[0170] Among them, the 100 data items in storage node 1 include: 60 data items from P1 (i.e., Figure 5 P1 => List <row>:60) and 40 data points of P4 (i.e. Figure 5 P4 => List <row>:40). The 100 data items in storage node 2 include: 50 data items from P2 (i.e. Figure 5 P2 => List <row>:50) and 50 data points of P5 (i.e. Figure 5 P5 => List <row>:50). The 100 data items in storage node 3 include: 40 data items in P3 (i.e. Figure 5 P3 => List <row>:40) and 60 data points of P6 (i.e. Figure 5 P6 => List <row>(60). Next, the query node can determine the order of the storage fragments P1, P2, P3, P4, P5, and P6 based on the second fragment identifiers sent by each storage node, i.e., the fragment identifiers P1, P2, P3, P4, P5, and P6. Assuming that P1, P2, P3, P4, P5, and P6 are the identifiers of the above 6 storage fragments, then the order of the storage fragments to which the response results sent by storage nodes 1-3 belong is P1, P2, P3, P4, P5, and P6.

[0171] Since the query node needs to retrieve 100 data items, and the response result sent by storage node 1 includes 60 data items from P1, and the response result sent by storage node 2 includes 50 data items from P2, according to the order of the storage shards to which the response results belong, the query node can retain the first 40 data items from the 60 data items from P1 and the 50 data items from P2, and determine these 100 data items as the query sub-result of the first query sub-request.

[0172] Correspondingly, the query node can discard the last 10 of the 50 data points in P2, as well as the data in P3, P4, P5, and P6.

[0173] Furthermore, the query node can use the 40th data point in P2 as a breakpoint, and determine the location information of this breakpoint data (including the shard identifier of P2 and the primary key of the data row of the 40th data point in P2) as the cursor position information for the first query sub-request (corresponding to the cursor position information for the second query sub-request mentioned earlier). Accordingly, since the query for P1 has ended, the shard identifier of P1 can be used as the shard identifier of the queried storage shard (corresponding to the first shard identifier mentioned earlier), and the location information of the breakpoint data of the query sub-result of the first query sub-request and the shard identifier of the queried storage shard can be sent to the SDK.

[0174] Next, the requesting end can call the SDK to send a second query sub-request to the query node (corresponding to the first query sub-request mentioned above). Since the SDK maintains the position information of the breakpoint data of the query sub-result of the first query sub-request and the fragment identifier of the queried storage fragment, the SDK can add this information to the second query sub-request and send the second query sub-request to the query node.

[0175] After receiving the second query sub-request, the query node can extract the location information of the breakpoint data and the fragment identifier of the queried storage fragment from the second query sub-request, and add this information to the broadcast request to broadcast the request to each storage node in the distributed database.

[0176] After receiving a broadcast request, each storage node can determine the storage fragment to be queried based on the fragment identifier of the queried storage fragment, and determine the starting position of the data to be queried based on the location information of the breakpoint data.

[0177] For example, for storage node 1, since the shard identifier of the queried storage shard includes the shard identifier of P1, storage node 1 can query 100 data points starting from the first data point of P4 (including 40 data points in P4 and 60 data points in P7) and return these 100 data points to the querying node.

[0178] For storage node 2, since the fragment identifiers of the queried storage fragments do not include the fragment identifiers of all storage fragments in storage node 2, and the position information of the breakpoint data is the 40th data in P2, storage node 2 can query 100 data starting from the 41st data in P2 (including data from the 41st to the 50th data in P2, data from the 50th data in P5, and data from the 40th data in P8), and return these 100 data to the query node.

[0179] For storage node 3, since the fragment identifiers of the queried storage fragments do not include the fragment identifiers of all storage fragments in storage node 3, and the location information of the breakpoint data is not in storage node 3, storage node 3 can query 100 data points starting from the first data point of P3 (including the 40th data point in P3 and the 60th data point in P6), and return these 100 data points to the query node.

[0180] Next, the query node can aggregate and trim the response results sent by each storage node using the same method to generate the location information of the breakpoint data of the second query sub-request and the fragment identifier of the queried storage fragment, and send the location information of the breakpoint data of the query sub-result of the second query sub-request and the fragment identifier of the queried storage fragment to the SDK.

[0181] Next, the requesting client can call the SDK to send a third query sub-request to the query node. Using the same method, the query node can retrieve the query sub-results corresponding to the third query sub-request. This process is repeated until the requesting client can retrieve the query sub-results for all query sub-requests.

[0182] The foregoing primarily describes the solutions of the embodiments of this application from a methodological perspective. It is understood that, in order to achieve the aforementioned functions, each of the various nodes includes corresponding hardware structures and / or software modules for executing each function. Those skilled in the art should readily recognize that, in conjunction with the units and algorithm steps of the various examples described in the embodiments disclosed herein, the embodiments of this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the embodiments of this application.

[0183] This application embodiment can divide the above-mentioned nodes into functional units according to the above method examples. For example, each function can be divided into a separate functional unit, or two or more functions can be integrated into one processing unit. The integrated unit can be implemented in hardware or as a software functional unit. It should be noted that the unit division in this application embodiment is illustrative and only represents one logical functional division. In actual implementation, there may be other division methods.

[0184] like Figure 11 As shown, this application embodiment provides a data query device, including: a communication unit 1101 and a processing unit 1102;

[0185] Communication unit 1101 is used to receive a first query sub-request in a query request; the first query sub-request includes cursor position information of the previous second query sub-request in the query request and a first fragment identifier; the first fragment identifier is used to identify the storage fragment that has been queried by the second query sub-request.

[0186] Communication unit 1101 is also used to broadcast a broadcast request to each storage node in the distributed database based on the cursor position information of the second query sub-request and the first fragment identifier; the broadcast request is used to request each storage node to query data based on the cursor position information of the second query sub-request and the first fragment identifier.

[0187] The communication unit 1101 is also used to receive response results sent by each storage node;

[0188] The processing unit 1102 is used to generate a query sub-result of the first query sub-request based on the response results sent by each storage node; the query sub-result of the first query sub-request is used to generate the query result of the query request.

[0189] In some embodiments, the processing unit 1102 is further configured to generate cursor position information of the first query sub-request based on the query sub-result of the first query sub-request;

[0190] The communication unit 1101 is also used to send the query sub-result of the first query sub-request and the cursor position information of the first query sub-request to the requesting end corresponding to the query request, so that the requesting end can determine the query result of the query request based on the cursor position information and query sub-results corresponding to each of the multiple query sub-requests, including the first query sub-request and the second query sub-request.

[0191] In some embodiments, the response result includes a second fragment identifier; the second fragment identifier is used to identify the storage fragment to which the response result belongs;

[0192] Processing unit 1102 is specifically used for:

[0193] Based on the second fragment identifier sent by each storage node, determine the order in which the response results sent by each storage node belong to the storage fragment;

[0194] According to the order in which the response results sent by each storage node belong to the storage shards, a preset number of data is obtained from the response results sent by each storage node in sequence, and the preset number of data is determined as the query sub-result of the first query sub-request; the preset number is the amount of data required by the first query sub-request.

[0195] In some embodiments, the processing unit 1102 is specifically used for:

[0196] Based on the load balancing strategy, data is obtained from the response results sent by each storage node to obtain the query sub-result of the first query sub-request.

[0197] In some embodiments, the processing unit 1102 is specifically used for:

[0198] The position information of the breakpoint data of the query sub-result of the first query sub-request is determined as the cursor position information of the first query sub-request.

[0199] In some embodiments, the cursor position information includes: a third fragment identifier and the primary key of the data row of the breakpoint data; the third fragment identifier is used to identify the storage fragment to which the breakpoint data belongs.

[0200] like Figure 12 As shown in the figure, this application embodiment also provides a data query device, including: a communication unit 1201 and a processing unit 1202;

[0201] The communication unit 1201 is used to receive a broadcast request; the broadcast request is broadcast by the query node in response to the first query sub-request in the query request, based on the cursor position information of the previous second query sub-request in the query request and the first fragment identifier; the first fragment identifier is used to identify the storage fragment that has been queried by the second query sub-request.

[0202] Processing unit 1202 is used to determine the response result of the first query sub-request based on the cursor position information of the second query sub-request and the first fragment identifier;

[0203] The communication unit 1201 is also used to send a response result to the query node, so that the query node generates a query sub-result of the first query sub-request based on the response results sent by each storage node in the distributed database; the query sub-result of the first query sub-request is used to generate the query result of the query request.

[0204] In some embodiments, the processing unit 1202 is specifically used for:

[0205] The storage fragment to be queried is determined based on the first fragment identifier;

[0206] Based on the cursor position information of the second query sub-request, a preset number of data is retrieved from the storage shard to be queried, and the preset number of data is determined as the response result of the first query sub-request; the preset number is the amount of data required by the first query sub-request.

[0207] In some embodiments, the processing unit 1202 is specifically used for:

[0208] If the cursor position information of the second query sub-request is empty, query a preset number of data from the starting position of the data storage of the storage shard to be queried.

[0209] In some embodiments, the processing unit 1202 is specifically used for:

[0210] If the cursor position information of the second query sub-request includes the primary key of the data row of the breakpoint data and the third shard identifier, the query start position of the first query sub-request is determined according to the primary key of the data row of the breakpoint data and the third shard identifier; the third shard identifier is used to identify the storage shard to which the breakpoint data belongs.

[0211] Based on the query start position of the first query sub-request, query a preset number of data from the storage shard to be queried.

[0212] According to embodiments of the present disclosure, the present disclosure also provides an electronic device, including at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the data query method provided in the present disclosure.

[0213] According to embodiments of this disclosure, this disclosure also provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause an electronic device to execute the data query method provided in this disclosure.

[0214] According to embodiments of this disclosure, this disclosure also provides a computer program product, including a computer program that, when executed by a processor, implements the data query method provided in this disclosure.

[0215] Figure 13 A schematic block diagram of an example electronic device 1300 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0216] like Figure 13 As shown, the electronic device 1300 includes a computing unit 1301, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory 1302 or a computer program loaded from a storage unit 1308 into a random access memory 1303. The random access memory (RAM) 1303 may also store various programs and data required for the operation of the electronic device 1300. The computing unit 1301, the read-only memory (ROM) 1302, and the RAM 1303 are interconnected via a bus 1304. An input / output (I / O) interface 1305 is also connected to the bus 1304.

[0217] Multiple components in electronic device 1300 are connected to input / output interface 1305, including: input unit 1306, such as keyboard, mouse, etc.; output unit 1307, such as various types of monitors, speakers, etc.; storage unit 1308, such as disk, optical disk, etc.; and communication unit 1309, such as network card, modem, wireless transceiver, etc. Communication unit 1309 allows electronic device 1300 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0218] The computing unit 1301 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Examples of computing units 1301 include, but are not limited to, a central processing unit, a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors, and any suitable processor, controller, microcontroller, etc. The computing unit 1301 performs the various methods and processes described above, such as the data matching method. For example, in one embodiment, the data matching method can be implemented as a computer software program tangibly included in a machine-readable medium, such as storage unit 1308. In one embodiment, part or all of the computer program can be loaded and / or installed on the electronic device 1300 via ROM 1302 and / or communication unit 1309. When the computer program is loaded into RAM 1303 and executed by the computing unit 1301, one or more steps of the data matching method described above can be performed. Alternatively, in other embodiments, the computing unit 1301 may be configured to perform a data matching method by any other suitable means (e.g., by means of firmware).

[0219] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays, application-specific integrated circuits (ASICs), application-specific standard parts (ASSPs), systems-on-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0220] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0221] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory, read-only memory, erasable programmable read-only memory, optical fibers, portable compact disk read-only memory, optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0222] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user, such as a cathode ray tube (CRT) or liquid crystal display (LCD) monitor; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0223] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0224] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0225] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this disclosure can be achieved, and this is not limited herein.

[0226] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.< / row> < / row> < / row> < / row> < / row> < / row> < / row> < / row> < / row> < / row> < / row> < / row> < / row>

Claims

1. A data query method, characterized in that, include: Receive the first query sub-request in the query request; The first query sub-request includes the cursor position information and the first fragment identifier of the previous second query sub-request in the query request; The first shard identifier is used to identify the storage shard that the second query sub-request has queried; Based on the cursor position information of the second query sub-request and the first shard identifier, broadcast the broadcast request to each storage node in the distributed database; The broadcast request is used to request each storage node to query data based on the cursor position information of the second query sub-request and the first fragment identifier; Receive the response results sent by each storage node, and generate the query sub-result of the first query sub-request based on the response results sent by each storage node; The query sub-result of the first query sub-request is used to generate the query result of the query request.

2. The method according to claim 1, characterized in that, Also includes: Generate the cursor position information of the first query sub-request based on the query sub-result of the first query sub-request; The query result of the first query sub-request and the cursor position information of the first query sub-request are sent to the requesting end corresponding to the query request, so that the requesting end can determine the query result of the query request based on the cursor position information and query result corresponding to each of the multiple query sub-requests, including the first query sub-request and the second query sub-request.

3. The method according to claim 1, characterized in that, The response result includes a second fragment identifier; the second fragment identifier is used to identify the storage fragment to which the response result belongs; The step of generating the query sub-result of the first query sub-request based on the response results sent by each storage node includes: Based on the second fragment identifier sent by each storage node, determine the order in which the response results sent by each storage node belong to the storage fragment; According to the order in which the response results sent by each storage node belong, a preset number of data are sequentially obtained from the response results sent by each storage node, and the preset number of data is determined as the query sub-result of the first query sub-request; the preset number is the amount of data required by the first query sub-request.

4. The method according to claim 1, characterized in that, The step of generating the query sub-result of the first query sub-request based on the response results sent by each storage node includes: Based on the load balancing strategy, data is obtained from the response results sent by each of the storage nodes to obtain the query sub-result of the first query sub-request.

5. The method according to claim 2, characterized in that, The step of generating the cursor position information of the first query sub-request based on the query sub-result of the first query sub-request includes: The position information of the breakpoint data of the query sub-result of the first query sub-request is determined as the cursor position information of the first query sub-request.

6. The method according to claim 5, characterized in that, The cursor position information includes: the third shard identifier and the primary key of the data row of the breakpoint data; the third shard identifier is used to identify the storage shard to which the breakpoint data belongs.

7. A data query method, characterized in that, include: Receive broadcast requests; The broadcast request is a response from the query node to the first query sub-request in the query request, broadcast based on the cursor position information of the previous second query sub-request in the query request and the first fragment identifier; The first shard identifier is used to identify the storage shard that the second query sub-request has queried; Based on the cursor position information of the second query sub-request and the first fragment identifier, determine the response result of the first query sub-request; The response result is sent to the query node so that the query node generates the query sub-result of the first query sub-request based on the response results sent by each storage node in the distributed database; The query sub-result of the first query sub-request is used to generate the query result of the query request.

8. The method according to claim 7, characterized in that, The step of determining the response result of the first query sub-request based on the cursor position information of the second query sub-request and the first fragment identifier includes: The storage fragment to be queried is determined based on the first fragment identifier; Based on the cursor position information of the second query sub-request, a preset number of data is retrieved from the storage shard to be queried, and the preset number of data is determined as the response result of the first query sub-request; the preset number is the amount of data required by the first query sub-request.

9. The method according to claim 8, characterized in that, The step of querying a preset number of data from the storage shard to be queried based on the cursor position information of the second query sub-request includes: If the cursor position information of the second query sub-request is empty, the preset number of data is queried from the data storage start position of the storage segment to be queried.

10. The method according to claim 8, characterized in that, The step of querying a preset number of data from the storage shard to be queried based on the cursor position information of the second query sub-request includes: If the cursor position information of the second query sub-request includes the primary key of the data row of the breakpoint data and the third shard identifier, the query start position of the first query sub-request is determined according to the primary key of the data row of the breakpoint data and the third shard identifier; the third shard identifier is used to identify the storage shard to which the breakpoint data belongs. Based on the query start position of the first query sub-request, the preset number of data is queried from the storage shard to be queried.

11. A data query device, characterized in that, include: Communication unit and processing unit; The communication unit is used to receive the first query sub-request in the query request; The first query sub-request includes the cursor position information and the first fragment identifier of the previous second query sub-request in the query request; The first shard identifier is used to identify the storage shard that the second query sub-request has queried; The communication unit is further configured to broadcast a broadcast request to each storage node in the distributed database based on the cursor position information of the second query sub-request and the first fragment identifier; the broadcast request is used to request each storage node to query data based on the cursor position information of the second query sub-request and the first fragment identifier; The communication unit is also used to receive response results sent by each storage node; The processing unit is used to generate a query sub-result of the first query sub-request based on the response results sent by each storage node; The query sub-result of the first query sub-request is used to generate the query result of the query request.

12. A data query device, characterized in that, include: Communication unit and processing unit; The communication unit is used to receive broadcast requests; The broadcast request is a response from the query node to the first query sub-request in the query request, broadcast based on the cursor position information of the previous second query sub-request in the query request and the first fragment identifier; The first shard identifier is used to identify the storage shard that the second query sub-request has queried; The processing unit is configured to determine the response result of the first query sub-request based on the cursor position information of the second query sub-request and the first fragment identifier; The communication unit is further configured to send the response result to the query node, so that the query node generates the query sub-result of the first query sub-request based on the response results sent by each storage node in the distributed database; The query sub-result of the first query sub-request is used to generate the query result of the query request.

13. A data query system, characterized in that, include: Query nodes and storage nodes; The query node is used to perform the method as described in any one of claims 1-6; The storage node is used to perform the method as described in any one of claims 7-10.

14. An electronic device, characterized in that, The electronic device includes: processor; A memory configured to store processor-executable instructions; The processor is configured to execute the instructions to implement the method as described in any one of claims 1-6 or any one of claims 7-10.

15. A non-volatile storage medium, characterized in that, The storage medium stores a computer program, which, when read and executed, implements the method as described in any one of claims 1-6 or any one of claims 7-10.

16. A computer program product comprising instructions, characterized in that, When the instructions are executed on an electronic device, the electronic device causes the electronic device to perform the method as described in any one of claims 1-6 or any one of claims 7-10.

Images