Query request asynchronous processing method, device, computer equipment, and storage medium

Abstract

The present disclosure relates to a method, device, computer equipment, and storage medium for asynchronous processing of query requests, the method comprising: receiving a query request sent by a query end, generating a query task based on the query request and assigning a task number, and writing the query task into a query queue; obtaining the current query task from the query queue using a distributed execution node, and initiating a query to a cache based on the current query task; querying the first key-value pair in the cache based on the structured query language of the current query task to determine whether there is a target key-value pair, and obtaining a query result; generating a second key-value pair of the current query task based on the query result and the task number of the current query task, and storing the second key-value pair in the cache for reading by the query end. The present disclosure can make the response result of the query task more efficient, reduce the waiting time of the query end, and have wider adaptability.

Description

Technical Field

[0001] The present disclosure relates to the field of data query technology, and in particular to a method, device, computer equipment, and storage medium for asynchronously processing query requests. Background Art

[0002] With the maturity of various software and hardware service technologies and the development of the big data era, the amount of information data and the complexity of information query have greatly increased. At present, user query requests are usually responded to through background logic and databases, but in the case of high concurrency, the database may be blocked and the query request cannot be responded to in time. Users need to wait for the result to be returned. In serious cases, the database cannot be connected or even crashes.

[0003] In traditional technology, the database is often expanded, some cache mechanisms of the database itself are enabled, or the service architecture is changed to allow some slow query requests to be returned asynchronously. However, the current processing methods have the following problems: the database's own cache mechanism is limited by the server's single machine memory, which is very limited; and the traditional asynchronous query solution is not universal, and each service needs to be modified for itself, which is cumbersome. Summary of the invention

[0004] Based on this, it is necessary to provide a query request asynchronous processing method, device, computer equipment, storage medium and computer program product that can improve the response efficiency of query requests without changing or only slightly changing the original system business architecture to address the above technical problems.

[0005] In a first aspect, the present disclosure provides a method for asynchronously processing a query request. The method comprises:

[0006] Receive a query request sent by a query end, generate a query task based on the query request and assign a task number, and write the query task into a query queue;

[0007] The distributed execution node is used to obtain the current query task from the query queue, and a query is initiated to the cache library based on the current query task; the cache library includes a plurality of cache nodes, and the cache library is used to store a first key-value pair including a historical query task, wherein the first key-value pair is a key-value pair of a structured query language of the query task and a query result set;

[0008] According to the structured query language of the current query task, query whether there is a target key-value pair in the first key-value pair in the cache library to obtain a query result; the target key-value pair is the first key-value pair in the cache library that has the same structured query language as the current query task;

[0009] According to the query result and the task number of the current query task, a second key-value pair of the current query task is generated accordingly, and the second key-value pair is stored in the cache library for reading by the query end; the second key-value pair is a key-value pair of the task number of the query task and the query result set.

[0010] In one embodiment, the distributed execution node is used to obtain the current query task from the query queue, and a query is initiated to the cache library based on the current query task; the cache library includes a plurality of cache nodes, and the cache library is used to store a first key-value pair including a historical query task, wherein the first key-value pair is a key-value pair of a structured query language of the query task and a query result set including:

[0011] Based on the query task, the distributed execution node obtains the corresponding query task according to a preset rule.

[0012] In one embodiment, generating a second key-value pair of the current query task according to the query result and the task number of the current query task, and storing the second key-value pair in the cache library for reading by the query end includes:

[0013] When the query result shows that the target key-value pair exists in the cache, a second key-value pair is generated according to the target key-value pair and the task number of the current query task, and the second key-value pair is stored in the cache for reading by the query end.

[0014] In one of the embodiments, when the query result is that the target key-value pair exists in the cache library, generating a second key-value pair according to the target key-value pair and the task number of the current query task, and storing the second key-value pair in the cache library for the query end to read includes:

[0015] Acquire a query result set of the target key-value pair, and generate a second key-value pair of the current query task according to the task number of the current query task and the query result set of the target key-value pair;

[0016] Splitting the second key value of the current query task based on a hash algorithm to generate a plurality of data fragments;

[0017] The data shards are stored in several cache nodes of the cache library respectively.

[0018] In one embodiment, generating a second key-value pair of the current query task according to the query result and the task number of the current query task, and storing the second key-value pair in the cache library for reading by the query end includes:

[0019] When the query result is that the target key-value pair does not exist in the cache library, the database is queried to obtain the query result set of the current query task, the first key-value pair and the second key-value pair of the current query task are generated, and stored in the cache library for reading by the query end.

[0020] In one of the embodiments, when the query result is that the target key-value pair does not exist in the cache library, querying the database to obtain the query result set of the current query task, generating the first key-value pair and the second key-value pair of the current query task, and storing them in the cache library for the query end to read includes:

[0021] Initiate a query to a database according to the structured query language of the current query task, and receive a query result set of the current query task returned by the database;

[0022] Generate a first key-value pair according to the structured query language of the current query task and the query result set;

[0023] Generate a second key-value pair according to the task number of the current query task and the query result set;

[0024] Based on the hash algorithm, the first key-value pair and the second key value of the current query task are respectively split into data to generate a plurality of data fragments;

[0025] The data shards are stored in several cache nodes of the cache library respectively.

[0026] In one embodiment, storing the data shards on a plurality of cache nodes of the cache library respectively includes:

[0027] The data shards calculate and output node numbers according to a preset algorithm, and the node numbers are used to correspond one-to-one with the cache nodes;

[0028] The data fragments are stored in the corresponding cache nodes according to the node numbers.

[0029] In one embodiment, storing the data shards on a plurality of cache nodes of the cache library respectively further comprises:

[0030] Generate a shard copy of the data shard, and store the shard copy in a cache node that does not correspond to the node number of the data shard.

[0031] In one embodiment, providing the query end with the information to read includes:

[0032] A service interface is provided to the query end so that the query end can query the corresponding query result set in the cache based on the task number of the query task.

[0033] In one embodiment, providing the query end with the information to read includes:

[0034] The distributed execution node sends the acquired query result set of the current query task to the query end.

[0035] In a second aspect, the present disclosure further provides a query request asynchronous processing device. The device comprises:

[0036] A query queue module, used to receive a query request sent by a query end, generate a query task based on the query request and assign a task number, and write the query task into a query queue;

[0037] A distributed execution node module, used to obtain the current query task from the query queue using a distributed execution node, and to initiate a query to a cache based on the current query task; the cache includes a plurality of cache nodes, and the cache is used to store a first key-value pair including a historical query task, wherein the first key-value pair is a key-value pair of a structured query language of the query task and a query result set;

[0038] A cache query module, used to query whether there is a target key-value pair in the first key-value pair in the cache according to the structured query language of the current query task, and obtain a query result; the target key-value pair is the first key-value pair in the cache that has the same structured query language as the current query task;

[0039] The key-value pair generation module is used to generate a second key-value pair of the current query task according to the query result and the task number of the current query task, and store the second key-value pair in the cache library for reading by the query end.

[0040] In one embodiment, the distributed execution node module includes:

[0041] The acquisition unit is used for the distributed execution node to acquire the corresponding query task according to a preset rule based on the query task.

[0042] In one of the embodiments, the key-value pair generation module is used to generate a second key-value pair based on the target key-value pair and the task number of the current query task when the query result shows that the target key-value pair exists in the cache library, and store the second key-value pair in the cache library for reading by the query end.

[0043] In one embodiment, the key-value pair generation module includes:

[0044] A second key-value pair unit, used to obtain a query result set of the target key-value pair, and generate a second key-value pair of the current query task according to the task number of the current query task and the query result set of the target key-value pair;

[0045] A data sharding unit, used for performing data splitting on the second key value of the current query task based on a hash algorithm to generate a plurality of data shards;

[0046] The cache node unit is used to store the data fragments on several cache nodes of the cache library respectively.

[0047] In one of the embodiments, the key-value pair generation module is used to query the database to obtain the query result set of the current query task when the query result is that the target key-value pair does not exist in the cache library, generate the first key-value pair and the second key-value pair of the current query task, and store them in the cache library for the query end to read.

[0048] In one embodiment, the key-value pair generation module includes:

[0049] A database query unit, configured to initiate a query to a database according to the structured query language of the current query task, and receive a query result set of the current query task returned by the database;

[0050] A first key-value pair unit, configured to generate a first key-value pair according to the structured query language of the current query task and the query result set;

[0051] A second key-value pair unit, used to generate a second key-value pair according to the task number of the current query task and the query result set;

[0052] A data sharding unit, configured to perform data splitting on the first key-value pair and the second key value of the current query task based on a hash algorithm to generate a plurality of data shards;

[0053] The cache node unit stores the data slices on several cache nodes of the cache library respectively.

[0054] In one embodiment, the cache node unit includes:

[0055] A node numbering subunit, used for calculating and outputting a node number for the data shard according to a preset algorithm, wherein the node number is used for one-to-one correspondence with the cache node;

[0056] The storage subunit is used to store the data fragments in the corresponding cache nodes according to the node numbers.

[0057] In one of the embodiments, the cache node unit further includes:

[0058] The replica subunit is used to generate a shard replica of the data shard, and store the shard replica in a cache node that does not correspond to the node number of the data shard.

[0059] In one embodiment, the device further comprises:

[0060] The service interface unit is used to provide a service interface to the query end, so that the query end can query the corresponding query result set in the cache based on the task number of the query task.

[0061] In one embodiment, the device further comprises:

[0062] A sending unit is used to instruct the distributed execution node to send the acquired query result set of the current query task to the query end.

[0063] In a third aspect, the present disclosure further provides a computer device, wherein the computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the above-mentioned method for asynchronously processing query requests are implemented.

[0064] In a fourth aspect, the present disclosure further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the above-mentioned method for asynchronously processing query requests are implemented.

[0065] In a fifth aspect, the present disclosure further provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, the steps of the above-mentioned method for asynchronously processing query requests are implemented.

[0066] The above-mentioned query request asynchronous processing method, apparatus, computer device, storage medium and computer program product have at least the following beneficial effects:

[0067] The present invention obtains query tasks through distributed execution nodes for querying, so that the response results of the query tasks are more efficient and the waiting time of the query end is reduced; at the same time, the cache library can include multiple cache nodes, breaking through the cache limitation of the server's single-machine memory; in addition, by connecting with the query end to receive query requests, decoupling is achieved from the service system of the query end, and there is no need to add or modify an asynchronous query module in the original service system of the query end; and it can serve multiple query ends at the same time, and only needs to accept the query request sent by the query end to implement the query, which has wider adaptability, can well support high concurrency and high availability, and is easy to expand horizontally. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure or the conventional technology, the drawings required for use in the embodiments or the conventional technology descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0069] Figure 1 An application environment diagram of a query request asynchronous processing method in one embodiment;

[0070] Figure 2 A flowchart of a method for asynchronously processing a query request in one embodiment;

[0071] Figure 3 A flowchart of the steps of obtaining a query result set in one embodiment;

[0072] Figure 4 Another flowchart of the steps of obtaining a query result set in one embodiment;

[0073] Figure 5 A schematic diagram of a flow chart of steps for storing data slices in one embodiment;

[0074] Figure 6 Another flowchart of the steps of storing data slices in one embodiment;

[0075] Figure 7 A schematic diagram of data flow for reading by the query end in one embodiment;

[0076] Figure 8 It is a structural block diagram of a query request asynchronous processing device in one embodiment;

[0077] Fig. 9 A structural block diagram of a distributed execution node module in one embodiment;

[0078] Fig.10 A structural block diagram of a key-value pair generation module in an embodiment;

[0079] Fig.11 Another structural block diagram of a key-value pair generation module in one embodiment;

[0080] Fig.12 is a structural block diagram of a cache node unit in one embodiment;

[0081] Fig.13 is another structural block diagram of a cache node unit in one embodiment;

[0082] Fig.14Another structural block diagram of a query request asynchronous processing device in one embodiment;

[0083] Fig.15 Another structural block diagram of a query request asynchronous processing device in one embodiment;

[0084] Fig.16 FIG. 4 is a block diagram of the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0085] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0086] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which the present disclosure belongs. The terms used herein in the specification of the present disclosure are only for the purpose of describing specific embodiments and are not intended to limit the present disclosure.

[0087] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described here can be implemented in an order other than those illustrated or described here. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. On the contrary, they are only examples of devices and methods consistent with some aspects of the present disclosure as detailed in the attached claims. The terms "include", "comprise" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, product or device including a series of elements includes not only those elements, but also other elements that are not explicitly listed, or also includes elements inherent to such process, method, product or device. In the absence of more restrictions, it is not excluded that there are other identical or equivalent elements in the process, method, product or device including the elements. For example, if the words first, second, etc. are used to represent the name, they do not represent any specific order.

[0088] When used herein, the singular forms "a", "an", and "said / the" may also include plural forms, unless the context clearly indicates otherwise. It should also be understood that the terms "include / comprise" or "have" and the like specify the presence of stated features, wholes, steps, operations, components, parts, or combinations thereof, but do not exclude the possibility of the presence or addition of one or more other features, wholes, steps, operations, components, parts, or combinations thereof. At the same time, in this specification, the term "and / or" includes any and all combinations of the relevant listed items.

[0089] The query request asynchronous processing method provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown. Among them, the query end 102 communicates with the asynchronous query system 104 through the network. Among them, the asynchronous query server can be implemented with an independent server or a server cluster composed of multiple servers. The data storage system can store the data that the asynchronous query server needs to process. The data storage system can be integrated on the asynchronous query server, or it can be placed on the cloud or other network servers. The cache library can store the historical query data of the asynchronous query server. The cache library can be integrated on the asynchronous query server, or it can be placed on the cloud or other network servers. The cache library can also be a multi-node cache library.

[0090] In some embodiments of the present disclosure, Figure 2 As shown, a query request asynchronous processing method is provided, which is applied to Figure 1 The asynchronous query system in is used as an example to illustrate the process, which includes the following steps:

[0091] Step S10: receiving a query request sent by the query end, generating a query task based on the query request and assigning a task number, and writing the query task into a query queue.

[0092] Specifically, in this embodiment, the query end may generally refer to the device end on the user side that issues the query request, for example, it may be a user terminal or other device that provides services. After receiving the query request initiated by the query end, a query task is established according to the query request. The established query tasks are assigned a unique task number, and finally the query tasks with the task number are written into the query queue. In this embodiment, the query queue may generally refer to a linear table, which is stored in a continuous storage space and is managed by two pointers. The query task enters the queue from the back end of the query queue and exits the queue from the front end of the query queue.

[0093] Step S20: Use the distributed execution nodes to obtain the current query task from the query queue, and initiate a query to the cache library based on the current query task; the cache library includes a number of cache nodes, and the cache library is used to store a first key-value pair including historical query tasks, and the first key-value pair is a key-value pair of a structured query language of the query task and a query result set.

[0094] Specifically, in this embodiment, the number of distributed execution nodes is greater than or equal to 2. A traditional single node may be a single physical machine that includes all services and databases. Compared to a traditional single node, a distributed execution node can generally be understood as a plurality of nodes that together include all services and databases. Each distributed execution node contains its own processor and memory, and each has an independent function of processing data. Usually, they are equal in status, without distinction of primary and secondary, and can work autonomously, and can use shared communication lines to transmit information and coordinate task processing. At the same time, different distributed execution nodes can execute several subtasks respectively to complete a large overall task together.

[0095] In this embodiment, query tasks are pulled from the query queue respectively through multiple distributed execution nodes, and each distributed execution node initiates a query to the cache library based on its own current query task. The cache library includes a number of cache nodes, which can expand the cache of a single machine to multiple service devices. The cache library can be used to store the first key-value pairs of historical query tasks. A key-value pair generally refers to an organizational form of data storage, and the "value" corresponding to the "key" can be obtained based on the query "key". In this embodiment, the first key-value pair includes the structured query language of the query task and the query result set, that is, the structured query language of the query task is used as the "key", and the query result set of the query task is used as the "value".

[0096] Structured Query Language can represent the entire SQL statement, for example:

[0097] select col1, col2 from table1 where condition.

[0098] It can also be the query language of the query engine in JSON format, for example:

[0099] query=[{"queryType":"scan","dataSource":{"type":"table","name":"table2"},"intervals":{"type":"intervals","inte rvals":["2021-09-08T08:23:32.096Z / 2021-09-18T15:36:27.903Z"]},"descending":false,"granularity":{"type":"all"}}.

[0100] After the subsequent distributed execution nodes receive the same structured query language, they can directly obtain the "value" with the structured query language as the "key" through the cache library, that is, the query result set of the query task.

[0101] Step S30: According to the structured query language of the current query task, query whether there is a target key-value pair in the first key-value pair in the cache library to obtain a query result; the target key-value pair is the first key-value pair in the cache library that has the same structured query language as the current query task.

[0102] Specifically, after receiving the query task, according to the structured query language of the current query task, the distributed execution node can query the cache library to see whether there is a target key-value pair in the first key-value pair in the cache library, and the target key-value pair is the first key-value pair in the cache library that has the same structured query language as the current query task. After searching the target key-value pair in the cache library, the query result of this embodiment can be that the target key-value pair exists in the cache library, or it can be that the target key-value pair does not exist in the cache library.

[0103] Step S40: Based on the query result and the task number of the current query task, a second key-value pair of the current query task is generated accordingly, and the second key-value pair is stored in the cache library for reading by the query end; the second key-value pair is a key-value pair of the task number of the query task and the query result set.

[0104] Specifically, according to the query result of the above step S30, the distributed execution node performs different task operations according to different query results, and finally generates the second key-value pair of the current query task according to the query result and the task number of the current query task. Here, the "key" of the second key-value pair is the task number of the query task, and the "value" of the second key-value pair is the query result set of the query task. The second key-value pair is also stored in the cache library, so that the query end can quickly obtain the query result set of the query task according to the task number of the query task.

[0105] In the above-mentioned asynchronous processing method of query requests, query tasks are obtained through distributed execution nodes for query, so that the response results of query tasks are more efficient and the waiting time of the query end is reduced; at the same time, the cache library can include multiple cache nodes, breaking through the cache limitation of the server's single-machine memory; in addition, by connecting with the query end to receive query requests, decoupling is achieved from the service system of the query end, and there is no need to add or modify asynchronous query modules in the original service system of the query end; and it can serve multiple query ends at the same time, and it only needs to accept the query request sent by the query end to implement the query, which has wider adaptability, can well support high concurrency and high availability, and is easy to expand horizontally.

[0106] In some embodiments of the present disclosure, step S20 includes:

[0107] Based on the query task, the distributed execution node obtains the corresponding query task according to a preset rule.

[0108] Specifically, when a distributed execution node pulls a query task from a query queue, each distributed execution node obtains a query task corresponding to itself, so that the query tasks in the query queue can be evenly distributed among the distributed execution nodes, avoiding congestion caused by a certain distributed execution node.

[0109] The task number of the query task is calculated according to the number of the distributed execution nodes according to the preset algorithm. For example, if the number of distributed execution nodes is 3, it can be set to take the remainder of 3 in the order in which each query task enters the query queue, and the remainder result will only have three values: 0, 1, and 2, and 0, 1, and 2 are respectively assigned to the three distributed execution nodes. Each distributed execution node pulls the query task corresponding to its own number.

[0110] In this embodiment, query actions are executed by distributed execution tasks, which shares the query pressure. Especially in high concurrency scenarios, query actions can be executed stably and quickly. At the same time, the distributed execution nodes pull the query tasks corresponding to themselves according to the set algorithm. The whole process is orderly and efficient, reducing the risk of query congestion.

[0111] In some embodiments of the present disclosure, step S40 includes:

[0112] Step S41: When the query result is that the target key-value pair exists in the cache library, a second key-value pair is generated according to the target key-value pair and the task number of the current query task, and the second key-value pair is stored in the cache library for reading by the query end.

[0113] Specifically, when the distributed execution node searches for the target key-value pair in the cache library, if the query result is that the target key-value pair exists in the cache library, the query result set of the current query task can be directly obtained from the cache library. Then, a second key-value pair is generated according to the task number of the current query task and the query result set, and stored in the cache library.

[0114] This embodiment preferentially queries the cache library through distributed execution nodes. When the first key-value pair of the current query task exists in the cache library, that is, the current query task is a repeated query task, the query result set can be obtained directly from the cache library without querying the database. At the same time, the second key-value pair is generated according to the current query task and stored in the cache library, so that the query end can obtain the query result set of the query task according to the task number.

[0115] In some embodiments of the present disclosure, Figure 3 As shown, step S41 includes:

[0116] Step S412: Acquire the query result set of the target key-value pair, and generate a second key-value pair of the current query task according to the task number of the current query task and the query result set of the target key-value pair.

[0117] Specifically, the query result set of the current query task can be obtained by finding the target key-value pair of the current query task in the cache library. The second key-value pair is generated according to the task number of the current query task and the query result set of the target key-value pair.

[0118] Step S414: split the second key value of the current query task based on a hash algorithm to generate a number of data fragments.

[0119] Specifically, the second key-value pair is split into a plurality of data shards, so that the second key-value pair can be stored in a dispersed manner on multiple cache nodes.

[0120] Step S416: Store the data shards on several cache nodes of the cache library respectively.

[0121] Specifically, several data shards are stored in several cache nodes of the cache library respectively, so that the number of data shards stored in each storage node is as close as possible.

[0122] In some embodiments of the present disclosure, step S40 includes:

[0123] Step S42: When the query result is that the target key-value pair does not exist in the cache library, query the database to obtain the query result set of the current query task, generate the first key-value pair and the second key-value pair of the current query task, and store them in the cache library for the query end to read.

[0124] Specifically, when the target key-value pair of the current query task does not exist in the cache library, the distributed execution node needs to initiate a query to the database to obtain a query result set and generate a first key-value pair and a second key-value pair of the current query task.

[0125] This embodiment preferentially queries the cache library through distributed execution nodes. When the first key-value pair of the current query task does not exist in the cache library, that is, the current query task is a new query task, it is necessary to query the database; at the same time, the first key-value pair and the second key-value pair are generated according to the query result set obtained from the database and the current query task and stored in the cache library, which is convenient for the query end to repeat the query and obtain the query result set of the query task according to the task number.

[0126] In some embodiments of the present disclosure, Figure 4 As shown, step S42 includes:

[0127] Step S421: Initiate a query to a database according to the structured query language of the current query task, and receive a query result set of the current query task returned by the database.

[0128] Step S423: Generate a first key-value pair according to the structured query language of the current query task and the query result set;

[0129] Step S425: Generate a second key-value pair according to the task number of the current query task and the query result set;

[0130] Step S427: splitting the first key-value pair and the second key value of the current query task respectively based on a hash algorithm to generate a plurality of data fragments;

[0131] Step S429: Store the data shards on several cache nodes of the cache library respectively.

[0132] Specifically, the distributed execution node initiates a query to the database according to the structured query language of the current query task to obtain the query result set of the current query task. Finally, the generated first key-value pair and the second key-value pair are stored in the cache library. When storing, the specific steps are the same as the above step S41, which will not be repeated here.

[0133] In some embodiments of the present disclosure, Figure 5 As shown, the aforementioned step S416 or step S429 includes:

[0134] Step A10: The data shards are calculated according to a preset algorithm to output node numbers, and the node numbers are used to correspond one-to-one with the cache nodes.

[0135] Specifically, the divided data shards are hashed so that the node number can be calculated based on each data shard. The node number corresponds to the cache node one-to-one. The node number of each data shard generated by the first key-value pair or the second key-value pair is unique. Usually, the number of data shards generated by each first key-value pair or the second key-value pair can be set to be the same as the number of cache nodes. For example, the cache library includes 3 cache nodes. Therefore, the first key-value pair or the second key-value pair to be saved can be split into 3 groups of data shards, and these 3 groups of data shards are calculated to obtain 3 different node numbers, and each node number corresponds to a cache node.

[0136] Step A20: Store the data shards in corresponding cache nodes according to the node numbers.

[0137] Specifically, according to the node number of each data shard, the data shard is stored in the corresponding cache node according to the node number.

[0138] This embodiment uses data sharding storage and multi-node cache libraries to store the query result set of the query task in the memory of multiple service devices. When a complete data needs to be read, compared with reading a file on a single node, the data sharding method supports more visitors and can increase the number of concurrency.

[0139] In some embodiments of the present disclosure, Figure 6 As shown, the aforementioned step S416 or step S429 also includes:

[0140] Step A30: Generate a shard copy of the data shard, and store the shard copy in a cache node that does not correspond to the node number of the data shard.

[0141] Specifically, each data shard is backed up to generate a shard copy. When storing, the shard copy of the data shard needs to be stored on different cache nodes respectively with the data shard. For example, the cache library includes a first cache node, a second cache node, and a third cache node; the first key-value pair or the second key-value pair to be saved is split into a first data shard, a second data shard, and a third data shard. After calculating these three groups of data shards, three different node numbers are obtained, and each node number corresponds to a cache node. At the same time, a shard copy of each data shard is also stored on a cache node that does not correspond to the node number of the data shard. It can be one of the following situations:

[0142] The first cache node stores shard copies of the first data shard and the second data shard;

[0143] The second cache node stores shard copies of the second data shard and the third data shard;

[0144] The third cache node stores a third data shard and a shard copy of the first data shard.

[0145] This embodiment combines data sharding and shard replicas to not only support more visitors and increase the number of concurrent users, but also enables visitors to query the full amount of data even when a cache node is down, thereby improving fault tolerance and high availability. The above characteristics can be better achieved by adjusting the number of shard replicas, the number of data shards, the number of cache nodes, etc.

[0146] In some embodiments of the present disclosure, the step S40 for the query end to read includes:

[0147] A service interface is provided to the query end so that the query end can query the corresponding query result set in the cache based on the task number of the query task.

[0148] Specifically, combined Figure 7 , by providing an asynchronous service interface to the query end, the query end can actively read the query result set of the query task from the cache library. It can be set that after the query end initiates a query request, the query end actively reads the query result set from the cache library after a set time. According to the aforementioned steps of storing data shards in the cache library, it can be known that when reading data shards from the cache library to form complete data, it is also necessary to obtain the node number based on the query task calculation based on the hash algorithm, so that the query end can read data from the corresponding cache node. Here, I will not go into details.

[0149] This embodiment provides a service interface for the query end to read data from the cache library, which can reduce the change of the database architecture of the query end, make the query and reading process more stable, and reduce the development cost.

[0150] In some embodiments of the present disclosure, the step S40 for the query end to read includes:

[0151] The distributed execution node sends the acquired query result set of the current query task to the query end.

[0152] Specifically, when the distributed execution node obtains the query result set of the current query task from the cache or database, the distributed execution node actively pushes it to the query end. Correspondingly, the query end needs to monitor the push interface.

[0153] This embodiment improves the timeliness of query response by actively pushing the query result set of the current query task through the distributed execution node.

[0154] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0155] Based on the same inventive concept, the embodiment of the present disclosure also provides a query request asynchronous processing device for implementing the query request asynchronous processing method involved above. The implementation solution provided by the device to solve the problem is similar to the implementation solution recorded in the above method, so the specific limitations in the one or more query request asynchronous processing device embodiments provided below can refer to the limitations of the query request asynchronous processing method above, and will not be repeated here.

[0156] The device may include a system (including a distributed system), software (application), module, component, server, client, etc. using the method described in the embodiments of this specification and a device in combination with necessary implementation hardware. Based on the same innovative concept, the device in one or more embodiments provided in the embodiments of the present disclosure is as described in the following embodiments. Since the implementation scheme and method for solving the problem of the device are similar, the implementation of the specific device in the embodiments of this specification can refer to the implementation of the aforementioned method, and the repetitions will not be repeated. As used below, the term "unit" or "module" can implement a combination of software and / or hardware for predetermined functions. Although the device described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceived.

[0157] In some embodiments of the present disclosure, Figure 8 As shown, a query request asynchronous processing device is provided, and the device Z00 can be the aforementioned terminal, or a server, or a module, component, device, unit, etc. integrated in the terminal. The device may include:

[0158] The query queue module Z10 is used to receive the query request sent by the query end, generate a query task based on the query request and assign a task number, and write the query task into the query queue;

[0159] The distributed execution node module Z20 is used to obtain the current query task from the query queue using the distributed execution node, and initiate a query to the cache library based on the current query task; the cache library includes a plurality of cache nodes, and the cache library is used to store a first key-value pair including a historical query task, wherein the first key-value pair is a key-value pair of a structured query language of the query task and a query result set;

[0160] The cache query module Z30 is used to query whether there is a target key-value pair in the first key-value pair in the cache according to the structured query language of the current query task, and obtain a query result; the target key-value pair is the first key-value pair in the cache that has the same structured query language as the current query task;

[0161] The key-value pair generation module Z40 is used to generate a second key-value pair of the current query task according to the query result and the task number of the current query task, and store the second key-value pair in the cache library for reading by the query end.

[0162] In some embodiments of the present disclosure, Fig. 9 As shown, the distributed execution node module Z20 includes:

[0163] The acquisition unit Z22 is used for the distributed execution node to acquire the corresponding query task according to a preset rule based on the query task.

[0164] In some embodiments of the present disclosure, the key-value pair generation module Z40 is used to generate a second key-value pair based on the target key-value pair and the task number of the current query task when the query result is that the target key-value pair exists in the cache library, and store the second key-value pair in the cache library for reading by the query end.

[0165] In some embodiments of the present disclosure, Fig.10 As shown, the key-value pair generation module Z40 includes:

[0166] A second key-value pair unit Z41 is used to obtain a query result set of the target key-value pair, and generate a second key-value pair of the current query task according to the task number of the current query task and the query result set of the target key-value pair;

[0167] A data sharding unit Z43 is used to perform data splitting on the second key value of the current query task based on a hash algorithm to generate a plurality of data shards;

[0168] The cache node unit Z45 is used to store the data fragments on several cache nodes of the cache library respectively.

[0169] In some embodiments of the present disclosure, the key-value pair generation module Z40 is used to query the database to obtain the query result set of the current query task when the query result is that the target key-value pair does not exist in the cache library, generate the first key-value pair and the second key-value pair of the current query task, and store them in the cache library for reading by the query end.

[0170] In some embodiments of the present disclosure, Fig.11 As shown, the key-value pair generation module Z40 includes:

[0171] A database query unit Z47, configured to initiate a query to a database according to the structured query language of the current query task, and receive a query result set of the current query task returned by the database;

[0172] A first key-value pair unit Z49, configured to generate a first key-value pair according to the structured query language of the current query task and the query result set;

[0173] A second key-value pair unit Z41, used to generate a second key-value pair according to the task number of the current query task and the query result set;

[0174] A data sharding unit Z43 is used to perform data splitting on the first key-value pair and the second key value of the current query task based on a hash algorithm to generate a plurality of data shards;

[0175] The cache node unit Z45 stores the data fragments on several cache nodes of the cache library respectively.

[0176] In some embodiments of the present disclosure, Fig.12 As shown, the cache node unit Z45 includes:

[0177] A node numbering subunit Z451 is used for calculating and outputting a node number for the data shard according to a preset algorithm, and the node number is used to correspond one-to-one with the cache node;

[0178] The storage subunit Z453 is used to store the data fragments to the corresponding cache nodes according to the node numbers.

[0179] In some embodiments of the present disclosure, Fig.13 As shown, the cache node unit Z45 also includes:

[0180] The replica subunit Z455 is used to generate a shard replica of the data shard, and store the shard replica in a cache node that does not correspond to the node number of the data shard.

[0181] In some embodiments of the present disclosure, Fig.14 As shown, the device Z00 also includes:

[0182] The service interface unit Z50 is used to provide a service interface to the query end, so that the query end can query the corresponding query result set in the cache based on the task number of the query task.

[0183] In some embodiments of the present disclosure, Fig.15 As shown, the device Z00 also includes:

[0184] The sending unit Z60 is used to instruct the distributed execution node to send the acquired query result set of the current query task to the query end.

[0185] Each module in the above-mentioned query request asynchronous processing device can be implemented in whole or in part by software, hardware and a combination thereof. Each of the above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory in the computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules. It should be noted that the division of modules in the embodiments of the present disclosure is schematic and is only a logical function division. There may be other division methods in actual implementation.

[0186] Based on the description of the embodiment of the asynchronous processing method for query requests, in another embodiment provided by the present disclosure, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Fig.16 As shown. The computer device includes a processor, a memory and a network interface connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a method for asynchronous processing of query requests is implemented.

[0187] Those skilled in the art will understand that Fig.16 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0188] Based on the description of the embodiment of the asynchronous processing method for query requests, in another embodiment provided in the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0189] Based on the description of the embodiment of the asynchronous processing method for query requests, in another embodiment provided in the present disclosure, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0190] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties.

[0191] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0192] In the description of this specification, the description with reference to the terms "some embodiments", "other embodiments", "ideal embodiments", etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic descriptions of the above terms do not necessarily refer to the same embodiment or example.

[0193] It is understandable that the various embodiments of the above method in this specification are described in a progressive manner, and the same / similar parts between the various embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. For related parts, refer to the description of other method embodiments.

[0194] The technical features of the above-mentioned embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features of the above-mentioned embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0195] The above-described embodiments only express several implementation methods of the present disclosure, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the patent application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present disclosure, and these all belong to the protection scope of the present disclosure. Therefore, the protection scope of the patent of the present disclosure shall be subject to the attached claims.

Claims

1. A method for asynchronously processing a query request, characterized in that: The method comprises: Receive a query request sent by a query end, generate a query task based on the query request and assign a task number, and write the query task into a query queue; The distributed execution node is used to obtain the current query task from the query queue, and a query is initiated to the cache library based on the current query task; the cache library includes a plurality of cache nodes, and the cache library is used to store a first key-value pair including a historical query task, wherein the first key-value pair is a key-value pair of a structured query language of the query task and a query result set; According to the structured query language of the current query task, query whether there is a target key-value pair in the first key-value pair in the cache library to obtain a query result; the target key-value pair is the first key-value pair in the cache library that has the same structured query language as the current query task; According to the query result and the task number of the current query task, a second key-value pair of the current query task is generated accordingly, and the second key-value pair is stored in the cache library for reading by the query end; the second key-value pair is a key-value pair of the task number of the query task and the query result set.

2. The method according to claim 1, characterized in that: The distributed execution node is used to obtain the current query task from the query queue, and a query is initiated to the cache based on the current query task; the cache includes a plurality of cache nodes, and the cache is used to store a first key-value pair including a historical query task, wherein the first key-value pair is a structured query language of the query task and a key-value pair of a query result set including: Based on the query task, the distributed execution node obtains the corresponding query task according to a preset rule.

3. The method according to claim 1, characterized in that The step of generating a second key-value pair of the current query task according to the query result and the task number of the current query task, and storing the second key-value pair in the cache library for reading by the query end includes: When the query result shows that the target key-value pair exists in the cache, a second key-value pair is generated according to the target key-value pair and the task number of the current query task, and the second key-value pair is stored in the cache for reading by the query end.

4. The method according to claim 3, characterized in that When the query result indicates that the target key-value pair exists in the cache, generating a second key-value pair according to the target key-value pair and the task number of the current query task, and storing the second key-value pair in the cache for reading by the query end includes: Acquire a query result set of the target key-value pair, and generate a second key-value pair of the current query task according to the task number of the current query task and the query result set of the target key-value pair; Splitting the second key value of the current query task based on a hash algorithm to generate a plurality of data fragments; The data shards are stored in several cache nodes of the cache library respectively.

5. The method according to claim 1, characterized in that The step of generating a second key-value pair of the current query task according to the query result and the task number of the current query task, and storing the second key-value pair in the cache library for reading by the query end includes: When the query result is that the target key-value pair does not exist in the cache library, the database is queried to obtain the query result set of the current query task, the first key-value pair and the second key-value pair of the current query task are generated, and stored in the cache library for reading by the query end.

6. The method according to claim 5, characterized in that When the query result is that the target key-value pair does not exist in the cache library, querying the database to obtain the query result set of the current query task, generating the first key-value pair and the second key-value pair of the current query task, and storing them in the cache library for the query end to read includes: Initiate a query to a database according to the structured query language of the current query task, and receive a query result set of the current query task returned by the database; Generate a first key-value pair according to the structured query language of the current query task and the query result set; Generate a second key-value pair according to the task number of the current query task and the query result set; Based on the hash algorithm, the first key-value pair and the second key value of the current query task are respectively split into data to generate a plurality of data fragments; The data shards are stored in several cache nodes of the cache library respectively.

7. The method according to claim 4 or 6, characterized in that: The storing the data shards respectively on a plurality of cache nodes of the cache library comprises: The data shards calculate and output node numbers according to a preset algorithm, and the node numbers are used to correspond one-to-one with the cache nodes; The data fragments are stored in the corresponding cache nodes according to the node numbers.

8. The method according to claim 7, characterized in that The storing the data shards respectively on a plurality of cache nodes of the cache library further comprises: Generate a shard copy of the data shard, and store the shard copy in a cache node that does not correspond to the node number of the data shard.

9. The method according to claim 1, characterized in that: The providing the query end with the information to read includes: A service interface is provided to the query end so that the query end can query the corresponding query result set in the cache based on the task number of the query task.

10. The method according to claim 1, characterized in that The providing the query end with the information to read includes: The distributed execution node sends the acquired query result set of the current query task to the query end.

11. A query request asynchronous processing device, characterized in that: The device comprises: A query queue module, used to receive a query request sent by a query end, generate a query task based on the query request and assign a task number, and write the query task into a query queue; A distributed execution node module, used to obtain the current query task from the query queue using a distributed execution node, and to initiate a query to a cache based on the current query task; the cache includes a plurality of cache nodes, and the cache is used to store a first key-value pair including a historical query task, wherein the first key-value pair is a key-value pair of a structured query language of the query task and a query result set; A cache query module, used to query whether there is a target key-value pair in the first key-value pair in the cache according to the structured query language of the current query task, and obtain a query result; the target key-value pair is the first key-value pair in the cache that has the same structured query language as the current query task; The key-value pair generation module is used to generate a second key-value pair of the current query task according to the query result and the task number of the current query task, and store the second key-value pair in the cache library for reading by the query end.

12. The device according to claim 11, characterized in that The distributed execution node module includes: The acquisition unit is used for the distributed execution node to acquire the corresponding query task according to a preset rule based on the query task.

13. The device according to claim 11, characterized in that The key-value pair generation module is used to generate a second key-value pair based on the target key-value pair and the task number of the current query task when the query result shows that the target key-value pair exists in the cache library, and store the second key-value pair in the cache library for reading by the query end.

14. The device according to claim 13, characterized in that The key-value pair generation module includes: A second key-value pair unit, used to obtain a query result set of the target key-value pair, and generate a second key-value pair of the current query task according to the task number of the current query task and the query result set of the target key-value pair; A data sharding unit, used for performing data splitting on the second key value of the current query task based on a hash algorithm to generate a plurality of data shards; The cache node unit is used to store the data fragments on several cache nodes of the cache library respectively.

15. The device according to claim 11, characterized in that The key-value pair generation module is used to query the database to obtain the query result set of the current query task when the query result is that the target key-value pair does not exist in the cache library, generate the first key-value pair and the second key-value pair of the current query task, and store them in the cache library for the query end to read.

16. The device according to claim 15, characterized in that The key-value pair generation module includes: A database query unit, configured to initiate a query to a database according to the structured query language of the current query task, and receive a query result set of the current query task returned by the database; A first key-value pair unit, configured to generate a first key-value pair according to the structured query language of the current query task and the query result set; A second key-value pair unit, used to generate a second key-value pair according to the task number of the current query task and the query result set; A data sharding unit, configured to perform data splitting on the first key-value pair and the second key value of the current query task based on a hash algorithm to generate a plurality of data shards; The cache node unit stores the data slices on several cache nodes of the cache library respectively.

17. The device according to claim 14 or 16, characterized in that The cache node unit comprises: A node numbering subunit, used for calculating and outputting a node number for the data shard according to a preset algorithm, wherein the node number is used for one-to-one correspondence with the cache node; The storage subunit is used to store the data fragments in the corresponding cache nodes according to the node numbers.

18. The device according to claim 17, characterized in that The cache node unit also includes: The replica subunit is used to generate a shard replica of the data shard, and store the shard replica in a cache node that does not correspond to the node number of the data shard.

19. The device according to claim 11, characterized in that The device also includes: The service interface unit is used to provide a service interface to the query end, so that the query end can query the corresponding query result set in the cache based on the task number of the query task.

20. The device according to claim 11, characterized in that The device also includes: A sending unit is used to instruct the distributed execution node to send the acquired query result set of the current query task to the query end.

21. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 10 are implemented.

22. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 10 are implemented.

23. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 10 are implemented.

Images