Data query method and device, electronic equipment and storage medium

Abstract

The present disclosure provides a data query method and device, electronic equipment and storage medium, relates to the technical field of cloud computing, cloud storage and big data, and specific implementation schemes are as follows: a plurality of indexes matched with a query condition are queried from a distributed index library, and the plurality of indexes are written into a distributed message queue in a distributed cluster; a first index without a mark is acquired from the distributed message queue, and first business data matched with the first index is queried from a distributed database; and in response to the first business data being queried, the first index in the distributed message queue is marked through the distributed cluster. Thus, the situation that the business data is repeatedly queried and acquired due to the acquisition of indexes with marks from the distributed message queue when a device is down or an instance is migrated can be avoided, so that the query efficiency of data can be improved, and resource occupation can be reduced.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of artificial intelligence, in particular to the technical field of cloud computing, cloud storage and big data, and more particularly to a data query method and device, an electronic device and a storage medium. BACKGROUND

[0002] With the continuous development of Internet technology and information technology, the business data generated by enterprises is increasingly expanding. Massive business data can be stored in a distributed database, for example, structured business data (such as business data in the form of key-value, abbreviated as KV) can be stored in a distributed database, which can also be referred to as a KV library.

[0003] In order to improve the query efficiency of business data, how to quickly query the required business data from the distributed database is very important. SUMMARY

[0004] The present disclosure provides a data query method, device, electronic device and storage medium.

[0005] According to an aspect of the present disclosure, a data query method is provided, comprising:

[0006] obtaining a query condition and querying a plurality of indexes matching the query condition from a distributed index library; wherein the distributed index library is used to store indexes corresponding to each business data in a distributed database;

[0007] writing the plurality of indexes into a distributed message queue in a distributed cluster;

[0008] obtaining a first index without a mark from the distributed message queue, and querying a first business data matching the first index from the distributed database;

[0009] In response to querying the first business data, marking the first index in the distributed message queue through the distributed cluster.

[0010] According to another aspect of the present disclosure, a data query device is provided, comprising:

[0011] a query module configured to obtain a query condition and query a plurality of indexes matching the query condition from a distributed index library; wherein the distributed index library is used to store indexes corresponding to each business data in a distributed database;

[0012] a writing module configured to write the plurality of indexes into a distributed message queue in a distributed cluster;

[0013] an acquisition module, configured to acquire an unmarked first index from the distributed message queue, and query first service data matching the first index from the distributed database;

[0014] a sending module, configured to mark the first index in the distributed message queue by the distributed cluster in response to the query of the first service data.

[0015] According to yet another aspect of the present disclosure, an electronic device is provided, comprising:

[0016] at least one processor; and

[0017] a memory in communication with the at least one processor; wherein

[0018] the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the data query method according to the aspect of the present disclosure.

[0019] According to still another aspect of the present disclosure, a non-transitory computer-readable storage medium of computer instructions is provided, the computer instructions being used to cause the computer to perform the data query method according to the aspect of the present disclosure.

[0020] According to yet another aspect of the present disclosure, a computer program product is provided, comprising a computer program which, when executed by a processor, implements the data query method according to the aspect of the present disclosure.

[0021] It should be understood that the content described in this section is not intended to identify key or important features of the embodiments of the present disclosure, nor is it used to limit the scope of the present disclosure. Other features of the present disclosure will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0022] The accompanying drawings are used to better understand the present scheme, and do not limit the present disclosure. Among them:

[0023] Figure 1 a data query and transmission process schematic diagram in the related art;

[0024] Figure 2 a flowchart of the data query method provided by the first embodiment of the present disclosure;

[0025] Figure 3 a flowchart of the data query method provided by the second embodiment of the present disclosure;

[0026] Figure 4 a flowchart of the data query method provided by the third embodiment of the present disclosure;

[0027] Figure 5 A flowchart of a data query method provided by Embodiment Four of the present disclosure is shown in FIG. 4.

[0028] Figure 6 A flowchart of a data query method provided by Embodiment Five of the present disclosure is shown in FIG. 5.

[0029] Figure 7 An implementation principle diagram of any embodiment of the present disclosure is shown in FIG. 6.

[0030] Figure 8 A structure diagram of a data query apparatus provided by Embodiment Six of the present disclosure is shown in FIG. 7.

[0031] Figure 9 A schematic block diagram of an example electronic device that can be used to implement embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0032] Exemplary embodiments of the present disclosure are described below with reference to the accompanying drawings, which include various details of the embodiments of the present disclosure to assist in understanding, and should be considered as merely exemplary. Thus, those skilled in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Also, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0033] At present, for a KV type distributed database (DB for short), the business data stored in the distributed database can be queried through the rowkey of each business data. Specifically, the query method of the business data can be as shown in FIG. 1. When scanning each page data in the distributed database, the page data can be scanned according to the rowkey returned by the previous page data. Figure 1

[0034] However, the above method can only acquire data in single thread, and each data in the distributed database can only be scanned sequentially, so the data query efficiency is low, and the QPS (Queries-per-second) of data delivery cannot be improved.

[0035] Moreover, if instance migration or machine downtime occurs during data scanning, data transmission will be interrupted, and a retry mechanism needs to be performed to scan the data from the beginning to the end and send the scanned data to other resource parties, which is low in efficiency.

[0036] That is, the following problems exist in the prior art: Figure 1

[0037] ​​First, when a large amount of business data is obtained from a distributed database and the obtained business data is transmitted from one business department to another business department, the QPS is uncontrollable, and is large or small;

[0038] Second, when the data volume of the business data to be queried is large, not only is the data transmission time-consuming, but if the machine responsible for transmission migrates or crashes, all business data needs to be retransmitted, resulting in low transmission efficiency of business data, repeated transmission of business data, and waste of resources.

[0039] Therefore, in view of at least one of the above problems, the present disclosure provides a data query method, device, electronic equipment and storage medium.

[0040] The data query method, device, electronic equipment and storage medium of the embodiments of the present disclosure are described below with reference to the accompanying drawings.

[0041] Figure 2 The flowchart of the data query method provided by Embodiment One of the present disclosure.

[0042] Embodiments of the present disclosure take the data query method configured in the data query device as an example, which can be applied to any electronic equipment to enable the electronic equipment to perform data query functions.

[0043] Among them, the electronic equipment can be any device with computing power, such as a computer, a mobile terminal, a server, etc. The mobile terminal can be, for example, a vehicle-mounted device, a mobile phone, a tablet computer, a personal digital assistant, a wearable device, etc. Hardware devices with various operating systems, touch screens and / or display screens.

[0044] As shown in Figure 2 The data query method can include the following steps:

[0045] Step 201, obtain a query condition and query a plurality of indexes matching the query condition from a distributed index library; wherein the distributed index library is used to store indexes corresponding to each business data in the distributed database.

[0046] In the embodiments of the present disclosure, the query condition can be provided by a relevant person. For example, the query condition can be used to indicate which period of business data to query, or to indicate which business data corresponding to the business to query, or to indicate which field of business data to query, etc. The present disclosure does not limit this.

[0047] In the embodiments of the present disclosure, a plurality of indexes matching the query condition can be queried from the distributed index library according to the query condition. The distributed index library is used to store indexes corresponding to each business data in the distributed database.

[0048] As an example, a database in a distributed database in a KV type (KV database for short) is taken as an example, that is, the index of business data can be rowkey.

[0049] In step 202, multiple indexes are written into a distributed message queue in a distributed cluster.

[0050] In the embodiment of the present disclosure, multiple indexes matched with the query condition can be written into a distributed message queue in a distributed cluster.

[0051] In step 203, an unmarked first index is obtained from the distributed message queue, and first business data matched with the first index is obtained from the distributed database.

[0052] In the embodiment of the present disclosure, an unmarked index (denoted as a first index in the present disclosure) can be obtained from the distributed message queue, and business data (denoted as first business data in the present disclosure) matched with the first index can be queried from the distributed database.

[0053] In step 204, in response to the first business data being queried, the first index in the distributed message queue is marked through the distributed cluster.

[0054] In the embodiment of the present disclosure, in the case where the first business data is queried or obtained, the first index in the distributed message queue can be marked through the distributed cluster. For example, a confirmation response can be sent to a service node in the distributed cluster where the distributed message queue is located, and correspondingly, the service node can mark the first index in the distributed message queue after receiving the confirmation response.

[0055] The data query method of the embodiment of the present disclosure queries multiple indexes matched with a query condition from a distributed index database, and writes the multiple indexes into a distributed message queue in a distributed cluster; obtains an unmarked first index from the distributed message queue, and queries first business data matched with the first index from a distributed database; and in response to the first business data being queried, marks the first index in the distributed message queue through the distributed cluster. Thus, only an unmarked index is obtained from the distributed message queue, and business data is queried from the distributed database according to the obtained unmarked index, and after the business data is successfully queried, the index corresponding to the business data that has been queried is marked, which can avoid the case that the index (that is, the index carrying the mark) corresponding to the business data that has been queried is obtained from the distributed message queue when the device is down or the instance is migrated, and the business data is repeatedly queried and obtained, so that the data query efficiency can be improved, and the resource occupation can be reduced.

[0056] It should be noted that in the technical solutions of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information are all carried out on the premise of obtaining the consent of the user, and all comply with the relevant legal regulations and do not violate public order and good customs.

[0057] To clearly illustrate the above-mentioned embodiments, the present disclosure further provides a data query method.

[0058] Figure 3 A flowchart of the data query method provided in Embodiment Two of the present disclosure.

[0059] As shown in Figure 3 the data query method can include the following steps:

[0060] Step 301: Obtain a query condition and query a plurality of indexes matching the query condition from a distributed index database; wherein the distributed index database is used to store indexes corresponding to each business data in a distributed database.

[0061] Step 302: Write the plurality of indexes into a distributed message queue in a distributed cluster.

[0062] The explanation and description of steps 301-302 can be referred to the related description in any embodiment of the present disclosure, which will not be repeated here.

[0063] Step 303: Obtain, by a plurality of first threads in at least one first instance, an unmarked first index from the distributed message queue.

[0064] In the embodiments of the present disclosure, the number of first instances (or first consumer instances) can be one or multiple, and the present disclosure does not limit this. Each first instance can include at least one thread (or consumer thread), which is referred to as a first thread in the present disclosure.

[0065] In the embodiments of the present disclosure, in order to improve the query efficiency of data, the unmarked first index can be obtained from the distributed message queue by a plurality of first threads in at least one first instance.

[0066] Step 304: Query, by each first thread, a first business data matching the first index obtained by the corresponding first thread from the distributed database.

[0067] In the embodiments of the present disclosure, the first business data matching the first index obtained by the corresponding first thread can be queried from the distributed database by each first thread. That is, for any first thread, the first business data matching the first index obtained by the first thread can be queried from the distributed database according to the first index obtained by the first thread.

[0068] Step 305, in response to querying the first service data, marking the first index in the distributed message queue through the distributed cluster.

[0069] The explanation of step 305 can refer to the related description in any embodiment of the present disclosure, which will not be repeated here.

[0070] The data query method of the embodiment of the present disclosure, through the multiple first threads in the at least one first instance, acquires the first index without marking from the distributed message queue, and through each first thread, queries the first service data matching the first index acquired by the corresponding first thread from the distributed database. Therefore, through the multiple threads in the at least one instance, the index without marking is acquired from the distributed index library, and the service data is queried from the distributed database according to the acquired index, which can further improve the query efficiency of data.

[0071] In order to clearly illustrate how the first index in the distributed message queue is marked through the distributed cluster in any embodiment of the present disclosure, the present disclosure further provides a data query method.

[0072] Figure 4 The flowchart of the data query method provided in the third embodiment of the present disclosure.

[0073] As shown in Figure 4 , the data query method can include the following steps:

[0074] Step 401, acquiring a query condition and querying multiple indexes matching the query condition from a distributed index library; wherein the distributed index library is used to store indexes corresponding to each service data in a distributed database.

[0075] Step 402, writing the multiple indexes into a distributed message queue in a distributed cluster.

[0076] Step 403, through multiple first threads in at least one first instance, acquiring the first index without marking from the distributed message queue.

[0077] Step 404, through each first thread, querying the first service data matching the first index acquired by the corresponding first thread from the distributed database.

[0078] The explanation of steps 401 to 404 can refer to the related description in any embodiment of the present disclosure, which will not be repeated here.

[0079] Step 405, in response to each first thread querying the first service data, combining the first service data queried by each first thread.

[0080] In the embodiments of the present disclosure, after each first thread queries or acquires the first business data, the first business data queried or acquired by each first thread can be combined (or assembled) to obtain combined first business data.

[0081] In step 406, the combined first business data is sent to the specified client.

[0082] The specified client can be a data receiving end or a data demand end.

[0083] In the embodiments of the present disclosure, the combined first business data can be sent to the specified client.

[0084] In step 407, in response to receiving the first confirmation response sent by the specified client, the first indexes acquired by each first thread in the distributed message queue are marked by the distributed cluster.

[0085] The first confirmation response is used to indicate that the combined first business data is received by the specified client.

[0086] In the embodiments of the present disclosure, in the case of receiving the first confirmation response sent by the specified client, it can be determined that the combined first business data is successfully received by the specified client, and at this time, the first indexes acquired by each first thread in the distributed message queue can be marked by the distributed cluster. For example, the confirmation response can be sent to the service node where the distributed message queue is located in the distributed cluster, and correspondingly, the first indexes acquired by each first thread in the distributed message queue can be marked by the service node after receiving the confirmation response.

[0087] The data query method of the embodiments of the present disclosure combines the first business data queried by each first thread in response to each first thread querying the first business data, sends the combined first business data to the specified client, marks the first indexes acquired by each first thread in the distributed message queue by the distributed cluster in response to receiving the first confirmation response sent by the specified client, and the first confirmation response is used to indicate that the combined first business data is received by the specified client. Therefore, only in the case where it is determined that the data receiving end (i.e., the specified client) has successfully received the business data, the indexes corresponding to the business data that has been successfully received in the distributed message queue are marked, which can avoid marking the indexes corresponding to the business data in the case where the data receiving end has not successfully received the business data, and if instance downtime or migration occurs, the business data will be lost.

[0088] In order to clearly illustrate how the first indexes in the distributed message queue are marked by the distributed cluster in any embodiment of the present disclosure, the present disclosure further provides a data query method.

[0089] Figure 5 A flowchart of the data query method provided in Embodiment Four of the present disclosure is shown.

[0090] As shown in Figure 5 , the data query method can include the following steps:

[0091] Step 501: Obtain a query condition, and query a plurality of indexes matching the query condition from a distributed index library; wherein the distributed index library is used to store indexes corresponding to each business data in a distributed database.

[0092] In any one of the embodiments of the present disclosure, the distributed database is used to store each business data and the index corresponding to each business data. The distributed index library is also used to periodically obtain a third index corresponding to a third business data from the distributed database according to a set period, and store the third index. The third business data is business data that has been updated and / or newly added in the distributed database.

[0093] Thus, the indexes of each business data in the distributed database can be timely and effectively updated to the distributed index library, so as to ensure the comprehensiveness of data query and avoid missing query of data.

[0094] Step 502: Write the plurality of indexes into a distributed message queue in a distributed cluster.

[0095] Step 503: Obtain an unmarked first index from the distributed message queue through a plurality of first threads in at least one first instance.

[0096] Step 504: Query, through each first thread, a first business data matching the first index obtained by the corresponding first thread from the distributed database.

[0097] The explanation and description of steps 501 to 504 can be referred to the related description in any one of the embodiments of the present disclosure, which will not be repeated here.

[0098] Step 505: In response to any first thread querying the first business data, send the first business data queried by any first thread to a specified client.

[0099] The specified client can be a data receiving end or a data demand end.

[0100] In the embodiments of the present disclosure, when any one of the first threads queries or obtains the first business data, the first business data queried or obtained by the first thread can be sent to the specified client.

[0101] Step 506, in response to receiving the second confirmation response sent by the specified client, marking the first index acquired by any first thread in the distributed message queue by the distributed cluster.

[0102] The second confirmation response is used to indicate that the specified client receives the first service data queried by any first thread.

[0103] In the embodiment of the present disclosure, in the case of receiving the second confirmation response sent by the specified client, it can be determined that the specified client has successfully received the first service data queried or acquired by any first thread. At this time, the first index acquired by any first thread in the distributed message queue can be marked by the distributed cluster. For example, a confirmation response can be sent to the service node where the distributed message queue is located in the distributed cluster. Correspondingly, after receiving the confirmation response, the service node can mark the first index acquired by any first thread in the distributed message queue.

[0104] In any one of the embodiments of the present disclosure, in the case that all indexes in the distributed message queue are marked, it indicates that the required query service data has been successfully acquired. In order to reduce resource occupation, the indexes in the distributed message queue can be deleted. Thus, the occupied storage resources can be released after the required service data is successfully acquired.

[0105] The data query method of the embodiment of the present disclosure, in response to any first thread querying the first service data, sends the first service data queried by any first thread to the specified client; in response to receiving the second confirmation response sent by the specified client, marking the first index acquired by any first thread in the distributed message queue by the distributed cluster; wherein the second confirmation response is used to indicate that the specified client receives the first service data queried by any first thread. Thus, the service data acquired by each thread can be sent to the data demand end (i.e. the specified client) in time, and the waiting time of the specified client is reduced.

[0106] In a possible implementation manner of the embodiment of the present disclosure, when a down instance or a migrated instance (referred to as a second instance in the present disclosure) occurs in each first instance, the second instance can be restarted, and the unmarked index in the distributed index library can be acquired by the restarted second instance, so as to achieve the purpose of breakpoint continuation. The above process will be described in detail below. Figure 6

[0107] Figure 6 The flowchart of the data query method provided in the fifth embodiment of the present disclosure is shown.

[0108] As shown in Figure 6 , in​Figures 2-5 On any of the embodiments, the data query method can further include the following steps:

[0109] At step 601, in response to the second instance of the at least one first instance being down or migrated, the second instance is restarted.

[0110] In the embodiments of the present disclosure, the number of the second instances (or referred to as second consumer instances) can be one or multiple, and the present disclosure does not limit this.

[0111] In the embodiments of the present disclosure, when the second instance of the at least one first instance is down or migrated, the second instance can be restarted.

[0112] At step 602, the unmarked second indexes are obtained from the distributed message queue by the multiple second threads in the restarted second instance.

[0113] In the embodiments of the present disclosure, the unmarked second indexes can be obtained from the distributed message queue by the multiple second threads in the restarted second instance.

[0114] At step 603, the second business data matching the second indexes obtained by the corresponding second threads is queried from the distributed database by the second threads.

[0115] In the embodiments of the present disclosure, the second business data matching the second indexes obtained by the corresponding second threads can be queried from the distributed database by the second threads. That is, for any second thread, the second business data matching the second indexes obtained by the second thread can be queried from the distributed database according to the second indexes obtained by the second thread.

[0116] In summary, after the instance is down or migrated, the instance is restarted, the restarted instance can be used to speed up the data query process, and thus the QPS of data delivery can be improved.

[0117] In a possible implementation manner of the embodiments of the present disclosure, after the second business data is queried or obtained by the second threads, the second business data queried or obtained by the second threads can be combined (or assembled) to obtain combined second business data, and the combined second business data is sent to the specified client.

[0118] In a case where the third confirmation response sent by the specified client is received, it can be determined that the specified client has successfully received the combined second service data, and at this time, the second indexes obtained by the second threads in the distributed message queue can be marked by the distributed cluster. For example, the confirmation response can be sent to the service node where the distributed message queue is located in the distributed cluster, and correspondingly, after receiving the confirmation response, the service node can mark the second indexes obtained by the second threads in the distributed message queue.

[0119] Thus, only in a case where it is determined that the data receiving end (i.e., the specified client) has successfully received the service data, the indexes corresponding to the service data that has been successfully received in the distributed message queue are marked, which can avoid marking the indexes corresponding to the service data in a case where the data receiving end has not successfully received the service data, and if instance downtime or migration occurs, the service data will be lost.

[0120] In another possible implementation of the embodiments of the present disclosure, when any one of the second threads queries or obtains the second service data, the second service data queried or obtained by the second thread can be sent to the specified client.

[0121] In a case where the fourth confirmation response sent by the specified client is received, it can be determined that the specified client has successfully received the second service data queried or obtained by any one of the second threads, and at this time, the second indexes obtained by any one of the second threads in the distributed message queue can be marked by the distributed cluster. For example, the confirmation response can be sent to the service node where the distributed message queue is located in the distributed cluster, and correspondingly, after receiving the confirmation response, the service node can mark the second indexes obtained by any one of the second threads in the distributed message queue.

[0122] Thus, the service data obtained by each thread can be sent to the data demand end (i.e., the specified client) in time, and the waiting time of the specified client can be reduced.

[0123] The data query method of the embodiments of the present disclosure can restart the second instance in response to the downtime or migration of the second instance in the at least one first instance, obtain the unmarked second indexes from the distributed message queue by the multiple second threads in the restarted second instance, and query the second service data matching the second indexes obtained by the corresponding second threads from the distributed database by each second thread. Thus, after the instance downtime or migration, the instance can be restarted, so that the data query process can be accelerated by the restarted instance, and the QPS of data delivery can be improved.

[0124] As an application scenario, taking a distributed database as a KV library, the index of business data as a rowkey, and a distributed index library as an ES (ElasticSearch, a distributed open-source search and analysis engine) library as an example, the implementation principle of any embodiment of the present disclosure can be as shown in the following. Figure 7

[0125] When new data or updated data is added in the KV library, the rowkey in the KV library can be extracted and written into the ES library, and the index of all the business data in the KV library is established through the ES. The KV library and the ES library can be periodically synchronized in full.

[0126] Taking a distributed message queue as a bp (bigpipe, a distributed data transmission pipeline, a kind of distributed message queue) as an example, when it is needed to synchronize the business data in the KV library to other business departments, all the rowkeys in the ES library that meet the query conditions can be queried through an instance, and the rowkeys queried from the ES library are written into the bp.

[0127] A plurality of consumption instances are started to consume the rowkeys in the bp, the consumption instance can obtain an unmarked rowkey from the bp, and then batch query the KV library according to the obtained unmarked rowkey to obtain the required business data, and the queried business data is assembled and sent to the client (denoted as a specified client in the present disclosure) of other business departments. After the data is sent successfully, an ACK can be fed back to the service node where the bp is located, and the ACK is used to indicate that the rowkey obtained by the consumption instance has been consumed, so that the consumed rowkey can be marked.

[0128] In the rowkey consumption process, if the consumption instance is down or migrated and no ACK is fed back to the service node where the bp is located, after the consumption instance is restarted, the rowkey in the bp can still be consumed at the original position, that is, the unmarked rowkey in the bp, so as to achieve the purpose of breakpoint continuation and avoid the problem of repeated data sending. Not only can the data query and sending efficiency be improved, but also the resource occupation can be reduced.

[0129] In addition, in the present disclosure, the number of consumption instances and the number of consumption threads in the consumption instance are configurable, which can achieve the purpose of controlling QPS according to actual needs.

[0130] Corresponding to the data query method provided in the above Figures 2 to 6 embodiment, the present disclosure also provides a data query device. Since the data query device provided in the embodiments of the present disclosure corresponds to the data query method provided in the above Figures 2 to 6 ​The data query method provided by the embodiments corresponds, and therefore the implementation of the data query method is also applicable to the data query apparatus provided by the embodiments of the present disclosure, which will not be described in detail in the embodiments of the present disclosure.

[0131] Figure 8 A structural schematic diagram of the data query apparatus provided by the sixth embodiment of the present disclosure.

[0132] As Figure 8 shown, the data query apparatus 800 can include a query module 801, a write module 802, an acquisition module 803, and a marking module 804.

[0133] The query module 801 is configured to acquire a query condition and query a plurality of indexes matching the query condition from a distributed index library, where the distributed index library is configured to store indexes corresponding to business data in a distributed database.

[0134] The write module 802 is configured to write the plurality of indexes into a distributed message queue in a distributed cluster.

[0135] The acquisition module 803 is configured to acquire an unmarked first index from the distributed message queue and query first business data matching the first index from the distributed database.

[0136] The marking module 804 is configured to mark the first index in the distributed message queue through the distributed cluster in response to the query of the first business data.

[0137] In a possible implementation of the present disclosure, the acquisition module 803 is configured to acquire the unmarked first index from the distributed message queue through a plurality of first threads in at least one first instance, and acquire the first business data matching the first index acquired by each first thread from the distributed database through each first thread.

[0138] In a possible implementation of the present disclosure, the marking module 804 is configured to combine the first business data queried by each first thread in response to the query of the first business data by each first thread, send the combined first business data to a specified client, mark the first index acquired by each first thread in the distributed message queue through the distributed cluster in response to receiving a first confirmation response sent by the specified client, and the first confirmation response is configured to indicate that the specified client receives the combined first business data.

[0139] In a possible implementation of the embodiment of the present disclosure, the marking module 804 is configured to: in response to any first thread inquiring the first business data, send the first business data inquired by any first thread to the specified client; and in response to receiving a second confirmation response sent by the specified client, mark the first index acquired by any first thread in the distributed message queue through the distributed cluster; the second confirmation response is used to indicate that the specified client receives the first business data inquired by any first thread.

[0140] In a possible implementation of the embodiment of the present disclosure, the data query apparatus 800 can further include:

[0141] The restart module is configured to restart the second instance in response to the second instance in the at least one first instance being down or migrated.

[0142] The acquisition module 803 is further configured to acquire, by the multiple second threads in the restarted second instance, the second index without the mark from the distributed message queue.

[0143] The acquisition module 803 is further configured to acquire, by each second thread, the second business data matching the second index acquired by the corresponding second thread from the distributed database.

[0144] In a possible implementation of the embodiment of the present disclosure, the data query apparatus 800 can further include:

[0145] The combination module is configured to combine the second business data inquired by each second thread in response to each second thread inquiring the second business data.

[0146] The first sending module is configured to send the combined second business data to the specified client.

[0147] The marking module 804 is further configured to mark the second index acquired by each second thread in the distributed message queue through the distributed cluster in response to receiving a third confirmation response sent by the specified client.

[0148] The third confirmation response is used to indicate that the specified client receives the combined second business data.

[0149] In a possible implementation of the embodiment of the present disclosure, the data query apparatus 800 can further include:

[0150] The second sending module is configured to send the second business data inquired by any second thread to the specified client in response to any second thread inquiring the second business data.

[0151] The marking module 804 is further configured to mark the second index obtained by any second thread from the distributed message queue by the distributed cluster in response to receiving the fourth confirmation response sent by the specified client.

[0152] The fourth confirmation response is used to indicate that the specified client receives the second service data queried by any second thread.

[0153] In a possible implementation of the embodiment of the present disclosure, the distributed database is configured to store each service data and an index of each service data; the distributed index library is further configured to periodically obtain a third index corresponding to third service data from the distributed database according to a set period, and store the third index; wherein the third service data is service data that is updated and / or newly added in the distributed database.

[0154] In a possible implementation of the embodiment of the present disclosure, the data query apparatus 800 can further include:

[0155] The deleting module is configured to delete each index in the distributed message queue when each index in the distributed message queue is marked.

[0156] The data query apparatus of the embodiment of the present disclosure queries a plurality of indexes matching the query condition from the distributed index library, and writes the plurality of indexes into the distributed message queue in the distributed cluster; obtains a first index without a mark from the distributed message queue, and queries first service data matching the first index from the distributed database; in response to querying the first service data, marks the first index in the distributed message queue by the distributed cluster. Thus, only the index without a mark is obtained from the distributed message queue, and the service data is queried from the distributed database according to the obtained index without a mark, and after successfully querying the service data, the index corresponding to the queried service data is marked, which can avoid the situation that the index corresponding to the service data that has been queried is obtained from the distributed message queue (i.e., the index with a mark) when the device is down or the instance is migrated, and the service data is repeatedly queried and obtained, thereby improving the data query efficiency and reducing the resource occupation.

[0157] In order to implement the above-mentioned embodiments, the present disclosure further provides an electronic device, which can include at least one processor; and a memory connected with the at least one processor in communication; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the data query method proposed in any one of the embodiments of the present disclosure.

[0158] To achieve the above-mentioned embodiments, the present disclosure further provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to make a computer execute the data query method proposed in any one of the above-mentioned embodiments of the present disclosure.

[0159] To achieve the above-mentioned embodiments, the present disclosure further provides a computer program product, which comprises a computer program, and the computer program, when executed by a processor, implements the data query method proposed in any one of the above-mentioned embodiments of the present disclosure.

[0160] According to embodiments of the present disclosure, the present disclosure further provides an electronic device, a readable storage medium and a computer program product.

[0161] Figure 9 A schematic block diagram of an example electronic device that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptops, desktops, tablets, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular telephones, smartphones, wearable devices, and other similar computing devices. The components shown here, their connections and relationships, and their functions, are meant to be examples only, and are not meant to limit implementations of the present disclosure described and / or claimed in this document.

[0162] As shown in Figure 9 The electronic device 900 includes a computing unit 901 that can perform various appropriate actions and processes in accordance with a computer program stored in a ROM (Read-Only Memory) 902 or a computer program loaded into a RAM (Random Access Memory) 903 from a storage unit 908. Various programs and data required for the operation of the electronic device 900 can also be stored in the RAM 903. The computing unit 901, the ROM 902, and the RAM 903 are connected to each other through a bus 904. An I / O (Input / Output) interface 905 is also connected to the bus 904.

[0163] Various components in the electronic device 900 are connected to the I / O interface 905, including an input unit 906, such as a keyboard, a mouse, etc., an output unit 907, such as various types of displays, a speaker, etc., a storage unit 908, such as a magnetic disk, an optical disk, etc., and a communication unit 909, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 909 allows the electronic device 900 to exchange information / data with other devices through a computer network, such as the Internet, and / or various telecommunication networks.

[0164] The computing unit 901 can be various general and / or special purpose processing components with processing and computing capabilities. Some examples of the computing unit 901 include, but are not limited to, a CPU (Central Processing Unit), a GPU (Graphic Processing Units), various special-purpose AI (Artificial Intelligence) computing chips, various computing units running machine learning model algorithms, a DSP (Digital Signal Processor), and any appropriate processor, controller, microcontroller, etc. The computing unit 901 performs various methods and processes described above, such as the data query method described above. For example, in some embodiments, the data query method described above can be implemented as a computer software program, which is tangibly embodied in a machine-readable medium, such as the storage unit 908. In some embodiments, part or all of the computer program can be loaded and / or installed onto the electronic device 900 via the ROM 902 and / or the communication unit 909. When the computer program is loaded onto the RAM 903 and executed by the computing unit 901, one or more steps of the data query method described above can be performed. Alternatively, in other embodiments, the computing unit 901 can be configured to perform the data query method described above by other any appropriate means, such as by means of firmware.

[0165] Various implementations of the systems and techniques described above herein can be realized in digital electronic circuitry, integrated circuitry, a Field Programmable Gate Array (FPGA), an Application-Specific Integrated Circuit (ASIC), an Application Specific Standard Product (ASSP), a System on Chip (SOC), a Complex Programmable Logic Device (CPLD), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0166] Program code for carrying out methods of the present disclosure can be written in any combination of one or more programming languages. The program code can be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the program code, when executed by the processor or controller, produces the functions / operations specified in the flowcharts and / or block diagrams. The program code can be executed entirely on a machine, partially on a machine, partially on a machine as a stand-alone software package, partially on a machine and partially on a remote machine or entirely on a remote machine or server.

[0167] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable storage medium can include, without limitation, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the machine-readable storage medium will include one or more lines of electrical connections, portable computer disks, hard disk, RAM, ROM, EPROM (Electrically Programmable Read-Only-Memory), or flash memory, fiber optics, CD-ROM (Compact Disc Read-Only Memory), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0168] To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having a display device (e.g., a CRT (Cathode-Ray Tube) or LCD (Liquid Crystal Display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0169] The systems and techniques described herein can be implemented in a computing system that includes a back end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front end component, e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described herein, or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a LAN (Local Area Network), a WAN (Wide Area Network), the Internet, and a blockchain network.

[0170] The computer system can include clients and servers. The clients and servers are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system, to solve the defects of large management difficulty and weak business scalability in traditional physical host and VPS service (Virtual Private Server). The server can also be a server of a distributed system, or a server combined with a blockchain.

[0171] It should be noted that artificial intelligence is a discipline that studies enabling computers to simulate some thinking processes and intelligent behaviors of people (such as learning, reasoning, thinking, planning, etc.), both hardware and software technologies. Artificial intelligence hardware technologies generally include technologies such as sensors, special artificial intelligence chips, cloud computing, distributed storage, big data processing, etc.; artificial intelligence software technologies mainly include computer vision technology, speech recognition technology, natural language processing technology, and machine learning / deep learning, big data processing technology, knowledge graph technology, etc. several major directions.

[0172] Cloud computing refers to accessing elastic and scalable shared physical or virtual resource pools through a network, resources can include servers, operating systems, networks, software, applications, and storage devices, etc., and resources can be deployed and managed in a demand-driven and self-service manner. Through cloud computing technology, efficient and powerful data processing capabilities can be provided for artificial intelligence, blockchain, and other technology applications, model training.

[0173] According to the technical scheme of the embodiment of the present disclosure, a plurality of indexes matched with the query condition are queried from the distributed index library, and the plurality of indexes are written into the distributed message queue in the distributed cluster; the first index without a mark is acquired from the distributed message queue, and the first business data matched with the first index is queried from the distributed database; and in response to the first business data being queried, the first index in the distributed message queue is marked by the distributed cluster. Thus, only the index without a mark is acquired from the distributed message queue, and the business data is queried from the distributed database according to the acquired index without a mark, and after the business data is successfully queried, the index corresponding to the queried business data is marked, which can avoid the situation that the index corresponding to the business data that has been queried (i.e., the index with a mark) is acquired from the distributed message queue when the device is down or the instance is migrated, and the business data is repeatedly queried and acquired, so that the query efficiency of the data can be improved, and the resource occupation can be reduced.

[0174] It should be understood that the various forms of flow shown above can be reordered, steps added or removed. For example, the steps described in the present disclosure can be executed in parallel, sequentially, or in a different order, as long as the desired results of the technical scheme proposed in the present disclosure can be achieved, which is not limited herein.

[0175] The above detailed description does not constitute a limitation on the protection scope of the present disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modifications, equivalent replacements and improvements within the spirit and principles of the present disclosure should be included in the protection scope of the present disclosure.

Claims

1. A data query method, comprising: obtaining a query condition and querying a plurality of indexes matching the query condition from a distributed index library, wherein the distributed index library is configured to store indexes corresponding to business data in a distributed database; writing the plurality of indexes into a distributed message queue in a distributed cluster; obtaining an unmarked first index from the distributed message queue and querying first business data matching the first index from the distributed database; in response to querying the first business data, marking the first index in the distributed message queue through the distributed cluster, wherein an acknowledgement response is sent to a service node where the distributed message queue is located in the distributed cluster, and the service node marks the first index in the distributed message queue after receiving the acknowledgement response.

2. The method of claim 1, wherein, The step of obtaining an unmarked first index from the distributed message queue and querying first business data matching the first index from the distributed database comprises: obtaining the unmarked first index from the distributed message queue through a plurality of first threads in at least one first instance; querying first business data matching the first index obtained by the corresponding first thread from the distributed database through each first thread.

3. The method of claim 2, wherein, The step of marking the first index in the distributed message queue through the distributed cluster in response to querying the first business data comprises: combining the first business data queried by each first thread in response to each first thread querying the first business data; sending the combined first business data to a specified client; in response to receiving a first acknowledgement response sent by the specified client, marking the first index obtained by each first thread in the distributed message queue through the distributed cluster; wherein the first acknowledgement response is configured to indicate that the specified client receives the combined first business data.

4. The method of claim 2, wherein, The step of marking the first index in the distributed message queue through the distributed cluster in response to querying the first business data comprises: sending the first business data queried by any first thread to a specified client in response to any first thread querying the first business data; in response to receiving a second acknowledgement response sent by the specified client, marking the first index obtained by the any first thread in the distributed message queue through the distributed cluster; wherein the second acknowledgement response is configured to indicate that the specified client receives the first business data queried by the any first thread.

5. The method of claim 2, wherein, The method further comprises: in response to a second instance in the at least one first instance being down or migrated, restarting the second instance; obtaining an unmarked second index from the distributed message queue through a plurality of second threads in the restarted second instance; querying second business data matching the second index obtained by the corresponding second thread from the distributed database through each second thread.

6. The method of claim 5, wherein, The method further comprises: in response to each of the second threads querying the second business data, combining the second business data queried by each of the second threads; sending the combined second business data to a specified client; in response to receiving a third confirmation response sent by the specified client, marking, by the distributed cluster, the second indexes obtained by each of the second threads in the distributed message queue; wherein the third confirmation response is used to indicate that the specified client has received the combined second business data.

7. The method of claim 5, wherein, The method further comprises: in response to any of the second threads querying the second business data, sending the second business data queried by the any of the second threads to a specified client; in response to receiving a fourth confirmation response sent by the specified client, marking, by the distributed cluster, the second index obtained by the any of the second threads in the distributed message queue; wherein the fourth confirmation response is used to indicate that the specified client has received the second business data queried by the any of the second threads.

8. The method of any one of claims 1-7, wherein, The distributed database is configured to store each of the business data and an index of each of the business data. The distributed index library is further configured to periodically obtain a third index corresponding to third business data from the distributed database at a set period, and store the third index; wherein the third business data is business data that has been updated and / or newly added in the distributed database.

9. The method of any one of claims 1-7, wherein, The method further comprises: in a case where each of the indexes in the distributed message queue is marked, deleting each of the indexes in the distributed message queue.

10. A data query apparatus, comprising: a query module configured to obtain a query condition, and query a plurality of indexes matching the query condition from a distributed index library; wherein the distributed index library is configured to store indexes corresponding to business data in a distributed database; a write-in module configured to write the plurality of indexes into a distributed message queue in a distributed cluster; an obtaining module configured to obtain an unmarked first index from the distributed message queue, and query first business data matching the first index from the distributed database; a marking module configured to, in response to querying the first business data, mark, by the distributed cluster, the first index in the distributed message queue, wherein an acknowledgment response is sent to a service node where the distributed message queue in the distributed cluster is located, and the service node marks the first index in the distributed message queue after receiving the acknowledgment response.

11. The apparatus of claim 10, wherein, The obtaining module is configured to: obtain, by a plurality of first threads in at least one first instance, the unmarked first index from the distributed message queue; obtain, by each of the first threads, first business data matching the first index obtained by the corresponding first thread from the distributed database.

12. The apparatus of claim 11, wherein, The marking module is configured to: in response to each of the first threads querying the first business data, combine the first business data queried by each of the first threads; send the combined first business data to a specified client; In response to receiving the first confirmation response sent by the specified client, marking, by the distributed cluster, the first indexes obtained by the first threads in the distributed message queue; The first confirmation response is used to indicate that the specified client receives the combined first business data.

13. The apparatus of claim 11, wherein, The marking module is configured to: In response to any of the first threads querying the first business data, sending the first business data queried by the any of the first threads to the specified client; In response to receiving the second confirmation response sent by the specified client, marking, by the distributed cluster, the first index obtained by the any of the first threads in the distributed message queue; The second confirmation response is used to indicate that the specified client receives the first business data queried by the any of the first threads.

14. The apparatus of claim 11, wherein, The apparatus further comprises: The restarting module is configured to restart a second instance in the at least one first instance in response to the second instance being down or migrated; The obtaining module is further configured to obtain, by a plurality of second threads in the restarted second instance, second indexes without being marked from the distributed message queue; The obtaining module is further configured to obtain, by each of the second threads, second business data matching the second index obtained by the corresponding second thread from the distributed database.

15. The apparatus of claim 14, wherein, The apparatus further comprises: The combining module is configured to combine the second business data queried by each of the second threads in response to each of the second threads querying the second business data; The first sending module is configured to send the combined second business data to the specified client; The marking module is further configured to mark, by the distributed cluster, the second indexes obtained by each of the second threads in the distributed message queue in response to receiving a third confirmation response sent by the specified client; The third confirmation response is used to indicate that the specified client receives the combined second business data.

16. The apparatus of claim 14, wherein, The apparatus further comprises: The second sending module is further configured to send the second business data queried by the any of the second threads to the specified client in response to any of the second threads querying the second business data; The marking module is further configured to mark, by the distributed cluster, the second index obtained by the any of the second threads in the distributed message queue in response to receiving a fourth confirmation response sent by the specified client; The fourth confirmation response is used to indicate that the specified client receives the second business data queried by the any of the second threads.

17. The apparatus of any one of claims 10-16, wherein, The distributed database is configured to store each of the business data and an index of each of the business data; The distributed index library is further configured to periodically obtain, from the distributed database, a third index corresponding to third business data according to a set period, and store the third index; The third business data is business data that is updated and / or newly added in the distributed database.

18. The apparatus of any one of claims 10-16, wherein, The apparatus further comprises: The deleting module is configured to delete each of the indexes in the distributed message queue in a case where each of the indexes in the distributed message queue is marked.

19. An electronic device, comprising: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-9.

20. A non-transitory computer readable storage medium having stored thereon computer instructions, wherein, The computer instructions are for causing the computer to perform the method of any one of claims 1-9.

21. A computer program product comprising a computer program which, when executed by a processor, implements the steps of the method of any one of claims 1-9.

Images