Distributed data query method and device, equipment and medium

Abstract

The invention provides a distributed data query method and device, equipment and a medium, which can be applied to the technical fields of computers, big data and databases. The method comprises the following steps: in response to a query request received from a user, determining N initial data nodes associated with the query request, so that the N initial data nodes respectively perform data query to obtain a query result; based on the available memory amounts corresponding to the N initial data nodes, M intermediate data nodes are screened from the N initial data nodes, M is smaller than or equal to N, and M is a positive integer; determining a target data node from the M intermediate data nodes based on the memory occupancy of the query results corresponding to the M intermediate data nodes; and sending query results corresponding to other initial data nodes except the target data node to the target data node, so that the target data node combines the N query results and returns the combined query results to the user.

Description

Technical Field

[0001] The present invention relates to the technical field of computers, big data and databases, and more specifically, to a method, device, equipment and medium for distributed data query. Background Art

[0002] Distributed data query becomes more and more important in big data and cloud computing environments as the amount of data and query complexity that distributed data query can process continue to increase. Distributed data query allows users to access and retrieve data across multiple physically or logically separated nodes.

[0003] In the process of realizing the concept of the present invention, the inventors found that there are at least the following problems in the related art: when exporting the query results of distributed data queries, multiple data nodes are required to cache their query results locally, and finally the task initiating node among the multiple data nodes acts as a data collection node to store the query results of other data nodes, and merge the query results of all data nodes into a complete query result before exporting. When the available space of the task initiating node is insufficient to store the query results of all data nodes, the query results cannot be exported normally. Summary of the invention

[0004] In view of this, the present invention provides a method, apparatus, device, and medium for distributed data query.

[0005] One aspect of the present invention provides a method for distributed data query, comprising: in response to receiving a query request from a user, determining N initial data nodes associated with the query request, so that the N initial data nodes each perform a data query to obtain a query result, wherein the N initial data nodes are part or all of the nodes in a distributed cluster, and N is an integer greater than or equal to 2; based on the amount of available memory corresponding to each of the N initial data nodes, screening M intermediate data nodes from the N initial data nodes, wherein M≤N, and M is a positive integer; based on the memory occupancy of the query results corresponding to each of the M intermediate data nodes, determining a target data node from the M intermediate data nodes; and sending the query results corresponding to the other initial data nodes except the target data node to the target data node, so that the target data node merges the N query results and returns them to the user.

[0006] According to an embodiment of the present invention, a target data node is determined from the M intermediate data nodes based on the memory usage of the query results obtained by each of the M intermediate data nodes when performing data queries, including: sorting the M intermediate data nodes in descending order based on the memory usage of the query results obtained by each of the M intermediate data nodes when performing data queries to obtain a sorting result; and determining the intermediate data node ranked first in the sorting result as the target data node.

[0007] According to an embodiment of the present invention, the method for distributed data query also includes: after determining the target data node, locking the target data node; setting a delay time for the target data node, wherein the target data node is prohibited from serving as a data collection node for other query requests during the delay time.

[0008] According to an embodiment of the present invention, based on the amount of available memory corresponding to each of the N initial data nodes, M intermediate data nodes are screened from the N initial data nodes, including: when it is determined that the amount of available memory corresponding to each of the N initial data nodes is greater than or equal to a memory threshold, initial data nodes whose corresponding amount of available memory is greater than or equal to the memory threshold are screened from the N initial data nodes as intermediate data nodes.

[0009] According to an embodiment of the present invention, the distributed data query method also includes: when it is determined that the available memory corresponding to each of the N initial data nodes is less than the memory threshold, the query results obtained by performing data queries on each of the N initial data nodes are sent to a remote server, so that the remote server merges the N query results and returns them to the user.

[0010] According to an embodiment of the present invention, the distributed data query method further includes: in the case of determining that multiple query requests are received, processing the multiple query requests in parallel.

[0011] According to an embodiment of the present invention, the method for distributed data query also includes: when it is determined that the target data nodes corresponding to at least two query requests are the same target data node, determining the order for feeding back at least two merged query results corresponding to the at least two query requests based on the time when the at least two query requests are received.

[0012] Another aspect of the present invention provides a distributed data query device, including: a first determination module, used to determine N initial data nodes associated with the query request in response to receiving a query request from a user, so that the N initial data nodes each perform a data query to obtain a query result, wherein the N initial data nodes are part or all of the nodes in the distributed cluster, and N is an integer greater than or equal to 2; a screening module, used to screen M intermediate data nodes from the N initial data nodes based on the available memory corresponding to each of the N initial data nodes, wherein M≤N, and M is a positive integer; a second determination module, used to determine a target data node from the M intermediate data nodes based on the memory occupancy of the query results corresponding to each of the M intermediate data nodes; and a first sending module, used to send the query results corresponding to other initial data nodes except the target data node to the target data node, so that the target data node merges the N query results and returns them to the user.

[0013] Another aspect of the present invention provides an electronic device, comprising: one or more processors; a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the above-mentioned method for distributed data query.

[0014] Another aspect of the present invention provides a computer-readable storage medium storing computer-executable instructions, which are used to implement the above-mentioned distributed data query method when executed.

[0015] Another aspect of the present invention provides a computer program product, which includes computer executable instructions, and the instructions are used to implement the above distributed data query method when executed.

[0016] According to an embodiment of the present invention, based on the amount of available memory, multiple intermediate data nodes are screened from multiple initial data nodes associated with the query request. Since the data nodes are screened based on the amount of available memory of the data nodes, the intermediate data nodes screened thereby can meet the demand for the amount of available memory, thereby at least partially avoiding the situation where the intermediate data nodes are used as target data nodes to save the query results of all data nodes when the memory overflow occurs. Based on the memory footprint of the query results obtained by each of the multiple intermediate data nodes performing data queries, the target data node is screened from the multiple intermediate data nodes. On the basis of meeting the demand for the amount of available memory, combined with the memory footprint of the query results obtained by each of the intermediate data nodes performing data queries, the query results corresponding to other initial data nodes can be sent to the target data node. The consumed resources can be reduced as much as possible, and the transmission efficiency can be accelerated. By determining the target data node through two screenings, not only can the target data node be prevented from having a memory overflow when saving the query results of all data nodes, but also the consumed transmission resources can be reduced, the transmission efficiency can be accelerated, and the user's query request can be responded to in a timely manner, and the user experience can be enhanced. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The above and other objects, features and advantages of the present invention will become more apparent through the following description of the embodiments of the present invention with reference to the accompanying drawings, in which:

[0018] Figure 1 A schematic diagram of query result export of a related example is shown;

[0019] Figure 2 The application scenarios of the distributed data query method, apparatus, device, medium and program product according to the embodiments of the present invention are shown;

[0020] Figure 3 A flow chart of a distributed data query method according to an embodiment of the present invention is shown;

[0021] Figure 4 A flow chart of a method for publishing data query according to another embodiment of the present invention is shown;

[0022] Figure 5 It shows a structural block diagram of a distributed data query device according to an embodiment of the present invention; and

[0023] Figure 6 A block diagram of an electronic device suitable for implementing a distributed data query method according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0024] Below, embodiments of the present invention will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present invention. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of embodiments of the present invention. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of concepts of the present invention.

[0025] The terms used herein are only for describing specific embodiments and are not intended to limit the present invention. The terms "comprise", "include", etc. used herein indicate the existence of features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.

[0026] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0027] When using expressions such as "at least one of A, B, and C, etc.", they should generally be interpreted according to the meaning of the expression commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0028] In the embodiments of the present invention, the collection, updating, analysis, processing, use, transmission, provision, invention, storage and other aspects of the data involved (for example, including but not limited to user personal information) are in compliance with the provisions of relevant laws and regulations, are used for legitimate purposes, and do not violate public order and good morals. In particular, necessary measures are taken for user personal information to prevent illegal access to user personal information data and maintain the security of user personal information and network security.

[0029] In the embodiment of the present invention, before obtaining or collecting the user's personal information, the user's authorization or consent is obtained.

[0030] For databases that support massively parallel processing, query results can be exported to a server in the cluster or to an external storage service that it supports.

[0031] Figure 1 A schematic diagram of query result export of a related example is shown.

[0032] like Figure 1As shown, the distributed cluster may include a management node 110 and a data node 120. When a user performs a distributed data query, the user may send a query statement to data nodes 1201, 1202, and 1203 through the management node 1101 or the management node 1102 in the distributed cluster. The data nodes 1201, 1202, and 1203 perform data query and write the query results of each data node into a temporary file corresponding to each data node. Then, the first data node to which the management node 1101 or the management node 1102 sends the query statement is used as the task initiating node, that is, the storage of the exported query results, such as the data node 1201. Other data nodes, such as the data node 1202 and the data node 1203, send the query results written into the temporary file to the task initiating node, such as the data node 1201. Finally, the task initiating node, such as the data node 1201, generates the export file and saves it in the specified path of the node.

[0033] In the relevant examples, when exporting the query results of distributed data queries, multiple data nodes need to cache their query results locally, and finally the task initiating node among the multiple data nodes acts as a data collection node to store the query results of other data nodes, and merge the query results of all data nodes into a complete query result for export. Therefore, when the available space of the task initiating node is insufficient to store the query results of all data nodes, the query results cannot be exported normally.

[0034] Based on this, an embodiment of the present invention provides a method, apparatus, device, medium and program product for distributed data query. The method includes: in response to receiving a query request from a user, determining N initial data nodes associated with the query request, so that the N initial data nodes each perform a data query to obtain a query result, the N initial data nodes are part or all of the nodes in the distributed cluster, and N is an integer greater than or equal to 2; based on the available memory corresponding to each of the N initial data nodes, M intermediate data nodes are selected from the N initial data nodes, M≤N, and M is a positive integer; based on the memory occupancy of the query results corresponding to each of the M intermediate data nodes, a target data node is determined from the M intermediate data nodes; and query results corresponding to other initial data nodes except the target data node are sent to the target data node, so that the target data node merges the N query results and returns them to the user.

[0035] Figure 2 The application scenarios of the distributed data query method, apparatus, device, medium and program product according to the embodiments of the present invention are shown.

[0036] like Figure 2As shown, the application scenario 200 according to this embodiment may include a first terminal device 201, a second terminal device 202, a third terminal device 203, a network 204, and a server 205. The network 204 is used to provide a medium for a communication link between the first terminal device 201, the second terminal device 202, the third terminal device 203, and the server 205. The network 204 may include various connection types, such as wired and / or wireless communication links, etc.

[0037] The user may use the first terminal device 201, the second terminal device 202, and the third terminal device 203 to interact with the server 205 through the network 204 to receive or send messages, etc. Various communication client applications may be installed on the first terminal device 201, the second terminal device 202, and the third terminal device 203, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, and / or social platform software, etc. (only for example).

[0038] The first terminal device 201, the second terminal device 202, and the third terminal device 203 may be various electronic devices having display screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.

[0039] The server 205 may be a server that provides various services, such as a background management server (only as an example) that provides support for websites browsed by users using the first terminal device 201, the second terminal device 202, and the third terminal device 203. The background management server may analyze and process the received data such as user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal device.

[0040] The server 205 may include multiple servers and multiple management servers, and the multiple servers may serve as data nodes of a distributed cluster to store and process data.

[0041] It should be noted that the distributed data query method provided in the embodiment of the present invention can generally be executed by the server 205. Accordingly, the distributed data query device provided in the embodiment of the present invention can generally be set in the server 205. The distributed data query method provided in the embodiment of the present invention can also be executed by a server or server cluster that is different from the server 205 and can communicate with the first terminal device 201, the second terminal device 202, the third terminal device 203 and / or the server 205. Correspondingly, the distributed data query device provided in the embodiment of the present invention can also be set in a server or server cluster that is different from the server 205 and can communicate with the first terminal device 201, the second terminal device 202, the third terminal device 203 and / or the server 205. Alternatively, the distributed data query method provided in the embodiment of the present invention can also be executed by the first terminal device 201, the second terminal device 202 or the third terminal device 203, or by other terminal devices that are different from the first terminal device 201, the second terminal device 202 or the third terminal device 203. Correspondingly, the distributed data query device provided in the embodiment of the present invention can also be set in the first terminal device 201, the second terminal device 202 or the third terminal device 203, or in other terminal devices different from the first terminal device 201, the second terminal device 202 or the third terminal device 203.

[0042] It should be understood that Figure 2 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0043] Figure 3 A flow chart of a distributed data query method according to an embodiment of the present invention is shown.

[0044] like Figure 3 As shown, the distributed data query method of this embodiment includes operations S310 to S340.

[0045] In operation S310, in response to receiving a query request from a user, N initial data nodes associated with the query request are determined so that each of the N initial data nodes performs a data query to obtain a query result.

[0046] According to an embodiment of the present invention, the N initial data nodes are part or all of the nodes in the distributed cluster, and N is an integer greater than or equal to 2. The distributed cluster may include multiple data nodes. The query data may be divided and distributed on different data nodes. The data query may be performed between the N initial data nodes.

[0047] According to an embodiment of the present invention, a user may send a query request to a server through a client device. After receiving the query request, the server determines N initial data nodes based on the query request. For example, a query statement may be determined through the query request, and data nodes associated with the query statement may be determined based on the query statement.

[0048] In operation S320 , based on the available memory amounts corresponding to the N initial data nodes, M intermediate data nodes are selected from the N initial data nodes.

[0049] According to an embodiment of the present invention, M≤N, and M is a positive integer.

[0050] According to an embodiment of the present invention, the available memory amount can be used to characterize the free memory amount of the data node. For example, the available memory amount corresponding to each of the N initial data nodes can be obtained based on the difference between the inherent memory amount corresponding to each of the N initial data nodes and the occupied memory amount. The inherent memory amount can be used to characterize the maximum memory amount used by the data node to store data.

[0051] According to an embodiment of the present invention, the total memory usage required to store the query results corresponding to the N initial data nodes can be determined based on the memory usage of the query results corresponding to the N initial data nodes. Data nodes whose available memory is greater than or equal to the total memory usage are selected from the N initial data nodes as intermediate data nodes.

[0052] In operation S330 , a target data node is determined from the M intermediate data nodes based on the memory usage of the query results corresponding to each of the M intermediate data nodes.

[0053] According to an embodiment of the present invention, the memory usage of the query result can be used to characterize the memory usage required to store the query result. The target data node can be used as a data collection node for the query result.

[0054] According to an embodiment of the present invention, a data node with a memory footprint greater than a preset memory footprint can be selected from the M intermediate data nodes as a target data node. The preset memory footprint can be determined according to actual needs. For example, if it is necessary to consume the least transmission resources, a larger preset memory footprint can be set, and the data node with the largest memory footprint can be selected from the M intermediate data nodes as the target data node.

[0055] In operation S340, query results corresponding to other initial data nodes except the target data node are sent to the target data node, so that the target data node merges the N query results and returns them to the user.

[0056] According to an embodiment of the present invention, the target data node may splice N query results into a complete file format and return it to the user.

[0057] According to an embodiment of the present invention, based on the amount of available memory, multiple intermediate data nodes are screened from multiple initial data nodes associated with the query request. Since the data nodes are screened based on the amount of available memory of the data nodes, the intermediate data nodes screened thereby can meet the demand for the amount of available memory, thereby at least partially avoiding the situation where the intermediate data nodes are used as target data nodes to save the query results of all data nodes when the memory overflow occurs. Based on the memory footprint of the query results corresponding to each of the multiple intermediate data nodes, the target data node is screened from the multiple intermediate data nodes. On the basis of meeting the demand for the amount of available memory, combined with the memory footprint of the query results corresponding to each of the intermediate data nodes, the query results corresponding to other initial data nodes can be sent to the target data node. The resources consumed can be reduced as much as possible, and the transmission efficiency can be accelerated. By determining the target data node through two screenings, not only can the target data node be prevented from having a memory overflow when saving the query results of all data nodes, but also the consumed transmission resources can be reduced, the transmission efficiency can be accelerated, and the user's query request can be responded to in a timely manner, and the user experience can be enhanced.

[0058] According to an embodiment of the present invention, Figure 3 The operation S330 shown, determining the target data node from the M intermediate data nodes based on the memory usage of the query results corresponding to the M intermediate data nodes, may include the operation of: sorting the M intermediate data nodes in descending order based on the memory usage of the query results corresponding to the M intermediate data nodes, to obtain a sorting result. Determine the intermediate data node that is at the first position in the sorting result as the target data node.

[0059] According to an embodiment of the present invention, the larger the memory usage of the query result corresponding to the intermediate data node, the higher the position of the corresponding sorted result.

[0060] According to an embodiment of the present invention, since the query result corresponding to the intermediate data node ranked first in the sorting result has the largest memory occupancy, determining the target data node thereby can reduce the resources consumed in sending the query results corresponding to other initial data nodes to the target data node, thereby increasing transmission efficiency.

[0061] According to an embodiment of the present invention, the distributed data query method may include the following: Figure 3 In addition to the operations S310 to S340 shown, the operation may also include: after determining the target data node, locking the target data node; and setting a delay time of the target data node.

[0062] According to an embodiment of the present invention, the target data node is prohibited from being used as a data collection node for other query requests during the delay time. Other query requests can be understood as requests that are not responded to at the same time as the query request. The data collection node can be understood as other target data nodes determined according to other query requests.

[0063] According to an embodiment of the present invention, the delay time can be determined according to the network transmission speed and the memory occupied by the query result to be transmitted. For example, the faster the network transmission speed, the smaller the memory occupied, and the shorter the delay time. On the contrary, the slower the network transmission speed, the larger the memory occupied, and the longer the delay time.

[0064] According to an embodiment of the present invention, the target data node can be unlocked after the delay time ends, and the target data node can be used as a data collection node for other query requests again.

[0065] According to the embodiment of the present invention, by locking the target data node, the operation in response to the query request can monopolize the resources of the target data node, thereby increasing the rate at which the target data node receives and sends query results, and reducing the impact on system performance. In addition, the integrity and availability of data can be guaranteed during the process of receiving and sending query results.

[0066] Figure 4 A flow chart of a published data query method according to another embodiment of the present invention is shown.

[0067] like Figure 4 As shown, the distributed data query method of this embodiment includes operations S410 to S460.

[0068] In operation S410, in response to receiving a query request from a user, N initial data nodes associated with the query request are determined so that each of the N initial data nodes performs a data query to obtain a query result.

[0069] In operation S420, it is determined whether the available memory amounts corresponding to the N initial data nodes are greater than or equal to the memory threshold.

[0070] According to an embodiment of the present invention, when it is determined that the available memory amounts corresponding to the N initial data nodes are all less than the memory threshold, operation S440 is performed. When it is determined that the available memory amounts corresponding to the N initial data nodes are all less than the memory threshold, operation S430 is performed.

[0071] In operation S430, query results obtained by querying data on each of the N initial data nodes are sent to a remote server, so that the remote server merges the N query results and returns them to the user.

[0072] In operation S440, initial data nodes corresponding to which available memory amounts are greater than or equal to a memory threshold are selected from the N initial data nodes as intermediate data nodes.

[0073] In operation S450, a target data node is determined from the M intermediate data nodes based on the memory usage of the query results corresponding to each of the M intermediate data nodes.

[0074] In operation S460, query results corresponding to other initial data nodes except the target data node are sent to the target data node, so that the target data node merges the N query results and returns them to the user.

[0075] According to an embodiment of the present invention, the memory threshold may be determined based on the sum of the memory occupancy of the query results corresponding to each initial data node.

[0076] According to an embodiment of the present invention, based on the available memory amount being greater than or equal to the memory threshold, intermediate data nodes are screened, which can meet the demand for available memory amount, thereby at least partially avoiding the memory overflow that occurs when the intermediate data nodes are used as target data nodes to save the query results of all data nodes.

[0077] According to an embodiment of the present invention, the distributed data query method may be executed by a management node of a distributed cluster.

[0078] According to an embodiment of the present invention, a remote server can execute a remote export command by connecting to a management node of a distributed cluster. The remote server can create a listening thread and an export thread, and the listening thread opens a listening port and waits for the connection of the data node. The management node of the distributed cluster can insert the address and port number of the remote server into the execution command and send it to the data node. The data node can generate a query result according to the query statement. The data node can establish a connection to the remote server according to the received remote server address and port number. After the remote server establishes a connection with the data node, it is handed over to the listening thread for maintenance. The data node starts to send query results to the remote server. When the remote server listening thread polls the event and finds that there is data sent, it receives the data of the query result and puts it into the buffer. The remote server export thread continuously obtains the query results from the buffer and writes the query results to the local file. When the data node sends all query results to the remote server, it sends an end mark to the remote server and disconnects. After receiving the disconnection notification from the data node, the remote server listening thread closes the connection. After receiving the data of all data nodes and closing the connection, the listening thread notifies the export thread that the export operation is over and stops the listening thread. After receiving the notification, the export thread of the remote server stops the export thread and the export operation ends.

[0079] According to an embodiment of the present invention, by querying data on each of the N initial data nodes to obtain their own corresponding query results and sending them to the remote server, the disk space of the nodes in the distributed cluster will not be occupied. In addition, the remote server can be flexibly deployed, and data can be exported to different servers without using other storage services or transmission protocols.

[0080] According to an embodiment of the present invention, the distributed data query method may include the above Figure 3 In addition to the operations S310 to S340 shown, the following operation may be further included: when it is determined that multiple query requests are received, the multiple query requests are processed in parallel.

[0081] According to an embodiment of the present invention, if multiple query requests are continuously received within a predetermined time period, different initial data nodes can be configured for each query request to perform data query, so as to realize parallel processing of multiple query requests. During the parallel processing, if there are at least two query requests whose target data nodes determined during the parallel processing are the same, the N query results can be merged and returned to the user according to the order of the query request received, or the query results with less memory usage can be returned to the user first according to the sum of the memory usage of the N query results.

[0082] According to an embodiment of the present invention, in a scenario with a large amount of data and high concurrent requests, the amount of query requests is large. If each query request is processed one by one in a queue, the processing efficiency is low, which can easily cause network congestion, and the user's request cannot be responded to in time, resulting in a poor user experience. Based on this, the present invention can respond to user requests faster and reduce waiting time by processing multiple query requests in parallel and processing multiple queries at the same time, that is, improving the response speed and enhancing the user experience. In addition, the parallel processing capability enables the system to process more requests at the same time, thereby improving the overall data processing capability, that is, improving the system throughput.

[0083] According to an embodiment of the present invention, the distributed data query method may include the above Figure 3 In addition to the operations S310 to S340 shown, the following operation may also be included: when it is determined that the target data nodes corresponding to at least two query requests are the same target data node, determining, based on the time when the at least two query requests are received, an order for feeding back at least two merged query results corresponding to the at least two query requests.

[0084] According to an embodiment of the present invention, during the parallel processing, each query request will determine the target data node corresponding to each query request. If the target data node corresponding to at least two query requests is the same target data node, the order in which the query results are returned to the user can be determined based on the time when the management node of the distributed cluster receives the query request. While returning the query results, the storage records in the target data node are cleared to release the storage memory.

[0085] According to an embodiment of the present invention, based on the time sequence of query requests, when there are at least two query requests corresponding to the same target data node, the query results are returned to the user. This can not only avoid data overflow of the target data node, but also speed up data processing efficiency, respond to user requests in a timely manner, and enhance user experience.

[0086] Figure 5 A structural block diagram of a distributed data query device according to an embodiment of the present invention is shown.

[0087] like Figure 5 As shown, the distributed data query device 500 includes a first determination module 510 , a screening module 520 , a second determination module 530 and a first sending module 540 .

[0088] The first determination module 510 is used to determine the N initial data nodes associated with the query request in response to receiving a query request from the user, so that the N initial data nodes each perform a data query to obtain a query result, wherein the N initial data nodes are part or all of the nodes in the distributed cluster, and N is an integer greater than or equal to 2. The screening module 520 is used to screen M intermediate data nodes from the N initial data nodes based on the available memory corresponding to each of the N initial data nodes, wherein M≤N, and M is a positive integer. The second determination module 530 is used to determine the target data node from the M intermediate data nodes based on the memory usage of the query results corresponding to each of the M intermediate data nodes. The first sending module 540 is used to send the query results corresponding to the other initial data nodes except the target data node to the target data node, so that the target data node merges the N query results and returns them to the user.

[0089] According to an embodiment of the present invention, the distributed data query device 500 may further include a locking module and a time setting module. The locking module is used to lock the target data node after determining the target data node. The time setting module is used to set the delay time of the target data node, wherein the target data node is prohibited from being used as a data collection node for other query requests during the delay time.

[0090] According to an embodiment of the present invention, the distributed data query device 500 may further include a second sending module. The second sending module is used to send the query results obtained by performing data queries on the N initial data nodes to the remote server when it is determined that the available memory amounts corresponding to the N initial data nodes are all less than the memory threshold, so that the remote server merges the N query results and returns them to the user.

[0091] According to an embodiment of the present invention, the distributed data query device 500 may further include a parallel processing module. The parallel processing module is used to process the multiple query requests in parallel when it is determined that multiple query requests are received.

[0092] According to an embodiment of the present invention, the distributed data query device 500 may further include a third determination module. The third determination module is used to determine the sequence of feeding back at least two merged query results corresponding to at least two query requests according to the time when the at least two query requests are received, when it is determined that the target data nodes corresponding to at least two query requests are the same target data node.

[0093] According to the embodiments of the present invention, any one or more of the modules, submodules, units, and subunits, or at least part of the functions of any one of them can be implemented in one module. According to the embodiments of the present invention, any one or more of the modules, submodules, units, and subunits can be split into multiple modules for implementation. According to the embodiments of the present invention, any one or more of the modules, submodules, units, and subunits can be at least partially implemented as hardware circuits, such as field programmable gate arrays (FPGAs), programmable logic arrays (PLAs), systems on chips, systems on substrates, systems on packages, application specific integrated circuits (ASICs), or can be implemented by hardware or firmware in any other reasonable way of integrating or packaging the circuit, or by any one of the three implementation methods of software, hardware, and firmware, or by a proper combination of any of them. Alternatively, according to the embodiments of the present invention, one or more of the modules, submodules, units, and subunits can be at least partially implemented as computer program modules, and when the computer program modules are run, the corresponding functions can be executed.

[0094] For example, any multiple of the first determination module 510, the screening module 520, the second determination module 530, and the first sending module 540 can be combined in one module / unit / sub-unit for implementation, or any one of the modules / units / sub-units can be split into multiple modules / units / sub-units. Alternatively, at least part of the functions of one or more of these modules / units / sub-units can be combined with at least part of the functions of other modules / units / sub-units and implemented in one module / unit / sub-unit. According to an embodiment of the present invention, at least one of the first determination module 510, the screening module 520, the second determination module 530, and the first sending module 540 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or implemented in any one of the three implementation methods of software, hardware, and firmware, or in any appropriate combination of any of them. Alternatively, at least one of the first determination module 510, the screening module 520, the second determination module 530 and the first sending module 540 may be at least partially implemented as a computer program module, and when the computer program module is executed, a corresponding function may be performed.

[0095] It should be noted that the part of the distributed data query device in the embodiment of the present invention corresponds to the part of the distributed data query method in the embodiment of the present invention. The description of the part of the distributed data query device specifically refers to the part of the distributed data query method, which will not be repeated here.

[0096] Figure 6 A block diagram of an electronic device suitable for implementing a distributed data query method according to an embodiment of the present invention is shown. Figure 6 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0097] like Figure 6As shown, the electronic device 600 according to an embodiment of the present invention includes a processor 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage part 608 to a random access memory (RAM) 603. The processor 601 may include, for example, a general-purpose microprocessor (such as a CPU), an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (for example, an application-specific integrated circuit (ASIC)), etc. The processor 601 may also include an onboard memory for caching purposes. The processor 601 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present invention.

[0098] In RAM 603, various programs and data required for the operation of electronic device 600 are stored. Processor 601, ROM 602 and RAM 603 are connected to each other via bus 604. Processor 601 performs various operations of the method flow according to the embodiment of the present invention by executing the programs in ROM 602 and / or RAM 603. It should be noted that the program can also be stored in one or more memories other than ROM 602 and RAM 603. Processor 601 can also perform various operations of the method flow according to the embodiment of the present invention by executing the programs stored in one or more memories.

[0099] According to an embodiment of the present invention, the electronic device 600 may further include an input / output (I / O) interface 605, which is also connected to the bus 604. The electronic device 600 may further include one or more of the following components connected to the input / output (I / O) interface 605: an input portion 606 including a keyboard, a mouse, etc.; an output portion 607 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage portion 608 including a hard disk, etc.; and a communication portion 609 including a network interface card such as a LAN card, a modem, etc. The communication portion 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the input / output (I / O) interface 605 as needed. A removable medium 611, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 610 as needed, so that a computer program read therefrom is installed into the storage portion 608 as needed.

[0100] According to an embodiment of the present invention, the method flow according to an embodiment of the present invention can be implemented as a computer software program. For example, an embodiment of the present invention includes a computer program product, which includes a computer program carried on a computer-readable storage medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 609, and / or installed from the removable medium 611. When the computer program is executed by the processor 601, the above-mentioned functions defined in the system of the embodiment of the present invention are executed. According to an embodiment of the present invention, the system, equipment, device, module, unit, etc. described above can be implemented by a computer program module.

[0101] The present invention also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiment; or may exist independently without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the method according to the embodiment of the present invention is implemented.

[0102] According to an embodiment of the present invention, the computer-readable storage medium may be a non-volatile computer-readable storage medium. For example, it may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present invention, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, an apparatus or a device.

[0103] For example, according to an embodiment of the present invention, the computer-readable storage medium may include the ROM 602 and / or the RAM 603 described above and / or one or more memories other than the ROM 602 and the RAM 603 .

[0104] An embodiment of the present invention also includes a computer program product, which includes a computer program, and the computer program contains program code for executing the method provided by the embodiment of the present invention. When the computer program product runs on an electronic device, the program code is used to enable the electronic device to implement the method provided by the embodiment of the present invention.

[0105] When the computer program is executed by the processor 601, the above functions defined in the system / device of the embodiment of the present invention are executed. According to the embodiment of the present invention, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0106] In one embodiment, the computer program may rely on tangible storage media such as optical storage devices, magnetic storage devices, etc. In another embodiment, the computer program may also be transmitted and distributed in the form of signals on a network medium, and downloaded and installed through the communication part 609, and / or installed from a removable medium 611. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0107] According to an embodiment of the present invention, the program code for executing the computer program provided by the embodiment of the present invention can be written in any combination of one or more programming languages. Specifically, these computing programs can be implemented using high-level process and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, Java, C++, python, "C" language or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on the remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (e.g., using an Internet service provider to connect through the Internet).

[0108] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present invention. In this regard, each box in the flowchart or block diagram may represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of the code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box may also occur in an order different from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions. It can be understood by those skilled in the art that the features recorded in the various embodiments of the present invention can be combined and / or combined in various ways, even if such a combination or combination is not explicitly recorded in the present invention. In particular, without departing from the spirit and teachings of the present invention, the features recorded in the various embodiments of the present invention can be combined and / or combined in various ways. All these combinations and / or combinations fall within the scope of the present invention.

[0109] The embodiments of the present invention are described above. However, these embodiments are only for the purpose of illustration, and are not intended to limit the scope of the present invention. Although each embodiment is described above, it does not mean that the measures in each embodiment cannot be used in combination advantageously. Without departing from the scope of the present invention, those skilled in the art may make various substitutions and modifications, which should all fall within the scope of the present invention.

Claims

1. A distributed data query method, characterized in that: The method comprises: In response to receiving a query request from a user, determine N initial data nodes associated with the query request, so that the N initial data nodes each perform a data query to obtain a query result, wherein the N initial data nodes are part or all of the nodes in the distributed cluster, and N is an integer greater than or equal to 2; Based on the available memory amounts corresponding to the N initial data nodes, M intermediate data nodes are selected from the N initial data nodes, where M≤N and M is a positive integer; Determine a target data node from the M intermediate data nodes based on the memory usage of the query results corresponding to each of the M intermediate data nodes; and The query results corresponding to the other initial data nodes except the target data node are sent to the target data node, so that the target data node merges the N query results and returns them to the user.

2. The method according to claim 1, characterized in that The determining the target data node from the M intermediate data nodes based on the memory usage of the query results respectively corresponding to the M intermediate data nodes includes: Based on the memory usage of the query results corresponding to each of the M intermediate data nodes, sorting the M intermediate data nodes in descending order to obtain a sorting result; and The intermediate data node located first in the sorting result is determined as the target data node.

3. The method according to claim 1, characterized in that The method further comprises: After determining the target data node, locking the target data node; and A delay time is set for the target data node, wherein the target data node is prohibited from being used as a data collection node for other query requests during the delay time.

4. The method according to claim 1, characterized in that The selecting M intermediate data nodes from the N initial data nodes based on the available memory amounts corresponding to the N initial data nodes respectively includes: When it is determined that the amount of available memory corresponding to each of the N initial data nodes is greater than or equal to a memory threshold, the initial data nodes whose corresponding amount of available memory is greater than or equal to the memory threshold are selected from the N initial data nodes as the intermediate data nodes.

5. The method according to claim 4, characterized in that The method further comprises: When it is determined that the available memory amounts corresponding to each of the N initial data nodes are less than the memory threshold, the query results obtained by performing data queries on each of the N initial data nodes are sent to a remote server, so that the remote server merges the N query results and returns them to the user.

6. The method according to any one of claims 1 to 5, characterized in that The method further comprises: When it is determined that a plurality of query requests are received, the plurality of query requests are processed in parallel.

7. The method according to claim 6, characterized in that The method further comprises: When it is determined that the target data nodes corresponding to at least two query requests are the same target data node, the order of feeding back at least two merged query results corresponding to the at least two query requests is determined according to the time when the at least two query requests are received.

8. A distributed data query device, characterized in that: The device comprises: A first determination module is used to determine, in response to receiving a query request from a user, N initial data nodes associated with the query request, so that the N initial data nodes each perform a data query to obtain a query result, wherein the N initial data nodes are part or all of the nodes in the distributed cluster, and N is an integer greater than or equal to 2; A screening module, configured to screen M intermediate data nodes from the N initial data nodes based on the available memory amounts corresponding to the N initial data nodes, where M≤N and M is a positive integer; A second determination module is configured to determine a target data node from the M intermediate data nodes based on the memory usage of the query results corresponding to each of the M intermediate data nodes; and The first sending module is used to send the query results corresponding to other initial data nodes except the target data node to the target data node, so that the target data node merges the N query results and returns them to the user.

9. An electronic device, comprising: one or more processors; a memory for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, enables the processor to implement the method according to any one of claims 1 to 7.

Images