Data query method, data query device, storage medium and electronic equipment

Abstract

The present disclosure provides a data query method and device, a storage medium and an electronic device, and relates to the technical field of data processing. The method comprises the following steps: obtaining query information; distributing the query information to multiple query nodes, obtaining first statistical information by querying a local database to count information of a field to be queried, and obtaining first data according to the query information; obtaining second statistical information by summarizing the first statistical information, dividing a value interval of the field to be queried into multiple first intervals according to the second statistical information; dividing and distributing the first data into multiple first aggregation nodes according to the first intervals, and obtaining second data by aggregating the first data; dividing the value interval of the field to be queried into second intervals according to the second statistical information; dividing and distributing the second data into second aggregation nodes according to the second intervals, and obtaining third data by aggregating the second data; and outputting a final query result corresponding to the query information according to the third data. The present disclosure improves the joint query speed.

Description

Technical Field

[0001] This disclosure relates to the field of data processing technology, and in particular to a data query method, a data query device, a computer-readable storage medium, and an electronic device. Background Technology

[0002] As more and more enterprises, educational institutions, and other organizations adopt distributed data and business systems, they often need to perform joint queries based on multiple distributed data tables when querying data. A common problem with these technologies is slow query speed. Summary of the Invention

[0003] This disclosure provides a data query method, a data query device, a computer-readable storage medium, and an electronic device to at least partially solve the problem of slow query speed in distributed systems.

[0004] According to a first aspect of this disclosure, a data query method is provided for querying data from multiple locally deployed databases in a distributed manner. The method includes: obtaining query information, the query information including a field to be queried; distributing the query information to multiple query nodes, obtaining first statistical information by statistically analyzing the information of the field to be queried from the associated local databases through the query nodes, and querying first data according to the query information; summarizing the first statistical information to obtain second statistical information, dividing the numerical range of the field to be queried into multiple first intervals according to the second statistical information; dividing the first data according to the first intervals and distributing it to multiple first aggregation nodes, and aggregating the first data through the first aggregation nodes to obtain second data; dividing the numerical range of the field to be queried into one or more second intervals according to the second statistical information; the number of second intervals is less than the number of first intervals; dividing the second data according to the second intervals and distributing it to one or more second aggregation nodes, and aggregating the second data through the second aggregation nodes to obtain third data; and outputting the final query result corresponding to the query information according to the third data.

[0005] According to a second aspect of this disclosure, a data query apparatus is provided for querying data from multiple locally deployed databases; the apparatus includes: a query information acquisition module configured to acquire query information, the query information including a field to be queried; a query node processing module configured to distribute the query information to multiple query nodes, obtain first statistical information by statistically analyzing the information of the field to be queried from the associated local databases through the query nodes, and query first data according to the query information; a first partitioning module configured to summarize the first statistical information to obtain second statistical information, and divide the numerical range of the field to be queried into multiple first intervals according to the second statistical information; and a first clustering module. The first data aggregation module is configured to divide and distribute the first data according to the first interval to multiple first aggregation nodes, and aggregate the first data through the first aggregation nodes to obtain second data; the second partitioning module is configured to divide the numerical range of the field to be queried into one or more second intervals according to the second statistical information; the number of second intervals is less than the number of first intervals; the second aggregation module is configured to divide and distribute the second data according to the second interval to one or more second aggregation nodes, and aggregate the second data through the second aggregation nodes to obtain third data; the query result output module is configured to output the final query result corresponding to the query information according to the third data.

[0006] According to a third aspect of this disclosure, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the data query method of the first aspect described above and its possible implementations.

[0007] According to a fourth aspect of this disclosure, an electronic device is provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to perform the data query method of the first aspect and possible implementations thereof by executing the executable instructions.

[0008] The technical solution disclosed herein has the following beneficial effects:

[0009] This paper presents a federated query scheme for distributed databases. The query information is distributed to multiple query nodes, each querying from its own local database. The scheme divides the data into numerical ranges based on statistical information, performs multiple aggregations on the first set of data obtained, and rapidly outputs the data downstream through a small-scale, multi-stage aggregation process to obtain the final query result. This reduces the data processing load and storage pressure on the first and second aggregation nodes, improves the speed of federated queries, reduces query time, and enhances data processing and management efficiency. Attached Figure Description

[0010] Figure 1A system architecture diagram of the operating environment of this exemplary embodiment is shown.

[0011] Figure 2 A flowchart illustrating a data query method in this exemplary embodiment is shown.

[0012] Figure 3 A flowchart illustrating the processing of the SourceFunction operator in this exemplary embodiment is shown.

[0013] Figure 4 A flowchart of one exemplary embodiment is shown.

[0014] Figure 5 A flowchart illustrating the processing of a query node in this exemplary embodiment is shown.

[0015] Figure 6 A flowchart of the process1 operator in this exemplary embodiment is shown.

[0016] Figure 7 The flowchart of the process3 operator in this exemplary embodiment is shown.

[0017] Figure 8 The flowchart of the process4 and 5 operators in this exemplary embodiment is shown.

[0018] Figure 9 The flowchart of the processing of the first partition node and the second partition node in this exemplary embodiment is shown.

[0019] Figure 10 Another process architecture diagram is shown in this exemplary embodiment.

[0020] Figure 11 This diagram illustrates the structure of a data query device according to an exemplary embodiment of the present invention.

[0021] Figure 12 A schematic diagram of the structure of an electronic device in this exemplary embodiment is shown. Detailed Implementation

[0022] Exemplary embodiments of this disclosure will be described more fully below with reference to the accompanying drawings.

[0023] The accompanying drawings are schematic illustrations of this disclosure and are not necessarily drawn to scale. Some block diagrams shown in the drawings may be functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities may be implemented in software, in hardware modules or integrated circuits, or in networks, processors, or microcontrollers. Implementations can be carried out in various forms and should not be construed as limited to the examples set forth herein. The features, structures, or characteristics described in this disclosure can be combined in any suitable manner in one or more embodiments. Numerous specific details are provided in the following description to give a thorough description of embodiments of this disclosure. However, those skilled in the art will recognize that one or more specific details may be omitted when implementing the technical solutions of this disclosure, or other methods, components, apparatuses, steps, etc., may be used to replace one or more specific details.

[0024] In related technologies, when performing multi-table join queries in a distributed environment, it is necessary to aggregate data from local tables distributed across multiple nodes into an intermediate table on a single node before executing aggregation statements. This results in slow aggregation query speeds. For example, if a local table has a large amount of data, slow queries on that local table can increase the overall query time.

[0025] In view of the above problems, an exemplary embodiment of this disclosure provides a data query method for querying data from multiple local databases deployed in a distributed manner.

[0026] Figure 1A system architecture diagram of the operating environment of this exemplary embodiment is shown. This system architecture 100 may include a messaging system 110, a streaming processing system 120, and a database system 130. The messaging system 110 is the party initiating the query, and may be deployed on the data request side. In one embodiment, the messaging system 110 may be a Kafka system. The streaming processing system 120 is the party executing the query, and can perform data querying and aggregation processing through a streaming framework. In one embodiment, the streaming processing system 120 may be a Flink system. The database system 130 includes multiple local databases, which may be deployed on different physical nodes to form a distributed data system. The data stored in different local databases may differ. In one embodiment, data can be divided according to one or more specific fields, and stored in the corresponding local database according to the division results. For example, local databases are deployed in different regions, and data is divided according to the "region" field and stored in the local database of the corresponding region. In one embodiment, the database system 130 may use an SQL (Structured Query Language) database. The messaging system 110, streaming processing system 120, and database system 130 can be connected via wired or wireless communication links to achieve data transmission. The messaging system 110 can respond to user-initiated query requests, generate query information based on user-input query conditions, and send the query information to the streaming processing system 120. The streaming processing system 120 can execute the data query method of this exemplary embodiment, query data from different local databases of the database system 130, and obtain the final query result through processing.

[0027] Figure 2 An exemplary flow of this data query method is shown, which may include the following steps S210 to S270:

[0028] Step S210: Obtain query information, which includes the fields to be queried;

[0029] Step S220: Distribute the query information to multiple query nodes, obtain the first statistical information by statistically analyzing the information of the field to be queried from the associated local database through the query nodes, and obtain the first data according to the query information;

[0030] Step S230: Summarize the first statistical information to obtain the second statistical information, and divide the numerical range of the field to be queried into multiple first intervals based on the second statistical information;

[0031] Step S240: Divide the first data according to the first interval and distribute it to multiple first aggregation nodes, and aggregate the first data through the first aggregation nodes to obtain the second data;

[0032] Step S250: Divide the numerical range of the field to be queried into one or more second intervals according to the second statistical information; the number of second intervals is less than the number of first intervals.

[0033] Step S260: Divide the second data according to the second interval and distribute it to one or more second aggregation nodes, and aggregate the second data through the second aggregation nodes to obtain the third data;

[0034] Step S270: Output the final query result corresponding to the query information based on the third data.

[0035] based on Figure 2 This method provides a joint query scheme for distributed databases. It distributes query information to multiple query nodes, each querying from its own local database. The system then divides the data into numerical ranges based on statistical information, performs multiple aggregations on the first set of data obtained, and rapidly outputs the data downstream through a small-scale, multi-stage aggregation process to obtain the final query result. This reduces the data processing load and storage pressure on the first and second aggregation nodes, improves the speed of joint queries, reduces query time, and enhances data processing and management efficiency.

[0036] The following is about Figure 2 Each step in the process will be explained in detail.

[0037] refer to Figure 2 In step S210, query information is obtained, which includes the field to be queried.

[0038] The query information is used to retrieve the required data and may include user-inputted query conditions. For example, the query information can be an SQL query statement. The field to be queried refers to the field specified in the query information that needs to be queried. For instance, if a user queries data using the query condition "call duration," then the "call duration" field is the field to be queried.

[0039] In one implementation, the query information may originate from the messaging system 110. For example, the messaging system 110 may be a Kafka system. When a user initiates a query request, the messaging system 110 generates query information based on the query conditions input by the user, and the streaming processing system 120 obtains the query information from the messaging system 110.

[0040] Continue to refer to Figure 2 In step S220, the query information is distributed to multiple query nodes. The query nodes collect information on the fields to be queried from the associated local database to obtain the first statistical information, and the first data is obtained by querying according to the query information.

[0041] In this exemplary embodiment, the streaming processing system 120 can be configured with nodes of different functions, such as query nodes, first aggregation nodes, second aggregation nodes, etc., which are instances used to perform data streaming processing. For example, the streaming processing system 120 is a Flink system, and the nodes can be operators of the Flink system or parallel subtasks of operators. Query nodes are instances used to execute query information, and different query nodes are in parallel relationship. Each query node is associated with one or more local databases. In one embodiment, the number of local databases is n, and the number of query nodes is also n. Each query node is associated with one local database, and different query nodes are associated with different local databases.

[0042] Query nodes can retrieve data from their associated local databases based on query information and compile statistics for the queried fields. The first data is the data retrieved by the query node; it can be raw data retrieved from the local database, which is then aggregated and processed to form the second, third, and so on data. The first statistical information is the information compiled by the query node from the local database, providing a reference for downstream aggregation and other processing. The query node can perform statistics on each queried field or only on the aggregated fields to obtain the first statistical information. Aggregated fields are the fields that need to be aggregated, such as fields used for range queries.

[0043] For example, the query information can be an SQL query statement. After receiving the SQL query statement, the query node can extract the aggregated fields as the fields to be queried, and execute statements such as select MIN(aggregate field), select MAX(aggregate field), and COUNT(1) From table_name(local database name) in its associated local database to obtain the first statistical information. The query node can also query data in its associated local database according to the query information to obtain the first data.

[0044] Continue to refer to Figure 2 In step S230, the first statistical information is summarized to obtain the second statistical information, and the numerical range of the field to be queried is divided into multiple first intervals based on the second statistical information.

[0045] It should be understood that since different query nodes are associated with different local databases, the first data and first statistical information obtained by different query nodes will also be different. The first statistical information is the statistical information of different local databases. After aggregating the first statistical information, the statistical information of all local databases is obtained, which is called the second statistical information. For example, the first statistical information includes the maximum and minimum values ​​of the field to be queried in each local database, as well as the number of data (or rows) in the local database. The second statistical information includes the maximum and minimum values ​​of the field to be queried in all local databases, as well as the total number of data (or rows). Therefore, the second statistical information includes the numerical range of the field to be queried, which can be divided into multiple first intervals to facilitate the rapid aggregation of the first data according to the first intervals. For example, if the numerical range of the field to be queried is [min, max], it can be divided into 4 first intervals, namely...

[0046] Continue to refer to Figure 2 In step S240, the first data is divided into a first interval and distributed to multiple first aggregation nodes, and the first data is aggregated through the first aggregation nodes to obtain the second data.

[0047] Here, a first aggregation node is an instance used to perform data aggregation. Different first aggregation nodes are in parallel relationship, and each first aggregation node can correspond to one or more first intervals. In one implementation, the number of first intervals is p, and the number of first aggregation nodes is also p. Each first aggregation node corresponds to one first interval, and different first aggregation nodes correspond to different first intervals. Alternatively, if the number of first aggregation nodes is determined first, in step S230, the numerical range of the field to be queried can be divided into the same number of first intervals.

[0048] The first aggregation node receives the first data within its corresponding first interval and aggregates it to form the second data. It should be understood that the first data of each query node may be distributed to different first aggregation nodes, and each first aggregation node may receive the first data from different query nodes.

[0049] Continue to refer to Figure 2 In step S250, the numerical range of the field to be queried is divided into one or more second intervals according to the second statistical information; the number of second intervals is less than the number of first intervals.

[0050] The second interval is a range defined by a different scale than the first interval. The first interval is a relatively finely divided interval, while the second interval is a relatively larger range. In one implementation, each second interval can be the union of multiple first intervals. For example, if the numerical range of the field to be queried is [min, max], it can be divided into two second intervals, namely...

[0051] Continue to refer to Figure 2 In step S260, the second data is divided according to the second interval and distributed to one or more second aggregation nodes, and the second data is aggregated through the second aggregation nodes to obtain the third data.

[0052] In this system, the second aggregation node is an instance used to perform data aggregation. Different second aggregation nodes operate in parallel, and each second aggregation node can correspond to one or more second intervals. In one implementation, the number of second intervals is q, and the number of second aggregation nodes is also q. Each second aggregation node corresponds to one second interval, and different second aggregation nodes correspond to different second intervals. Alternatively, if the number of second aggregation nodes is determined beforehand, in step S250, the numerical range of the field to be queried can be divided into the same number of second intervals. The aggregation degree of the second aggregation node is higher than that of the first aggregation node; the higher the aggregation degree of the second aggregation node, the better.

[0053] The second aggregation node receives the second data within its corresponding second interval and aggregates it to form the third data. It should be understood that the second data of each first aggregation node may be distributed to different second aggregation nodes, and each second aggregation node may receive second data from different first aggregation nodes.

[0054] The first aggregation node performs the first aggregation, and the second aggregation node performs the second aggregation. Through dual aggregation, the scattered data retrieved is gradually aggregated into the whole, and the aggregation speed is improved by a small number of multiple aggregations.

[0055] Continue to refer to Figure 2 In step S270, the final query result corresponding to the query information is output based on the third data.

[0056] It should be understood that if the number of elements in the second interval is 1, then after aggregating the second data into the third data, the data aggregation is complete, and the third data can be output as the final query result. If the number of elements in the second interval is greater than 1, then after obtaining the third data, the third data can be fully aggregated to obtain the final query result and output it.

[0057] In one implementation, after obtaining the query information, the data query method may further include the following steps:

[0058] The identifier for the retrieved information;

[0059] The identifier of the query information is added to the first data by querying the node.

[0060] The identifier for the query information is used to distinguish different query information. In one implementation, the query information originates from the message system 110; the identifier for the query information can be generated based on the storage address offset of the query information in the message system 110 and the current timestamp. For example, if the message system 110 is a Kafka system and the query information is an SQL query statement, the storage address offset of the SQL query statement in the Kafka system combined with the current timestamp can be used as the identifier for the SQL query statement. Alternatively, the identifier for the query information can also be generated based on other information, such as extracting the MD5 hash or other digest information of the query information and generating an identifier based on the combination of the digest information and the current timestamp. Alternatively, the message system 110 can assign an identifier to the query information; for example, when generating the query information, the message system 110 can generate a string based on incrementing rules as the identifier for the query information.

[0061] The query node can add the identifier of the query information to the first data. Correspondingly, in step S240, the first data with the same identifier can be aggregated through the first aggregation node to obtain the second data, and the identifier of the query information is retained in the second data. In step S260, the second data with the same identifier can be aggregated through the second aggregation node to obtain the third data, and the identifier of the query information can also be retained in the third data. In the streaming processing system 120, there is a large amount of query information, and based on the characteristics of the streaming framework, there may be no boundaries between different batches of data. Aggregating the first or second data using the identifier of the query information can avoid aggregating data from different batches and ensure the accuracy of the query results.

[0062] In one implementation, the data query method may further include the following steps before obtaining the query information:

[0063] Get the address of the local database;

[0064] The connection information of the query node is set according to the address of the local database, so that the query node accesses the local database according to the address of the local database when executing the query information.

[0065] The local database address can be an IP address, etc. Based on the local database address, query nodes can be initialized, and their connection information can be set to connect to the corresponding IP address. For example, to obtain the addresses of n local databases, n query nodes can be set up, and the connection information of each query node can be set to a different address. This way, when subsequent query nodes execute queries, they can access the local database according to its address, with each query node accessing its associated local database.

[0066] In one implementation, both the first aggregation node and the second aggregation node are provided with containers; wherein, the process of aggregating the first data through the first aggregation node to obtain the second data may include the following steps:

[0067] The first data is aggregated in the container of the first aggregation node to obtain the second data;

[0068] The above-mentioned method of aggregating second data through a second aggregation node to obtain third data may include the following steps:

[0069] The second data is aggregated in the container of the second aggregation node to obtain the third data.

[0070] Therefore, container technology allows multiple first aggregation nodes and / or second aggregation nodes to be set up on a single physical machine. It also enables data isolation between different first aggregation nodes, between different second aggregation nodes, and between first and second aggregation nodes.

[0071] In one implementation, the data query method may further include the following steps:

[0072] Based on the second statistical information, set the container size of the first aggregation node and the container size of the second aggregation node.

[0073] The second statistical information reflects the total amount of data in the local database (e.g., the total number of rows), providing a reference for the amount of data aggregation to be performed by the first and second aggregation nodes. Generally, the larger the total amount of data in the local database, the larger the containers of the first and second aggregation nodes should be. In one implementation, the container size of the first and second aggregation nodes can also be determined by considering the number of local databases. Generally, the more local databases there are, the greater the probability of data duplication with the same aggregation field value, and the container size of the first and second aggregation nodes can be appropriately reduced.

[0074] By setting an appropriate container size, it is possible to adapt to different levels of data aggregation, meet the needs of joint data queries, and avoid excessively large containers, thus saving storage space resources.

[0075] In one implementation, when aggregating first data through a first aggregation node to obtain second data, the aggregated second data is output through the first aggregation node when one or more of the following conditions are met:

[0076] ① Aggregate the first data with the same field value until a first preset number of aggregations is reached. The first preset number of aggregations can be determined based on experience and specific circumstances, such as... K represents the parallelism of the upstream node (such as the query node). If the first data with the same field value has been aggregated a preset number of times, it means that most of the first data with the same aggregation field value has been aggregated. It is unnecessary to wait until all the first data with the same aggregation field value is aggregated before sending it to the downstream node, thus saving container space.

[0077] ② The containers of the first aggregation node have been filled to the first preset capacity. The first preset capacity refers to a capacity that is full or nearly full, such as 90% of the total container capacity, and can be determined empirically. When the containers of the first aggregation node have been filled to the preset capacity, there is not enough space to continue data aggregation. At this point, the obtained second data is output to the downstream nodes, freeing up space to continue data aggregation. This is equivalent to converting from streaming processing to micro-batch processing, which can still process data quickly while reducing the storage pressure on the containers.

[0078] ③ The number of end markers in the aggregated first data reaches a first preset number. The end marker can be a character representing an end flag or an end flag line, etc., added to the first data by the query node. For example, after retrieving the first data, the query node adds an end flag line to the end of the first data. The first preset number can be determined based on experience, or it can be equal to the number of query nodes or the local database. When the number of end markers in the aggregated first data reaches the preset number, it indicates that the same batch of query data has been aggregated and can be output to downstream nodes.

[0079] In one implementation, when aggregating second data to obtain third data through a second aggregation node, the aggregated third data is output through the second aggregation node when one or more of the following conditions are met:

[0080] ④ Aggregate second data with the same field values ​​to reach a second preset number of times. The second preset number of times can be determined based on experience and specific circumstances, such as... K represents the parallelism of the upstream node (such as the first aggregation node). The second preset number of iterations can differ from the first preset number of iterations. If the second data with the same field value has been aggregated to the preset number of iterations, it means that most of the second data with the same aggregation field value has been aggregated. It is not necessary to wait until all the second data with the same aggregation field value has been aggregated before sending it to the downstream node, thus saving container space.

[0081] ⑤ The containers of the second aggregation node have been stored to the second preset capacity. The second preset capacity refers to a capacity that is full or nearly full, such as 90% of the total container capacity, and can be determined empirically. The second preset capacity may differ from the first preset capacity. When the containers of the second aggregation node have been stored to the preset capacity, there is insufficient space to continue data aggregation. In this case, the obtained third data is output to the downstream node, freeing up space to continue data aggregation.

[0082] ⑥ The number of end markers in the aggregated second data reaches a second preset number. The end markers are added to the first data by the query node, and the first aggregation node can retain the end markers in the second data when aggregating the first data. The second preset number can be determined based on experience, or it can be equal to the number of query nodes or the local database. When the number of end markers in the aggregated second data reaches the preset number, it indicates that the same batch of query data has been fully aggregated and can be output to downstream nodes.

[0083] In one implementation, the streaming system 120 can be a Flink system, and the functional nodes in the streaming system 120 can be operators of the Flink system. The specific implementation of data querying in the Flink system is described below.

[0084] The node for retrieving query information can be a SourceFunction operator. The process for the SourceFunction operator can be found in [reference needed]. Figure 3 As shown, it includes:

[0085] Step S301: Pull Kafka topic data;

[0086] Step S302: Obtain the Kafka message offset;

[0087] Step S303: Obtain the message based on the Kafka message offset, and determine whether the message is IP data (i.e., the address of the local database). If yes, proceed to step S304; otherwise, proceed to step S305.

[0088] Step S304: Send IP data to downstream nodes, such as to the query node, so that the query node accesses the local database according to the address of the local database when executing the query information.

[0089] Step S305: Obtain the current timestamp, integrate the message offset, current timestamp, and aggregation fields into the SQL message, and copy it n times (n being the number of local databases and query nodes), sending it to downstream nodes. The partition nodes associated with the query information retrieval node can use the Rebalance operator (balanced partition operator) to send the query information to the downstream process1 operator (i.e., the query node) in a round-robin fashion.

[0090] refer to Figure 4 As shown, query nodes are set up, which can be the `process1` operator. The number of local databases is n (e.g., 5), and the parallelism of the `process1` operator is also set to n. Each subtask of the `process1` operator is a query node. n query nodes simultaneously receive query information sent from upstream. The processing of query nodes can be referenced. Figure 5 As shown, the received query information includes the SQL query statement, the storage address offset of the SQL query statement in Kafka, and the current timestamp. The query node obtains the first data by querying the local database. The output of the first data includes: row number, field value combination, and query information identifier (Uid, which can be a combination of storage address offset and current timestamp). The query node can also perform statistical analysis on the queried fields in the local database to obtain the first statistical information. In addition, the query node can add an end marker row to the end of the first data.

[0091] In one implementation, the processing flow of the process1 operator is referenced. Figure 6 As shown, it includes:

[0092] Step S601: Obtain information sent by the upstream node, which is query information (such as SQL query statement) or the IP address of the local database;

[0093] Step S602: Determine whether the obtained information is an IP address. If yes, proceed to step S603; otherwise, proceed to step S604.

[0094] Step S603: Configure the connection to the local database based on the IP address;

[0095] Step S604: Obtain the SQL query statement;

[0096] Step S605: Obtain the aggregated field and perform statistics, then send the first statistical information to the process2 operator;

[0097] Step S606: Query the local database to obtain the first data;

[0098] Step S607: Append the identifier of the SQL query statement to the first data, send the data to process3 one by one, and add an end marker line when the dataset is sent.

[0099] Set up a statistical summary node, which can be the process2 operator. It collects the first statistical information, obtains the second statistical information, and sends it to the downstream node.

[0100] The information integration node can be the process3 operator, with a parallelism of n. The process1 operator sends the first data to the process3 operator using the ForwardPartitioner method. The processing flow of the process3 operator can be found in [reference needed]. Figure 7 As shown, it includes:

[0101] Step S701: Obtain information sent by the upstream node, which is either second statistical information or first data;

[0102] Step S702: Determine whether the information comes from the process2 operator. If yes, proceed to step S703; otherwise, proceed to step S704.

[0103] Step S703: Determine the information as the second statistical information, and save the second statistical information (mainly including the minimum and maximum values ​​of the aggregated fields, the total number of rows in the local database, and may also include Uid), such as by storing it in a Map;

[0104] Step S704: Determine the information as first data, retrieve the corresponding second statistical information from the saved information based on the identifier (such as Uid) in the first data, add it to the first data, and send it to the downstream node.

[0105] The first partition node is set up, which can be a custom Partitioner operator, corresponding one-to-one with the information integration node. The first partition node divides the numerical range of the queried field into multiple first intervals based on the second statistical information, and sets an index value for each first interval. For example, the index values ​​for the four first intervals are 0, 1, 2, and 3, corresponding to the downstream first aggregation node. First data with the same partition index are routed to the same downstream node (i.e., the same subtask in process4) for processing.

[0106] The first aggregation node can be the process4 operator with a parallelism of p, meaning that process4 has p parallel subtasks, and each subtask is a first aggregation node.

[0107] A second partition node is set up, which can be a custom Partitioner operator, corresponding one-to-one with the first aggregation node. The second partition node divides the numerical range of the query field into multiple second intervals based on the second statistical information, and sets an index value for each second interval. For example, the index values ​​for two second intervals might be 0 and 1, corresponding to downstream second aggregation nodes. Second data with the same partition index are routed to the same downstream node (i.e., the same subtask in process5) for processing. This ensures that data with the same aggregation field value is aggregated at the same downstream node, efficiently speeding up the aggregation process and reducing the amount of data in subsequent aggregations.

[0108] The second aggregation node can be the process5 operator with a parallelism of q, meaning that process5 has q parallel subtasks, and each subtask is a second aggregation node.

[0109] In one implementation, the processing flow of the process4 and 5 operators can be referred to Figure 8 As shown, it includes:

[0110] Step S801: Extract input data (first data or second data);

[0111] Step S802: Determine whether the number of end marker lines is a preset number (first preset number or second preset number); if yes, proceed to step S803; if no, proceed to step S804.

[0112] Step S803: Output all data in the container to the downstream node and add an end flag line;

[0113] Step S804: Aggregate the input data with the data in the same Uid container;

[0114] Step S805: Determine whether the container has been stored to the preset capacity. If yes, proceed to step S806; otherwise, proceed to step S807.

[0115] Step S806: Output all data in the container to the downstream node;

[0116] Step S807: Determine whether the number of aggregations of the input data has reached the preset number. If yes, proceed to step S808; otherwise, proceed to step S809.

[0117] Step S808: Output the aggregated data to the downstream node;

[0118] Step S809: Store the data in a container with the same Uid.

[0119] In one implementation, the processing flow for the first partition node and the second partition node can be referred to Figure 9As shown, it includes:

[0120] Step S901: Read the input data;

[0121] Step S902: Obtain SQL aggregation information (such as first statistical information and second statistical information);

[0122] Step S903: Allocate data partition indexes based on the maximum and minimum values ​​of the aggregated fields (such as the maximum or minimum value in the first or second statistical information).

[0123] Set up a full aggregation node, which can be a process6 operator with a parallelism of 1, to fully aggregate the third-party data sent by the upstream node and obtain the final query result.

[0124] The `process4`, `5`, and `6` operators can dynamically adjust their container sizes based on the total number of rows in the local database, adapting to data aggregation queries with varying row counts and saving memory. When the data aggregated from the upstream node reaches the container's set value, the data is then sent to the downstream node to prevent data backlog. Since the data is randomly and evenly distributed across the local databases, the more local databases there are, the greater the probability of duplicate rows with the same aggregation field value. Assuming the number of local databases is T, the container sizes for `process4`, `5`, and `6` can be set as follows: COUNT(1) represents the total number of rows in the second set of statistics. The Process4 and 5 operators have relatively small container capacities and are used for micro-batch aggregation of the first or second data before sending it to downstream nodes. The Process6 operator has a relatively large container capacity and is used for full aggregation of all data. If the container capacity of the Process6 operator is insufficient, it can be expanded.

[0125] Figure 10 The flowchart illustrates the data query method's architecture. The input SQL statement (query information) is retrieved from Kafka, identified by the SourceFunction operator, and distributed to a subtask of the process1 operator via partitioning operator 1. This subtask performs queries and statistics (i.e., local table aggregation) on the local database (e.g., local table 1, local table 2). Process2 outputs the second set of statistics. Process3 integrates the first and second sets of statistics. Processes 4, 5, and 6 implement the group by function for multiple aggregations. Process 7 implements the order by function. Finally, the sink operator outputs the final query result.

[0126] Exemplary embodiments of this disclosure also provide a data query apparatus. (See reference...) Figure 11 As shown, the data query device 1100 may include the following program modules:

[0127] The query information acquisition module 1110 is configured to acquire query information, which includes the fields to be queried.

[0128] The query node processing module 1120 is configured to distribute query information to multiple query nodes, obtain first statistical information by statistically analyzing the information of the field to be queried from the associated local database through the query nodes, and obtain first data by querying according to the query information.

[0129] The first partitioning module 1130 is configured to summarize the first statistical information to obtain the second statistical information, and divide the numerical range of the field to be queried into multiple first intervals based on the second statistical information;

[0130] The first aggregation module 1140 is configured to divide the first data according to the first interval and distribute it to multiple first aggregation nodes, and aggregate the first data through the first aggregation nodes to obtain the second data.

[0131] The second partitioning module 1150 is configured to divide the numerical range of the field to be queried into one or more second intervals based on the second statistical information; the number of second intervals is less than the number of first intervals.

[0132] The second aggregation module 1160 is configured to divide the second data according to the second interval and distribute it to one or more second aggregation nodes, and aggregate the second data through the second aggregation nodes to obtain the third data.

[0133] The query result output module 1170 is configured to output the final query result corresponding to the query information based on the third data.

[0134] In one embodiment, the query information acquisition module 1110 is further configured to: acquire the identifier of the query information after acquiring the query information; the query node processing module 1120 is further configured to: add the identifier of the query information to the first data through the query node; the above-mentioned aggregating the first data through the first aggregation node to obtain the second data includes: aggregating the first data with the same identifier through the first aggregation node to obtain the second data, and retaining the identifier of the query information in the second data; the above-mentioned aggregating the second data through the second aggregation node to obtain the third data includes: aggregating the second data with the same identifier through the second aggregation node to obtain the third data.

[0135] In one implementation, the query information originates from a messaging system; obtaining the identifier of the query information includes generating an identifier for the query information based on the storage address offset of the query information in the messaging system and the current timestamp.

[0136] In one embodiment, the query information acquisition module 1110 is further configured to: acquire the address of the local database before acquiring the query information; the query node processing module 1120 is further configured to: set the connection information of the query node according to the address of the local database, so that the query node accesses the local database according to the address of the local database when executing the query information.

[0137] In one embodiment, both the first aggregation node and the second aggregation node are provided with containers; wherein, aggregating the first data through the first aggregation node to obtain the second data includes: aggregating the first data in the container of the first aggregation node to obtain the second data; aggregating the second data through the second aggregation node to obtain the third data includes: aggregating the second data in the container of the second aggregation node to obtain the third data.

[0138] In one embodiment, the data query device 1100 further includes a container size setting module, configured to: set the container size of the first aggregation node and the container size of the second aggregation node according to the second statistical information.

[0139] In one implementation, when aggregating first data through a first aggregation node to obtain second data, the aggregated second data is output through the first aggregation node when one or more of the following conditions are met: the first data with the same field value is aggregated a first preset number of times; the container of the first aggregation node has been stored to a first preset capacity; the number of end markers in the aggregated first data reaches a first preset number; and the end marker is added to the first data by a query node.

[0140] The specific details of each part of the above-mentioned device have been described in detail in the method section of the implementation plan. For any undisclosed details, please refer to the implementation plan of the method section, and therefore will not be repeated here.

[0141] Exemplary embodiments of this disclosure also provide a computer-readable storage medium that can be implemented as a program product including program code, which, when run on an electronic device, causes the electronic device to perform the steps described in the "Exemplary Methods" section of this specification according to various exemplary embodiments of this disclosure. In an alternative embodiment, the program product can be implemented as a portable compact disc read-only memory (CD-ROM) including program code and can run on an electronic device, such as a personal computer. However, the program product of this disclosure is not limited thereto. In this document, the readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0142] The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (a non-exhaustive list) of readable storage media include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0143] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, capable of sending, propagating, or transmitting programs for use by or in conjunction with an instruction execution system, apparatus, or device.

[0144] The program code contained on the readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0145] Program code for performing the operations of this disclosure can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing devices can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0146] Exemplary embodiments of this disclosure also provide an electronic device. The electronic device may include a processor and a memory. The memory stores executable instructions for the processor, such as program code. The processor executes the executable instructions to perform the methods of this exemplary embodiment.

[0147] The following is for reference. Figure 12The electronic device is illustrated by way of a general-purpose computing device. It should be understood that... Figure 12 The electronic device 1200 shown is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments disclosed herein.

[0148] like Figure 12 As shown, the electronic device 1200 may include: a processor 1210, a memory 1220, a bus 1230, an I / O (input / output) interface 1240, and a network adapter 1250.

[0149] Memory 1220 may include volatile memory, such as RAM 1221 and cache unit 1222, and may also include non-volatile memory, such as ROM 1223. Memory 1220 may also include one or more program units 1224, such program units 1224 including, but not limited to, an operating system, one or more application programs, other program units, and program data. Each or some combination of these examples may include an implementation of a network environment. For example, program unit 1224 may include the modules in the above-described apparatus.

[0150] Bus 1230 is used to connect different components of electronic device 1200, and may include data bus, address bus and control bus.

[0151] Electronic device 1200 can communicate with one or more external devices 1300 (e.g., keyboard, mouse, external controller, etc.) through I / O interface 1240.

[0152] Electronic device 1200 can communicate with one or more networks via network adapter 1250. For example, network adapter 1250 can provide mobile communication solutions such as 3G / 4G / 5G, or wireless communication solutions such as wireless LAN, Bluetooth, and near-field communication. Network adapter 1250 can communicate with other units of electronic device 1200 via bus 1230.

[0153] although Figure 12 Other hardware and / or software units may also be provided in the electronic device 1200, including but not limited to: display, microcode, device driver, redundant processor, external disk drive array, RAID system, tape drive and data backup storage system, etc.

[0154] It should be noted that although several units or modules of the device for performing actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to exemplary embodiments of this disclosure, the features and functions of two or more units or modules described above can be embodied in one unit or module. Conversely, the features and functions of one unit or module described above can be further divided and embodied by multiple units or modules.

[0155] Those skilled in the art will understand that various aspects of this disclosure can be implemented as systems, methods, or program products. Therefore, various aspects of this disclosure can be embodied in entirely hardware implementations, entirely software implementations (including firmware, microcode, etc.), or implementations combining hardware and software aspects, collectively referred to herein as “circuit,” “unit,” or “system.” Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and embodiments are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the claims. It should be understood that this disclosure is not limited to the precise structures described above and shown in the drawings, and various modifications and changes can be made without departing from its scope.

Claims

1. A data query method, characterized in that, The method is used to query data from multiple local databases deployed in a distributed manner; the method includes: Obtain query information, which includes the field to be queried; The query information is distributed to multiple query nodes. The query nodes collect information about the field to be queried from the associated local database to obtain first statistical information, and then query the first data according to the query information. The first statistical information is summarized to obtain the second statistical information, and the numerical range of the field to be queried is divided into multiple first ranges based on the second statistical information; The first data is divided and distributed to multiple first aggregation nodes according to the first interval, and the first data is aggregated through the first aggregation nodes to obtain the second data; Based on the second statistical information, the numerical range of the field to be queried is divided into one or more second intervals; the number of second intervals is less than the number of first intervals; The second data is divided according to the second interval and distributed to one or more second aggregation nodes. The second data is then aggregated through the second aggregation nodes to obtain the third data. The final query result corresponding to the query information is output based on the third data.

2. The method according to claim 1, characterized in that, After obtaining the query information, the method further includes: Obtain the identifier of the query information; The identifier of the query information is added to the first data through the query node; The process of aggregating the first data through the first aggregation node to obtain the second data includes: The first data with the same identifier is aggregated by the first aggregation node to obtain the second data, and the identifier of the query information is retained in the second data; The process of aggregating the second data through the second aggregation node to obtain the third data includes: The third data is obtained by aggregating the second data with the same identifier through the second aggregation node.

3. The method according to claim 2, characterized in that, The query information originates from the messaging system; the identifier for obtaining the query information includes: An identifier for the query information is generated based on the storage address offset of the query information in the messaging system and the current timestamp.

4. The method according to claim 1, characterized in that, Before obtaining the query information, the method further includes: Get the address of the local database; The connection information of the query node is set according to the address of the local database, so that the query node accesses the local database according to the address of the local database when executing the query information.

5. The method according to claim 1, characterized in that, Both the first aggregation node and the second aggregation node are equipped with containers; wherein, the step of aggregating the first data through the first aggregation node to obtain the second data includes: The first data is aggregated in the container of the first aggregation node to obtain the second data; The process of aggregating the second data through the second aggregation node to obtain the third data includes: The second data is aggregated in the container of the second aggregation node to obtain the third data.

6. The method according to claim 5, characterized in that, The method further includes: Based on the second statistical information, set the container size of the first aggregation node and the container size of the second aggregation node.

7. The method according to claim 5, characterized in that, When aggregating the first data through the first aggregation node to obtain the second data, the aggregated second data is output through the first aggregation node when one or more of the following conditions are met: Aggregate the first data with the same field value to reach the first preset number of times; The container of the first aggregation node has been stored to the first preset capacity; The number of end markers in the aggregated first data reaches a first preset number; the end markers are added to the first data by the query node.

8. A data query device, characterized in that, For querying data from multiple local databases deployed in a distributed manner; the apparatus includes: The query information acquisition module is configured to acquire query information, which includes the field to be queried. The query node processing module is configured to distribute the query information to multiple query nodes, obtain first statistical information by statistically analyzing the information of the field to be queried from the associated local database through the query nodes, and obtain first data by querying according to the query information; The first partitioning module is configured to summarize the first statistical information to obtain second statistical information, and divide the numerical range of the field to be queried into multiple first intervals based on the second statistical information; The first aggregation module is configured to divide the first data according to the first interval and distribute it to multiple first aggregation nodes, and aggregate the first data through the first aggregation nodes to obtain the second data. The second partitioning module is configured to divide the numerical range of the field to be queried into one or more second intervals based on the second statistical information; the number of second intervals is less than the number of first intervals. The second aggregation module is configured to divide the second data according to the second interval and distribute it to one or more second aggregation nodes, and aggregate the second data through the second aggregation nodes to obtain the third data; The query result output module is configured to output the final query result corresponding to the query information based on the third data.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method described in any one of claims 1 to 7.

10. An electronic device, characterized in that, include: processor; as well as Memory for storing the executable instructions of the processor; The processor is configured to execute the method of any one of claims 1 to 7 by executing the executable instructions.

Images