Method for data query and method for distributed federated analysis
By pre-storing historical query results in the cache of worker nodes, the network I/O performance bottleneck in the distributed federated analysis system is solved, improving query performance and cache hit rate, and increasing data query efficiency.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- BEIJING BAIDU NETCOM SCI & TECH CO LTD
- Filing Date
- 2022-11-21
- Publication Date
- 2026-07-14
AI Technical Summary
In distributed federated analysis systems, network I/O performance bottlenecks and high network transmission costs lead to a decline in query performance.
By pre-storing historical query results in the cache of worker nodes, and using the pre-existing query results in the target cache to determine the current query result, the process of repeatedly traversing the target data source is reduced, thereby improving query performance.
It significantly improves cache hit rate and overall performance of distributed systems when query conditions change frequently, reduces network I/O overhead, and improves data query efficiency.
Smart Images
Figure CN115964437B_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates to the field of artificial intelligence technology, and in particular to the fields of cloud computing and big data technology, and can be applied to intelligent cloud scenarios. Background Technology
[0002] With the development of big data technology and the further evolution of traditional database technology, enterprises have more and more choices in data architecture. Currently, data in an enterprise is usually stored in multiple data sources, and for enterprises with complex businesses, there may even be dozens or hundreds of data sources. For data querying, mining and analysis, it is often necessary to cross multiple business systems. At this time, a distributed computing engine with federated query capabilities is needed to use multiple parallel nodes to obtain data from different data sources for federated analysis. Summary of the Invention
[0003] This disclosure provides a method for data querying and a method for distributed federated analysis.
[0004] According to one aspect of this disclosure, a data query method is provided, applied to worker nodes of a distributed system, including:
[0005] Determine the target data source corresponding to the current query command, wherein the target attribute data stored in the target data source is divided into multiple intervals; and
[0006] If the first interval among multiple intervals is determined to be within the data query range of the current query instruction and has a corresponding target cache, the first query result is determined based on the current query instruction and the target cache.
[0007] According to another aspect of this disclosure, a method for distributed federated analysis is provided, applied to a scheduling node in a distributed system, comprising:
[0008] Based on the user's query command, determine multiple worker nodes from the distributed system;
[0009] The user query command is broken down into multiple current query commands, which are then sent one-to-one to multiple worker nodes. Each worker node performs a data query based on one of the multiple current query commands, according to the method of any embodiment of this disclosure.
[0010] Based on the query results fed back from multiple worker nodes, the federated analysis query results of the user's query command are determined.
[0011] According to another aspect of this disclosure, a data query apparatus is provided, comprising:
[0012] The first determining module is used to determine the target data source corresponding to the current query command, wherein the target attribute data stored in the target data source is divided into multiple intervals; and
[0013] The second determining module is used to determine the first query result based on the current query instruction and the target cache, provided that the first interval among multiple intervals is located within the data query range of the current query instruction and a corresponding target cache exists.
[0014] According to another aspect of this disclosure, an apparatus for distributed federated analysis is provided, applied to a scheduling node in a distributed system, comprising:
[0015] The node determination module is used to determine multiple worker nodes from the distributed system based on user query commands;
[0016] The sending module is configured to split a user query instruction into multiple current query instructions and send them one-to-one to multiple worker nodes, wherein the multiple worker nodes perform data queries based on the multiple current query instructions according to the method of any embodiment of this disclosure; and
[0017] The result determination module is used to determine the federated analysis query result of the user's query command based on the query results fed back by multiple worker nodes.
[0018] According to another aspect of this disclosure, an electronic device is provided, comprising:
[0019] At least one processor; and
[0020] The memory is communicatively connected to the at least one processor; wherein,
[0021] The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the methods of any embodiment of the present disclosure.
[0022] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are used to cause the computer to perform a method according to any embodiment of this disclosure.
[0023] According to another aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements a method according to any embodiment of this disclosure.
[0024] According to the technology disclosed herein, the efficiency of data retrieval can be improved.
[0025] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description
[0026] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:
[0027] Figure 1 This is a schematic diagram of a data query method according to an embodiment of the present disclosure;
[0028] Figure 2 This is a schematic diagram illustrating an application scenario according to an embodiment of this disclosure;
[0029] Figure 3 This is a schematic diagram illustrating an application scenario according to an embodiment of this disclosure;
[0030] Figure 4 This is a schematic diagram illustrating an application scenario according to an embodiment of this disclosure;
[0031] Figure 5 This is a schematic diagram of a distributed federated analysis method according to an embodiment of the present disclosure;
[0032] Figure 6 This is a schematic diagram of a data query apparatus according to an embodiment of the present disclosure;
[0033] Figure 7 This is a schematic diagram of an apparatus for distributed federated analysis according to an embodiment of the present disclosure;
[0034] Figure 8 This is a block diagram of an electronic device used to implement the data query method and / or the distributed federated analysis method of the embodiments of this disclosure. Detailed Implementation
[0035] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.
[0036] like Figure 1 As shown, this disclosure provides a data query method applied to a worker node in a distributed system, including:
[0037] Step S101: Determine the target data source corresponding to the current query command, wherein the target attribute data stored in the target data source is divided into multiple intervals.
[0038] Step S102: If the first interval among multiple intervals is determined to be within the data query range of the current query instruction and a corresponding target cache exists, determine the first query result based on the current query instruction and the target cache.
[0039] According to the embodiments of this disclosure, it should be noted that:
[0040] A distributed system can be understood as any distributed federated analysis engine in existing technology. A worker node can be understood as a node in the distributed federated analysis engine used to perform data querying, mining, analysis, and computation. Worker nodes can be assigned current query instructions by the scheduling node in the distributed federated analysis engine, enabling them to execute data query tasks based on the assigned current query instructions. The execution entity of the data query method in this embodiment can be understood as a worker node. The architecture of the distributed federated analysis engine can be referenced. Figure 2 , Figure 2 Worker node 1, worker node 2, and worker node 3 can all serve as the execution entities for the data query method in this embodiment of the disclosure. It should be noted that... Figure 2 This is only used as an example to illustrate the architecture of the distributed federated analysis engine. The specific distributed federated analysis engine used can be adjusted as needed. The number of worker nodes, the connection relationships between worker nodes, and the connection relationships between worker nodes and scheduling nodes in the distributed federated analysis engine can all be adjusted as needed.
[0041] The data query method of this disclosure can be applied to any business scenario, and is not specifically limited herein.
[0042] The target data source can be understood as a database of any structure. For example, the target data source can be a MySQL (relational database management system), a Hive (data warehouse tool), or an Oracle database. The target data source is connected to the worker node, and it stores at least the business data related to the current query command.
[0043] The current query instruction can be understood as a query instruction sent directly by the user, or as a sub-query instruction obtained by the scheduling node by splitting the query instruction sent by the user.
[0044] The target data source corresponding to the current query instruction can be understood as the data source that stores the business data that the query instruction needs to retrieve.
[0045] Target attribute data can be understood as multiple data points related to a specific attribute stored in the target data source. For example, target attribute data could be multiple data points related to dates, multiple data points related to age groups, etc. The target attribute data in the target data source that needs to be divided into intervals can be one or more, without specific limitations. For example, target attribute data frequently queried by query commands can be used as the objects to be divided into intervals. The rules for dividing target attribute data into intervals are not specifically limited here and can be adjusted as needed. For example, when the target attribute data consists of multiple data points related to dates, the data for each month can be divided into one interval, or the data for each month of a quarter can be divided into one interval. When the target attribute data consists of multiple data points related to age groups, intervals can be divided every 10 years, or intervals can be divided by age (teenager, young adult, middle-aged). In one example, such as... Figure 3 As shown, the target data source can store data related to sales operations. This same business data can be stored in the form of a data table, which includes columns for Customer Key, Order Status, Total Price, and Order Date. Each column stores data related to the sales operation. The target attribute data can be understood as the data stored in the Order Date column. Figure 4 As shown, the order date column is divided into multiple intervals by month, with each interval containing order date data for one month. Specifically, partition 1 corresponds to the data in the interval from January 1, 2021 to January 31, 2021; partition 2 corresponds to the data in the interval from February 1, 2021 to February 28, 2021; partition 3 corresponds to the data in the interval from March 1, 2021 to March 31, 2021; and partition 4 corresponds to the data in the interval from April 1, 2021 to April 30, 2021.
[0046] The target cache can be understood as a cache in the worker node's memory or on the worker node's disk. The target cache stores historical query results obtained from queries performed on the complete data of a specific partition, as well as the original data of that partition in the target data source (i.e., target attribute data). The historical query results data can include the final or intermediate query results obtained when historical query commands query the entire interval. For example, the final query result of summing the associated data of the entire interval, the final query result of averaging the associated data of the entire interval, intermediate results of summing the first part of the associated data of the entire interval, and intermediate results of summing the second part of the associated data of the entire interval.
[0047] According to embodiments of this disclosure, for query commands with a large query range, the query results of the current query command can be determined by fully utilizing the query results of complete partitions falling within the query range, based on historical query commands, stored in the target cache. This significantly reduces the process of repeatedly traversing the target data source to obtain data and perform calculations, thus significantly improving the query performance of worker nodes. Even when the user modifies the query range of the query command, the cache can still be hit to the maximum extent for complete partitions falling within the query range. This effectively improves the cache hit rate and the overall performance of the distributed system when query conditions change frequently. It should be noted that in specific business scenarios, the probability of a certain complete interval in the query command being queried is very high; therefore, the method implemented using this disclosure is more likely to hit the cache.
[0048] In one example, with the development of big data technology and the further evolution of traditional database technology, enterprises have an increasing number of choices in data architecture. Currently, data within an enterprise is typically stored in multiple data sources; for enterprises with complex businesses, this can range from dozens to hundreds. Data mining and analysis often requires crossing multiple business systems, necessitating a distributed computing engine with federated query capabilities to perform federated analysis on the heterogeneous data sources underlying these business systems. The architecture of a distributed federated analysis engine is as follows: Figure 2 As shown, each worker node performs parallel computation based on the query instructions allocated by the scheduling node, utilizing data obtained from different data sources to improve query performance. However, this architecture also has some performance bottlenecks, the most significant of which is network I / O (input / output). A single worker node pulls data from multiple data sources for local computation. The varying load capacities of different data sources can become a bottleneck for queries. Furthermore, since the worker nodes of the distributed federated analysis engine need to obtain data from different data sources, and most data sources are not on the same host as the worker nodes, the data scanned to complete the query often needs to be transmitted over the network. For data sources that are across physical networks from the distributed federated analysis engine, the cost of network transmission is very high. Network I / O also affects overall query performance. Therefore, caching becomes a key optimization point for improving the performance of the distributed federated analysis engine. The data query method of this embodiment fully utilizes the data in the local cache of the worker nodes, saving the performance overhead of querying all the data required for query instructions from the data sources.
[0049] In one example, a data query method is applied to worker nodes in a distributed system, including:
[0050] When the current query is related to apparel sales, the target data source for the corresponding apparel sales business is determined based on the specific content of the query. The target attribute data (order date column) in the target data source is divided into multiple intervals by month, with each interval containing order date data for one month. Specifically, for example... Figure 4 As shown, partition 1 corresponds to the data from January 1, 2021 to January 31, 2021; partition 2 corresponds to the data from February 1, 2021 to February 28, 2021; partition 3 corresponds to the data from March 1, 2021 to March 31, 2021; and partition 4 corresponds to the data from April 1, 2021 to April 30, 2021.
[0051] The current query command searches for apparel sales data from February 1, 2021 to February 28, 2021, and partition 2 falls within this data search range.
[0052] Determine if a target cache containing partition 2 data exists in each cache. If it exists, retrieve the target attribute data and / or historical query data for partition 2 from that target cache.
[0053] If the historical query data matches the query conditions for partition 2 in the current query command, then the historical query data for partition 2 stored in the target cache will be used as the first query result.
[0054] If the historical query data is inconsistent with the query conditions for partition 2 in the current query command, the target attribute data of partition 2 stored in the target cache is obtained, and the first query result is calculated based on the current query command and the target attribute data of partition 2.
[0055] In one embodiment, the data query method of this disclosure includes steps S101 and S102, wherein step S102: when it is determined that a first interval among multiple intervals is entirely within the data query range of the current query instruction and a corresponding target cache exists, determining a first query result based on the current query instruction and the target cache may include:
[0056] Step S1021: Determine the target interval among multiple intervals that is entirely within the data query range of the current query instruction.
[0057] Step S1022: If it is determined that there is a corresponding target cache in the target interval, the target interval is determined as the first interval.
[0058] Step S1023: Based on the current query instruction, and based on the target attribute data corresponding to the first interval stored in the target cache, and / or the query results of the historical query instructions corresponding to the first interval, determine the first query result.
[0059] According to the embodiments of this disclosure, it should be noted that:
[0060] If there is no corresponding target cache for the target range, it means that the target range has not been queried by historical query instructions based on the data of the entire target range. Therefore, the corresponding historical query results have not been stored in the cache, so there is no corresponding target cache.
[0061] The query results of the historical query instructions corresponding to the first interval can be understood as the final query results or intermediate query results obtained after the historical query instructions query the data of the entire first interval.
[0062] According to embodiments of this disclosure, it is possible to quickly and accurately determine whether a first interval that can hit the target cache exists. Simultaneously, by utilizing the target attribute data corresponding to the first interval stored in the target cache, and / or the query results of historical query instructions corresponding to the first interval, the first query result for the first interval can be quickly obtained. For query instructions with a large query range, the query results of the complete partitions falling within its query range can be fully utilized based on the query results pre-stored in the target cache according to historical query instructions. Using the pre-stored query results in the target cache to determine the query result of the current query instruction can significantly reduce the process of repeatedly traversing the target data source to obtain data and perform calculations, thus significantly improving the query performance of worker nodes. Even when the user modifies the query range of the query instruction, the cache can still be hit to the maximum extent for complete partitions falling within the query range. This effectively improves the cache hit rate and the overall performance of the distributed system when query conditions change frequently.
[0063] In one embodiment, the data query method of this disclosure includes steps S101 and S102, and steps S1021 to S1023, wherein step S1021: determining a target interval among multiple intervals that is entirely within the data query range of the current query instruction, including:
[0064] The data query range of the current query instruction is compared with the first and last data of each of the multiple intervals to determine the target interval that is located within the data query range of the current query instruction.
[0065] According to the embodiments of this disclosure, it should be noted that:
[0066] The first data point of each interval can be understood as the first data point in that interval. The last data point of each interval can be understood as the last data point in that interval.
[0067] According to the embodiments of this disclosure, by using the first and last data of each interval, the data range of each interval can be accurately determined. Thus, when compared with the data query range of the current query instruction, the target interval that is located within the data query range of the current query instruction can be quickly determined.
[0068] In one example, when the current query is related to apparel sales, the target data source for the corresponding apparel sales business is determined based on the specific content of the query. The target attribute data (order date column) in the target data source is divided into multiple intervals by month, with each interval containing order date data for one month. Specifically, for example... Figure 4 As shown, partition 1 corresponds to the data from January 1, 2021 to January 31, 2021; partition 2 corresponds to the data from February 1, 2021 to February 28, 2021; partition 3 corresponds to the data from March 1, 2021 to March 31, 2021; and partition 4 corresponds to the data from April 1, 2021 to April 30, 2021.
[0069] In this context, January 1, 2021, can be understood as the beginning of the data in partition 1, and January 31, 2021, can be understood as the end of the data in partition 1.
[0070] The current query command's data query range is apparel sales data from January 1, 2021 to February 28, 2021. By using the first and last data of partition 1, we can quickly determine whether partition 1 falls within this data query range.
[0071] In one embodiment, the data query method of this disclosure includes steps S101 and S102, and further includes:
[0072] Step S103: If it is determined that some intervals of the second interval among multiple intervals are within the data query range, the second query result is determined based on the target attribute data corresponding to the partial intervals according to the current query instruction.
[0073] Step S104: Determine the total query result of the current query instruction based on the first query result and the second query result.
[0074] According to the embodiments of this disclosure, it should be noted that:
[0075] The second interval contains a portion of the data query range. This can be understood as a portion of the data within the complete second interval falling within the data query range, while a portion does not. For example, ... Figure 4As shown, partition 1 corresponds to the data from January 1, 2021 to January 31, 2021; partition 2 corresponds to the data from February 1, 2021 to February 28, 2021; and partition 3 corresponds to the data from March 1, 2021 to March 31, 2021. The data query range is from January 18, 2021 to March 18, 2021, so partitions 1 and 3 are the second interval. Specifically, only a portion of the time period in both partitions 1 and 3 falls within the data query range.
[0076] The second query result is determined based on the target attribute data corresponding to a portion of the interval. This can be understood as calculating the second query result based on a portion of the data falling within the data query range of the second interval and its associated data in the target data source, according to the current query instruction.
[0077] According to the embodiments of this disclosure, while obtaining the first query result of the first interval using the target cache, the query result of the second interval is obtained using the target data source. By using parallel querying, the efficiency of obtaining the query result of the current query instruction can be improved.
[0078] In one example, such as Figure 4 As shown, partition 1 corresponds to the data from January 1, 2021 to January 31, 2021; partition 2 corresponds to the data from February 1, 2021 to February 28, 2021; and partition 3 corresponds to the data from March 1, 2021 to March 31, 2021. The data query range is from January 18, 2021 to March 18, 2021, so partitions 1 and 3 are determined as the second interval. Meanwhile, partition 2, which falls entirely within the data query range, has a corresponding target cache, so partition 2 is determined as the first interval. The data of partition 2 stored in the target cache is used to determine the first query result, and the target data source is used to determine the second query result for a portion of partitions 1 and 3. The first and second query results are then combined to obtain the total query result for the current query instruction.
[0079] In one embodiment, the data query method of this disclosure includes steps S101 and S102, and further includes:
[0080] Step S105: If it is determined that the third interval among multiple intervals is entirely within the data query range of the current query instruction and there is no corresponding target cache, then based on the target attribute data corresponding to the third interval, determine the third query result according to the current query instruction.
[0081] Step S106: Determine the total query result of the current query instruction based on the first query result and the third query result.
[0082] According to the embodiments of this disclosure, it should be noted that:
[0083] The absence of a corresponding target cache in the third interval can be interpreted as the fact that previous historical query data did not perform any relevant calculations or query analyses on the overall data of the third interval.
[0084] According to the embodiments of this disclosure, while obtaining the first query result of the first interval using the target cache, the query result of the third interval is obtained using the target data source. By using parallel querying, the efficiency of obtaining the query result of the current query instruction can be improved.
[0085] In one example, partition 1 corresponds to data from January 1, 2021 to January 31, 2021; partition 2 corresponds to data from February 1, 2021 to February 28, 2021; and partition 3 corresponds to data from March 1, 2021 to March 31, 2021. The data query range is from January 1, 2021 to February 28, 2021. If partition 2, which falls entirely within the data query range, has a corresponding target cache, then partition 2 is determined as the first partition. If partition 1, which falls entirely within the data query range, does not have a corresponding target cache, then partition 1 is determined as the third partition. The first query result is determined using the data of partition 2 stored in the target cache, and the third query result is determined using the target data source. The first query result and the third query result are combined to obtain the total query result of the current query instruction.
[0086] In one embodiment, the data query method of this disclosure includes steps S101 and S102, and further includes:
[0087] Step S103: If it is determined that some intervals of the second interval among multiple intervals are within the data query range, the second query result is determined based on the target attribute data corresponding to the partial intervals according to the current query instruction.
[0088] Step S105: If it is determined that the third interval among multiple intervals is entirely within the data query range of the current query instruction and there is no corresponding target cache, then based on the target attribute data corresponding to the third interval, determine the third query result according to the current query instruction.
[0089] Step S107: Determine the total query result of the current query instruction based on the first query result, the second query result, and the third query result.
[0090] According to the embodiments of this disclosure, while obtaining the first query result of the first interval using the target cache, the query results of the third interval and the second interval are obtained using the target data source. By using parallel querying, the efficiency of obtaining the query results of the current query instruction can be improved.
[0091] In one implementation, step S103: If it is determined that a portion of the second interval among multiple intervals falls within the data query range, a second query result is determined based on the target attribute data corresponding to the portion of the intervals, according to the current query instruction, including:
[0092] The data query range of the current query instruction is compared with the first and last data of each of the multiple intervals to determine the fourth interval, which is not entirely within the data query range of the current query instruction, and the second interval, which contains some intervals within the data query range.
[0093] Based on the first and fourth intervals, the first data query task is updated to a second data query task. The first data query task queries all target attribute data stored in the target data source and / or the associated data with all target attribute data in the target data source. The second data query task queries only the target attribute data corresponding to a subset of intervals.
[0094] Based on the second data query task, obtain the target attribute data corresponding to a portion of the interval.
[0095] Based on the current query command and the target attribute data corresponding to a portion of the intervals, determine the second query result.
[0096] According to the embodiments of this disclosure, data traversal can be performed using the target data source only for a portion of the second interval, without needing to access all target attribute data in the target data source, thus effectively improving the speed of data querying from the target data source.
[0097] In one implementation, step S106: If it is determined that the third interval among multiple intervals is entirely within the data query range of the current query instruction and there is no corresponding target cache, then, based on the target attribute data corresponding to the third interval, a third query result is determined according to the current query instruction, including:
[0098] The data query range of the current query instruction is compared with the first and last data of each of the multiple intervals to determine the fourth interval, which is not within the data query range of the current query instruction, and the third interval, which is within the data query range but does not have a corresponding target cache.
[0099] Based on the first and fourth intervals, the first data query task is updated to the third data query task. The first data query task queries all target attribute data stored in the target data source and / or associated data with all target attribute data in the target data source. The third data query task queries only the target attribute data corresponding to the third interval.
[0100] Based on the third data query task, obtain the target attribute data corresponding to the third interval.
[0101] Based on the current query command and the target attribute data corresponding to the third interval, determine the third query result.
[0102] According to the embodiments of this disclosure, data traversal can be performed using the target data source only for the third interval, without needing to access all target attribute data in the target data source, thus effectively improving the speed of data querying from the target data source.
[0103] In one embodiment, the data query method of this disclosure includes steps S101 and S102, and steps S105 and S106, and further includes:
[0104] Step S108: Store the third query result and / or the target attribute data corresponding to the third interval into a preset cache.
[0105] According to the embodiments of this disclosure, it should be noted that:
[0106] The preset cache can be understood as a cache in the memory of the worker node or a cache on the disk of the worker node.
[0107] According to the embodiments of this disclosure, when a subsequent query command queries the data of the entire third interval, the data of the third interval can be quickly hit from the preset cache without using the target data source to obtain the data of the third interval, thereby improving query efficiency and cache hit rate.
[0108] In one application example, the data query method of this disclosure embodiment includes:
[0109] 1. Partition the hot columns (target attribute data) of the data table storing specific business data in the target data source.
[0110] 2. Collect statistical information (head data and tail data) for each partition.
[0111] 3. During the query, cache the intermediate data, final query data and / or original data of the complete partition obtained from the leaf nodes (working nodes) of the historical query plan.
[0112] 4. Based on the current query command, the query results include:
[0113] 1) If some partitions fall entirely within the query conditions (data query range) of the current query command, then the range corresponding to these partitions will be removed from the query conditions, the execution plan will be modified, and these partitions will be enumerated and cached during subsequent processing.
[0114] 2) If some partitions are not included in the query conditions at all, remove the range corresponding to these partitions from the query conditions and modify the execution plan.
[0115] 3) If some partitions fall within the query conditions, apply the query conditions to these partitions and merge them with the cached query results from step 2) for subsequent processing.
[0116] In one application example, such as Figure 3 , Figure 4 As shown, the target data source (test_table) can store data related to sales operations. This data can be stored in the form of a data table, which includes columns for Customer Key, Order Status, Total Price, and Order Date. Each column stores data related to the sales operations.
[0117] Assuming the user has already executed the query:
[0118] SELECT sum(o_total price) FROM test_table WHERE o_order date BETWEEN'2021-02-01' AND'2021-03-01'. At this point, the sum(o_total price) calculation result for the partition of 2021-02 has already been cached.
[0119] For a newly submitted query: SELECT sum(o_total price) FROM test_table WHERE o_order date BETWEEN'2021-01-10' AND'2021-03-15', without optimization, the engine will scan the entire table and filter records that satisfy o_order date between 2021-01-10 and 2021-03-15.
[0120] Using this method, the engine compares the statistics (start date, end date) of each partition of o_order date with the query conditions BETWEEN '2021-01-16' AND '2021-03-15'. The conclusion is that partitions matching the query conditions include: 2021-01, 2021-02, and 2021-03; partitions not matching the query conditions include: 2021-04.
[0121] Based on this: Partition 2 is entirely within the scope of the query conditions, so it is removed from the query conditions, and the execution plan is modified. Subsequent processing will directly use the intermediate result cache of Partition 2. Partition 4 is not within the scope of the query conditions, so it is removed from the query conditions. Partitions 1 and 3 partially fall within the query conditions. The query conditions are applied to Partitions 1 and 3, and sum(o_total price) is calculated for each. The calculated results are then merged with the cached sum(o_total price) for Partition 2, and the final result is returned.
[0122] like Figure 5 As shown, this disclosure provides a method for distributed federated analysis, applied to the scheduling node of a distributed system, including:
[0123] Step S501: Based on the user's query command, determine multiple worker nodes from the distributed system.
[0124] Step S502: The user query instruction is split into multiple current query instructions, and each current query instruction is sent to multiple worker nodes in a corresponding manner. The multiple worker nodes perform data queries based on the multiple current query instructions according to the method of any embodiment of this disclosure.
[0125] Step S503: Based on the query results fed back by multiple working nodes, determine the federated analysis query results of the user's query command.
[0126] According to the embodiments of this disclosure, it should be noted that:
[0127] A distributed system can be understood as any distributed federated analysis engine in existing technology. A worker node can be understood as a node in the distributed federated analysis engine used to perform data querying, mining, analysis, and computation. Worker nodes can be assigned current query instructions by the scheduling node in the distributed federated analysis engine, enabling them to execute data query tasks based on the assigned current query instructions. The execution entity of the data query method in this embodiment can be understood as a worker node. The architecture of the distributed federated analysis engine can be referenced. Figure 2 , Figure 2 Worker node 1, worker node 2, and worker node 3 can all serve as the execution entities for the data query method in this embodiment of the disclosure. It should be noted that... Figure 2 This is only used as an example to illustrate the architecture of the distributed federated analysis engine. The specific distributed federated analysis engine used can be adjusted as needed. The number of worker nodes, the connection relationships between worker nodes, and the connection relationships between worker nodes and scheduling nodes in the distributed federated analysis engine can all be adjusted as needed.
[0128] The distributed federated analysis method of this disclosure can be applied to any business scenario, and is not specifically limited herein.
[0129] The user's query command is broken down into multiple current query commands, which are then sent one-to-one to multiple worker nodes. This can be understood as each worker node receiving a current query command.
[0130] According to embodiments of this disclosure, each working node obtains query results using the data query method of any embodiment of this disclosure, which can improve the data query efficiency of each working node. With the performance improvement of each working node, the overall data query efficiency of the distributed system can be improved.
[0131] like Figure 6 As shown, this disclosure provides a data query apparatus applied to a worker node of a distributed system, comprising:
[0132] The first determining module 610 is used to determine the target data source corresponding to the current query instruction, wherein the target attribute data stored in the target data source is divided into multiple intervals.
[0133] The second determining module 620 is used to determine the first query result based on the current query instruction and the target cache when the first interval among multiple intervals is determined to be within the data query range of the current query instruction and a corresponding target cache exists.
[0134] In one implementation, the second determining module 620 includes:
[0135] The first determination submodule is used to determine the target interval among multiple intervals that is entirely within the data query range of the current query instruction.
[0136] The second determination submodule is used to determine the target interval as the first interval if a corresponding target cache exists in the target interval.
[0137] The third determining submodule is used to determine the first query result based on the target attribute data corresponding to the first interval stored in the target cache and / or the query results of the historical query instructions corresponding to the first interval, according to the current query instruction.
[0138] In one implementation, the first determining submodule is used to:
[0139] The data query range of the current query instruction is compared with the first and last data of each of the multiple intervals to determine the target interval that is located within the data query range of the current query instruction.
[0140] In one embodiment, the data query apparatus further includes:
[0141] The third determining module is used to determine the second query result based on the target attribute data corresponding to the partial intervals when a portion of the second interval among multiple intervals is located within the data query range, according to the current query instruction.
[0142] The fourth determining module is used to determine the total query result of the current query instruction based on the first query result and the second query result.
[0143] In one embodiment, the data query apparatus further includes:
[0144] The fifth determination module is used to determine the third query result based on the target attribute data corresponding to the third interval when the third interval among multiple intervals is located within the data query range of the current query instruction and there is no corresponding target cache.
[0145] The sixth determination module is used to determine the total query result of the current query instruction based on the first query result and the third query result.
[0146] In one embodiment, the data query apparatus further includes:
[0147] The seventh determination module is used to determine the second query result based on the target attribute data corresponding to the partial intervals, when a portion of the second interval in a set of multiple intervals is determined to be within the data query range, according to the current query instruction.
[0148] The eighth determination module is used to determine the third query result based on the target attribute data corresponding to the third interval when the third interval among multiple intervals is located within the data query range of the current query instruction and there is no corresponding target cache.
[0149] The ninth determination module is used to determine the total query result of the current query instruction based on the first query result, the second query result, and the third query result.
[0150] In one implementation, the third determining module includes:
[0151] The fourth determination submodule is used to compare the data query range of the current query instruction with the first and last data of each of the multiple intervals to determine the fourth interval that is not entirely within the data query range of the current query instruction and the second interval that partially falls within the data query range.
[0152] The first update submodule is used to update the first data query task to a second data query task based on the first interval and the fourth interval. The first data query task queries all target attribute data stored in the target data source and / or the associated data with all target attribute data in the target data source. The second data query task queries only the target attribute data corresponding to a subset of intervals.
[0153] The first acquisition submodule is used to obtain target attribute data corresponding to a certain range based on the second data query task.
[0154] The fifth determination submodule is used to determine the second query result based on the target attribute data corresponding to a portion of the current query instruction.
[0155] In one implementation, the fifth determining module includes:
[0156] The sixth determination submodule is used to compare the data query range of the current query instruction with the first and last data of each of the multiple intervals, and determine the fourth interval which is not located in the data query range of the current query instruction and the third interval which is located in the data query range but does not have a corresponding target cache.
[0157] The second update submodule is used to update the first data query task to a third data query task based on the first and fourth intervals. The first data query task queries all target attribute data stored in the target data source and / or associated data with all target attribute data in the target data source. The third data query task queries only the target attribute data corresponding to the third interval.
[0158] The second acquisition submodule is used to obtain the target attribute data corresponding to the third interval based on the third data query task.
[0159] The sixth determination submodule is used to determine the third query result based on the target attribute data corresponding to the third interval according to the current query instruction.
[0160] In one embodiment, the data query apparatus further includes:
[0161] The caching module is used to store the third query result and / or the target attribute data corresponding to the third interval into a preset cache.
[0162] The specific functions and examples of each module and submodule of the apparatus in this disclosure can be found in the relevant descriptions of the corresponding steps in the above method embodiments, and will not be repeated here.
[0163] like Figure 7 As shown, this disclosure provides an apparatus for distributed federated analysis, applied to a scheduling node in a distributed system, comprising:
[0164] The node determination module 710 is used to determine multiple worker nodes from the distributed system based on user query instructions.
[0165] The sending module 720 is used to split a user query instruction into multiple current query instructions and send them one-to-one to multiple working nodes, wherein the multiple working nodes perform data queries based on the multiple current query instructions according to the method of any embodiment of this disclosure.
[0166] The result determination module 730 is used to determine the federated analysis query result of the user's query instruction based on the query results fed back by multiple working nodes.
[0167] The specific functions and examples of each module and submodule of the apparatus in this disclosure can be found in the relevant descriptions of the corresponding steps in the above method embodiments, and will not be repeated here.
[0168] The acquisition, storage, and application of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.
[0169] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.
[0170] Figure 8 A schematic block diagram of an example electronic device 800 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.
[0171] like Figure 8 As shown, device 800 includes a computing unit 801, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 802 or a computer program loaded from storage unit 808 into random access memory (RAM) 803. RAM 803 may also store various programs and data required for the operation of device 800. The computing unit 801, ROM 802, and RAM 803 are interconnected via bus 804. Input / output (I / O) interface 805 is also connected to bus 804.
[0172] Multiple components in device 800 are connected to I / O interface 805, including: input unit 806, such as keyboard, mouse, etc.; output unit 807, such as various types of monitors, speakers, etc.; storage unit 808, such as disk, optical disk, etc.; and communication unit 809, such as network card, modem, wireless transceiver, etc. Communication unit 809 allows device 800 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.
[0173] The computing unit 801 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 801 performs the various methods and processes described above, such as data querying methods and / or distributed federated analysis methods. For example, in some embodiments, the data querying methods and / or distributed federated analysis methods can be implemented as computer software programs tangibly contained in a machine-readable medium, such as storage unit 808. In some embodiments, part or all of the computer program can be loaded and / or installed on device 800 via ROM 802 and / or communication unit 809. When the computer program is loaded into RAM 803 and executed by the computing unit 801, one or more steps of the data querying methods and / or distributed federated analysis methods described above can be performed. Alternatively, in other embodiments, computing unit 801 may be configured by any other suitable means (e.g., by means of firmware) to perform methods of data querying and / or distributed federated analysis.
[0174] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.
[0175] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.
[0176] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.
[0177] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).
[0178] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.
[0179] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.
[0180] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.
[0181] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the principles of this disclosure should be included within the scope of protection of this disclosure.
Claims
1. A data query method, applied to worker nodes of a distributed system, comprising: Among the multiple data sources connected to the working node, the data source storing the business data required by the current query instruction is determined as the corresponding target data source. The target attribute data stored in the target data source is divided into multiple intervals, and the target attributes include hot columns from data tables specific to the business. Based on the data query range of the current query instruction, the first interval, the second interval, and the third interval among the multiple intervals are identified; a portion of the second interval is located within the data query range; the entire third interval is located within the data query range of the current query instruction and has no corresponding target cache; different intervals are processed in parallel, wherein: For the first interval, if it is determined that the first interval as a whole is within the data query range of the current query instruction and a corresponding target cache exists, a first query result is determined according to the current query instruction and the target cache, including: if the historical query data in the target cache is consistent with the query conditions of the current query instruction for the first interval, then the historical query data is used as the first query result; the historical query data is obtained by querying the complete data of the first interval based on the historical query instruction. For the second interval, based on the current query instruction and the target attribute data corresponding to the partial interval, the second query result is determined; For the third interval, based on the current query instruction and the target attribute data corresponding to the third interval, the third query result is determined; Based on the first query result of the first interval, the second query result of the second interval, and the third query result of the third interval, the total query result of the current query instruction is determined.
2. The method according to claim 1, wherein, Determining the first interval includes: Determine the target interval among the plurality of intervals that is entirely within the data query range of the current query instruction; If it is determined that there is a corresponding target cache in the target interval, the target interval is determined as the first interval; Determining the first query result based on the current query instruction and the target cache also includes: If the historical query data in the target cache is inconsistent with the query conditions of the current query instruction for the first interval, the first query result is determined based on the target attribute data corresponding to the first interval stored in the target cache according to the current query instruction.
3. The method according to claim 2, wherein, Determining the target interval among the plurality of intervals that is entirely within the data query range of the current query instruction includes: The data query range of the current query instruction is compared with the first and last data of each of the plurality of intervals to determine the target interval that is located within the data query range of the current query instruction.
4. The method according to claim 1, wherein, The step of determining the second query result based on the target attribute data corresponding to the partial interval according to the current query instruction includes: The data query range of the current query instruction is compared with the first and last data of each of the plurality of intervals to determine the fourth interval that is not entirely within the data query range of the current query instruction and the second interval that is partially within the data query range; Based on the first interval and the fourth interval, the first data query task is updated to the second data query task, wherein the first data query task is used to query all target attribute data stored in the target data source and / or the associated data with all target attribute data in the target data source; the second data query task is used to query only the target attribute data corresponding to the partial interval; Based on the second data query task, obtain the target attribute data corresponding to the partial interval; Based on the current query instruction and the target attribute data corresponding to the partial interval, the second query result is determined.
5. The method according to claim 1, wherein, The step of determining the third query result based on the target attribute data corresponding to the third interval according to the current query instruction includes: The data query range of the current query instruction is compared with the first and last data of each of the multiple intervals to determine the fourth interval which is not located in the data query range of the current query instruction and the third interval which is located in the data query range but does not have a corresponding target cache. Based on the first interval and the fourth interval, the first data query task is updated to the third data query task, wherein the first data query task is used to query all target attribute data stored in the target data source and / or the associated data of the target data source with all target attribute data; the third data query task is used to query only the target attribute data corresponding to the third interval; Based on the third data query task, obtain the target attribute data corresponding to the third interval; Based on the current query instruction and the target attribute data corresponding to the third interval, the third query result is determined.
6. The method according to claim 1, further comprising: Store the third query result and / or the target attribute data corresponding to the third interval into a preset cache.
7. A method for distributed federated analysis, applied to the scheduling node of a distributed system, comprising: Based on the user's query command, multiple working nodes are determined from the distributed system; The user query instruction is split into multiple current query instructions, which are sent one-to-one to the multiple working nodes. The multiple working nodes perform data queries based on the multiple current query instructions according to any one of claims 1 to 6. as well as Based on the query results fed back by the multiple working nodes, the federated analysis query result of the user query instruction is determined.
8. A data query apparatus, applied to a worker node of a distributed system, comprising: The first determining module identifies, from among the multiple data sources connected to the working node, the data source storing the business data required for the current query instruction as the corresponding target data source. The target attribute data stored in the target data source is divided into multiple intervals, and the target attributes include hotspot columns from data tables specific to the business. Based on the data query range of the current query instruction, the first interval, the second interval, and the third interval among the multiple intervals are identified; a portion of the second interval is located within the data query range; the entire third interval is located within the data query range of the current query instruction and has no corresponding target cache; different intervals are processed in parallel, wherein: The second determining module is used to determine a first query result for a first interval, provided that the first interval as a whole is within the data query range of the current query instruction and a corresponding target cache exists. This includes: if the historical query data in the target cache matches the query conditions of the current query instruction for the first interval, then the historical query data is used as the first query result; the historical query data is obtained by querying the complete data of the first interval based on the historical query instruction. The third determining module is used to determine the second query result for the second interval based on the target attribute data corresponding to the partial interval, according to the current query instruction. The fifth determining module is used to determine the third query result for the third interval based on the target attribute data corresponding to the current query instruction; The ninth determining module is used to determine the total query result of the current query instruction based on the first query result of the first interval, the second query result of the second interval, and the third query result of the third interval.
9. The apparatus according to claim 8, wherein, The second determining module includes: The first determining submodule is used to determine the target interval among the plurality of intervals that is entirely within the data query range of the current query instruction; The second determining submodule is used to determine the target interval as the first interval when it is determined that there is a corresponding target cache in the target interval; The third determining submodule is used to determine the first query result based on the target attribute data corresponding to the first interval stored in the target cache, if the historical query data in the target cache is inconsistent with the query conditions of the current query instruction for the first interval.
10. The apparatus according to claim 9, wherein, The first determining submodule is used for: The data query range of the current query instruction is compared with the first and last data of each of the plurality of intervals to determine the target interval that is located within the data query range of the current query instruction.
11. The apparatus according to claim 8, wherein, The third determining module includes: The fourth determining submodule is used to compare the data query range of the current query instruction with the first and last data of each of the plurality of intervals, and determine the fourth interval that is not entirely within the data query range of the current query instruction and the second interval that partially falls within the data query range; The first update submodule is used to update the first data query task to a second data query task based on the first interval and the fourth interval. The first data query task is used to query all target attribute data stored in the target data source and / or the associated data with all target attribute data in the target data source. The second data query task is used to query only the target attribute data corresponding to the partial interval. The first acquisition submodule is used to acquire the target attribute data corresponding to the partial interval based on the second data query task; The fifth determining submodule is used to determine the second query result based on the target attribute data corresponding to the partial interval according to the current query instruction.
12. The apparatus according to claim 8, wherein, The fifth determining module includes: The sixth determining submodule is used to compare the data query range of the current query instruction with the first and last data of each of the plurality of intervals, and determine the fourth interval which is not located in the data query range of the current query instruction and the third interval which is located in the data query range and does not have a corresponding target cache. The second update submodule is used to update the first data query task to a third data query task based on the first interval and the fourth interval. The first data query task is used to query all target attribute data stored in the target data source and / or the associated data with all target attribute data in the target data source. The third data query task is used to query only the target attribute data corresponding to the third interval. The second acquisition submodule is used to acquire the target attribute data corresponding to the third interval based on the third data query task; The sixth determining submodule is used to determine the third query result based on the target attribute data corresponding to the third interval according to the current query instruction.
13. The apparatus according to claim 8, further comprising: The caching module is used to store the third query result and / or the target attribute data corresponding to the third interval into a preset cache.
14. An apparatus for distributed federated analysis, applied to a scheduling node in a distributed system, comprising: The node determination module is used to determine multiple working nodes from the distributed system based on user query instructions; The sending module is used to split the user query instruction into multiple current query instructions and send them one-to-one to the multiple working nodes, wherein the multiple working nodes perform data queries based on the multiple current query instructions according to any one of claims 1 to 6; as well as The result determination module is used to determine the federated analysis query result of the user query instruction based on the query results fed back by the multiple working nodes.
15. An electronic device comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1 to 7.
16. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1 to 7.
17. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1 to 7.