A data query method, device, apparatus and storage medium

Abstract

This application discloses a data query method, apparatus, device, and storage medium, which can be applied to a data processing system including a coordinating node and worker nodes. The coordinating node sends a task to the worker nodes to perform query operations on a first file and a second file. This task includes query conditions for the first and second files, and the data volume of the first file is greater than that of the second file. Based on the query conditions in the task and the corresponding data information in the first and second files, the worker nodes estimate the proportion of data in the first file that meets the query conditions. When this proportion exceeds a preset threshold, the worker nodes read the first and second files and perform the query operation. In this way, when the estimated proportion of data meeting the query conditions in the first file is large, the problem of decreased data query efficiency and increased query overhead caused by generating filtering conditions is avoided.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a data query method, apparatus, device, and storage medium. Background Technology

[0002] In the era of big data, with the expansion of data collection methods, the cost of data collection is decreasing, and correspondingly, the amount of data stored in data sources is increasing. When querying and analyzing massive amounts of data, query efficiency and query overhead have become core issues in the field of data processing.

[0003] Currently, when data processing systems perform data queries based on user-input queries, they typically generate dynamic filters (DFs) to reduce the amount of data read from the data source, thereby improving query efficiency and reducing query overhead. However, in practical applications, the filter conditions generated by the data processing system may not achieve good data filtering results; that is, the amount of data before and after filtering is not significantly different. In this case, the data query efficiency and data query overhead are not significantly optimized. On the contrary, the processes involved in generating, calculating, and transmitting the filter conditions may actually reduce the data query efficiency and increase the query overhead. Summary of the Invention

[0004] This application provides a data query method, apparatus, device, storage medium, and computer program product to maintain the data query efficiency of the data processing system at a high level and the query overhead at a low level.

[0005] Firstly, embodiments of this application provide a data query method. This method can be applied to a data processing system, which includes a coordinating node and worker nodes. The coordinating node can send a task to the worker nodes to perform query operations on a first file and a second file. The sent task includes query conditions for the first and second files, and the data volume of the first file is greater than that of the second file. For example, the task sent by the coordinating node may be a logical plan tree generated by the coordinating node. Based on the query conditions in the received task and the data information corresponding to the first and second files, the worker nodes estimate the proportion of data in the first file that meets the query conditions. When the proportion is greater than a preset threshold, the worker nodes read the first and second files to the worker nodes and perform query operations on the read first and second files.

[0006] If the proportion of data matching the query criteria is large in the first file, it indicates that the filtering effect of the filter criteria to be generated is poor. In this case, the data processing system can directly read the first and second files and query the data that matches the query criteria, without generating filter criteria to filter the data in the first file. This avoids the problem of lower data query efficiency and increased query overhead caused by generating filter criteria, thus keeping the data query overhead of the data processing system at a low level.

[0007] In one possible implementation, when the estimated proportion of data in the first file that meets the query conditions is less than or equal to a preset threshold, the worker node can read the second file and query the first data in the second file that meets the query conditions. Then, the worker node can generate filtering conditions based on the first data and send the filtering conditions to the data source where the first file is located. The filtering conditions can be used to instruct the data source to query the second data that matches the first data from the first file. Thus, the worker node can receive the second data sent by the data source and perform query operations on the first data and the second data.

[0008] In this implementation, when the proportion of data in the first file that meets the query conditions is small, it indicates that the filtering effect of the filter conditions to be generated is better. At this time, the data processing system can reduce the amount of data that the working node reads from the first file of the data source by generating filter conditions, thereby enabling the data query efficiency of the data processing system to reach a high level. Furthermore, when the working node queries data that meets the query conditions, it does not need to read a large amount of data that does not meet the query conditions from the data source in the first file, thereby effectively reducing the data query overhead of the data processing system.

[0009] In one possible implementation, the data information corresponding to the first file and the second file can specifically be sampled data from the first file and sampled data from the second file, respectively. When the working node estimates the proportion of data in the first file that meets the query conditions, it can specifically sample the data in both the first and second files to obtain the sampled data in the first and second files. Then, the working node can determine the identifier of the target sampled data that meets the query conditions in the sampled data of the second file and calculate the proportion of sampled data with that identifier in the sampled data of the first file. In this way, by sampling a small amount of data from the first and second files, it is possible to predict whether the filter conditions to be generated have a good data filtering effect. This allows the data processing system to evaluate the filtering effect of the filter conditions with minimal overhead, improving the feasibility of the solution implementation.

[0010] In one possible implementation, the data information corresponding to the first file and the second file can specifically be the data statistics corresponding to the first file and the second file, respectively. When the working node estimates the proportion of data in the first file that meets the query conditions, it can specifically determine the identifier of the data in the second file that meets the query conditions based on the data statistics corresponding to the second file, and further calculate the proportion of data with that identifier in the first file based on the data statistics corresponding to the first file. In this way, the proportion that can be used to evaluate the filtering effect of the filtering conditions can be obtained using the data statistics corresponding to each file, so as to further determine whether to generate filtering conditions based on this proportion.

[0011] The statistical information for each file can be pre-generated collaboratively by the worker nodes and the coordinating node. For example, a worker node can pre-read the first and second files from the data source and have the coordinating node perform statistical analysis on them, obtaining and saving the statistical information for each file. This allows the worker node to determine whether to generate a filter condition for the current query based on the locally saved statistical information. Alternatively, in other examples, while the first and second files are stored in the data source, the data source can generate corresponding statistical information for each file and store it locally. When a worker node determines whether to generate a filter condition, it can retrieve the statistical information for the first and second files from the data source to estimate the proportion of data used to determine the appropriate level of the filter condition.

[0012] In one possible implementation, the data statistics information corresponding to the first file can specifically be any one or more of the following: the data range, the data distribution interval, and duplicate data of the first file. Of course, other possible implementations are also possible. Thus, the working node can determine whether to generate filtering conditions based on the aforementioned information such as the data range, the data distribution interval, and duplicate data.

[0013] In one possible implementation, when a worker node determines and generates a filter condition and uses that condition to query data from a first file, the worker node can calculate the proportion of the retrieved data that meets the query condition within the already traversed data of the first file. If this proportion exceeds a filter threshold, it indicates that the filter condition does not have a superior data filtering effect during the actual data query process. In this case, the worker node can stop using the filter condition to query the remaining data and instead query the remaining data that meets the query condition from the untraversed data of the first file. This allows for timely termination of the query for remaining data when the filter condition does not have the expected data filtering effect, thereby minimizing the additional overhead associated with its application.

[0014] Secondly, based on the same inventive concept as the method embodiments of the first aspect, this application provides a data query device. This device has the functions corresponding to the various embodiments of the first aspect described above. These functions can be implemented by hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the aforementioned functions.

[0015] Thirdly, embodiments of this application provide an apparatus, including: a processor and a memory; the memory is used to store instructions, and when the computing device is running, the processor executes the instructions stored in the memory to cause the device to perform the data query method in the first aspect or any implementation thereof. It should be noted that the memory can be integrated into the processor or can be independent of the processor. The apparatus may also include a bus. The processor is connected to the memory via the bus. The memory may include readable storage and random access memory.

[0016] Fourthly, embodiments of this application also provide a readable storage medium storing a program or instructions that, when run on a computer, cause the data query method in the first aspect or any implementation thereof to be executed.

[0017] Fifthly, embodiments of this application also provide a computer program product containing instructions that, when run on a computer, cause the computer to execute any data query method in the first aspect or any implementation thereof.

[0018] Furthermore, the technical effects of any of the implementation methods in aspects two through six can be found in the technical effects of different implementation methods in aspect one, or in the technical effects of different implementation methods in aspect two, and will not be repeated here. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments recorded in this application. For those skilled in the art, other drawings can be obtained based on these drawings.

[0020] Figure 1 A schematic diagram of the structure of an exemplary data processing system provided in an embodiment of this application;

[0021] Figure 2 This is a diagram illustrating the people and orders tables included in data source 103.

[0022] Figure 3 This is a schematic diagram of the structure of another exemplary data processing system provided in the embodiments of this application;

[0023] Figure 4 A flowchart illustrating a data query method provided in an embodiment of this application;

[0024] Figure 5 This is a schematic diagram of the query input interface provided in an embodiment of this application;

[0025] Figure 6 This is a schematic diagram of the structure of a data query device provided in an embodiment of this application;

[0026] Figure 7 This is a schematic diagram of the hardware structure of a device provided in an embodiment of this application. Detailed Implementation

[0027] See Figure 1 This is a schematic diagram of the structure of an exemplary data processing system. Figure 1 As shown, the data processing system 100 may include a coordinator node 101 and worker nodes 102. The worker node 102 can access data in the data source 103, which may include one or more data sources, such as... Figure 1 The data sources shown include Hive and Oracle.

[0028] The data processing system 100 can provide a client 104 for human-computer interaction with users, enabling the execution of corresponding data query processes based on user-input query statements. The coordination node 101 can receive the SQL statement input by the user through the client 104 and perform syntax and semantic analysis on the SQL statement. Syntax analysis refers to the coordination node 101 using the syntax rules of the SQL language to verify whether the SQL statement has syntax errors; semantic analysis refers to the coordination node 101 analyzing whether the semantics of the SQL statement are valid. Once the SQL statement is syntactically and semantically valid, the coordination node 101 can generate a logical plan tree based on the SQL statement. This logical plan tree indicates the logical execution plan for operations such as data calculation, analysis, and access. Then, the coordination node 101 can optimize the plan tree using one or more optimizers and send the optimized logical plan tree to the worker node 102 for execution. Specifically, a scheduler can determine which executor in the worker node 102 will execute the logical plan tree. The worker node 102 can include one or more executors. Figure 1 (Taking executor 1 and executor 2 as examples), it can execute the corresponding plan according to the received logical plan tree, and return the query results obtained after executing the plan to the client 104 through the coordination node 101, so that the query results can be presented to the user on the client 104.

[0029] In the era of big data, the amount of data in data source 103 is usually enormous, with the number of rows potentially reaching tens or even hundreds of millions, which may result in a large table A, such as a fact table. If worker node 102 directly traverses table A, the efficiency of data processing system 100 in querying data will decrease due to the large amount of data to be traversed. Therefore, currently, a common practice is to join table A with another table B (such as a dimension table) with a smaller amount of data. Table A and table B must contain at least some of the same columns, and the data in the two tables must have at least some of the same column values. Then, worker node 102 can first query table B to find data that meets the query conditions. This data can then be used as a filter to filter out the data in table A that meets the query conditions (i.e., the data the user needs). For example, the filter conditions could be some column values ​​from the same columns in tables A and B.

[0030] For example, suppose data source 103 has a large people table and a smaller orders table, respectively... Figure 2 As shown, both the `people` and `orders` tables have the same `id` (identity) column, and further share some identical column values, such as... Figure 2The two tables shown have some identical values ​​in their `id` columns, ranging from 120001 to 149999. Assume a user enters the SQL statement on client 104: `select people.name from people join orders on people.id=orders.id where orders.sum>10`, which queries data from data source 103 where `orders.sum` is greater than 10. To reduce the amount of data read from the `people` table, worker node 102 can first collect the values ​​of the `id` column corresponding to data in the `orders` table that meet the query conditions "people.id=orders.id" and "orders.sum>10". Figure 2 The `orders` table contains `id` values ​​such as 120001, 120002, ..., 149999, which are used as filter conditions. Worker node 102 can then read data from the `people` table based on these filter conditions, specifically reading rows with `id` values ​​of 120001, 120002, ..., 149999, without needing to read rows with `id` values ​​of 150000, 150001, etc. Thus, during data querying, worker node 102 does not need to read all data from the entire `people` table; instead, it can read only a portion of the data based on the generated filter conditions, thereby reducing the amount of data read from the `people` table and consequently reducing the amount of data involved in the Join operation between the two tables.

[0031] When the worker node 102 in the data processing system includes multiple executors, the Join operation can be distributed across multiple executors for execution, such as... Figure 1 The shown worker node 102 includes executor 1 and executor 2, and join operations can be assigned to join node_1 and join node_2 for execution. A join node is a logical node used to perform join operations on tables A and B. Each executor can send the filter conditions generated based on its corresponding table B to the coordinating node 101, such as... Figure 1 In this process, executor 1 can send its partial filter generated based on table B_1 to coordinating node 101, and executor 2 can also send its partial filter generated based on table B_2 to coordinating node 101. Coordinating node 101 then merges the filter conditions sent by the multiple executors. Finally, coordinating node 101 can distribute the merged filter conditions to executors 1 and 2, which are performing the join operation, so that each executor can read the corresponding data from its respective table A based on the merged filter conditions.

[0032] However, in practical applications, the filtering conditions generated by worker node 102 based on table B may not achieve a good data filtering effect. That is, after filtering (or merging the filtering conditions), the amount of data contained in the filtering conditions is large. Consequently, the amount of data read from table A based on these filtering conditions is not significantly different from the total amount of data in table A. This means that compared to the implementation method where worker node 102 directly reads all the data in table B, the data processing system 100 saves less data overhead due to the reduced amount of data read by worker node 102. At the same time, worker node 102 incurs considerable additional overhead (such as computational resource consumption) in generating filtering conditions and using them to filter data in table B, which may exceed the overhead savings of the data processing system 100. Specifically, when the data processing system 100 queries data using multiple worker nodes 102, each worker node 102 sends its generated filtering conditions to the coordinating node 101. The coordinating node 101 then merges the multiple filtering conditions and distributes the merged filtering conditions to the worker nodes 102. This further consumes more resources of the data processing system 100 (including network transmission, system computing, and storage resources). Thus, when the filtering effect of the filtering conditions is poor, compared to the worker nodes 102 directly reading all the data in the entire table B, the data processing system 100's use of filtering conditions not only fails to reduce the overall overhead of the data processing system 100, but also increases the overall overhead of the data processing system 100 due to the additional network transmission, system computing, and storage resources consumed by the filtering conditions. At the same time, the processes of generating, transmitting, merging, and applying filtering conditions also reduce data query efficiency.

[0033] Based on this, this application provides a data query method to maintain the data query efficiency of the data processing system 100 at a high level while keeping the query overhead at a low level. Specifically, the working node 102 can pre-estimate the proportion of data that meets the query conditions in table A, which has a larger data volume, based on the query conditions and the data information corresponding to the two tables involved in the Join operation. When the proportion is large, it indicates that the generated filtering conditions may not achieve a good data filtering effect, that is, the difference in data volume before and after filtering is not significant. In this case, the working node 102 can directly query the data that meets the query conditions (hereinafter referred to as the data to be queried) from tables A and B without generating filtering conditions and perform the query operation. When the proportion of data is small, it indicates that the filtering conditions can effectively filter out a large amount of irrelevant data in table A (that is, non-data to be queried, which does not need to be read to the working node 102). In this case, the working node 102 can generate the filtering conditions using the query conditions and table B, and query the data to be queried from table A based on the filtering conditions. Therefore, when the filtering condition is effective, worker node 102 can use it to reduce the amount of data read from table A, improving data query efficiency and reducing query overhead. Conversely, when the filtering condition is ineffective, data processing system 100 can avoid the problems of decreased data query efficiency and increased query overhead caused by generating filtering conditions by not generating them. In this way, data processing system 100 can maintain both high data query efficiency and low query overhead.

[0034] For example, such as Figure 3 As shown, the Join operation is still broken down and executed in Executor 1 and Executor 2. Both Executor 1 and Executor 2 contain a decision module (i.e.,... Figure 3 The executor includes two decision modules (Module 1 and Module 2), which determine whether to generate filter conditions. Each decision module in the executor can estimate the data filtering effect of the filter conditions to be generated based on the query conditions and the data information corresponding to the two tables to be joined. Specifically, it determines whether the proportion of the data is large. If the estimated data filtering effect is good, the decision module can instruct its executor to generate the filter conditions. Conversely, if the estimated data filtering effect is poor, the decision module can instruct its executor not to generate the filter conditions.

[0035] Thus, for executors 1 and executor 2, if both executors generate filter conditions, they can each send their generated (partial) filter conditions to the coordinating node 101, which will then merge and distribute the conditions. If only executor 1 generates filter conditions, it can directly read data from table A_1 using those conditions, while executor 2 can directly read all the data in table A_2 and then find the data that meets the query conditions. If neither executor generates filter conditions, they can each read their respective corresponding table A and find the data that meets the query conditions from the read table A.

[0036] It is worth noting that, Figure 1 as well as Figure 3 The system architecture shown is merely an example and is not intended to limit its specific implementation to this example. For example, in other possible system architectures, the data processing system may not include the client 104; or, the number of executors included in the worker nodes may not be limited to two; or, in addition to coordinating nodes and worker nodes, the data processing system may also integrate data sources, etc. In practical applications, the data processing system can... Figure 1 as well as Figure 3 The architecture shown may be adapted to add or remove corresponding components, but this embodiment does not limit this.

[0037] To make the above-mentioned objectives, features, and advantages of this application more apparent and understandable, various non-limiting embodiments of the present application will be described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0038] like Figure 4 The diagram shown is a flowchart illustrating a data query method according to an embodiment of this application. This method can be applied to, for example... Figure 1 or Figure 3 The data processing system 100 shown can be executed by the data processing system 100 or by corresponding nodes within the data processing system 100. In this embodiment, the method is illustrated by taking the execution of the method by the coordination node 101 and the worker node 102 in the data processing system 100 as an example. The method specifically includes:

[0039] S401: Coordinating node 101 receives a data query statement, which includes query conditions for the data to be queried and query operations for the first file and the second file, wherein the data volume of the first file is greater than the data volume of the second file.

[0040] In practical applications, users can provide data query statements to the data processing system 100, so that the data processing system 100 can locate the data to be queried based on the data query statement. For ease of description, the data required by the user in this embodiment is referred to as the data to be queried.

[0041] As an example of receiving data query statements, the data processing system 100 includes a client 104, which can present data to the user as follows: Figure 5 The query input interface shown is shown. Users can enter their query statements in a specific area of ​​the query input interface presented by client 104, so that data processing system 100 can provide the desired query results. Then, client 104 can send the data query statement to coordinating node 101, for example, by sending it as a data query request, so that coordinating node 101 can retrieve the data required by the user. The data query statement includes query conditions and query operations on two files (such as two tables), such as a join operation, and the query conditions are used to locate the data to be queried. For example, the data query statement entered by the user could be an SQL statement: `select people.name from people joinorders on people.id=orders.id where orders.sum>10`, used to query... Figure 2 The data in the corresponding id value row of the people table shown is shown. In the SQL statement, "people.id=orders.id" and "orders.sum>10" are the query conditions used to locate the data to be queried, that is, the row data in the aforementioned people table with id values ​​of 120001, 120002, ..., 149999.

[0042] S402: Coordinating node 101 performs syntactic and semantic analysis on the data query statement to determine whether the data query statement entered by the user is valid.

[0043] In practical applications, if the user-inputted data query statement is invalid, the data processing system 100 can terminate the data query task and prompt the user to input a data query statement with correct syntax / semantics. If the user-inputted data query statement is determined to be valid, the coordinating node 101 can continue to execute subsequent data query steps.

[0044] S403: After determining that the data query statement is valid, the coordinating node 101 sends a task to the worker node to perform query operations on the first file and the second file. The task includes the query conditions for the first file and the second file.

[0045] As an example, the task sent by the coordinating node 101 to the worker node may be, for example, a logical plan tree, which includes the query conditions for the data to be queried and the identifiers (such as file names) of the two files involved in the query operation. In a specific implementation, the coordinating node can generate a logical plan tree based on the data query statement and send the logical plan tree to the worker node 102 so that the worker node 102 can be scheduled to execute the tasks in the logical plan tree.

[0046] S404: Based on the query conditions in the received task and the data information corresponding to the first file and the second file respectively, worker node 102 estimates the proportion of data in the first file that meets the query conditions in the first file.

[0047] The data source 103 may include a first file and a second file, and the worker node 102 can obtain data that meets the query conditions by accessing the data in the first file and the second file. In practical applications, the first file and the second file may be formed from different data in the data source 103, and specifically, the first file and the second file may record data in the form of tables, such as... Figure 2 The table shown includes the people table and the orders table, etc.

[0048] In some implementations, the data in the first file and the second file can be related. For example, the first file and the second file can contain the same columns, and the values ​​included in that column in the first file and the values ​​included in that column in the second file are at least partially the same. Figure 2 The people and orders tables shown contain some identical id values ​​in the id column, such as 120001, 120002, ..., 149999, etc.

[0049] In this embodiment, the amount of data included in the first file and the second file may differ; for example, the amount of data in the first file may be greater than that in the second file. In practical applications, the first file may specifically be a fact table, while the second file may specifically be a dimension table, etc. The data recorded in the fact table is usually quite rich, and may include information from multiple dimensions. For example, a fact table used for work records may include information on work dates, employees, work hours, overtime hours, job nature, job content, and job supervisor, etc. It may include information from three dimensions: time dimension (work dates, work hours, overtime hours), personnel dimension (employees, job supervisor), and job attributes (job nature, job content). Accordingly, because the fact table records a lot of data, its data volume is usually large. A dimension table can be used to record some dimension data. For example, a time dimension table may only record time dimension data, such as the aforementioned work dates, work hours, and overtime hours, while data on employees, job nature, job content, and job supervisor may not be recorded in this time dimension table, but may be recorded in other dimension tables. A dimension table can be viewed as a window into the data being analyzed, containing data characteristics from the fact table across certain dimensions. Typically, the dimension table contains less data than the fact table.

[0050] In this embodiment, the first file may contain a large amount of data, while the data that meets the query conditions (i.e., the data to be queried) may be a small portion of the data in the first file. Therefore, if the worker node 102 directly reads the first file from the data source 103 and queries the data to be queried from it, the worker node 102 will need to read a large amount of useless data in the first file, excluding the data to be queried. Therefore, in this embodiment, the worker node 102 can use the filtering conditions generated based on the second file to reduce the amount of data read from the first file. Since the generated filtering conditions may not necessarily have a good data filtering effect, the worker node 102 can parse the query conditions and the two files involved in the Join operation from the logical execution plan tree, and obtain the data information corresponding to the first file and the second file respectively. Thus, the worker node 102 can estimate the proportion of the data to be queried in the first file based on the query conditions and the data information corresponding to the two files respectively, so as to use this proportion to measure whether the filtering conditions can have a better data filtering effect.

[0051] Specifically, when the proportion is large, it indicates that most of the data in the first file is the data to be queried. In this case, even if the data in the first file is filtered using filtering conditions, the difference between the amount of data to be queried in the first file read by worker node 102 and the amount of data read from the entire first file is not significant. This indicates that the data filtering effect of the filtering conditions is poor. Conversely, when the proportion is small, it indicates that a small portion of the data in the first file is the data to be queried. In this case, the filtering conditions can effectively filter most of the data in the first file, and worker node 102 only needs to read a small portion of the data in the first file to obtain the required data to be queried. This indicates that the data filtering effect of the filtering conditions is good.

[0052] As an example of an estimated proportion implementation, the data information corresponding to the first and second files can specifically be the statistical information of the data corresponding to the first and second files, respectively. For example, the statistical information corresponding to the first file (or the second file) can be any one or more of the following: data range, data distribution interval, and duplicate data. Here, data range refers to the range of values ​​for the data in the first file (or the second file), specifically the range of values ​​for each column in the first file, i.e., the minimum to maximum value of each column; data distribution interval is used to describe the distribution of data in the first file (or the second file), for example, it can be represented by a bar chart; duplicate data can be used to indicate the existence of identical data in the first file (or the second file), for example, it can be represented by the repetition rate or the number of duplicate entries.

[0053] Work node 102 can determine the identifier of the data in the second file that meets the query condition based on the data statistics information corresponding to the second file. Then, based on the data statistics information corresponding to the first file, it can calculate the proportion of data with that identifier in the first file. For example, assuming the data statistics information is specifically a data range, work node 102 can find the data range that meets the query condition in the second file based on the data range of the second file, and further determine the identifier corresponding to the data within the found data range. Then, work node 102 can find the data with that identifier in the first file based on the determined data identifier, and calculate the proportion of data with that identifier in the first file. For example, if the data range in the first file is 1 to 100,000, and the data with the above identifier in the first file is 1 to 1,000, then the proportion can be 1 / 10 (i.e., 1,000 / 10,000). Of course, the data processing system 100 can also calculate the proportion based on the above data distribution range, duplicate data, or any combination of the above three types of information; this embodiment does not limit this.

[0054] The statistical information corresponding to the first file (or the second file) can be pre-collected and stored locally by the worker node 102 and the coordinating node 101. For example, after the data processing system 100 connects to the data source 103, the worker node 102 can access each file in the data source 103 one by one and send the read files to the coordinating node 101. The coordinating node 101 can then generate and save the corresponding statistical information for each file. Subsequently, when querying data, the worker node 102 can obtain the corresponding statistical information for the file it wants to access from its local storage (or from the coordinating node 101) to determine whether to generate filtering conditions during the data query process. Of course, in practical applications, the worker node 102 can also obtain the statistical information for the first and second files in other ways. For example, when the first and second files are stored in the data source 103, the relevant devices in the data source 103 can pre-generate corresponding statistical information for the first and second files, allowing the worker node 102 to obtain the statistical information for the first and second files from the data source 103. In this embodiment, the specific implementation method for obtaining data statistics information by the working node is not limited.

[0055] In another example of estimating the proportion, worker node 102 can also determine the proportion through data sampling. In this case, the data information corresponding to the first and second files can specifically be sampled data from the first and second files. Specifically, worker node 102 can sample the data in the first and second files respectively, for example, through random sampling, equal-interval sampling, or proportional sampling. It then reads the sampled data from the first and second files from the data source 103. Worker node 102 can then determine the identifier of the target sampled data in the second file that meets the query condition, and count the sampled data in the first file that has that identifier. Therefore, it can calculate the proportion of the sampled data with that identifier in the first file. Thus, the proportion of the data to be queried in the first file can be predicted using the sampled data from the first and second files.

[0056] It should be noted that the above two methods of estimating the proportion are only examples. In actual applications, other possible methods can be used to estimate the proportion, and this embodiment does not limit this.

[0057] S405: When the proportion is greater than the preset threshold, worker node 102 reads the first file and the second file to worker node 102 and performs a query operation on the first file and the second file.

[0058] S406: When the proportion is less than the preset threshold, the working node 102 generates filtering conditions based on the query conditions and the second file, and queries the data that meets the query conditions from the first file according to the filtering conditions, and performs the corresponding query operation.

[0059] In this embodiment, since the proportion calculated in step S404 can reflect the data filtering effect of the filtering conditions, when the proportion is greater than the preset threshold, it indicates that the filtering conditions to be generated are difficult to filter out a large amount of data in the first file. If the working node 102 generates the filtering conditions, it may reduce the query efficiency of the data processing system 100. At this time, the working node 102 can read the first file and the second file from the data source 103 without generating the filtering conditions based on the second file, and query the data that meets the query conditions from the read first file and the second file according to the query conditions and perform corresponding operations, such as performing a join operation on the queried data. In this way, the total overhead of the working node 102 in generating the filtering conditions and applying the filtering conditions (including even the transmission and merging of the filtering conditions) can be avoided from exceeding the total overhead of the working node 102 directly reading the first file.

[0060] When the proportion is less than the preset threshold, it indicates that the generated filtering conditions can effectively filter out more data in the first file. Correspondingly, the overall overhead of worker node 102 in generating and applying the filtering conditions (including transmission and merging of the filtering conditions) is less than the overall overhead of worker node 102 directly reading the first file. At this time, worker node 102 can read the second file and query the first data that meets the query conditions. Then, worker node 102 can generate filtering conditions based on the first data and send these filtering conditions to data source 103, instructing data source 103 to query the second data that matches the first data in the first file and return it to worker node 102. The first and second data can have the same identifier, such as the first data in the first file and the second data in the second file having the same column value (e.g., the id value in the people table and the orders table in the previous example). In this way, worker node 102 can perform the above query operations on the received second data and the first data queried from the second file, such as performing a Join operation on the first and second data, to obtain the data required by the user. Because the amount of data read by worker node 102 from data source 103 is relatively small (compared to the amount of data read directly from the first file), the data query efficiency of worker node 102 can be maintained at a high level. The specific implementation process of worker node 102 generating dynamic query conditions based on the second file and query conditions can be found in the aforementioned descriptions and will not be repeated here.

[0061] The preset threshold can be calculated and determined by the working node 102 based on the amount of data in the first file; that is, the working node 102 can generate different preset thresholds when querying data in different files. Alternatively, the preset threshold can be a fixed value, such as an empirical value, and can be preset by relevant technical personnel. In this embodiment, the specific implementation method of setting the preset threshold is not limited.

[0062] It is worth noting that in this embodiment, the data source 103 includes a first file and a second file as an example for illustrative purposes. In actual applications, the data source 103 may also include more files, such as a third file. When multiple files in the data source 103 need to participate in the Join operation, the first file and the second file can be Joined first, and then the first file and the third file can be Joined, and so on. This embodiment will not elaborate on this.

[0063] It is worth noting that in some possible implementations, after determining the maximum data volume of each file in data source 103, the aforementioned proportion can also be characterized by the data volume included in the filtering conditions. For example, in some scenarios, when the data volume included in the filtering conditions is large, it can represent a large proportion; conversely, when the data volume included in the filtering conditions is small, it represents a small proportion. Therefore, when determining whether to generate filtering conditions based on the size of the proportion, it can also be determined based on the size of the data volume included in the filtering conditions. The specific implementation principle is similar to the above process, and this embodiment will not elaborate further.

[0064] In a further possible implementation, during the data query process, the working node 102 can determine whether to continue using the filtering conditions to query the remaining data based on the partially queried data. For example, if the filtering effect of the filtering conditions is predicted to be good by sampling the data of the first file and the second file as described above, there may be a problem that the filtering effect of the filtering conditions is poor in actual use. In this way, the working node 102 can stop using the filtering conditions to continue filtering the data in the first file in a timely manner.

[0065] As an example, after worker node 102 retrieves a portion of the data to be queried from the first file using filtering conditions, it can calculate the proportion of the retrieved data in the first file that has been traversed. The traversed data in the first file includes both the queried data and the data currently filtered out using the filtering conditions. When this proportion is large, specifically exceeding the filtering threshold, it indicates that the filtering conditions are ineffective in filtering the data in the first file during actual use. In this case, since filtering the data in the first file requires additional computational overhead, worker node 102 can directly read the untraversed data from the first file when subsequently querying the remaining data in the data to be queried (i.e., data other than the retrieved data), and continue querying the remaining data in the data to be queried from this untraversed data according to the query conditions, without needing to filter the data in the first file again. Of course, when the proportion is small, specifically when the proportion is less than the filtering threshold, it indicates that the filtering condition has a good filtering effect on the data in the first file during actual use. Therefore, the working node 102 can continue to use the filtering condition to query the remaining data in the data to be queried from the untraversed data of the first file.

[0066] In other implementations, the proportion of the queried data in the traversed data of the first file can be determined by data sampling. For example, when the amount of queried data in the data to be queried is large, the traversed data and the queried data of the first file can be sampled separately, and the proportion of the sampled data in the queried data to the sampled data in the traversed data can be used as the proportion. In this embodiment, the specific implementation of the proportion is not limited.

[0067] In practical applications, individual functional modules can be configured in the actuator, as described above. Figure 3 The module includes a judgment module, which determines the proportion of data in the first file through real-time monitoring or data sampling during the data query process, and determines whether to continue using filtering conditions to filter the data in the first file in subsequent queries based on this proportion. This module can be implemented in software or hardware, and this embodiment does not limit it.

[0068] The method for setting the filtering threshold can be similar to that for setting the preset threshold. For details on how to set the filtering threshold, please refer to the description of setting the preset threshold; it will not be repeated here. Furthermore, the value of the filtering threshold can be the same as or different from the preset threshold; this embodiment does not limit this.

[0069] The above text combines Figures 1 to 5This application describes in detail the data query method provided. The following section will combine... Figures 6 to 7 This application describes the apparatus and equipment provided in accordance with this application.

[0070] Similar to the above-described method, this application also provides a data query device that can achieve the above-described... Figure 4 The illustrated embodiment describes the functionality of a data processing system, which includes a coordination node and worker nodes. See also... Figure 6 As shown, the data query device 600 may include:

[0071] The first communication module 601 is used to send a task to perform a query operation on a first file and a second file to the working node. The task includes query conditions for the first file and the second file, wherein the data volume of the first file is greater than the data volume of the second file.

[0072] The estimation module 602 is used to estimate the proportion of data in the first file that meets the query conditions in the first file based on the query conditions of the received task and the data information corresponding to the first file and the second file respectively.

[0073] The reading module 603 is used to read the first file and the second file to the working node when the proportion is greater than a preset threshold, and to perform the query operation on the first file and the second file.

[0074] In one possible implementation, the reading module 603 is further configured to read the second file when the percentage is less than or equal to the preset threshold;

[0075] The device 600 further includes:

[0076] Query module 604 is used to query the first data in the second file that meets the query conditions;

[0077] Generation module 605 is used to generate filtering conditions based on the first data;

[0078] The second communication module 606 is used to send the filtering conditions to the data source where the first file is located. The filtering conditions are used to instruct the data source to query the second data that matches the first data from the first file, and to receive the second data sent by the data source, and to perform the query operation on the first data and the second data.

[0079] In one possible implementation, the data information corresponding to the first file and the second file respectively includes the data statistical information corresponding to the first file and the second file respectively;

[0080] The estimation module 602 is specifically used for:

[0081] Based on the data statistics information corresponding to the second file, determine the identifiers of the data in the second file that meet the query conditions;

[0082] Based on the data statistics information corresponding to the first file, calculate the proportion of data with the identifier in the first file.

[0083] In one possible implementation, the data information corresponding to the first file and the second file respectively includes the sampled data in the first file and the sampled data in the second file;

[0084] The estimation module 602 is specifically used for:

[0085] The working node samples the data in the first file and the data in the second file respectively to obtain sampled data in the first file and sampled data in the second file;

[0086] The working node determines the identifier of the target sampled data in the sampled data of the second file that meets the query conditions;

[0087] The working node calculates the proportion of sampled data with the identifier in the sampled data of the first file.

[0088] In one possible implementation, the data statistics information corresponding to the first file includes any one or more of the following: the data range, the data distribution interval, and the duplicate data of the first file.

[0089] In one possible implementation, when the proportion is less than the preset threshold, the reading module 603 is further configured to include:

[0090] During the process of querying the data to be queried from the first file, the proportion of the data that has been queried in the data to be queried to the traversed data in the first file is calculated;

[0091] When the percentage is greater than the filtering threshold, the remaining data in the data to be queried is retrieved from the untraversed data of the first file according to the query conditions.

[0092] The data query device 600 in this embodiment corresponds to Figure 4 The data query method shown herein; therefore, for the specific implementation of each functional module in the data query device 600 of this embodiment and its technical effects, please refer to [link to relevant documentation]. Figure 4 The relevant details in the illustrated embodiments are described in detail here, and will not be repeated here.

[0093] Furthermore, embodiments of this application also provide a device, such as... Figure 7 As shown, device 700 may include a communication interface 710 and a processor 720. Optionally, device 700 may also include a memory 730. The memory 730 may be located internally or externally to device 700. For example, the above... Figure 4 In the illustrated embodiment, each action can be implemented by the processor 720. The processor 720 can obtain the first and second files from the data source 103 through the communication interface 710 and use them to implement... Figure 4 Any method executed within it. During implementation, each step of the processing flow can be accomplished through integrated logic circuits in the hardware of the processor 720 or through software instructions. Figure 4 The method executed by the processor 720 is described below for brevity. The program code used by the processor 720 to implement the above method can be stored in the memory 730. The memory 730 and the processor 720 are connected, such as by a coupling connection.

[0094] Some features of the embodiments of this application can be implemented / supported by the processor 720 executing program instructions or software code in the memory 730. The software components loaded on the memory 730 can be summarized functionally or logically, for example... Figure 6 The module shown includes a prediction module 602, a reading module 603, a query module 604, and a generation module 605. The functions of the first communication module 601 and the second communication module 606 can be implemented through the communication interface 710.

[0095] Any communication interface involved in the embodiments of this application can be a circuit, a bus, a transceiver, or any other device that can be used for information exchange. For example, the communication interface 710 in device 700. Exemplarily, the other device can be a device connected to device 700, etc.

[0096] The processors involved in the embodiments of this application can be general-purpose processors, digital signal processors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs) or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components, and can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly manifested as being executed by a hardware processor, or executed by a combination of hardware and software modules within the processor.

[0097] The coupling in the embodiments of this application is an indirect coupling or communication connection between devices, modules, or modules, which can be electrical, mechanical, or other forms, and is used for information interaction between devices, modules, or modules.

[0098] The processor may work in conjunction with memory. Memory can be non-volatile memory, such as hard disk drives (HDDs) or solid-state drives (SSDs), or it can be volatile memory, such as random-access memory (RAM). Memory is any other medium capable of carrying or storing desired program code in the form of instructions or data structures, and accessible by a computer, but is not limited to this.

[0099] This application does not limit the specific connection medium between the communication interface, processor, and memory. For example, the memory, processor, and communication interface can be connected via a bus. The bus can be categorized as an address bus, data bus, control bus, etc.

[0100] Based on the above embodiments, this application also provides a computer storage medium storing a software program. When read and executed by one or more processors, the software program can implement the method executed by the data processing system 100 provided in any one or more of the above embodiments. The computer storage medium may include various media capable of storing program code, such as a USB flash drive, portable hard drive, read-only memory, random access memory, magnetic disk, or optical disk.

[0101] Based on the above embodiments, this application also provides a chip, which includes a processor for implementing the functions of the data processing system 100 involved in the above embodiments, for example, for implementing... Figure 4 The method executed in the processor. Optionally, the chip also includes a memory for storing program instructions and data necessary for the processor to execute. The chip may be composed of chips or may contain chips and other discrete devices.

[0102] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0103] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0104] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0105] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0106] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms can be used interchangeably where appropriate; this is merely a way of distinguishing objects with the same attributes in the embodiments of this application.

[0107] Obviously, those skilled in the art can make various modifications and variations to the embodiments of this application without departing from the scope of the embodiments of this application. Therefore, if these modifications and variations to the embodiments of this application fall within the scope of the claims of this application and their equivalents, this application also intends to include these modifications and variations.

Claims

1. A data query method, characterized by, The method is applied to a data processing system, which includes a coordination node and worker nodes, and the method includes: The coordinating node sends a task to the worker node to perform query operations on the first file and the second file. The task includes query conditions for the first file and the second file, wherein the data volume of the first file is greater than the data volume of the second file. The working node estimates the proportion of data in the first file that meets the query conditions in the first file based on the query conditions of the received task and the data information corresponding to the first file and the second file respectively. The data information corresponding to the first file and the second file respectively includes sampled data in the first file and sampled data in the second file, or the data information corresponding to the first file and the second file respectively includes data statistics information corresponding to the first file and the second file respectively. When the proportion is greater than a preset threshold, the working node reads the first file and the second file to the working node, and performs the query operation on the first file and the second file; When the percentage is less than the preset threshold, the working node generates filtering conditions based on the query conditions and the second file, and performs a query operation on the first file based on the filtering conditions.

2. The method according to claim 1, characterized in that, When the proportion is less than the preset threshold, the working node generates filtering conditions based on the query conditions and the second file, and performs a query operation on the first file based on the filtering conditions, including: When the percentage is less than or equal to the preset threshold, the working node reads the second file; The working node queries the first data in the second file that meets the query conditions; The working node generates filtering conditions based on the first data; The working node sends the filtering conditions to the data source where the first file is located. The filtering conditions are used to instruct the data source to query the second data that matches the first data from the first file. The working node receives the second data sent by the data source and performs the query operation on the first data and the second data.

3. The method according to claim 1 or 2, characterized in that, The data information corresponding to the first file and the second file respectively includes the sampled data in the first file and the sampled data in the second file; The working node estimates the proportion of data in the first file that meets the query conditions based on the received query conditions of the task and the data information corresponding to the first file and the second file, including: The working node samples the data in the first file and the data in the second file respectively to obtain sampled data in the first file and sampled data in the second file; The working node determines the identifier of the target sampled data in the sampled data of the second file that meets the query conditions; The working node calculates the proportion of sampled data with the identifier in the sampled data of the first file.

4. The method according to claim 1 or 2, characterized in that, The data information corresponding to the first file and the second file respectively includes the data statistics information corresponding to the first file and the second file respectively; The working node estimates the proportion of data in the first file that meets the query conditions based on the received query conditions of the task and the data information corresponding to the first file and the second file, including: The working node determines the identifier of the data in the second file that meets the query conditions based on the data statistics information corresponding to the second file; The working node calculates the proportion of data with the identifier in the first file based on the data statistics information corresponding to the first file.

5. The method of claim 4, wherein, The data statistics information corresponding to the first file includes any one or more of the following: the data range, the data distribution interval, and the duplicate data of the first file.

6. A data query apparatus, characterized by comprising: The device is used in a data processing system, which includes a coordination node and worker nodes, and the device includes: The first communication module is used to send a task to the working node to perform a query operation on the first file and the second file. The task includes query conditions for the first file and the second file, wherein the data volume of the first file is greater than the data volume of the second file. The estimation module is used to estimate the proportion of data in the first file that meets the query conditions in the first file based on the query conditions of the received task and the data information corresponding to the first file and the second file respectively. The data information corresponding to the first file and the second file respectively includes sampled data in the first file and sampled data in the second file, or the data information corresponding to the first file and the second file respectively includes data statistics information corresponding to the first file and the second file respectively. The reading module is used to read the first file and the second file to the working node when the proportion is greater than a preset threshold, and to perform the query operation on the first file and the second file; when the proportion is less than the preset threshold, the working node generates filtering conditions based on the query conditions and the second file, and performs a query operation on the first file based on the filtering conditions.

7. The apparatus of claim 6, wherein, The reading module is specifically used to read the second file when the proportion is less than or equal to the preset threshold; and to query the first data in the second file that meets the query conditions. Generate filtering conditions based on the first data; The filtering conditions are sent to the data source where the first file is located. The filtering conditions are used to instruct the data source to query the second data that matches the first data from the first file, and to receive the second data sent by the data source, and to perform the query operation on the first data and the second data.

8. The apparatus of claim 6 or 7, wherein, The data information corresponding to the first file and the second file respectively includes the data statistics information corresponding to the first file and the second file respectively; The prediction module is specifically used for: Based on the data statistics information corresponding to the second file, determine the identifiers of the data in the second file that meet the query conditions; Based on the data statistics information corresponding to the first file, calculate the proportion of data with the identifier in the first file.

9. An apparatus, comprising: The device includes a processor and a memory; The processor is configured to execute instructions stored in the memory to cause the device to perform the method of any one of claims 1 to 5.

10. A computer-readable storage medium, characterized in that, Includes instructions for implementing the method of any one of claims 1 to 5.

Images