Data query method and device and electronic equipment
By parsing multiple SQL statements and building an aggregated field collection, querying them in the corresponding data partition based on these collections, and deploying operators that support query functions, the problem of slow aggregated query speed in database queries is solved, and a more efficient query process is achieved.
Patent Information
- Application Number
- CN202411998151.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-31
- Publication Date
- 2025-05-30
AI Technical Summary
In database query scenarios, aggregation query speeds of multiple SQL statements are slow, especially when a large amount of data is needed to query from the local tables of multiple storage nodes.
By parsing multiple SQL statements, the same aggregate fields are constructed into an aggregate field collection and querying in the corresponding data partition based on these collections. At the same time, operators that support various query functions are deployed to speed up the query process.
By querying similar data partitions with the same aggregate fields, the efficiency of the query process is improved, the query time is reduced, and the entire query process is smoother through the deployment of operators.
Smart Images
Figure CN120067153A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of database technology, and in particular, to a data query method, apparatus, and electronic device. Background Art
[0002] In the scenario of database query, it is possible to receive multiple aggregate query statements constructed based on Structured Query Language (SQL) simultaneously. The SQL statements may contain multiple aggregate fields, and the content corresponding to different aggregate fields may be a large amount, and it is necessary to query from multiple local tables in different storage nodes.
[0003] Currently, the query solution for the above scenario is to gather the local tables distributed on multiple nodes on one node and then execute the SQL statement. This method has a slow query speed. Summary of the Invention
[0004] Embodiments of this application provide a data query method, apparatus, and electronic device, which can accelerate the aggregate query speed.
[0005] In a first aspect, an embodiment of this application provides a data query method, and the method includes:
[0006] Obtain N query statements; where each query statement includes at least one aggregate field;
[0007] Classify the aggregate fields corresponding to the N query statements according to the functions of the respective aggregate fields, and obtain M aggregate field sets; where the types of the aggregate fields included in each aggregate field set are the same;
[0008] Query in the first data partition based on the first aggregate field set to obtain a corresponding first query result; where the first aggregate field set is any one of the M aggregate field sets; the first data partition is obtained by partitioning the database to be queried according to the functions of the respective aggregate field sets;
[0009] Determine the query result of the first query statement according to the query results corresponding to at least one aggregate field set included in the first query statement; where the first query statement is any one of the N query statements.
[0010] In a possible implementation manner, the method includes:
[0011] Obtain the query description information corresponding to each first aggregate field included in the first aggregate field set; where the query description information indicates the data size and location in the database to be queried;
[0012] Determine a first data partition corresponding to the first aggregation field set according to the query description information.
[0013] In a possible implementation manner, determining the query result of the first query statement according to the query results corresponding to at least one aggregation field set included in the first query statement includes:
[0014] Obtain the query results corresponding to at least one aggregation field set included in the first query statement;
[0015] Concatenate the query results in the order of the field contents corresponding to the respective query results in the first query statement to obtain the query result of the first query statement.
[0016] In a possible implementation manner, at least one operator is deployed on the storage node;
[0017] The querying in the first data partition based on the first aggregation field set to obtain a corresponding first query result includes:
[0018] In the at least one operator, determine a first operator for executing the first set of aggregation fields according to the correspondence between the operator function and the representation function of the aggregation field set;
[0019] Apply the first operator to query in the first data partition to obtain a corresponding first query result.
[0020] In a possible implementation manner, each query statement has a corresponding query identifier; the query identifier is determined according to the offset and timestamp of the query statement in the stream processing platform.
[0021] In a possible implementation manner, the method further includes:
[0022] Determine that the N query statements are SQL statements.
[0023] In a second aspect, an embodiment of the present application provides a data query device, and the device includes:
[0024] An acquisition module, configured to acquire N query statements; where each query statement includes at least one aggregation field;
[0025] A classification module, configured to classify the aggregation fields corresponding to the N query statements according to the functions of the respective aggregation fields to obtain M aggregation field sets; where the types of the aggregation fields included in each aggregation field set are the same;
[0026] A query module, configured to perform a query in a first data partition based on a first set of aggregation fields to obtain a corresponding first query result; wherein, the first set of aggregation fields is any one of the M sets of aggregation fields; the first data partition is obtained by partitioning a database to be queried based on the functions of each set of aggregation fields.
[0027] A summarization module, configured to determine the query result of the first query statement according to the query results respectively corresponding to at least one set of aggregation fields included in the first query statement; wherein, the first query statement is any one of the N query statements.
[0028] In a third aspect, an embodiment of the present application provides an electronic device, which includes:
[0029] At least one processor; and a memory communicatively connected to the at least one processor; wherein, the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method in the first aspect as described above.
[0030] In a fourth aspect, an embodiment of the present application provides a computer storage medium, which stores a computer program, and the computer program is used to cause a computer to execute the method in the first aspect as described above.
[0031] In a fifth aspect, an embodiment of the present application provides a computer program product, which implements the method in the first aspect as described above when the computer program is executed by a processor.
[0032] An embodiment of the present application provides a data query method. By parsing different SQL statements, constructing sets of aggregation fields for the same aggregation fields, and then querying based on the sets of aggregation fields, it is possible to query similar data partitions based on the fields in the same set of aggregation fields, improving the efficiency of the query process; moreover, operators supporting various query functions are also deployed on the storage nodes, and each operator executes the corresponding function, making the entire query process smoother; in addition, the aggregation fields and their corresponding query description information are also identified, so as to be able to determine the query results corresponding to different query statements subsequently. BRIEF DESCRIPTION OF THE DRAWINGS
[0033] Figure 1 FIG. is a schematic flowchart of a data query method according to an exemplary embodiment of the present invention;
[0034] Figure 2 FIG. is a schematic flowchart of the execution of the first operator according to an exemplary embodiment of the present invention;
[0035] Figure 3Schematic diagram of the execution process of the second operator according to an exemplary embodiment of the present invention;
[0036] Figure 4 Schematic diagram of the execution process of the third operator according to an exemplary embodiment of the present invention;
[0037] Figure 5 Schematic diagram of the execution process of the fourth operator according to an exemplary embodiment of the present invention;
[0038] Figure 6 Schematic diagram of the execution process of the fifth operator according to an exemplary embodiment of the present invention;
[0039] Figure 7 Schematic diagram of the execution process of the sixth operator according to an exemplary embodiment of the present invention;
[0040] Figure 8 Schematic diagram of the execution process of the seventh operator according to an exemplary embodiment of the present invention;
[0041] Figure 9 Schematic diagram of a data query device according to an exemplary embodiment of the present invention;
[0042] Figure 10 Schematic diagram of an electronic device according to an exemplary embodiment of the present invention. Detailed implementation manners
[0043] To make the objectives, technical solutions, and advantages of the present application clearer and more understandable, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. Apparently, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present application without making creative efforts fall within the scope of protection of the present application. Without conflict, the embodiments in the present application and the features in the embodiments may be combined with each other arbitrarily. And although the logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than here.
[0044] In the embodiments of the present application, the acquisition, storage, use, processing, etc. of data all comply with the relevant provisions of national laws and regulations.
[0045] Based on the problem of slow query speed in the aggregation query scenario in the prior art, the embodiments of the present application provide a data query method, as Figure 1 shown, the method includes:
[0046] S101: Obtain N query statements.
[0047] In the embodiments of the present application, the N query statements may come from a stream processing platform, such as Kafka. The query statements may be SQL statements. For example, Select*from table_a A
[0048] Where id in(select id from table_b B where item_id in(select id fromtable_cC where name like‘%wang%’)).
[0049] Among them, each query statement includes at least one aggregation field, such as table_a A, table_b B, table_c, %wang%, where in the above embodiments.
[0050] Different query statements may query data in the same table. For example, the first query statement queries table_aA and table_b B, and the second query statement queries table_a A and table_c C, that is, both the first query statement and the second query statement query table_a A.
[0051] S102: Classify the aggregation fields corresponding to the N query statements according to the functions of the respective aggregation fields to obtain M aggregation field sets.
[0052] The function of the aggregation field may represent the content to be queried, such as table_a A in S101 above; or it may represent the query condition, such as "where" in S101 above.
[0053] Classify all the obtained aggregation fields according to the functions of the aggregation fields in the N query statements. For example, if both the first query statement and the second query statement query table_a A, then table_a A is used as an aggregation field set.
[0054] S103: Query in the first data partition based on the first aggregation field set to obtain the corresponding first query result.
[0055] Among them, the first aggregation field set is any one of the M aggregation field sets; the first data partition is obtained by partitioning the database to be queried based on the functions of the respective aggregation field sets. For example, if the aggregation field in the aggregation field set is table_a A, then the partition corresponding to table_a A is determined from the database to be queried.
[0056] Further, obtain the query description information corresponding to each first aggregation field included in the first aggregation field set; wherein, the query description information indicates the data size and location in the database to be queried; determine the first data partition corresponding to the first aggregation field set according to the query description information.
[0057] Since the specific data queried by different query statements is different, that is, the query description information is different. That is, both the first query statement and the second query statement query table_a A, but the first query statement queries the 1st to 100th data in table_a A, and the second query statement queries the 200th to 500th data in table_a A.
[0058] In order to distinguish which query statement the fields in the first aggregation field set come from and ensure the accuracy of the query results of subsequent query statements, the embodiments of the present application identify the aggregation fields parsed from each query statement.
[0059] Specifically, the above identification can be constructed using the offset and timestamp of the query statement in Kafka. The offset and timestamp are the offset and timestamp when the query statement arrives at Kafka, and its format can be {SQL, Kafka offset, timestamp}. When the storage node obtains an SQL statement, it can obtain the identification of the query statement, and after determining the aggregation fields of the query statement, identify each aggregation field, so as to summarize the query results with the same identification subsequently.
[0060] S104: Determine the query result of the first query statement according to the query results corresponding to each at least one aggregation field set included in the first query statement.
[0061] In the scenario of aggregate query, it may be that multiple different query statements are executed simultaneously. During the process of executing multiple different query statements simultaneously, the present application constructs the same aggregate field set for the aggregate fields with the same function in different query statements, queries for each different aggregate sub-set, obtains the query results corresponding to each aggregate query field set, and when summarizing the query results, splices the query results according to the order of the field contents corresponding to each query result in the first query statement to obtain the query result of the first query statement. For example, the aggregate fields corresponding to the first query statement are table_a A and table_b B. table_a A is in the first aggregate field set, and table_b B is in the second aggregate field set. Based on the identifier of the first query, determine the corresponding table_a A data from the first aggregate field set, determine the corresponding table_b B data from the first aggregate field set, and then splice the queried table_a A data and table_b B data according to the front-back order of the contents of "table_a A" and "table_b B" in the first query statement. For example, if the content corresponding to table_a A is in the front, the obtained result is {table_a A data, table_b B data}.
[0062] In a possible implementation manner, the query statement can also be a query statement of the IP address type. If it is of the IP address type, it can be directly connected to the corresponding storage node.
[0063] In a possible implementation manner, in order to ensure the smooth and fast progress of the query process, the embodiments of the present application provide different operators. Different operators perform different functions. For example, they can be distinguished according to the processing function. For example, the first operator performs the information extraction function, the second operator performs the query data parsing function, and the third operator performs the data summarization function; an operator can also be constructed for each aggregate field set. The first operator performs the query of the table_a A aggregate field set, and the second operator performs the query of the table_b B aggregate field set.
[0064] The operator execution process provided by the embodiments of the present application is as Figure 2 shown.
[0065] (1) Operator 1 can be a SourceFuntion operator, which is used to pull N query statements from Kafka. The specific process is as Figure 3 shown.
[0066] S301: Obtain the query statement and its offset. The query statement can be pulled from kafka, and the offset is also the offset of the query statement in kafka. The query statement can be N, and each query statement corresponds to the first offset;
[0067] S302: Determine whether the obtained statement is an SQL statement. If so, execute S303; otherwise, directly obtain the data. That is, when the query statement contains IP address information, the data can be directly obtained from the corresponding storage node.
[0068] S303: Obtain the timestamp of the query statement. There can be N query statements, and each query statement corresponds to a first timestamp.
[0069] S304: Add the offset and timestamp to the SQL statement and copy the SQL statement.
[0070] S305: Send the data, that is, send the SQL statement with the added offset and timestamp to the subsequent operator (operator 2).
[0071] (2) Figure 2 In this case, operator 2 can be the process1 operator, and the process1 operator can read the aggregated query data of each local table according to the SQL statement. The execution process of the process1 operator is as Figure 4 shown
[0072] Receive the query statement of operator 1 (SourceFuntion operator). When it is determined that the query statement is an SQL statement, obtain the aggregated fields in the SQL statement and construct a set (i.e., the aggregated field set), and then send the set to the downstream operator (operator 3); attach the SQL additional fields to each query result data and send them one by one to the downstream operator (operator 4). The data stream structure output by the process1 operator is {row, aggregated field set, identifier}; after the data set is sent, send an additional end flag row data. That is, the process1 operator queries the data according to the aggregated field set and sends the queried data to the downstream operator. When it is determined that the query statement is not an SQL statement, that is, a query statement containing an IP address, connect to the local table according to the IP address.
[0073] (3) Figure 2 In this case, operator 3 can be the process2 operator, and the process2 operator can summarize each aggregated field set and the query description information corresponding to each aggregated field, such as the maximum value, the total number of rows, that is, 1 to 500 pieces of data.
[0074] (4) Figure 2 In this case, operator 4 can be the process3 operator, and the execution process of the process4 operator is as Figure 5 shown.
[0075] The process3 operator receives data, including the set of aggregated fields, identifiers, and queried data from process1, as well as the set of aggregated fields summarized by process2 and the query description information corresponding to each aggregated field. When the data comes from process1, the maximum and minimum values and the total number of rows are read from memory (or map), and each set of aggregated fields and the corresponding identifiers are carried in the output of the maximum and minimum values and the total number of rows. When the data comes from process2, the obtained maximum and minimum values, identifiers, and total number of rows are stored for subsequent queries. The process3 operator sends the identifiers of the aggregated fields, each aggregated field and its corresponding query description information (i.e., the maximum and minimum values, the total number of rows) to operators 5-7. In addition, the process3 also sends a large amount of data queried based on the aggregated fields to operators 5-7.
[0076] (5) Operators 5-7 can be KeyBy operators, and the execution process of the KeyBy operator is as Figure 6 shown.
[0077] Obtain the data of the process3 operator, extract the identifier and the partition value, and output to operator 8. The partition values (the number of partitions) corresponding to different KeyBy operators are different, that is, the return values of getkey() are different. The KeyBy operator divides the number of partitions according to the amount of data transmitted for the queries sent through presses3, so that the data is evenly distributed to each node for processing. For example, according to the data traffic (i.e., the amount of data queried based on multiple sets of aggregated fields) and the deployed resources, every 10,000 pieces of data are set as a partition. Multiple SQL statements are evenly distributed on the aggregation operator according to the identifier and the query range to quickly output the query results of multiple SQL statements.
[0078] Specifically, as in the embodiment in S103 above, it will not be elaborated here.
[0079] (6) Operator 8 can be a process4 operator, and the execution method of the process4 operator Figure 7 is shown.
[0080] Receives data and caches the received data. The process4 operator receives the maximum and minimum values (including the maximum value and the minimum value) and the partition value sent by the KeyBy operator. After determining that the number of maximum and minimum values reaches the preset number, arrange them in numerical order and output (output the queried data).
[0081] (7) Operator 9 can be a process5 operator, and operator 10 can be a process6 operator. The execution processes of the process5 operator and the process6 operator can be as Figure 8 shown.
[0082] Receive data and cache the received data. The process5 operator and the process6 operator receive the partition values sent by the process4 operator. After determining that the number of start values and end values for each output partition has reached the preset number, the data is output in the order of the partition values (the queried data is output), and then the process ends.
[0083] (8) The operator 11 can be a sink operator. The function of the sink operator is to summarize the output query information according to the identifier to obtain the query result corresponding to each query statement. The specific summarization process is as described in S104 above and will not be elaborated here.
[0084] The embodiments of the present application do not specifically limit the number and execution type of operators.
[0085] The embodiments of the present application provide a data query method. Compared with the prior art, in the embodiments of the present application, different identifiers are generated for each SQL statement. When an operator receives the specified number of end flag lines with the same identifier from the upstream, the operator processes the same batch of data sets and then sends them to the downstream operator, realizing the aggregation function in the distributed table SQL query, and multiple SQL queries can be executed simultaneously according to the identifier. Based on the maximum and minimum values and the total number of rows of the aggregation fields of the SQL query; the division interval is dynamically set through the keyby operator, which speeds up the aggregation processing speed and reduces the amount of data for subsequent aggregation. The multiple aggregation operators partition the data according to keyby, and the number of partitions is determined according to the amount of data, so that the data is evenly distributed to each node for processing. Multiple SQL statements are evenly distributed on the aggregation operator according to the identifier and the query interval, realizing the fast output of the query results of multiple SQL statements. Through keyby data partitioning and classification, the data with the same key value is aggregated and ordered in a small interval, and the data within the interval is orderly aggregated and the overall data flow is orderly output in the process4 operator. The Process5 and 6 operators output the SQL query results in a streaming manner, greatly increasing the data processing speed.
[0086] The embodiments of the present application first obtain the identifier of the SQL as a category distinction, then obtain the maximum and minimum values and the total data volume, divide the data to obtain its own interval value, and obtain an identifier and interval value information as the key of the keyby operator. By the numerical values of each node in the pre-query, the number and range of partitions of the keyby operator are dynamically adjusted. The multiple aggregation operators partition the data according to keyby, and the number of partitions is determined according to the amount of data, so that the data is evenly distributed to each node for processing, reducing the storage pressure on a single node, and the data is quickly output to the downstream.
[0087] In the embodiments of the present application, node data divided by each local table is extracted, and these nodes are divided into multiple statistical intervals. When the amount of node data with the maximum value in this interval reaches the previously statistically obtained data volume, it indicates that all the data in this interval has arrived, and it is output to the next aggregation operator. By summarizing the maximum and minimum values of the local table, the local table is divided, and the number of values at the start value and end value of the interval is obtained. When the number of values at the start value and end value reaches the statistical value, it indicates that all the data in this interval has arrived, and this interval can be statistically output. Through keyby data partitioning and classification, data with the same key value is aggregated and ordered in a small interval. In the process4 operator, when all the data in the interval has arrived, that is, when the number of values at the start value and end value reaches the statistical value, it indicates that all the data in this interval has arrived, and an ordered set of data within the interval and an overall ordered output of the data stream can be achieved. In the process5 and 6 operators, when the data of the previous interval in the continuous small intervals arrives, it can be streamed out without waiting for all the data in all the small intervals to arrive, and the SQL query result is streamed out, greatly increasing the data processing speed.
[0088] Based on the same inventive concept, the embodiments of the present application further provide a data query device, as Figure 9 shown. The device includes:
[0089] An acquisition module 901, configured to acquire N query statements; wherein, each query statement includes at least one aggregation field;
[0090] A classification module 902, configured to classify the aggregation fields corresponding to the N query statements according to the functions of the respective aggregation fields, to obtain M aggregation field sets; wherein, the types of the aggregation fields included in each aggregation field set are the same;
[0091] A query module 903, configured to perform a query in a first data partition based on a first aggregation field set to obtain a corresponding first query result; wherein, the first aggregation field set is any one of the M aggregation field sets; the first data partition is obtained by partitioning the database to be queried according to the functions of the respective aggregation field sets;
[0092] A summary module 904, configured to determine the query result of the first query statement according to the query results corresponding to at least one aggregation field set included in the first query statement; wherein, the first query statement is any one of the N query statements.
[0093] In a possible implementation manner, the query module 903 is configured to:
[0094] Obtain the query description information corresponding to each first aggregation field included in the first aggregation field set; wherein, the query description information indicates the data size and location in the database to be queried.
[0095] Determine the first data partition corresponding to the first aggregation field set according to the query description information.
[0096] In a possible implementation manner, the summarization module 904 is used for:
[0097] Obtain the query results corresponding to each of at least one aggregation field set included in the first query statement.
[0098] Concatenate the query results in the order of the field contents corresponding to the query results in the first query statement to obtain the query result of the first query statement.
[0099] In a possible implementation manner, at least one operator is deployed on the storage node.
[0100] The query module 905 is used for: among the at least one operator, determine the first operator for executing the first set field set according to the corresponding relationship between the operator function and the aggregation field set representation function.
[0101] Apply the first operator to perform a query in the first data partition to obtain the corresponding first query result.
[0102] In a possible implementation manner, the summarization module 904 is used for:
[0103] Determine that each query statement has a corresponding query identifier; the query identifier is determined according to the offset and timestamp of the query statement in the stream processing platform.
[0104] In a possible implementation manner, the query module 903 is used for:
[0105] Determine that the N query statements are SQL statements.
[0106] Based on the same inventive concept, an embodiment of the present application further provides an electronic device, which includes:
[0107] At least one processor; and a memory communicatively connected to the at least one processor; wherein, the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute any data query method provided by the embodiments of the present application.
[0108] Next, refer to Figure 10 to describe the electronic device 100 according to this embodiment of the present application.Figure 10 The displayed electronic device 100 is merely an example and should not impose any restrictions on the functions and usage scope of the embodiments of this application.
[0109] As Figure 10 shown, the electronic device 100 is presented in the form of a general electronic device. The components of the electronic device 100 may include but are not limited to: at least one of the above-mentioned processors 101, at least one of the above-mentioned memories 102, and a bus 103 connecting different system components (including the memory 102 and the processor 101).
[0110] The processor 101 is configured to read and execute instructions in the memory 102, so that the at least one processor can execute a data query method provided by the above embodiments.
[0111] The bus 103 represents one or more of several types of bus structures, including a memory bus or a memory controller, a peripheral bus, a processor, or a local bus using any bus structure in a variety of bus structures.
[0112] The memory 102 may include a readable medium in the form of volatile memory, such as a random access memory (RAM) 1021 and / or a cache memory 1022, and may further include a read-only memory (ROM) 1023.
[0113] The memory 102 may also include a program / utility 1025 having a set (at least one) of program modules 1024. Such program modules 1024 include but are not limited to: an operating system, one or more application programs, other program modules, and program data. Each or some combination of these examples may include the implementation of a network environment.
[0114] The electronic device 100 may also communicate with one or more external devices 104 (such as a keyboard, a pointing device, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 100, and / or communicate with any device (such as a router, a modem, etc.) that enables the electronic device 100 to communicate with one or more other electronic devices. Such communication may be carried out through an input / output (I / O) interface 105. And, the electronic device 100 may also communicate with one or more networks (such as a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) through a network adapter 106. As shown in the figure, the network adapter 106 communicates with other modules for the electronic device 100 through the bus 103. It should be understood that although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 100, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, etc.
[0115] In some possible embodiments, various aspects of a data query method provided in this application can also be implemented in the form of a program product, which includes program code. When the program product runs on a computer device, the program code is used to cause the computer device to execute the steps of a data query method according to various exemplary embodiments of this application described above in this specification.
[0116] In addition, this application also provides a computer-readable storage medium, and the computer storage medium stores a computer program, and the computer program is used to cause a computer to execute the method described in any one of the above embodiments.
[0117] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured article including an instruction device, and the instruction device implements the functions specified in one process Figure 1 one process or multiple processes and / or blocks Figure 1 specified in one block or multiple blocks.
[0118] These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to generate a computer-implemented process, and thus the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one process Figure 1 one process or multiple processes and / or blocks Figure 1 specified in one block or multiple blocks.
[0119] Although the preferred embodiments of this application have been described, those skilled in the art can make additional changes and modifications once they learn the basic creative concept. Therefore, the appended claims are intended to be construed to include the preferred embodiments and all changes and modifications falling within the scope of this application.
[0120] Obviously, those skilled in the art can make various changes and variations to this application without departing from the spirit and scope of this application. Thus, if these modifications and variations of this application fall within the scope of the claims of this application and their equivalent technologies, this application is also intended to include these modifications and variations.
Claims
1. A data query method, characterized in that: Applied to a storage node, the method comprises: Obtain N query statements, wherein each query statement includes at least one aggregate field; According to the function of each aggregation field, the aggregation fields corresponding to each of the N query statements are classified to obtain M aggregation field sets; wherein the types of the aggregation fields included in each aggregation field set are the same; Based on the first aggregate field set, a query is performed in the first data partition to obtain a corresponding first query result; wherein the first aggregate field set is any one of the M aggregate field sets; the first data partition is obtained by partitioning the database to be queried based on the functions of each aggregate field set; The query result of the first query statement is determined according to the query results corresponding to each of at least one aggregated field set included in the first query statement; wherein the first query statement is any one of the N query statements.
2. The method according to claim 1, characterized in that The method comprises: Obtain query description information corresponding to each first aggregation field included in the first aggregation field set; wherein the query description information indicates the size and location of data in the database to be queried; A first data partition corresponding to the first aggregated field set is determined according to the query description information.
3. The method according to claim 1, characterized in that: The determining the query result of the first query statement according to the query results corresponding to each of the at least one aggregated field set included in the first query statement includes: Obtain query results corresponding to each of at least one aggregate field set included in the first query statement; The query results are concatenated according to the order of the field contents corresponding to the query results in the first query statement to obtain the query result of the first query statement.
4. The method according to claim 1, characterized in that At least one operator is deployed on the storage node; The querying in the first data partition based on the first aggregate field set to obtain the corresponding first query result includes: In the at least one operator, according to the correspondence between the operator function and the aggregated field set representation function, determine a first operator for executing the first set of fields; The first operator is applied to perform a query in the first data partition to obtain a corresponding first query result.
5. The method according to claim 1, characterized in that Each query statement has a one-to-one corresponding query identifier; the query identifier is determined according to the offset and timestamp of the query statement in the stream processing platform.
6. The method according to any one of claims 1 to 5, characterized in that: The method further comprises: It is determined that the N query statements are structured query language SQL statements.
7. A data query device, characterized in that: The device comprises: An acquisition module, used to acquire N query statements, wherein each query statement includes at least one aggregate field; A classification module, used for classifying the aggregation fields corresponding to the N query statements according to the functions of the aggregation fields, to obtain M aggregation field sets; wherein the types of the aggregation fields included in each aggregation field set are the same; A query module, configured to query in a first data partition based on a first aggregate field set to obtain a corresponding first query result; wherein the first aggregate field set is any one of the M aggregate field sets; and the first data partition is obtained by partitioning the database to be queried based on the functions of each aggregate field set; A summary module is used to determine the query result of the first query statement according to the query results corresponding to each of at least one aggregate field set included in the first query statement; wherein the first query statement is any one of the N query statements.
8. An electronic device, characterized in that: The electronic device comprises: At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor so that the at least one processor executes a method as claimed in any one of claims 1 to 6.
9. A computer storage medium, characterized in that The computer storage medium stores a computer program, and the computer program is used to make a computer execute any one of the methods according to claims 1-6.
10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, a method as claimed in any one of claims 1 to 66 is implemented.