Data query method and device

Abstract

A data query method and a device, for use in implementing the requirement of one data table supporting a plurality of data query scenarios, and improving the data query performance of a database. For example, the data query method comprises: receiving a first request, wherein the first request is used for requesting to acquire data from a database and comprises a data query condition, the database comprises a plurality of data tables, the plurality of data tables are used for a data analysis service, and the data analysis service is used for acquiring data from the database and analyzing the acquired data; determining information of at least one data block on the basis of description information of a first data table, wherein the at least one data block is a data block satisfying the data query condition among data blocks corresponding to the first data table, the first data table is determined on the basis of the first request, and the description information is used for indicating the data blocks corresponding to the first data table; and reading the at least one data block from the database on the basis of the information of the at least one data block.

Description

Data query method and device

[0001] Cross-reference to related applications

[0002] This application claims priority to the Chinese Patent Application No. 202410855954.4, filed on June 27, 2024, and entitled "A data query method and device", the content of which is incorporated herein by reference in its entirety. TECHNICAL FIELD

[0003] The present application relates to the technical field of big data, and in particular to a data query method and device. BACKGROUND

[0004] When querying data on a big data platform, a data query condition needs to be provided, which is used to filter the required data through one or more index fields. According to the number of index fields corresponding to the data query condition, the data query scene can be divided into point query, range query and multi-dimensional query scenes. Among them, point query refers to data query according to a single index field query condition; range query refers to data query according to a small number (for example, less than or equal to 3) of index field query conditions; multi-dimensional query refers to data query according to a plurality of (for example, greater than 3) index field query conditions. At present, the time delay requirements of different data query scenes are different. For example, point query requires millisecond-level time delay, range query requires sub-second (i.e., less than 1 second) time delay, and multi-dimensional query requires second-level time delay.

[0005] In order to meet the time delay requirements of different data query scenes, different big data technologies are currently used for different data query scenes, which also means that resources need to be deployed according to multiple big data technologies for different data query scenes, resulting in large resource consumption. SUMMARY

[0006] Embodiments of the present application provide a data query method and device for reducing resource consumption.

[0007] In a first aspect, a data query method is provided, which can be performed by a second node configured to perform a data query service. The second node can query data from a database including a plurality of data tables configured for a data analysis service configured to obtain data from the database and analyze the obtained data. The second node receives a first request for obtaining data from the database, the first request including a query statement. The second node determines information of at least one data block from description information of a first data table, the at least one data block being a data block of data blocks corresponding to the first data table and satisfying a data query condition, the first data table and the data query condition being determined according to the query statement, the description information being configured to indicate the data blocks corresponding to the first data table. The second node reads the at least one data block from the database according to the information of the at least one data block.

[0008] According to the embodiment, the second node can determine information of at least one data block satisfying the data query condition from the data blocks corresponding to the first data table according to the description information of the first data table. The at least one data block is a data block satisfying the data query condition, that is, the at least one data block includes data required to be obtained by the first request, and thus reading the at least one data block can obtain the data requested by the first request. On the one hand, the plurality of data tables in the database are originally configured for the data analysis service, and the data query service is implemented based on the data tables configured for the data analysis service according to the embodiment. The data analysis service is usually configured to query data according to a large number of dimensions and analyze the queried data, and thus the data analysis service can also be referred to as multi-dimensional data analysis or multi-dimensional query service. The data query service is configured to obtain data from the database, and the data query service is usually configured to query data according to a small number of dimensions, for example, the data query service can include point query, range query or multi-dimensional query service. It can be seen that the data tables originally configured for the data analysis service can be used in the point query, range query or multi-dimensional query service according to the embodiment, and thus the data tables of the same big data query technology can be shared by multiple data query services, so that different data query services do not need to use respective big data query technologies, and thus resources do not need to be deployed according to multiple big data query technologies, but resources can be deployed for the shared big data query technology, which is beneficial to reducing resource consumption.

[0009] In a possible implementation, the second node can determine the at least one data block satisfying the data query condition according to whether the description information is stored. When the second node stores the description information, the second node can determine the information of the at least one data block according to the stored description information. In this way, the second node can determine the at least one data block satisfying the data query condition locally, and the communication overhead of requesting other nodes to determine the at least one data block satisfying the data query condition can be saved, and the data query efficiency can be improved. Alternatively, when the second node does not store the description information, the second node can send a second request to the first node and receive a first response from the first node. The first node is configured to store the description information of the data table in the database and determine the information of the at least one data block satisfying the data query condition according to the description information. The second request includes the identifier of the first data table and the data query condition, and the first response includes the information of the at least one data block. In this way, the second node does not need to store the description information, and the storage resource occupation of the second node is reduced.

[0010] In a possible implementation, the information of the at least one data block includes one or more of the following. An identifier of a table file to which the at least one data block respectively belongs, an address offset of the at least one data block in the table file to which the at least one data block respectively belongs, or a length of the at least one data block. The storage location of the at least one data block in the database can be determined by using the information, and the at least one data block can be successfully read from the database.

[0011] In a possible implementation, when the type of the first data table is a first type, the second node obtains and stores the description information from the first node. The data amount of the data corresponding to the data table of the first type is less than or equal to a first threshold value, and therefore the time required for determining the data block satisfying the data query condition from the data block corresponding to the data table of the first type is less, and the influence on other functions in the second node is also less. After the second node stores the description information of the data table of the first type, the second node can determine the data block satisfying the data query condition locally, the communication overhead of requesting other nodes to determine the data query condition can be saved, and the data query efficiency can be improved.

[0012] In a possible implementation, the second node can receive the information of a first time length from the first node, and when the first time length is less than or equal to a second threshold value, determine that the type of the first data table is the first type. The first time length is the time length consumed by the first node for determining the at least one data block, and when the first time length is less than or equal to the second threshold value, that is, the time required for determining the data block satisfying the data query condition from the data block corresponding to the data table of the first type is less, it can be determined that the data amount of the data corresponding to the first data table is less, and therefore it can be determined that the type of the first data table is the first type.

[0013] In a possible implementation, the second threshold is determined according to a first round trip time. The first round trip time is a time length between a sending time of the second request and a receiving time of the first response. The second threshold is determined according to the first round trip time, so that a size relationship between the first time length and the first round trip time can be measured. For example, when the first time length is closer to the second round trip time, it indicates that most of the time in the first round trip time is used for determining the data block satisfying the data query condition, and the data transmission time length between the second node and the first node only occupies a small part of the time. Therefore, requesting the first node to perform the determination process can save more resource consumption of the second node. Conversely, when the first time length is closer to the second round trip time, it indicates that most of the time in the first round trip time is used for data transmission between the second node and the first node. Therefore, requesting the first node to perform the determination process can save less resource consumption of the second node. Therefore, by setting the second threshold according to the first round trip time, when the second node determines whether the first data table is the first type of data table, the amount of resource consumption saved by the second node is essentially considered as a factor, which is beneficial to balance the resource consumption and communication overhead of the second node.

[0014] In a possible implementation, the description information includes one or more of the following. Metadata, the metadata is used to indicate a first index field included in the first data table, wherein the first data table includes a plurality of partitions, and the plurality of partitions are determined according to values of the first index field. First indication information, used to indicate table files corresponding to the plurality of partitions respectively, wherein a table file corresponding to a first partition in the plurality of partitions includes data corresponding to the first partition, and one of the table files corresponding to the plurality of partitions includes one or more data blocks, and one data block includes values of one or more index fields. Second indication information, used to indicate a value range of an index field of a data block included in one of the table files corresponding to the plurality of partitions. Through the information, the second node can accurately determine at least one data block satisfying the data query condition from the data blocks corresponding to the first data table.

[0015] In a possible implementation, the data query condition comprises values of at least one index field. Then, after determining that the at least one index field comprises the first index field according to the metadata, the second node can determine, according to the first indication information, a second partition corresponding to the first data table, and finally determine, according to the second indication information, information of at least one data block, the at least one data block comprising values of one or more index fields in the at least one index field. Through this implementation, the second node can exclude partitions that do not meet the data query condition according to the first index field used to divide the partitions, thereby reducing the number of partitions that need to be queried and improving the data query efficiency.

[0016] In a possible implementation, the data query condition comprises values of at least one index field. Then, after determining that the at least one index field comprises the first index field according to the metadata, the second node can determine, according to the first indication information, a second partition corresponding to the first data table, and finally determine, according to the second indication information, information of at least one data block, the at least one data block comprising values of one or more index fields in the at least one index field. Through this implementation, the second node can exclude partitions that do not meet the data query condition according to the first index field used to divide the partitions, thereby reducing the number of partitions that need to be queried and improving the data query efficiency.

[0017] In a possible implementation, the data query condition comprises values of at least one index field. Then, after determining that the at least one index field comprises the first index field according to the metadata, the second node can determine, according to the first indication information, a second partition corresponding to the first data table, and finally determine, according to the second indication information, information of at least one data block, the at least one data block comprising values of one or more index fields in the at least one index field. Through this implementation, the second node can exclude partitions that do not meet the data query condition according to the first index field used to divide the partitions, thereby reducing the number of partitions that need to be queried and improving the data query efficiency.

[0018] Through the embodiment, the first node can determine information of at least one data block meeting the data query condition from the data block corresponding to the first data table according to the description information of the first data table. On the one hand, because the plurality of data tables in the database are originally used for data analysis services, the data query service is implemented based on the data table used for the data analysis service, and it can be seen that the data table originally used for the data analysis service can be used for the point query, the range query, or the multi-dimensional query, and a plurality of data query scenarios can share one data table of the database technology, so it is not necessary to deploy storage resources for storing data tables corresponding to a plurality of database technologies for different data query scenarios as in the related technology, thereby the embodiment reduces resource consumption required for storing the data table. On the other hand, the description information of the first data table can indicate the data block corresponding to the first data table, for example, indicate data included in the data block, so that the data block meeting the data query condition in the first data table can be determined in combination with the description information and the data query condition, the complexity of the data query process is low, and the data query performance is improved.

[0019] In a possible implementation, the first node is configured to store description information of data tables in the database. The first node can obtain the description information in a plurality of ways. In one way, the first node reads table files corresponding to a plurality of data tables in the database in response to a preloading instruction, and determines description information of the plurality of data tables according to the table files corresponding to the plurality of data tables, so that the first node can pre-load the description information of all the data tables. In another way, the first node reads a table file corresponding to a second data table according to a third request from a third node, and determines description information of the second data table according to the table file corresponding to the second data table, the third request being used to instruct the third node to update the second data table, and the third node being configured to maintain the database, so that the first node can update the description information in time when data in the database is updated, improve the accuracy of the description information, and reduce the probability of data query failure.

[0020] In a possible implementation, the information of the at least one data block includes one or more of the following: an identifier of a table file to which the at least one data block respectively belongs; an address offset of the at least one data block in the table file to which the at least one data block respectively belongs; or a length of the at least one data block.

[0021] In a possible implementation, the first node sends information of a first time length to the second node, the first time length being a time length consumed by the first node for determining the at least one data block.

[0022] In a possible implementation, the description information includes one or more of the following. Metadata, the metadata is used to indicate a first index field included in the first data table, wherein the first data table includes a plurality of partitions, and the plurality of partitions are determined according to values of the first index field. First indication information, used to indicate table files corresponding to the plurality of partitions respectively, wherein a table file corresponding to a first partition in the plurality of partitions includes data corresponding to the first partition, one of the table files corresponding to the plurality of partitions includes one or more data blocks, and one of the data blocks includes values of one or more index fields. Second indication information, used to indicate a value range of an index field of a data block included in one of the table files corresponding to the plurality of partitions.

[0023] In a possible implementation, the data query condition includes values of at least one index field. After determining, according to the metadata, that the at least one index field includes the first index field, the first node can determine a second partition from the plurality of partitions, the second partition is a partition in which a value of the first index field in the at least one index field is located, determine, according to the first indication information, a table file corresponding to the second partition, and finally determine information of at least one data block, the at least one data block including values of one or more index fields in the at least one index field.

[0024] In a possible implementation, the data query condition includes values of at least one index field. After determining, according to the metadata, that the at least one index field does not include the first index field, the first node can determine, according to the first indication information, a table file corresponding to the first data table, and finally determine information of at least one data block, the at least one data block including values of one or more index fields in the at least one index field.

[0025] In a third aspect, an embodiment of the present application further provides a data query apparatus, which has functions of the method embodiments in the first aspect, and the advantages can be referred to the description of the first aspect, which will not be repeated here. The functions can be realized by hardware, or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the functions.

[0026] In a possible design, the data query apparatus includes a receiving module, a determining module and a data reading module. These modules can perform the corresponding functions in the method embodiments of the first aspect, and the specific implementation can be referred to the detailed description in the method embodiments, which will not be repeated here.

[0027] For example, the receiving module is configured to receive a first request, the first request being configured to request to obtain data in a database, the first request comprising a query statement; wherein the database comprises a plurality of data tables, the plurality of data tables being configured to be used for a data analysis service, the data analysis service being configured to obtain data from the database and analyze the obtained data. The determining module is configured to determine information of at least one data block according to description information of a first data table, the at least one data block being a data block that meets a data query condition among data blocks corresponding to the first data table, the first data table and the data query condition being determined according to the query statement, and the description information being configured to indicate the data blocks corresponding to the first data table. The data reading module is configured to read the at least one data block from the database according to the information of the at least one data block.

[0028] Through the above apparatus, the information of at least one data block that meets the data query condition can be determined from the data blocks corresponding to the first data table according to the description information of the first data table. The at least one data block is a data block that meets the data query condition, that is, the at least one data block comprises data required to be obtained by the first request, and thus reading the at least one data block can obtain the data requested by the first request. On the one hand, the plurality of data tables in the database are originally configured to be used for the data analysis service, and the embodiment of the present application implements a data query service on the basis of the data tables used for the data analysis service. The data analysis service is usually configured to query data according to a large number of dimensions and analyze the queried data, and thus the data analysis service can also be referred to as multi-dimensional data analysis or multi-dimensional query service. The data query service is configured to obtain data from the database, and the data query service is usually configured to query data according to a small number of dimensions, for example, the data query service can be point query, range query or multi-dimensional query service, and the like. It can be seen that the embodiment of the present application can enable the data tables originally used for the data analysis service to be used in the point query, range query or multi-dimensional query scenarios, and thus the data tables of a plurality of data query scenarios can share one database technology, and thus it is not necessary to deploy storage resources for storing data tables corresponding to a plurality of database technologies for different data query scenarios as in the related art, and thus the embodiment of the present application reduces the resource consumption required for storing the data tables. On the other hand, the description information of the first data table can indicate the data blocks corresponding to the first data table, for example, indicate data included in the data blocks, and thus the data blocks that meet the data query condition in the first data table can be determined in combination with the description information and the data query condition, the complexity of the data query process is low, and the data query performance is improved.

[0029] In a fourth aspect, the embodiments of the present application further provide a data query apparatus, which has the functions of the method embodiments of the second aspect, and the beneficial effects can be referred to the description of the second aspect, which will not be repeated here. The functions can be implemented by hardware, or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions.

[0030] In a possible design, the data query apparatus includes a receiving module, a determining module and a sending module. The modules can perform the corresponding functions in the method embodiments of the second aspect, and the specific description can be referred to the detailed description in the method embodiments, which will not be repeated here.

[0031] For example, the receiving module is configured to receive a second request from a second node, the second request including an identifier of a first data table and a data query condition; wherein the first data table is included in a database, and the databases in the database system include a plurality of data tables, the plurality of data tables being used for data analysis services, the data analysis services being used for obtaining data from the database and analyzing the obtained data. The determining module is configured to determine at least one data block that meets the data query condition from data blocks corresponding to the first data table according to stored description information of the first data table, the description information being used to indicate the data blocks corresponding to the first data table. The sending module is configured to send a first response to the second node, the first response including first indication information, the first indication information being used to indicate the at least one data block.

[0032] Through the above apparatus, the data query apparatus can determine the information of at least one data block that meets the data query condition from the data blocks corresponding to the first data table according to the description information of the first data table. On the one hand, because the plurality of data tables in the database are originally used for data analysis services, the embodiments of the present application implement the data query services on the basis of the data tables used for data analysis services, and it can be seen that the embodiments of the present application can make the data tables originally used for data analysis services be used in the scenarios of point query, range query or multi-dimensional query, and implement that a plurality of data query scenarios can share a data table of a database technology, so that the embodiments of the present application do not need to deploy storage resources for storing data tables corresponding to a plurality of database technologies for different data query scenarios as in the related technologies, thereby reducing the resource consumption required for storing the data tables. On the other hand, the description information of the first data table can indicate the data blocks corresponding to the first data table, for example, indicate the data included in the data blocks, so that the data blocks meeting the data query condition in the first data table can be determined in combination with the description information and the data query condition, the complexity of the data query process is low, and the data query performance is improved.

[0033] In a fifth aspect, the present application provides a big data processing system, comprising the data query device of the third aspect and the fourth aspect. Optionally, the big data processing system further comprises a data analysis node configured to perform data analysis services and maintain the database.

[0034] In a sixth aspect, the present application provides a computer device, comprising a processor and a memory, and optionally comprising a communication interface. The processor executes program instructions in the memory to perform the method of the first aspect or any possible implementation of the first aspect. Alternatively, the processor executes program instructions in the memory to perform the method of the second aspect or any possible implementation of the second aspect. The memory is coupled to the processor and stores computer program instructions and data necessary for the data query process. The communication interface is configured to communicate with other devices, such as receiving the first request, the second request, the first response, or sending the second request, the first response, etc.

[0035] In a seventh aspect, the present application provides a computer device, comprising an acceleration device and a processor, and optionally comprising a memory and a communication interface. The processor cooperates with the acceleration device to perform the method of the first aspect or any possible implementation of the first aspect. Alternatively, the acceleration device performs the method of the first aspect or any possible implementation of the first aspect. The memory is coupled to the processor and stores some computer program instructions and data necessary for the data query process. The communication interface is configured to communicate with other devices, such as receiving the first request, the second request, the first response, etc. Alternatively, the processor cooperates with the acceleration device to perform the method of the second aspect or any possible implementation of the second aspect. Alternatively, the acceleration device performs the method of the second aspect or any possible implementation of the second aspect. The communication interface is configured to communicate with other devices, such as sending the second request, the first response, etc.

[0036] In an eighth aspect, the present application provides a computer system, comprising at least one computer device. Each computer device comprises a memory and a processor. The processor of the at least one computer device is configured to access the code in the memory to perform the method of the first aspect or any possible implementation of the first aspect. Alternatively, the processor of the at least one computer device is configured to access the code in the memory to perform the method of the second aspect or any possible implementation of the second aspect.

[0037] In a ninth aspect, the present application provides a computer readable storage medium, which, when executed by a computer device, causes the computer device to perform the method provided in the first aspect or any possible implementation of the first aspect, or perform the method provided in the second aspect or any possible implementation of the second aspect. The storage medium stores computer program instructions. The storage medium includes, but is not limited to, a volatile memory, such as a random access memory, a non-volatile memory, such as a flash memory, a hard disk drive (HDD), and a solid state drive (SSD).

[0038] In a tenth aspect, the present application provides a computer program product, which includes computer program instructions. When the computer program instructions are executed by a computer device, the computer device performs the method provided in the first aspect or any possible implementation of the first aspect, or performs the method provided in the second aspect or any possible implementation of the second aspect. The computer program product can be a software installation package. When the method provided in the first aspect or any possible implementation of the first aspect needs to be used, or when the method provided in the second aspect or any possible implementation of the second aspect needs to be used, the computer program product can be downloaded and executed on the computer device.

[0039] In an eleventh aspect, the present application also provides a computer chip or a chip system, which is connected to a memory. The chip or the chip system is configured to read and execute computer program instructions stored in the memory, and perform the method provided in the first aspect and each possible implementation of the first aspect, or perform the method provided in the second aspect and each possible implementation of the second aspect.

[0040] The advantages of the third aspect to the eleventh aspect and the implementation manners thereof can refer to the description of the advantages of the first aspect or the second aspect and any implementation manner thereof. BRIEF DESCRIPTION OF DRAWINGS

[0041] FIG. 1 is a schematic diagram of an application scenario of a scheme provided by an embodiment of the present application;

[0042] FIG. 2A and FIG. 2B are schematic diagrams of a data query process in the related art;

[0043] FIG. 3 is a schematic diagram of a system architecture of a database system provided by an embodiment of the present application;

[0044] FIG. 4 is a schematic diagram of a flow of a data query method provided by an embodiment of the present application;

[0045] FIG. 5 is another schematic diagram of a flow of a data query method provided by an embodiment of the present application;

[0046] FIG. 6 is a flow diagram illustrating a process of obtaining description information by a first node according to an embodiment of the present application;

[0047] FIG. 7 is another flow diagram illustrating a data query method according to an embodiment of the present application;

[0048] FIG. 8 is a flow diagram illustrating a process of storing description information of a first data table by a second node according to an embodiment of the present application;

[0049] FIG. 9 is a flow diagram illustrating a process of determining information of at least one data block according to an embodiment of the present application;

[0050] FIG. 10 is a diagram illustrating a process of determining a data block according to an embodiment of the present application;

[0051] FIG. 11 is a diagram illustrating a structure of a data query apparatus according to an embodiment of the present application;

[0052] FIG. 12 is a diagram illustrating another structure of a data query apparatus according to an embodiment of the present application;

[0053] FIG. 13 is another diagram illustrating a structure of a data query apparatus according to an embodiment of the present application. DETAILED DESCRIPTION

[0054] In order to make the objects, technical solutions and advantages of the embodiments of the present application clearer, the embodiments of the present application will be further described in detail below with reference to the accompanying drawings.

[0055] In the embodiments of the present application, the number of a noun, unless otherwise specified, represents "a singular noun or a plural noun", i.e. "one or more". "At least one" means one or more, and "a plurality of" means two or more. "And / or" describes the association relationship of the associated objects, which means that there can be three kinds of relationships, for example, A and / or B can represent the following cases: A exists alone, A and B exist together, B exists alone, where A and B can be singular or plural. The character " / " generally represents an "or" relationship between the associated objects. For example, A / B means A or B. "At least one of the following" or the like means any combination of these items, including any combination of single item or multiple items. For example, at least one of a, b, or c means a, b, c, a and b, a and c, b and c, or a and b and c, where a, b, and c can be single or multiple.

[0056] The ordinal numbers such as "first", "second" and the like mentioned in the embodiments of the present application are used to distinguish a plurality of objects, and are not used to limit the size, content, order, time sequence, priority or importance of the plurality of objects. For example, the first data block and the third data block can be the same data block, or can be different data blocks. In addition, the numbering of steps in each embodiment introduced in the present application is only to distinguish different steps, and is not used to limit the order of the steps.

[0057] Before introducing the specific schemes provided by the embodiments of the present application, some terms or concepts in the present application are explained and described, so as to facilitate the understanding of the skilled in the art.

[0058] (1), database, data table, table file, data block

[0059] The database is a warehouse for organizing, storing and managing data according to a certain data structure. The database is stored according to a specific format, and the user can perform operations such as adding, deleting, modifying and querying on the stored data. In a certain sense, the database is a file collection, and the database can also be understood as a file system in essence. In an example, the database refers to a data warehouse in a big data platform, which is used to store data involved in the big data platform.

[0060] The data table is an object used to store data in the database. The data table can also be understood as a structured data set, and is also a structured data organization form, which usually stores data in the form of rows and columns. Each row represents a record in the table, and each column represents the value of an index field or attribute in the record. For example, a user information table can include index fields such as user identity (ID), name, age or address, and each row is the specific information of a user.

[0061] Table files are physical files in a database that store data in data tables. They can also be referred to as data files. In a database, files generally refer to physical files in the file system that store various data structures and contents of the database, such as table data, indexes, logs, etc. Each database is usually composed of multiple files, each responsible for storing data of a specific type or specific data table, such as data files (storing table data), log files (recording log information of database operations), index files (storing information of database indexes), etc. In a database, each data table is usually stored as a file or a set of related files that contain all the actual data of the data table. In a database system, data is read and written through table files to achieve persistent storage and efficient access of data. The physical storage format and specific implementation of table files depend on the specific business scenario, and different business scenarios may have different file organization structures and storage strategies. For example, in a big data technology stack based on HBase (a key-value query and storage engine), the data table is an HBase format table, and the table file is an HFile file. The combination of HBase and HFile formats can achieve fast point and range queries. For another example, in a big data technology stack based on Spark (a multi-dimensional query and storage engine), the data table is a Parquet format table, and the table file is a Parquet file. The combination of Spark and Parquet formats can achieve efficient multi-dimensional data analysis.

[0062] Data blocks are a section of data in a table file. For example, a data block can be a RowGroup in a Parquet file.

[0063] (2) Query statement, index field

[0064] A query statement is a command or instruction used to retrieve data from a database. It specifies which data needs to be returned and how to filter, sort, and combine the data. Common query commands include the SELECT command for selecting data to be retrieved from the database. In addition, query commands also include clauses of the command, such as FROM and WHERE, which are two key clauses used to build query statements. FROM is used to specify the identification of the table or view to be queried, and WHERE is used to filter data rows that meet specific conditions.

[0065] Index fields, also known as fields or attributes of data in a data table. Index fields are one or more columns in a data table that are often indexed to speed up query operations. For example, in the aforementioned example, a user information table may include index fields such as user ID, name, age, or address. Index structure can be understood as a kind of data structure, similar to the table of contents of a book, which can quickly locate specific content. Common index data structures include B-tree, Hash, full-text index, etc., each of which is suitable for different query patterns and data characteristics.

[0066] (3) Multidimensional data analysis

[0067] Multidimensional data analysis, also known as on-line analysis processing (OLAP), is a technique and method for exploring and analyzing complex data. Its core concept is to analyze data from multiple angles to deeply understand the relationship, trend and pattern of data. Multidimensional data model is the basis of multidimensional data analysis, which consists of multiple dimensions and measures. Dimensions are descriptive attributes of data, and an index field can be a dimension. Measures are numerical data that need to be analyzed. Dimensions are the main basis for data analysis, usually descriptive attributes or categories such as time, geographic location, product category, etc. Different combinations of dimensions form a multidimensional data space. Measures are numerical data analyzed in multidimensional data analysis, such as sales, quantity, profit, etc. In multidimensional data analysis, data is usually organized in the form of a cube, also known as a data cube or OLAP cube.

[0068] (4) Data skipping

[0069] The process of data query includes query plan generation and query execution. The query plan refers to detailed steps and strategies generated when executing a SQL query, used to describe how to retrieve the required data, usually including access path, connection method, filter condition application sequence, and sorting method, etc. Data skipping is an optimization technique for query plan. When performing data skipping, based on the value of the index field, files that do not meet the data query condition can be filtered out. This optimization process is also called "pruning", which can be divided into partition pruning and data pruning. Partition pruning is to cut off partitions in the data table that do not meet the query condition, and data pruning is to cut off files in the partition that do not meet the query condition. In the embodiment of the present application, data skipping can directly filter out data blocks that do not meet the data query condition, which is more fine-grained than data skipping in the related art, and can effectively improve the speed of data query.

[0070] Please refer to FIG. 1, which shows a scene diagram applicable to the method of the present application. In this scenario, it can include a server 101 and a terminal device 102. In one example, referring to FIG. 1, the server 101 can be a server of a big data platform, and the big data platform can include a computing cluster composed of one or more computer devices 1011. Users can send corresponding service requests to the server 101 through the terminal device 102 to use the services provided by the big data platform, such as data query service requests or data analysis service requests, etc.

[0071] The big data platform can also be referred to as a big data processing system or a big data system, etc., which can effectively process, store and analyze massive data. The core of the big data platform is the big data technology stack, which refers to a collection of a series of technologies and tools used to process and analyze large-scale data. These tools can effectively collect, store, process and analyze large amounts of data to extract valuable information from them. Referring to FIG. 1, some common key components in the big data technology stack are as follows:

[0072] (1) Data collection and preprocessing service: This stage involves collecting data from various sources and performing necessary cleaning and conversion to ensure data quality and applicability.

[0073] (2) Data storage service: The big data platform needs to be able to efficiently store and manage massive data and support the storage of multi-source heterogeneous data. In one example, the data storage of the big data platform can be realized through Hadoop distributed file system (HDFS), etc.

[0074] (3) Data processing and computing services: Big data platforms can utilize various computing frameworks to perform in-depth analysis on data and support complex query and data processing tasks. In one example, the data processing and computing of the big data platform can be implemented through Apache Spark or Hadoop MapReduce, etc., where Apache Spark is a fast general-purpose big data processing engine that supports batch processing, interactive query and stream processing, or Hadoop MapReduce is a parallel distributed computing model for large-scale data sets.

[0075] (4) Data query and analysis services: The purpose of this service is to extract useful information and analyze from massive data. Common big data query and analysis tools include Apache Spark and Apache HBase, etc. Spark is a fast general-purpose big data processing engine that supports in-memory computing and provides a rich application program interface (API) that can be used for real-time data processing and complex analysis. HBase is a distributed, column-oriented NoSQL database suitable for storing large amounts of structured data and supporting real-time query and analysis.

[0076] (5) Data mining and application services: Through machine learning, deep learning and other technologies, potential correlations, patterns and trends are mined from data to support business decision-making.

[0077] In addition, big data platforms also need to consider security, scalability, high performance, etc. to ensure the stable operation of the system and the reliable protection of data. The choice of big data technology stack depends on specific business needs, data size, real-time requirements and scalability requirements. Enterprises will usually combine and integrate these technologies and tools according to their own circumstances to build solutions that meet their big data processing needs. With the development of technology, big data systems are becoming more and more full-process, visual and intelligent, providing strong support for enterprise-level applications.

[0078] The terminal device 102 can be a computer device used by an individual, such as a smartphone, a notebook computer, or a desktop computer, etc., or a server on the user side, and no limitation is made thereto. The various computer devices 1011 included in the server 101 can be servers, such as local servers, cloud servers, or servers in a data center.

[0079] In one example, the server 101 can be a server of a database system, and the server 101 can provide database-related business services to the terminal device 102.

[0080] In some examples, different big data technologies are currently adopted for different data query scenarios, which also means that when the latency requirements of multiple data query scenarios need to be met at the same time, device resources need to be deployed according to the requirements of multiple big data technologies, and resource consumption is large. Among them, different big data technologies are based on their own private file formats and provide different query engine capabilities. For example, the following examples illustrate two different database technology solutions.

[0081] The first one: a solution implemented by using key value (K / V) storage and retrieval technology

[0082] This solution uses key value pairs to store data, and each key corresponds to a unique value, which is very suitable for using primary keys as query conditions for data queries. However, when non-primary keys are used as query conditions, full table scanning is required, and the data query latency is very high. The value of the primary key can uniquely identify each row in the data table. Such a column or multiple columns of data is called the primary key of the data table.

[0083] Please refer to FIG. 2A, which is a flowchart of a data query process implemented by using K / V storage and retrieval technology. In this example, the combination of HBase engine and HFile data format is used to implement data query, HBase engine is a common K / V query engine, and HFile data format is its private data format. The combination of the two can realize fast Key point query or range query. The implementation process of this solution includes the following stages:

[0084] S10: HBase table creation stage. For example, when there is user data to be stored and there is no corresponding data table, the HBase engine creates an HBase table. In each Hbase table, the keys are globally sorted, and one Hbase table can be horizontally divided into multiple small tables, each of which corresponds to multiple HFile files.

[0085] S11: Data loading stage. In the data loading stage, the HBase engine in the Hbase server needs to store the data as HFile files in the HFile data format to facilitate subsequent data queries.

[0086] In order to improve the loading performance, referring to Fig. 2A, the HBase engine generates HFile files in batches with the help of the Spark engine. When the user exists data storage or update, that is, the loading request shown in Fig. 2A, the Spark engine generates HFile files, and sends the generated HFile files to the HBase engine, the HBase engine sends the HFile files to the HDFS for storage, and returns the corresponding response result to the client after the storage is completed. The data in each HFile file is sorted by key, and is divided into multiple blocks (for example, the size of each block is 64K), and the HFile file records the index of the block, and each block stores data in the K / V data format, that is, a key must be stored for each value.

[0087] S12: data query phase. When data needs to be queried, the HBase engine acquires data from the HDFS according to the query request of the user. In the implementation scheme, the correspondence between each small table and the server storing it is saved through a metadata (Meta) table.

[0088] Referring to Fig. 2A, when the Meta table is not cached by the client for the first time, the Meta table needs to be requested by the HBase engine to acquire and cache. When retrieving by key, the client first accesses the Meta table to determine the small table where the key is located, then acquires the Hbase server where the small table is located, and then sends a point query / range query request to the Hbase server to request the Hbase server to acquire the required data. The Hbase engine in the Hbase server reads and parses the HFile file, constructs a B-Tree index structure by using the block index, and then locates the block where the required data is located according to the B-Tree index structure. Finally, the data in the block is scanned, and the query result including the K / V in the data block that meets the condition is returned to the client.

[0089] In the database using the K / V storage technology, the key query (that is, the query condition is the primary key) can meet the delay requirement of the point query or the range query, so the scheme can be applied to the point query or the range query scene. However, the multi-dimensional query often involves non-key query (that is, the non-primary key is used as the query condition), and the non-key query needs to scan the data table in the entire database, so the delay is high, and the delay requirement of the multi-dimensional query cannot be met. In addition, when the relational data table data is stored by using the K / V data structure, the storage space is linearly expanded, and more storage resources are occupied.

[0090] The second kind: a scheme implemented by using on-line analysis processing (OLAP) technology

[0091] OLAP is a data processing technique for analyzing and querying large-scale datasets. OLAP processes analysis through a multidimensional data model, which is composed of multiple dimensions and measures. Data formats suitable for OLAP techniques include, for example, the Parquet data format, which is a columnar storage format. A Spark engine is a multidimensional query engine, and the combination of the Spark engine and a data table in the Parquet data format can implement an efficient OLAP process and also implement point or range queries. Please refer to FIG. 2B, which is a flowchart of a process of implementing point or range queries by combining a Spark engine and a Parquet data format. The flowchart includes the following stages:

[0092] S20: Parquet data format data table (referred to as a Parquet table) creation stage. In the table creation stage, the Spark engine creates a Parquet table, and in order to accelerate point or range queries, the Spark engine divides each data table into partitions / buckets, i.e., creates a partitioned / bucketed Parquet table. The table file corresponding to the Parquet table is referred to as a Parquet file, and one partition / bucket can correspond to multiple Parquet files.

[0093] S21: Data loading stage. In the data loading stage, when the user exists data storage or update, i.e., the loading request shown in FIG. 2B, the Spark engine performs a data loading process, writes data according to the key (such as a high-frequency query key) to the corresponding partition and / or bucket, sorts the data according to the key in each bucket, finally generates the corresponding Parquet file, and stores the Parquet file in the HDFS.

[0094] S22: Data query stage. The Spark adopts a distributed cluster architecture, including a Driver process and an Executor process, which can be deployed in different device nodes. In the data query process, the members in the architecture need to cooperate to complete the entire data query process. The data query process includes the following steps:

[0095] S221: The driver process generates a query plan. In generating the query plan, the driver process reads the index file from the HDFS, determines the Parquet files satisfying the query condition according to the index file, and thus generates a query plan including a list of Parquet files required for the query, which is equivalent to a process of rough selection of Parquet files satisfying the query condition. The query plan is then divided into multiple subtasks, each of which processes part of the Parquet files in the list, and the multiple subtasks are assigned to multiple executor processes for execution.

[0096] S222: The executor processes execute their own subtasks respectively to obtain data rows satisfying the query condition. Referring to FIG. 2B, in executing a subtask, the executor process queries the Parquet files in the subtask assigned to the executor process. Since the driver process only provides a list of Parquet files but does not provide any index information, the executor process further reads the footer of each Parquet file in the list from the HDFS to obtain metadata, and then obtains data blocks satisfying the query condition according to the metadata and the query condition. Subsequently, the executor process reads the data blocks and filters out data rows satisfying the query condition from the read data blocks.

[0097] Finally, the driver process collects the data rows obtained by the executor processes executing the subtasks and returns the data rows to the client.

[0098] Since the design of the Spark engine is originally intended for multi-dimensional data analysis, the Spark engine provides a large distributed task coordination framework and a complex query process to meet the computing requirements of the multi-dimensional data analysis process, but this results in a high latency for the Key point query / range query, which is usually in the order of seconds, and does not meet the latency requirements of the point query or range query.

[0099] According to the above two schemes, it can be seen that at present, different query engine capabilities are provided based on various big data technology private data formats. For example, the first scheme described above cannot identify the Parquet data format and does not support custom data sources, and cannot meet the demand for multidimensional query. The data query process of the second scheme described above is too cumbersome, and the query efficiency is low when point query or range query is performed. Therefore, if it is desired to meet the latency requirements of point query or range query and meet the latency requirements of multidimensional query, it is necessary to deploy device resources according to the needs of various big data technologies, for example, the database needs to store two different data format files at the same time, which obviously occupies too much device storage resources and consumes too much resources.

[0100] Based on this, the embodiment of the present application provides a data query method. When a first request is received, the information of at least one data block meeting the data query condition can be determined from the data block corresponding to the first data table according to the description information of the first data table. The at least one data block is a data block meeting the data query condition, that is, the at least one data block includes the data required to be obtained by the first request, so that reading the at least one data block can obtain the data requested by the first request. On the one hand, the data table originally used for data analysis business can be used for point query, range query or multidimensional query scene, and the data table of a variety of data query scenes can be shared by one database technology, so it is not necessary to deploy storage resources for storing data tables corresponding to various database technologies for different data query scenes as in related technologies, thereby reducing the resource consumption required for data table storage. On the other hand, the description information of the first data table can indicate the data block corresponding to the first data table, for example, the data included in the data block, so that the data block meeting the data query condition in the first data table can be determined in combination with the description information and the data query condition, the complexity of the data query process is low, and the data query performance is improved. For example, compared with the second database technology described above, the data query performance of point query or range query and other data query businesses can be effectively improved based on the data table of the data analysis business.

[0101] The method provided by the embodiment of the present application can be applied to a big data platform or a database system. Here, taking the big data platform as an example, Fig. 3 is a system architecture schematic diagram of the big data platform provided by the embodiment of the present application. In the system architecture, it includes a data query business node 301, a data analysis business node 302, a projection service node 303 and a database 304.

[0102] The data query service node 301 is configured to process data query service. The data query service is configured to query the required data from the database 304, which is stored in the database 304. As shown in FIG. 3, a client can send a data query request to the data query service node 301, the data query service node 301 performs a data query process to obtain corresponding data, and returns a response to the client, in which only the data queried from the database 304 is included. For example, the data query service can include point query or range query service.

[0103] In some examples, the data query service node 301 includes a data query engine, which can be referred to as a K / V Server. The K / V Server is a lightweight query engine provided by the embodiments of the present application to process data query service (such as point query / range query).

[0104] The data analysis service node 302 is configured to process data analysis service. The data analysis service is configured to perform data analysis according to the data in the database 304, so that when the data analysis service is executed, the corresponding data in the database 304 is also required to be obtained first, and then the data analysis is performed according to the obtained data, and finally the response returned to the client includes the result of the data analysis. For example, the data analysis service can include multi-dimensional data analysis (or OLAP) service.

[0105] The projection service node 303 is configured to perform data skipping. The projection service node 303 stores description information of each data table in the database, which is used to indicate the data in the data block corresponding to each data table in the database. The description information can also be referred to as index information. The projection service node 303 can pre-load and store the description information of the data table through a pre-loading process, and incrementally update the description information when the data in the data table is changed. As shown in FIG. 3, the data query service node 301 can include two processes of generating a query plan and executing the query plan after analyzing the received data query request. When generating the query plan, the projection service node 303 can be requested to perform a data skipping process. When performing the data skipping, the projection service node 303 can determine the data blocks satisfying the data query condition from the data blocks corresponding to the data table to be queried according to the description information of the data table to be queried stored in the projection service node 303, and return the information of the data blocks to the data query service node 301. The data query service node 301 accurately locates and reads the data blocks in the database 304 through the information of the data blocks, and analyzes each record satisfying the data query condition in the data blocks to the client. In this way, the number of data blocks required to be read and queried by the data query service node 301 when executing the query plan can be reduced, so as to improve the data query efficiency.

[0106] The database 304 is used to store data and metadata, and is a data warehouse of the big data platform. The data of the database 304 can be stored by one or more storage nodes. The database 304 stores table files corresponding to various data tables, and therefore the database can also be understood as a file system in nature. In some examples, the database 304 can adopt a distributed storage architecture, that is, the database 304 can be a distributed file system. The database 304 can include one or more data tables, which can be used to implement the basis of data query services and data analysis services. For example, the data tables in the database 304 can be used in point query, range query and multi-dimensional query scenarios, and therefore, for a piece of data in the database, services of multiple different query scenarios can be supported, and therefore, unlike the related art, if different query scenarios need to be compatible with the latency requirements, two different data formats of files need to be stored at the same time, and therefore, the scheme of the embodiments of the present application can effectively reduce the consumption of data storage resources. The embodiments of the present application do not limit the specific type of the database 304. The database 304 can store data or metadata under the indication of the data query service node 301, the data analysis service node 302 or other nodes, and can also transmit data to the data query service node 301 or the data analysis service node 302 under the indication of the data query service node 301 or the data analysis service node 302. For example, when the data query service node 301 queries a data block, the data query service node 301 can obtain the data block from the database 304 according to the address of the data block, and the database 304 will transmit the data block to the data query service node 301 in response. The database 304 can be a computer device or a cluster of computer devices with storage function. The database 304 can include components with storage function in the computer device, such as a hard disk, a magnetic disk and the like. Any device or apparatus with data storage function is suitable for the embodiments of the present application.

[0107] As shown in FIG. 3, the data table can be arranged in a partitioned and bucketed manner for storage in the database 304, a field used for partitioning can be referred to as a partition field, and the value of the partition field can be a possible value of an index field in the data table, such as a value of a time field, and a time period can be divided into a partition; or it can also be a value converted from a possible value of an index field in the data table, for example, the data table includes an index field of a user's mobile phone number, and the value of the partition field can be the last digit of the mobile phone number, and each digit can correspond to a partition. In one or more table files corresponding to a data table, each table file can sort data according to a sorting field, and each data block included in the table file can also sort data according to the sorting field, record Min-Max statistical information for all index fields, and create other types of indexes for other fields in addition to the sorting field. The sorting field refers to an index field used to sort data in the data table, for example, it can be a high-frequency index field in the data table, and the high-frequency index field refers to an index field with the highest query frequency, and the high-frequency index field can also be referred to as a high-frequency query key. Other fields are, for example, a secondary high-frequency index field, and the secondary high-frequency index field refers to an index field with a secondary highest query frequency. The Min-Max statistical information refers to the minimum value and the maximum value in the value of an index field.

[0108] In some examples, the number of each node in the system architecture can be one or more. In the case where the number of each node is multiple, the multiple nodes can adopt a distributed architecture. For example, the multiple data query service nodes 301 can adopt a distributed architecture, and then the data query request sent by the client can be distributed to appropriate data query service nodes 301 according to geographical location, load balancing, and the like, to improve the response speed of the data query request. For another example, the multiple projection service nodes 303 can adopt a distributed architecture, to implement distributed caching of description information, for example, the multiple projection service nodes 303 can implement distributed caching of description information based on a consistent hashing algorithm, or other algorithms can also be used to ensure data consistency between the projection service nodes 303.

[0109] In some examples, embodiments of the present application do not limit the specific form of the data query service node 301, the data analysis service node 302, and the projection service node 303. These nodes can be hardware devices, for example, a single computer device or a cluster including multiple computer devices. Alternatively, these nodes can be virtual machines. Alternatively, these nodes can be certain hardware components in a computing device, such as a processor (e.g., a central processing unit (CPU) or a data processing unit (DPU)) in the computing device, an offload card, an acceleration card, etc. Alternatively, these nodes can also be software devices, for example, software for managing files, such as query service software, data analysis service software, data skipping software, etc. The data query service node 301, the data analysis service node 302, and the projection service node 303 can be deployed in different computer devices or in the same computer device, and embodiments of the present application do not limit this. For example, the data query service node 301 and the data analysis service node 302 can be deployed in the same computer device.

[0110] Embodiments of the present application do not limit the deployment manner of the storage device 200 and the query device 100. The storage device 200 and the query device 100 can be deployed in the same computer device; for example, the query device 100 can be a processor (e.g., a CPU or a DPU) in a computing device, and the storage device 200 can be a hard disk in the computing device. The storage device 200 and the query device 100 can be deployed in different computer devices; for example, the storage device 200 and the query device 100 can be deployed in a storage system, the query device 100 can be a computing node (having a data computing function and undertaking a computing task in the storage system) in the storage system, and the storage device 200 can be a storage node (having a data storage function) in the storage system.

[0111] The data query method provided by embodiments of the present application will be described below in combination with the scenario of FIG. 1 and the system architecture of FIG. 3.

[0112] In the data query method in the embodiments of the present application, the data involved is stored in a database. The data in the database can be in a data format suitable for data analysis services, such as a parquet data format or any other possible data format. Before the data query method is used to query data, a data table needs to be created in the database and a data loading process needs to be performed. For example, when a parquet data format is used, a parquet table can be created, and a parquet table file can be generated through a data loading process and stored in the database. Since the table files in the database are in a data format suitable for data analysis services, these table files can be naturally applied to data analysis services. At the same time, the data query method of the embodiments of the present application provides a query process with relatively low complexity to improve the efficiency and performance of data query based on these table files. This also means that the data query method can achieve the effect of supporting the use of a piece of data in multiple query scenarios, so it is not necessary to deploy different data formats in related technologies to meet the needs of different query scenarios, which helps to reduce the occupation of storage resources.

[0113] Please refer to FIG. 4, which is a flowchart of the data query method provided by the embodiments of the present application. The method can be executed by a second node, which can be the data query service node 301 or the data analysis service node 302 shown in FIG. 3.

[0114] Step 401: The second node (K / V server) receives a first request, the first request being used to request data in a database, the first request including a query statement; wherein the database includes a plurality of data tables, the plurality of data tables being used for data analysis services, the data analysis services being used to obtain data from the database and analyze the obtained data.

[0115] In some examples, the first request can be a request triggered by a user's operation on a client. The client can be any client that has access to the database. The client can be a software client, a mini-program in a software client, or a web page in a browser. For example, a user inputs relevant conditions for filtering data when using a client and performs a confirmation operation. The client then generates a corresponding query statement and sends the first request to the second node.

[0116] In some examples, the first request can also be automatically triggered by a user when using a client. For example, when a user uses a client to open a page that needs to display data in the database, the client sends the first request to the second node. The first request can also be triggered in other situations, and the embodiments of the present application do not limit the triggering manner of the first request.

[0117] The first request and the type of the second node can be corresponding. In one example, the first request can be a request corresponding to a data query service, and correspondingly, the second node can be a data query service node. For example, the first request can be a request for point query or range query. In another example, the first request can also be a request corresponding to a data analysis service, and correspondingly, the second node can be a data analysis service node. For example, the first request can be a request for OLAP analysis. In some scenarios, the data query service and the data analysis service are two different services, and can correspond to different service pages. When the page opened in the client is a page of the data query service, the client sends the first request to the data query service node. When the page opened in the client is a page of the data analysis service, the client sends the first request to the data analysis service node.

[0118] When the first request is a request corresponding to the data query service, because the general distributed query architecture adopted by the engine (such as the Spark engine) for executing the data analysis service (such as OLAP analysis) is too complex, the embodiment of the present application provides an engine for data query in the second node, which can be referred to as a K / V server. The K / V server is a lightweight query engine provided by the embodiment of the present application for processing the data query service (such as point query / range query), and is intended to show that the scheme of the embodiment of the present application can enable the point query or the range query to meet the time delay requirement of the point query or the range query as in the related art based on the K / V technology. In FIG. 4, this is specifically taken as an example for illustration. When the data query service includes point query or range query, the types of index fields included in the data query condition included in the query statement included in the first request are usually less, for example, less than or equal to N types, and the value of N can be 3 or other values.

[0119] In one example, the database can be a data warehouse of a big data platform. The database includes a plurality of data tables, and the data table is an object in the database for storing data. The data table can also be understood as a structured data set, and is also a form of structured data organization, and usually stores data in the form of rows and columns. Each row represents a record in the table, and each column represents the value of an index field in the record. The first request is used to indicate that one or more records are obtained from the data table according to the data query condition, and the data query condition usually includes the value of one or more index fields in the data table.

[0120] In the embodiments of the present application, the plurality of data tables are used for data analysis services, in other words, the plurality of data tables adopt a data format suitable for data analysis services. For example, the data format of the data table can be a parquet format, and the data table can also be referred to as a parquet format table or a parquet table. The parquet format is a high-efficiency columnar storage file format, and is usually used in big data processing and analysis scenarios. In the scheme in the above-mentioned related art 2, the combination of the Spark engine and the parquet format table can realize efficient OLAP analysis. Compared with the data table based on the K / V storage technology, the parquet format table occupies less storage space, but the parquet format table has the disadvantage that the performance cannot meet the time delay requirement of point query or range query when used for point query or range query. However, through the scheme of the embodiments of the present application, the query efficiency of data query services such as point query or range query can be improved on the basis of using the parquet format table, so as to improve the probability of meeting the time delay requirement of point query or range query. The specific scheme will be introduced later. That is, the scheme of the embodiments of the present application reuses the data originally used for data analysis services to improve the data query efficiency. In this way, for each piece of data stored in the database, it can be used for data analysis services and data query services (such as point query or range query), that is, one piece of data supports multiple query scenarios, or multiple query scenarios can use the same big data technology data table, thereby effectively saving the resource consumption required for data storage.

[0121] In some examples, the first request can describe the query statement in a JS object notation (JSON) format. Alternatively, other formats can also be used for description, which is not limited in the embodiments of the present application.

[0122] Step 402: The second node (K / V server) determines the information of at least one data block according to the description information of the first data table, the at least one data block being a data block of the data blocks corresponding to the first data table that meets the data query condition, and the first data table being determined according to the first request, and the description information being used to indicate the data block corresponding to the first data table.

[0123] The first request comprises a query statement. By parsing the query statement, the data table and the data query condition that need to be queried can be determined, and the first data table is the data table determined according to the query statement. That is, after receiving the first request, the query statement in the first request can be parsed to determine the first data table that needs to be queried. The query statement comprises at least one query command and a clause. The query command can also be referred to as a query function, which is used to specify which data needs to be returned and how to filter, sort and combine the data. For example, one or more data tables are specified in the data query condition, and the one or more data tables are the first data table. Alternatively, the data query condition does not specify the data table, and all data tables in the database are the first data table.

[0124] The description information is used to indicate the data in the data block corresponding to each data table in the database. Therefore, according to the description information, the data block in the data block corresponding to the first data table that meets the data query condition can be determined, so as to reduce the number of data blocks that need to be read. The description information can also be referred to as index information or index. In an example, one data table corresponds to one description information, and the description information is used to indicate the data in the data block corresponding to the data table. The data of one data table is stored in one or more table files. One table file can comprise one or more data blocks, and one data block can comprise the value of one or more index fields, and the value of the index field is essentially the data in the data table. The data block corresponding to one data table is the data block included in the table file corresponding to the data table.

[0125] In an example, the description information is used to indicate the value range of the data in the data block, which can also be described as indicating the value range of the value of the index field included in the data block. By the value range, the data block that does not need to be queried can be filtered to reduce the number of data blocks that need to be read.

[0126] Step 403: The second node (K / V server) reads at least one data block from the database according to the information of the at least one data block.

[0127] The at least one data block comprises the data requested by the first request. The information of each data block in the information of the at least one data block can comprise one or more of the identifier of the table file to which the data block belongs, the address offset of the data block in the file to which the data block belongs, or the length of the data block. The identifier of the table file can be the name or ID of the table file. In this way, by the information of the at least one data block, the at least one data block can be read from the database.

[0128] In one example, the database can be implemented by HDFS, and in the subsequent introduction, HDFS is taken as an example. Then, according to the information of the at least one data block, the at least one data block can be read from the HDFS. For example, for each of the at least one data block, when reading the data block, the second node can call the seek interface and the read interface of the HDFS, and the offset is taken as a parameter when calling the seek interface, and the length is taken as a parameter when calling the read interface. Correspondingly, the HDFS sends the data block specified by the offset and the length to the second node. The HDFS sends the data block in the form of a binary stream, and the second node can combine the metadata of the data block to construct the binary stream into a data block.

[0129] In the related art as shown in FIG. 2B, Spark needs to first generate a query plan in the Driver process. When generating the query plan, the Driver process reads the index file from the HDFS, determines the Parquet files satisfying the query condition according to the index file, and thus the finally generated query plan only includes the Parquet file list required for the query, which is equivalent to the process of generating the query plan is a process of rough selection of Parquet files satisfying the query condition. Then, the query plan is divided into multiple subtasks when executed, each subtask processes part of the Parquet files in the Parquet file list, and multiple subtasks are assigned to multiple Executor processes for execution. Each Executor process executes its own subtask to obtain data rows satisfying the query condition. When an Executor process executes a subtask, the Executor process performs a query according to the Parquet file in the subtask assigned to the Executor process. Since the Driver process only provides the Parquet file list and does not provide any index information, the Executor process also reads the footer of each Parquet file in the Parquet file list from the HDFS to obtain metadata, and then obtains data blocks satisfying the query condition according to the metadata and the query condition. This is equivalent to needing to interact with the HDFS twice, and the screening process also needs to be executed twice, which is determined by the general distributed architecture of Spark, and thus the query efficiency is extremely low and the time delay is high when performing point query or range query.

[0130] The scheme of the embodiment of the present application preloads the description information to pre-store the index information dispersed in the table files, and the description information can indicate the data blocks of each table file, so that the data block meeting the data query condition can be directly located according to the description information, without the multiple interactions and data screening process as in the related art, and the complexity of the query process of the scheme of the embodiment of the present application is lower, so that the query efficiency is higher and the data query delay is lower.

[0131] Step 404: The second node (K / V server) filters the records meeting the data query condition in the at least one data block, and sends the records meeting the data query condition to the client.

[0132] For each of the at least one data block, the records meeting the condition can be filtered according to the data query condition, and one record is a row in the first data table. Then, the data included in the records meeting the data query condition is sent to the client. The data returned to the client can include all the data included in each record, or can include the data of part of the index fields included in each record.

[0133] Through the above method, the information of at least one data block meeting the data query condition can be determined from the data block corresponding to the first data table according to the description information of the first data table. The at least one data block is the data block meeting the data query condition, that is, the at least one data block includes the data required to be acquired by the first request, so that reading the at least one data block can obtain the data requested by the first request. On the one hand, the embodiment of the present application can make the data table originally used for data analysis business be able to be used in point query, range query or multi-dimensional query scene, so that multiple data query scenes can share the same kind of database technology data table, so that different data query scenes do not have to use respective database technologies, and therefore do not have to deploy resources according to multiple database technologies, but can deploy resources for the shared database technology, which is beneficial to reduce resource overhead. On the other hand, the description information of the first data table can indicate the data block corresponding to the first data table, for example, indicate the data included in the data block, so that the second node can determine the data block meeting the data query condition in the first data table in combination with the description information and the data query condition, the complexity of the data query process is low, and the data query performance is improved.

[0134] Please refer to FIG. 5, which shows another flow diagram of the data query method provided by the embodiment of the application. The difference between the flow shown in FIG. 5 and that shown in FIG. 4 is that the second node needs to parse the query statement in the first request to parse out the first data table to be queried, the data query condition, and the index field to be queried after receiving the first request. The first data table to be queried refers to the range of data to be queried. When the first data table is specified in the query statement, only the first data table will be queried. The data query condition can also be referred to as the data filtering condition, which is used to filter the data rows of the first data table that meet a specific condition. The index field to be queried is used to indicate the index field corresponding to the data to be returned to the client. The data query method includes the following steps:

[0135] Step 501: The second node (K / V server) receives a first request, and the first request is used to request to obtain data in a database. The first request includes a query statement.

[0136] Step 502: The second node (K / V server) parses the query statement in the first request to determine the first data table and the data query condition.

[0137] In one example, the query statement includes one or more of the following contents:

[0138] (1) A FROM clause, which is used to specify the identifier of the data table to be queried. The identifier can be the ID or name of the data table. The first data table is the data table specified by the FROM clause. The data table specified by the FROM clause can include one or more. When the specified data table includes multiple data tables, the number of first data tables is also multiple, or the first data table includes multiple data tables. In the case of multiple first data tables, each first data table can be queried in the following manner, and therefore, in the following description, one first data table is mainly taken as an example for description, and the following will not be described in detail. For example, a data table in the database has a name of "employees", and a clause of a SELECT command is "FROM employees", which is used to specify to query data from the employees table.

[0139] (2) A SELECT command, which is used to select the data to be queried from the database. For example, the employees table includes the following columns: id, name, position, and salary, each of which is an index field. A SELECT command is "SELECT name, salary", which indicates that the data of the name and salary columns will be queried, that is, the data returned to the client finally will include the data of the name and salary columns.

[0140] (3) WHERE clause, used to filter rows that meet certain conditions, or in other words, used to specify the conditions when selecting data. It is used to filter records and only return records from a data table that meet certain conditions. For example, a SELECT command with the clause "WHERE position = 'Manager'" indicates that the records with position as Manager are queried.

[0141] In addition to the above commands, the query statement can also include other commands, which are not limited.

[0142] For example, continuing to use the above example of the employees table, the query statement included in the first request is as follows:

[0143] "SELECT name, salary FROM employees

[0144] WHERE position = 'Manager'"

[0145] This command indicates that the data of the name and salary columns of the employees with position as Manager is obtained from the employees table.

[0146] Then, after receiving the first request, the second node parses the query statement included in the first request to obtain the first data table to be queried by the first request, the data query condition, and the index field to be queried.

[0147] Step 503: The second node (K / V server) determines the information of at least one data block according to the description information of the first data table, the at least one data block being the data block of the first data table that meets the data query condition, and the description information being used to indicate the data block corresponding to the first data table.

[0148] Step 504: The second node (K / V server) reads at least one data block from the database according to the information of the at least one data block.

[0149] Step 505: The second node (K / V server) filters the records that meet the data query condition in the at least one data block, and sends the records that meet the data query condition to the client.

[0150] In an example, the index field to be queried can also be obtained by parsing the query statement in the first request, and then the data corresponding to the index field to be queried in the records that meet the data query condition can be returned to the client.

[0151] The steps in part of Fig. 5 correspond to the steps in the embodiment of Fig. 4, and thus reference can be made to the description of Fig. 4 for these steps, which will not be repeated here.

[0152] Referring to the related art shown in Fig. 2B, in the related art, the data analysis node pulls the description information of the table file into the Driver process of the query engine for Data Skipping, but this will cause the process memory of the data analysis node to be occupied too much, and the data analysis node also needs memory for other functions, and the upper limit of the process memory limits the efficiency of the data skipping process, and finally leads to low data query efficiency. Therefore, the embodiments of the present application provide that the first node provides a special Data Skipping service, and the first node is, for example, the projection service node shown in Fig. 3. In order for the first node to implement the Data Skipping process, the first node needs to store the description information of a plurality of data tables included in the database.

[0153] Referring to Fig. 6, Fig. 6 is a flowchart of the first node obtaining the description information. Taking the database as an example of HDFS, the first node obtaining the description information can include, but is not limited to, the following two ways:

[0154] Method 1: The first node preloads the description information of the data table from the HDFS.

[0155] Referring to Fig. 6, in this way, the first node can perform step 611, that is, reading the table file from the HDFS, and perform step 612, determining and storing the description information of a plurality of data tables according to the table file corresponding to each data table. For example, the first node can read the table file from the HDFS to obtain the description information at the initial use. For another example, the first node can read the table file from the HDFS to obtain the description information at a regular time.

[0156] Through this way, the first node can preload the description information scattered in the multi-dimensional analysis file (i.e., the table file) into the first node, so that when the first node subsequently provides the Data Skipping service for the second node, the data query efficiency can be improved according to the description information.

[0157] Method 2: The first node can update the stored description information according to the update of the HDFS.

[0158] In this mode, the big data platform further comprises a third node, which can be used to maintain the HDFS. For example, the third node can be a data analysis service node (e.g., a Spark engine node) shown in FIG. 3. The third node can update the data of the HDFS according to the operation on the HDFS, and when the third node updates the HDFS, the first node can also be notified to update the description information. For example, referring to step 621 shown in FIG. 6, the third node can send a third request to the first node, and the first node receives the third request. The third request is used to indicate that the third node updates the second data table. Then, the first node performs step 622, that is, reads the table file of the second data table from the HDFS according to the third request, and performs step 623, that is, determines and stores the description information of the second data table according to the table file of the second data table.

[0159] In one example, a deployment software development kit (SDK) of the first node can be invoked in the third node, and when the third node updates the HDFS, the SDK can be invoked to send the third request to the first node.

[0160] In one example, the third request can include an operation on the HDFS by the third node and an operation object, such as an operation of inserting data, deleting data, or modifying data. In this way, the first node can determine the second data table according to the operation and the operation object, so as to update the description information of the second data table.

[0161] In one example, the first node determines the description information of the second data table, which can be reading the complete table file of the second data table, and re-determining the description information of the second data table. Alternatively, the first node can only read part of the data, part of the table file, or part of the data block of the second data table, and update the description information of the second data table according to the read content, to realize incremental update of the description information of the second data table. The incremental update mode can reduce the data transmission amount, reduce the load pressure of the first node, so as to have more resources for data skipping, which is helpful to improve the data query efficiency.

[0162] After the description information of the first data table is stored in the first node, the data skipping process can be performed according to the description information. However, due to the large difference in the data volume among different data tables in the database, the time consumed for performing the data skipping process on different data tables is different. Moreover, when the second node requests the first node to perform the data skipping, there is an interaction overhead between the first node and the second node. If the interaction overhead occupies most of the network round trip time (RTT) and the data skipping process only occupies a small part of the RTT, there is no advantage in separately setting the first node.

[0163] Therefore, the embodiment of the present application provides a scheme of adaptively selecting local data skipping or remote data skipping according to a size table. The local data skipping refers to performing the data skipping by the second node, and the remote data skipping refers to requesting the first node to perform the data skipping process by the second node. The size table mainly refers to the size of the data volume of the data table. When the data volume of a data table is greater than a first threshold value, the data table is a large table, and when the data volume of a data table is less than or equal to the first threshold value, the data table is a small table. The value of the first threshold value can be set according to the actual situation, for example, an empirical value or an experimental value. In the embodiment of the present application, the small table is referred to as a first type of data table, and the large table is referred to as a second type of data table.

[0164] Referring to FIG. 7, FIG. 7 is another flowchart of the data query method provided by the embodiment of the present application.

[0165] Step 701: The client sends a first request to the second node (K / V server). The second node (K / V server) receives the first request, and the first request is used to request to obtain data in the database, and the first request includes a query statement.

[0166] Step 702: The second node (K / V server) parses the query statement in the first request to determine the first data table and the data query condition.

[0167] Step 703: The second node (K / V server) determines whether the description information of the first data table is stored.

[0168] Step 704: If the result of the determination in step 703 is yes, i.e., in the case that the second node (K / V server) has stored the description information of the first data table, the second node (K / V server) determines the information of at least one data block according to the description information of the first data table. The at least one data block is a data block of the data blocks corresponding to the first data table and meeting the data query condition.

[0169] When the first data table is a first type of data table, the second node stores the description information of the first data table, and then the second node performs a local Data Skipping process, i.e., determines the information of at least one data block meeting the data query condition according to the description information stored by the second node. The first type of data table is the small table described above. The Data Skipping process of this type of data table consumes a relatively short time. If the first node is requested to perform the Data Skipping process, the interaction overhead between the second node and the first node occupies a large part of the RTT. When the second node itself performs the Data Skipping process, since the first data table is a small table, the resource consumption is relatively small, and the interaction overhead with the first node can be saved, the time of the Data Skipping process of the first data table is reduced, and the data query efficiency is improved.

[0170] In one example, the second node can send a request to the first node to obtain the description information of the first data table. Alternatively, the second node can read a table file corresponding to the first data table from the database, and then obtain the description information of the first data table according to the read table file.

[0171] Step 705: If the result of the determination in step 703 is no, i.e., in the case that the second node (K / V server) does not store the description information of the first data table, the second node (K / V server) sends a second request to the first node (projection service node).

[0172] In the embodiment of the present application, the second request includes the identifier of the first data table and the data query condition.

[0173] The second node does not store the description information of the first data table in the following cases:

[0174] Case 1: The first data table is a second type of data table, and then the second node does not store the description information of this type of data table. The first type of data table is the large table described above. The Data Skipping process of this type of data table consumes a relatively long time and occupies a large number of device resources of the second node. Therefore, in order to avoid affecting the execution of other functions in the second node, a dedicated first node is requested to perform the Data Skipping process.

[0175] Case 2: The second node queries the first data table for the first time. When the second node queries the first data table for the first time, the second node does not make a type judgment on the first data table before this time, and thus the second node does not store the description information of the first data table.

[0176] Step 706: The first node (the projection service node) determines the information of at least one data block according to the stored description information of the first data table.

[0177] Step 707: The first node (the projection service node) sends a first response to the second node (the K / V server). Correspondingly, the second node (the K / V server) receives the first response from the first node. The first response includes the information of at least one data block.

[0178] That is, when the second node does not store the description information of the first data table, the second node performs a remote Data Skipping process, i.e., sends a second request to the first node to request the first node to perform Data Skipping, and the first node determines at least one data block that meets the data query condition according to the stored description information of the first data table.

[0179] Step 708: The second node (the K / V server) reads at least one data block from the database according to the information of at least one data block.

[0180] Step 709: The second node (the K / V server) filters records in at least one data block that meet the data query condition.

[0181] Step 710: The second node (the K / V server) sends the records that meet the data query condition to the client.

[0182] Some steps in the above FIG. 7 are the same as the steps in the embodiments corresponding to FIG. 4 or FIG. 5, and thus reference can be made to the description of the corresponding parts of FIG. 4 or FIG. 5, and no further description is given here.

[0183] Please refer to FIG. 8, which shows a flowchart of the second node storing the description information of the first data table.

[0184] Step 801: The first node (the projection service node) sends information of a first time duration to the second node (the K / V server). The second node (the K / V server) receives the information of the first time duration from the first node (the projection service node).

[0185] The first time duration is the time duration consumed by the first node to determine at least one data block, i.e., the time duration consumed by the first node to perform the Data Skipping process.

[0186] In an example, the information of the first time length can be included in the first response. Alternatively, the first node can further send the information of the first time length to the second node before or after sending the first response to the second node.

[0187] Step 801 is an optional step, thus is shown in dashed line in FIG. 8.

[0188] Step 802: The second node (K / V server) determines that the type of the first data table is the first type.

[0189] In an example, the second node can determine whether the type of the first data table is the first type according to the data amount of the data corresponding to the first data table. If the data amount of the data corresponding to the first data table is less than or equal to a first threshold, the type of the first data table is the first type, otherwise, if the data amount of the data corresponding to the first data table is greater than the first threshold, the type of the first data table is the second type.

[0190] In an example, the second node can further determine whether the type of the first data table is the first type according to the information of the first time length. If the first time length is less than or equal to a second threshold, it is determined that the type of the first data table is the first type, otherwise, if the first time length is greater than the second threshold, it is determined that the type of the first data table is the second type.

[0191] The second threshold can be a fixed value, for example, a fixed value determined by experience or experiment. Alternatively, the second threshold can also be determined according to the first round-trip time, for example, the second threshold can be the product of the first round-trip time and a first proportion. The value of the first proportion can be configured according to requirements. The first round-trip time is the time length between the sending time of the second request and the receiving time of the first response, the sending time can be the starting sending time, the ending sending time or the median of the two, and the receiving time can be the starting receiving time, the ending receiving time or the median of the two. For example, the first round-trip time is the RTT when the second node requests the first node to perform Data Skipping.

[0192] Step 803: The second node (K / V server) sends a request for obtaining the description information of the first data table to the first node (projection service node).

[0193] Step 804: The first node (projection service node) sends the description information of the first data table to the second node (K / V server).

[0194] Step 805: The second node (K / V server) stores the description information.

[0195] Next, the Data Skipping process performed on the second node or the first node, i.e., a process of determining information of at least one data block according to the description information of the first data table, is introduced. Generally, the Data Skipping process includes two processes of partition (and / or bucket) pruning and data pruning. The partition (and / or bucket) pruning selects, from the cached file list, a result matching the partition (and / or bucket) condition contained in the query condition, and the data pruning determines, for each table file, whether the table file meets the data query condition based on the result of the partition (and / or bucket) pruning, and retains the table file if the table file meets the data query condition, or excludes the table file if the table file does not meet the data query condition.

[0196] In the related art, as shown in FIG. 2B, Spark generates a query plan in the Driver process, the query plan provides a table file list meeting the data query condition, and then distributes the table file list to distributed Executor processes to scan the files provided by the query plan in a distributed manner. Since the Executor processes do not include the description information, the Executor processes need to read the footer information of the table file from the database, and then determine the data block meeting the condition, which is equivalent to loading the description information twice and performing the Data Skipping process twice. This is determined by the general distributed architecture of Spark, and thus the query efficiency of point query or range query is extremely low, and the response time is high.

[0197] In the embodiments of the present application, the description information is pre-stored, and the data block where the data is located is directly determined according to the description information, so that the description information does not need to be loaded multiple times, and the Data Skipping process does not need to be performed multiple times, which can effectively improve the query efficiency of query modes such as point query or range query, and reduce the response time.

[0198] First, the content included in the description information of the first data table is introduced.

[0199] For example, for a first data table, the description information of the first data table can include one or more of the following:

[0200] (1) Metadata.

[0201] The metadata can also be referred to as key metadata. The metadata is used to indicate a set or list of index fields included in the first data table. The metadata also indicates a first index field in the first data table. The first data table includes a plurality of partitions, and the plurality of partitions are determined according to the value of the first index field. The first index field can include a partition field and / or a bucket field.

[0202] Referring to FIG. 3, the data table in the database adopts a partition and bucket manner for data layout. One data table (e.g., a first data table) includes one or more partitions. The partition of the data table can be determined according to the value of one index field in the data table or the value after conversion, the index field is used to divide the partition of the data table, and thus the index field can be referred to as a partition field. Different partitions of the data table correspond to different values or value ranges of the partition field. For example, the partition field is time, and each partition of the data table includes data of a continuous time period. In some examples, the partition can be understood as a storage area, and different partitions of one data table can correspond to different storage areas, and one partition can correspond to one storage area. One partition can be understood as a sub-table of the data table.

[0203] The partition fields of different data tables can be the same or different. The partition field can be any index field in the data table, or can be a conversion of any index field in the data table. For example, the partition field can be a specific field in the specified data table, or can be a high-frequency index field in the data table. The high-frequency index field refers to the index field with the highest query frequency. Then, one data table can horizontally split the data of the data table according to the value (or the value after conversion, such as a hash value) of the high-frequency index field to different partitions (or storage locations). For one data table, the high-frequency index field can not change after being determined. For example, the high-frequency index field can be determined from the index fields of the data table according to experience. Once the high-frequency index field is determined as the partition field, the partition field of the data table will not change. Alternatively, the high-frequency index field of one data table can change, and the partition of the data table can also change with the change of the high-frequency index field.

[0204] One partition of one data table can include one or more buckets. The bucket can be determined according to the value of one index field in the data table, the index field is used to divide the bucket of the data table, and thus the index field can also be referred to as a bucket field. Different buckets correspond to different value ranges of the bucket field. For example, the bucket field is time, and the data table is split into different buckets according to time. Each bucket includes data of a continuous time period, and each bucket can include data of a continuous sub-period. In some examples, the bucket can also be understood as a storage area, and different buckets of one partition can correspond to different sub-storage areas, and one bucket can correspond to one sub-storage area.

[0205] The bucketing field can be any index field in the data table. For example, the partition field can be a specific field in the specified data table, or it can also be a high-frequency index field, a second-highest frequency index field, or other fields in the data table. The second-highest frequency index field refers to an index field with the second-highest query frequency. The second-highest frequency index field can also be referred to as a second-highest query key. Then, the data in the data table is horizontally split into different storage locations according to the values (or converted values such as hash values) of the high-frequency index field or the second-highest frequency index field. If used in combination with partitioning, the data is split using the bucketing field in each partition.

[0206] (2) The first indication information, which can also be referred to as a partition file list or a partition file set.

[0207] The first indication information is used to indicate the table files corresponding to the multiple partitions. As described above, the first data table includes multiple partitions, and a table file corresponding to a first partition of the multiple partitions includes data corresponding to the first partition. The first partition can be any partition of the multiple partitions. One of the table files corresponding to the multiple partitions includes one or more data blocks, and one of the data blocks includes values of one or more index fields. The first indication information can indicate a table file corresponding to each partition of the multiple partitions of the first data table.

[0208] (3) The second indication information, which can also be referred to as min-max statistical information of a table file or min-max statistical information of a data block.

[0209] The second indication information is used to indicate the value range of the index field of the data block included in one of the table files corresponding to the multiple partitions. In one example, the value range can be represented by min-max statistical information. The min-max statistical information of a data block indicates the minimum value and the maximum value in the data block. Since the data in the data block is sorted according to the values of the sorting field, the values of the index field in the data block are all within the interval between the minimum value and the maximum value. One data block can include values of one or more index fields. When a data block includes values of multiple index fields, the min-max statistical information of the data block can also include min-max statistical information of the multiple index fields. The sorting field refers to an index field used to sort the data in the data table, such as a high-frequency query key, a second-highest frequency index field, and the like.

[0210] For example, the table file corresponding to the first data table can be a Parquet file. One Parquet file contains one or more data blocks (each data block has a size of 128 MB or more). Each data block stores multiple data rows field by field in columnar form. The Parquet file also records the min-max statistical information of each field in each data block.

[0211] (4) Bloom filter, the bloom filter can be used to assist in screening out the table file or data block that does not include the required query. The bloom filter can refer to the bloom filter of one or more index fields in the first data table.

[0212] (5) Bitmap, which can also be used to assist in determining the data block.

[0213] Alternatively, the description information can also include information of other any possible index mode, which is not limited by the embodiments of the present application.

[0214] The data query condition includes the value of at least one index field, and the process of determining at least one data block meeting the data query condition is introduced below. Please refer to FIG. 9, which is a flowchart of the process of determining the information of at least one data block. The process can be performed by the first node or the second node.

[0215] Step 901: Determine whether the at least one index field includes the first index field according to the metadata. The first index field can include the partition field and / or the bucket field, and the partition field is taken as an example in FIG. 9.

[0216] Step 902: If the result of step 901 is yes, that is, it is determined that the at least one index field includes the first index field, then determine the second partition from the multiple partitions of the first data table. The second partition is the partition where the value of the first index field in the at least one index field is located.

[0217] For example, the partition field is time, and the value of time in the data query condition is the first time period, then the partition where the first time period is located can be determined from the multiple partitions of the first data table.

[0218] Step 903: Determine the table file corresponding to the second partition according to the first indication information.

[0219] The first indication information can also be referred to as a partition file list, which is used to indicate the table file corresponding to each partition of the first data table. Then after the second partition is determined, the table file corresponding to the second partition can be determined according to the first indication information, so as to narrow the range of data query.

[0220] Step 904: Determine the information of at least one data block according to the second indication information. The at least one data block includes the value of one or more index fields in the at least one index field.

[0221] The second indication information can indicate the value range of the data blocks included in the table file corresponding to the first data table. According to the value range of the data blocks included in the table file corresponding to the second partition in the second indication information and the value of the at least one index field included in the data query condition, at least one data block satisfying the data query condition can be determined, and information of the at least one data block can be obtained.

[0222] In one example, the value range is min-max statistical information. For each data block to be queried, whether the value of each index field in the at least one index field is located between the values of min-max can be determined. If the value is not located between the values of min-max, the data block is filtered. Otherwise, if the value is located between the values of min-max, the data block can include the data to be queried.

[0223] For example, referring to FIG. 10, FIG. 10 is a schematic diagram of a process for determining a data block according to an embodiment of the present application.

[0224] The first request indicates that the data table to be queried is "tab table", and the query condition is "index field no = 1 and index field c1 = 'd'". The index field no is a partition field, and the index field c1 is a bucket field. The value "d" of the index field c1 can be converted to determine that the bucket number corresponding to the value is 1.

[0225] In the description information of the "tab table", the partition / bucket file list is represented as "tab:no = 1:1"->"file1, file2", "tab:no = 1:2"->"file3", "tab:no = 2:1"->"file4, file5". That is, the bucket 1 of the partition 1 corresponds to the table files "file1" and "file2", the bucket 2 of the partition 1 corresponds to the table file "file3", and the bucket 1 of the partition 2 corresponds to the table files "file4" and "file5". In the file1 and file2, the min-max information of the index field c1 is [‘a’,‘e’] and [‘x’,‘y’] respectively. That is, the minimum value of the index field c1 in the file1 is a, and the maximum value is e. The minimum value of the index field c1 in the file2 is x, and the maximum value is y. The file1 includes two data blocks, i.e., data block 1 and data block 2. The min-max information of the index field c1 in the two data blocks is [‘a’,‘b’] and [‘c’,‘e’] respectively.

[0226] So, when performing the Data Skipping process, the partition / bucket pruning result is the file list "file1, file2" corresponding to "tab: no=1:1", and the data pruning result is file1 ('d'> 'a' and 'd'< 'e'), and file2 is excluded (because 'x'> 'd' does not satisfy the query condition). Further, continue to investigate the data block, and only data block 2 satisfies the query condition c1= 'd', so the information of data block 2 is returned, for example, the table file file1 where data block 2 is located, the offset and length of data block 2 in the table file, and the like.

[0227] In the related art (for example, the scheme of the above-mentioned Spark), after the Data Skipping process is performed, only a table file list is returned, and then the seek and read interfaces of the HDFS need to be called to read the file footer, the index information of the footer is parsed, the query condition of the client is combined, and a process similar to the Data Skipping process is performed again to obtain the data block information in the file that satisfies the query condition, and then the target data block in the corresponding table file in the HDFS is read.

[0228] The related art needs to read the file again and analyze the target data block in the file when accessing the file after the Data Skipping, and the process is relatively cumbersome. Through the method of the embodiment of the application, the data block that satisfies the data query condition can be directly determined through one Data Skipping process, without reading the footer from the database again and determining the data block, the complexity of the entire query process is lower, and the input / output (I / O) overhead of scanning is reduced due to the smaller number of data blocks.

[0229] For example, according to the scheme of the related art, after the Driver process gives the file list (that is, file1), the Executor process reads the footer of the file file1 from the HDFS, analyzes that only data block 2 satisfies the condition according to the index information saved in the footer in combination with the query condition, and finally the Driver process reads data block 2, while the embodiment of the application can directly determine that data block 2 satisfies the data query condition through the K / V server, so that data block 2 can be directly read.

[0230] Step 905: If the judgment result of step 901 is no, that is, it is determined that the at least one index field does not include the first index field, then the table file corresponding to the first data table is determined according to the first indication information. That is, when the at least one index field does not include the partition field or the bucket field, the partition cannot be filtered, and then the data blocks included in the table file of all partitions of the first data table are queried.

[0231] Step 906: determining information of at least one data block according to the second indication information. The at least one data block includes a value of one or more index fields in the at least one index field.

[0232] Similar to step 904, according to the value range of the data block included in the table file of all partitions of the first data table and the value of the at least one index field included in the data query condition, at least one data block satisfying the data query condition can be determined, and information of the at least one data block is obtained.

[0233] In summary, in the embodiment of the present application, a lightweight query engine K / V server is provided, which receives a point query / range query request from a client, parses a query table and a query condition, and then performs a Data Skipping process on a table file of a data table based on multi-dimensional data analysis to determine a data block that meets the data query condition, so as to reduce the number of data blocks that need to be read, and finally reads the data block to obtain a query result. Considering that related technologies pull index information to the Driver process of the query engine for Data Skipping, since the query engine is also used to implement other functions, the device where the query engine is located will face the problem of upper limit of process memory. In the embodiment of the present application, a special Data Skipping service node, i.e., the first node, is set to alleviate the problem of upper limit of device memory, and considering the interaction overhead between nodes, a local or remote Data Skipping selection scheme with adaptive size table is adopted to balance between interaction overhead and resource consumption of Data Skipping.

[0234] Therefore, the data query method can perform efficient point query / range query based on data analysis business (such as OLAP analysis), and the method realizes that a data table in a data format supports multiple data query scenarios (such as point query, range query or data analysis business, etc.), so that only one data is needed. Compared with the related art, when multiple data formats of data need to be deployed to meet the time delay requirements of multiple query scenarios, the method can save nearly half of the storage resources, and reduces the cost of data management. Moreover, compared with the cumbersome query process in the related art of spark point query or range query scheme, the query process complexity of the data query method of the embodiment of the present application is lower, and the granularity of the returned result is finer (related art is file, and the method of the embodiment of the present application is data block), the time consumed in the query process is shorter, and the query efficiency is higher. For example, the query performance is better when performing point query or range query, which can be close to the query performance of the K / V query scheme in the related art.

[0235] Based on the same technical concept as the method embodiments, the embodiments of the present application further provide a data query device for executing the data query method in the method embodiments.

[0236] In an example, as shown in FIG. 11, the data query device 110 includes a receiving module 1101, a determining module 1102, and a data reading module 1103. Specifically, in the data query device 110, the modules are connected through a communication channel.

[0237] The receiving module 1101 is configured to receive a first request, the first request being used to request to obtain data in a database, the first request including a query statement; wherein the database includes a plurality of data tables, the plurality of data tables being used for a data analysis service, the data analysis service being used to obtain data from the database and analyze the obtained data. The determining module 1102 is configured to determine information of at least one data block according to description information of a first data table, the at least one data block being a data block in data blocks corresponding to the first data table and meeting a data query condition, the first data table and the data query condition being determined according to the first request, and the description information being used to indicate the data blocks corresponding to the first data table. The data reading module 1103 is configured to read the at least one data block from the database according to the information of the at least one data block.

[0238] As a possible implementation, the determining module 1102 is specifically configured to determine the information of the at least one data block according to the stored description information; or, in the case where the description information is not stored, send a second request to the first node and receive a first response from the first node, wherein the second request includes an identifier of the first data table and the data query condition, and the first response includes the information of the at least one data block.

[0239] As a possible implementation, the information of the at least one data block includes one or more of the following: identifiers of table files to which the at least one data block respectively belongs; address offsets of the at least one data block in the table files to which the at least one data block respectively belongs; or lengths of the at least one data block.

[0240] As a possible implementation, the determining module 1102 is further configured to, when a type of the first data table is a first type, obtain and store the description information from the first node; wherein the data amount of data corresponding to the data table of the first type is less than or equal to a first threshold value.

[0241] As a possible implementation, the receiving module 1101 is further configured to receive information of a first time length from the first node, the first time length being a time length consumed by the first node to determine the at least one data block. The determining module 1102 is further configured to, if the first time length is less than or equal to a second threshold value, determine that the type of the first data table is the first type.

[0242] As a possible implementation, the second threshold is determined according to a first round-trip time, the first round-trip time being a time length between a sending time of the second request and a receiving time of the first response.

[0243] As a possible implementation, the description information includes one or more of the following: metadata, the metadata being used to indicate a first index field included in the first data table, wherein the first data table includes a plurality of partitions, the plurality of partitions being determined according to values of the first index field; first indication information, the first indication information being used to indicate table files corresponding to the plurality of partitions respectively, wherein a table file corresponding to a first partition in the plurality of partitions includes data corresponding to the first partition, one of the table files corresponding to the plurality of partitions includes one or more data blocks, and one of the data blocks includes values of one or more index fields; and second indication information, the second indication information being used to indicate a value range of an index field of a data block included in the one of the table files corresponding to the plurality of partitions.

[0244] As a possible implementation, the data query condition includes values of at least one index field. The determining module 1102 is specifically configured to: determine, according to the metadata, that the at least one index field includes the first index field; determine, from the plurality of partitions, a second partition, the second partition being a partition in which a value of the first index field in the at least one index field is located; determine, according to the first indication information, a table file corresponding to the second partition; and determine, according to the second indication information, information of at least one data block, the at least one data block including values of one or more index fields in the at least one index field.

[0245] As a possible implementation, the data query condition includes values of at least one index field. The determining module 1102 is specifically configured to: determine, according to the metadata, that the at least one index field does not include the first index field; determine, according to the first indication information, a table file corresponding to the first data table; and determine, according to the second indication information, information of at least one data block, the at least one data block including values of one or more index fields in the at least one index field.

[0246] As a possible implementation, the determining module 1102 is further configured to determine data rows in the at least one data block that meet the data query condition.

[0247] In an example, as shown in FIG. 12, the data query apparatus 120 includes a receiving module 1201, a determining module 1202, and a sending module 1203. Specifically, in the data query apparatus 120, the modules are connected through a communication channel.

[0248] The receiving module 1201 is configured to receive a second request from a second node, the second request comprising an identifier of a first data table and a data query condition; wherein the first data table is included in a database, the database comprising a plurality of data tables, the plurality of data tables being used for a data analysis service, the data analysis service being configured to obtain data from the database and analyze the obtained data. The determining module 1202 is configured to determine, according to stored description information of the first data table, at least one data block that meets the data query condition in data blocks corresponding to the first data table, the description information being configured to indicate the data blocks corresponding to the first data table. The sending module 1203 is configured to send, to the second node, a first response comprising first indication information, the first indication information being configured to indicate the at least one data block.

[0249] As a possible implementation, the data query apparatus 120 further comprises a description information loading module 1204. The description information loading module 1204 is configured to: in response to a preloading instruction, read table files corresponding to the plurality of data tables in the database, and determine, according to the table files corresponding to the plurality of data tables, description information of the plurality of data tables; or read, according to a third request from a third node, a table file corresponding to a second data table, and determine, according to the table file corresponding to the second data table, description information of the second data table, the third request being configured to instruct the third node to update the second data table, and the third node being configured to maintain the database.

[0250] As a possible implementation, the information of the at least one data block comprises one or more of the following: identifiers of table files to which the at least one data block respectively belongs; address offsets of the at least one data block in the table files to which the at least one data block respectively belongs; or lengths of the at least one data block.

[0251] As a possible implementation, the sending module 1203 is further configured to send, to the second node, information of a first time length, the first time length being a time length consumed by the first node for determining the at least one data block.

[0252] As a possible implementation, the description information comprises one or more of the following. Metadata, the metadata being configured to indicate a first index field included in the first data table, wherein the first data table comprises a plurality of partitions, the plurality of partitions being determined according to values of the first index field. First indication information, the first indication information being configured to indicate table files corresponding to the plurality of partitions respectively, wherein a table file corresponding to a first partition of the plurality of partitions comprises data corresponding to the first partition, and one of the table files corresponding to the plurality of partitions comprises one or more data blocks, one of the data blocks comprising values of one or more index fields. Second indication information, the second indication information being configured to indicate a value range of an index field of a data block included in the one of the table files corresponding to the plurality of partitions.

[0253] As a possible implementation, the data query condition includes values of at least one index field. The determining module 1202 is specifically configured to: determine, according to the metadata, that the at least one index field includes the first index field, determine, from the multiple partitions, a second partition, the second partition being a partition in which a value of the first index field in the at least one index field is located, determine, according to the first indication information, a table file corresponding to the second partition, and finally determine, according to the second indication information, information of at least one data block, the at least one data block including values of one or more index fields in the at least one index field.

[0254] As a possible implementation, the data query condition includes values of at least one index field. The determining module 1202 is specifically configured to: determine, according to the metadata, that the at least one index field does not include the first index field, determine, according to the first indication information, a table file corresponding to the first data table, and finally determine, according to the second indication information, information of at least one data block, the at least one data block including values of one or more index fields in the at least one index field.

[0255] The division of the modules in the embodiments of the present application is illustrative, and is merely a logical function division. In actual implementation, another division manner can be used. In addition, the function modules in each embodiment of the present application can be integrated in one processor, or can be physically separated, or two or more modules can be integrated into one module. The integrated module can be realized in the form of hardware or in the form of a software function module. In the schematic diagram of FIGS. 11 and 12, the units represented by the dashed lines are optional function units.

[0256] The integrated module, if realized in the form of a software function module and sold or used as an independent product, can be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a mobile phone, or a network device, etc.) or a processor to perform all or part of the steps of the method of each embodiment of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various media that can store program codes.

[0257] In some embodiments, the data query apparatus 130 provided by the embodiments of the present application can have the structure as shown in FIG. 13, including a processor 1301 and a memory 1302 connected with the processor 1301. The processor 1301 and the memory 1302 can be connected with each other through a bus. The processor 1301 can serve as the main processor of the data query apparatus 130, that is, the control core of the data query apparatus 130. The bus can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. The specific connection medium between the processor 1301 and the memory 1302 is not limited in the embodiments of the present application.

[0258] The memory 1302 can include the memory described above. The memory 1302 can be a volatile memory, such as a random access memory; or the memory 1302 can be a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD) or a solid-state drive (SSD), or the memory 1302 can be any other medium capable of carrying or storing desired program codes in the form of instructions or data structures and capable of being accessed by a computer, but is not limited to this. The memory 1302 can be a combination of the memories described above. The memory 1302 can be used to store software programs and modules.

[0259] The processor 1301 can include one or more processors. The processor 1301 performs various function applications and data processing of the data query apparatus 130 by running the software programs and modules stored in the memory 1302, such as the data query method provided by the embodiments of the present application.

[0260] It can be understood that the structure shown in the embodiments of the present application does not constitute a specific limitation on the data query apparatus. In some other embodiments of the present application, the electronic device can include more or fewer components than those shown, or combine certain components, or split certain components, or different arrangement of components. The components shown can be implemented in hardware, software or a combination of software and hardware.

[0261] The method steps in the embodiments of the present application can be implemented in the form of hardware or by a processor executing computer programs or instructions. The computer programs or instructions can constitute a computer program product. The embodiments of the present application also provide a computer program product containing computer executable instructions. In an embodiment, the computer executable instructions are used to make a computer execute the functions in the above method embodiments.

[0262] The computer executable instructions can be stored in a computer readable storage medium, and the embodiments of the present application further provide a computer readable storage medium, which stores executable instructions. In an embodiment, the computer executable instructions are used to make a computer execute the functions in the above-mentioned method embodiments.

[0263] The computer readable storage medium provided by the embodiments of the present application can be a random access memory (RAM), a flash memory, a read-only memory (ROM), a programmable ROM (PROM), an erasable PROM (EPROM), an electrically ePROM (EEPROM), a register, a hard disk, a mobile hard disk, a CD-ROM, or any other form of computer readable storage medium known in the art.

[0264] The computer executable instructions can be stored in a computer readable storage medium, or transmitted from one computer readable storage medium to another computer readable storage medium, for example, the computer program or instructions can be transmitted from one website site, computer, server or data center to another website site, computer, server or data center through wired or wireless manner. The computer readable storage medium can be any available medium accessible by a computer or a data storage device such as a server, data center and the like integrated with one or more available media. The available media can be a magnetic medium, for example, a floppy disk, a hard disk, a magnetic tape; or an optical medium, for example, a digital video disc (DVD); or a semiconductor medium, for example, a solid state disk.

[0265] The embodiments of the present application further provide a computer program product, which includes instructions, when running on a computer, causes the computer to execute the method performed by the first node or the second node.

[0266] The embodiments of the present application provide a chip system, which includes a processor and an interface, the processor is used to call and run instructions from the interface, so that the chip system implements the method in the above-mentioned embodiments. The chip system can be composed of a chip, or can contain a chip and other discrete devices.

[0267] The embodiment of the present application further provides a chip comprising a processor, which is used to support the communication device to realize the functions of the first node or the second node involved in the method embodiments. In a possible design, the chip is connected with a memory or the chip comprises a memory, and the memory is used to store the computer programs or instructions and data necessary for the communication device.

[0268] It should be understood that the size of the sequence number of each process described above in various embodiments of the present application does not mean the order of execution, and the execution order of each process should be determined according to its function and inherent logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0269] Those skilled in the art can realize that the various illustrative logical blocks and steps described in conjunction with the embodiments disclosed herein can be implemented by electronic hardware, computer software or a combination of computer software and electronic hardware. Whether the functions are implemented in hardware or software depends on the specific application and design constraints of the technical solutions. The skilled person can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0270] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described system, device and unit can refer to the corresponding processes in the foregoing method embodiments, which will not be repeated here.

[0271] In several embodiments provided in the present application, it should be understood that the disclosed system, device and method can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of the units is only a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some interface, device or unit, and can be electrical, mechanical or other forms.

[0272] The units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, that is, they can be located in one place, or can be distributed on a plurality of network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment of the present application.

[0273] If the functions are implemented in the form of software function units and sold or used as independent products, they can be stored in a computer readable storage medium. Based on this understanding, the part of the technical solutions of the present application that essentially contributes or the part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in the various embodiments of the present application. The aforementioned storage medium includes a U disk, a mobile hard disk, a read-only memory (ROM), a RAM, a magnetic disk or an optical disk, and various media that can store program codes.

[0274] Obviously, those skilled in the art can make various modifications and variations to the present application without departing from the scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application also intends to include these modifications and variations.

Claims

1. A data query method, characterized by, The method comprises: receiving a first request for requesting data in a database, the first request comprising a query statement; wherein the database comprises a plurality of data tables, the plurality of data tables being used for a data analysis service, the data analysis service being used for obtaining data from the database and analyzing the obtained data; determining information of at least one data block according to description information of a first data table, the at least one data block being a data block that meets a data query condition among data blocks corresponding to the first data table, the first data table and the data query condition being determined according to the query statement, the description information being used for indicating the data blocks corresponding to the first data table; reading the at least one data block from the database according to the information of the at least one data block.

2. The method of claim 1, wherein, The determining the information of the at least one data block according to the description information of the first data table comprises: determining the information of the at least one data block according to the stored description information; or in a case where the description information is not stored, sending a second request to a first node and receiving a first response from the first node, wherein the second request comprises an identifier of the first data table and the data query condition, and the first response comprises the information of the at least one data block.

3. The method of claim 2, wherein, The information of the at least one data block comprises one or more of the following: identifiers of table files to which the at least one data block respectively belongs; address offsets of the at least one data block in the table files to which the at least one data block respectively belongs; or lengths of the at least one data block.

4. The method according to claim 2 or 3, characterized in that, The method further comprises: in a case where a type of the first data table is a first type, obtaining and storing the description information from the first node; wherein the data amount of data corresponding to the data table of the first type is less than or equal to a first threshold value.

5. The method of claim 4, wherein, The method further comprises: receiving information of a first time length from the first node, the first time length being a time length consumed by the first node for determining the at least one data block; in a case where the first time length is less than or equal to a second threshold value, determining that the type of the first data table is the first type.

6. The method of claim 5, wherein, The second threshold value is determined according to a first round-trip time, the first round-trip time being a time length between a sending time of the second request and a receiving time of the first response.

7. The method according to any one of claims 1 to 6, characterized in that, The description information comprises one or more of the following: metadata used for indicating a first index field included in the first data table, wherein the first data table comprises a plurality of partitions, the plurality of partitions being determined according to values of the first index field; first indication information used for indicating table files corresponding to the plurality of partitions respectively, wherein a table file corresponding to a first partition in the plurality of partitions comprises data corresponding to the first partition, one of the table files corresponding to the plurality of partitions comprises one or more data blocks, and one of the data blocks comprises values of one or more index fields; second indication information used for indicating a value range of an index field of a data block included in one of the table files corresponding to the plurality of partitions.

8. The method of claim 7, wherein, The data query condition comprises values of at least one index field. The information of the at least one data block determined according to the description information of the first data table comprises: The at least one index field determined according to the metadata comprises a first index field; A second partition is determined from the plurality of partitions, the second partition being a partition in which a value of the first index field in the at least one index field is located; According to the first indication information, a table file corresponding to the second partition is determined; According to the second indication information, information of the at least one data block is determined, the at least one data block comprising values of one or more index fields in the at least one index field.

9. The method of claim 7, wherein, The data query condition comprises a value of at least one index field; The information of the at least one data block determined according to the description information of the first data table comprises: The at least one index field determined according to the metadata does not comprise the first index field; According to the first indication information, a table file corresponding to the first data table is determined; According to the second indication information, information of the at least one data block is determined, the at least one data block comprising values of one or more index fields in the at least one index field.

10. The method according to any one of claims 1 to 9, characterized in that, After reading the at least one data block from the database according to the information of the at least one data block, the method further comprises: Determining a data row in the at least one data block that meets the data query condition.

11. A data query method, characterized by, The method comprises: Receiving a second request from a second node, the second request comprising an identification of a first data table and a data query condition; wherein the first data table is included in a database, the database comprising a plurality of data tables, the plurality of data tables being used for a data analysis service, the data analysis service being used for obtaining data from the database and analyzing the obtained data; Determining, according to stored description information of the first data table, at least one data block in a data block corresponding to the first data table that meets the data query condition, the description information being used to indicate the data block corresponding to the first data table; Sending a first response to the second node, the first response comprising first indication information, the first indication information being used to indicate the at least one data block.

12. The method of claim 11, wherein, The method further comprises: In response to a preloading indication, reading table files corresponding to the plurality of data tables in the database, and determining description information of the plurality of data tables according to the table files corresponding to the plurality of data tables; or According to a third request from a third node, reading a table file corresponding to a second data table, and determining description information of the second data table according to the table file corresponding to the second data table, the third request being used to indicate that the third node updates the second data table, the third node being used to maintain the database.

13. A data query apparatus, characterized by comprising: The data query device comprises a processing unit and a transceiver unit, the processing unit being coupled to the transceiver unit to perform the method of any one of claims 1-10, or to perform the method of any one of claims 11-12.

14. A computer device, comprising: The apparatus comprises a processor and a memory, the processor and the memory are connected, the memory stores a computer program, when the computer program stored in the memory is executed by the processor, the apparatus executes the method of any one of claims 1-10 or 11-12.

15. A computer-readable storage medium, characterized in that, The computer readable storage medium is used to store a computer program, when the computer program runs on a computer, the computer executes the method of any one of claims 1-10, or the computer executes the method of any one of claims 11-12.

16. A computer program product, characterised in that, The computer program product comprises a computer program, when the computer program runs on a computer, the computer executes the method of any one of claims 1-10, or the computer executes the method of any one of claims 11-12.

17. A chip system, characterized by The chip system comprises: A processor and an interface, the processor is used to call and run instructions from the interface, when the processor executes the instructions, the method of any one of claims 1-10 is realized, or the method of any one of claims 11-12 is realized.

Images