Data query method, distributed database system and management equipment

Abstract

The invention provides a data query method, a distributed database system and management equipment, and relates to the technical field of data query, the method comprises the following steps: for a data query request involving a plurality of filtering conditions, generating a plurality of query tasks by a management node in the distributed database system, and generating a read request based on the plurality of filtering conditions, the first working node executes the read request, and reads data meeting a plurality of filtering conditions into a memory pool of the system at one time, namely, reads data required for executing a plurality of query tasks into the memory pool at one time, so that when the working nodes executing the plurality of query tasks in the system execute respective query tasks, the data required for executing the plurality of query tasks is read into the memory pool at one time; according to the data query method and device, the data required for executing the corresponding query task can be obtained from the memory pool, redundant reading of the data queried by the data query request is avoided, the data reading overhead is reduced, and the data query efficiency is improved.

Description

Technical Field

[0001] The present application relates to the field of data query technology, and in particular to a data query method, a distributed database system, and a management device. Background Art

[0002] With the rapid development of technologies such as databases and big data, distributed database systems have emerged. Currently, in distributed database systems, the use of operator fusion can effectively improve data query speed. For example, the read (scan) operator and the filter (filter) operator involved in the data query process can be integrated to achieve reading and filtering.

[0003] In the related technology, the process of integrating the reading operator and the filtering operator in the distributed database system is, for example: assigning multiple query tasks created based on the data query request to each working node of the system, each query task corresponds to the processing of a part of the data, and when each working node executes the reading operator in its respective query task, it performs preliminary screening of the data in the file that does not meet the filtering conditions based on the filtering conditions of the filtering operator (for example, data identification id<10) and the statistical information of the file to be read (for example, the maximum and minimum values ​​of the data id in the file), thereby reducing the amount of data read.

[0004] However, in the above method, since each working node performs its own task independently, different working nodes will redundantly read the data queried by the data query request, resulting in a large data reading overhead and affecting the data query efficiency. Summary of the Invention

[0005] The embodiments of the present application provide a data query method, a distributed database system, and a management device, which can reduce the data reading overhead of the distributed database system and improve data query efficiency.

[0006] In a first aspect, the present application provides a data query method that can be applied to data analysis scenarios such as databases and big data. Schematically, the method is applied to a distributed database system, which includes a management node, multiple working nodes, and a memory pool. The method includes:

[0007] The management node receives a data query request and generates multiple query tasks based on the data query request, where the data query request includes multiple filtering conditions;

[0008] The management node generates a read request based on the multiple filter conditions, and sends the read request to a first working node among the multiple working nodes, where the read request includes the multiple filter conditions.

[0009] The first working node executes the read request and reads the data that meets the multiple filter conditions into the memory pool;

[0010] The worker nodes that execute multiple query tasks obtain the data required to execute the corresponding query tasks from the memory pool.

[0011] Among them, for data query requests involving multiple filtering conditions, the management node in the distributed database system generates multiple query tasks, and generates a read request based on the multiple filtering conditions. The first working node executes the read request, and reads the data that meets the multiple filtering conditions into the system's memory pool at one time, that is, the data required to execute multiple query tasks is read into the memory pool at one time. Based on this, the working nodes that execute multiple query tasks in the system can obtain the data required to execute the corresponding query tasks from the memory pool when executing their respective query tasks, avoiding redundant reading of the data queried by the data query request, reducing data reading overhead, and improving data query efficiency.

[0012] In some embodiments, the method also includes: after the first working node reads the data that meets multiple filtering conditions into the memory pool, the location information of the data corresponding to each filtering condition in the memory pool is stored in the memory pool; the working node that executes multiple query tasks obtains the data required to execute the corresponding query tasks from the memory pool, including: obtaining the data required to execute the corresponding query tasks from the memory pool based on the location information.

[0013] By storing the location information of the data corresponding to each filter condition in the memory pool, multiple working nodes can obtain the location information of the data corresponding to the filter condition from the memory pool based on the filter condition of the query task being executed when executing their respective query tasks, and obtain the required data from the memory pool based on the obtained location information.

[0014] In some embodiments, the data queried by the data query request includes multiple data units; the first working node stores the location information of the data corresponding to each filter condition in the memory pool in the memory pool, including: in the memory pool, each filter condition and the location information of the data unit corresponding to each filter condition in the memory pool are stored in correspondence; the working node that executes multiple query tasks obtains the data units required to execute the corresponding query tasks from the memory pool based on the location information.

[0015] In this way, when multiple working nodes in the system execute tasks related to data query requests, they can obtain the location information of the data units corresponding to the filter conditions in the memory pool based on the filter conditions corresponding to the executed tasks, and based on the obtained fine-grained location information, quickly and accurately obtain data units from the memory pool, thereby reducing the subsequent data processing volume and improving data query efficiency.

[0016] In some embodiments, if the distributed database system adopts a storage-computing integrated architecture, the first working node executes a read request and reads data that meets multiple filter conditions into the memory pool, including:

[0017] Execute a read request and classify the data that meets multiple filter conditions based on the data popularity information of the data query request to obtain data in multiple categories. The data popularity information indicates the frequency of data use during the execution of the data query request.

[0018] Multiple categories of data are stored separately in the storage space provided by different working nodes in the memory pool.

[0019] The frequency of data usage includes, for example, the number of joins, the number of filters, the number of broadcasts, etc., but the present application is not limited thereto.

[0020] In some embodiments, the second working node executing the first query task includes a processor and a computing unit, and the processor is used to offload the first operator in the first query task to the computing unit for execution. The first operator is an operator in the first query task other than the read operator.

[0021] By offloading the first operator to the computing unit for execution, the computing burden of the processor can be reduced.

[0022] In some embodiments, the first query task also includes a second operator in addition to the reading operator and the first operator, and the execution of the second operator depends on the execution result of the first operator. The computing unit is used to store the execution result of the first operator to the memory pool, and the processor is used to obtain the execution result of the first operator from the memory pool to execute the second operator.

[0023] In this way, data transmission between the computing unit and the processor can be avoided, saving communication resources.

[0024] In a second aspect, the present application provides a distributed database system, which includes a management node, multiple working nodes and a memory pool. The system is used to implement the data query method provided by the first aspect or any possible implementation of the first aspect.

[0025] In a third aspect, the present application provides a data query device, which is configured as a management node in a distributed database system. The system also includes multiple working nodes and a memory pool. The device includes at least one functional module for performing the function of the management node in the data query method provided by the first aspect or any possible implementation of the first aspect.

[0026] In a fourth aspect, the present application provides a management device, which includes a processor and a memory, and the processor is used to execute at least one piece of program code stored in the memory so that the management device can implement the function of the management node in the data query method provided in the first aspect or any possible implementation method of the first aspect.

[0027] In a fifth aspect, the present application provides a computer-readable storage medium for storing at least one program code segment. When the at least one program code segment is executed by a management device, the management device implements the functions of a management node in the data query method provided in the first aspect or any possible implementation of the first aspect. The storage medium includes, but is not limited to, volatile memory, such as random access memory, and non-volatile memory, such as flash memory, a hard disk drive (HDD), or a solid state drive (SSD).

[0028] In a sixth aspect, the present application provides a computer program product that, when executed on a management device, enables the management device to implement the functions of a management node in the data query method provided in the first aspect or any possible implementation of the first aspect. The computer program product may be a software installation package, and when it is necessary to implement the aforementioned data query method, the computer program product may be downloaded and executed on the management device. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 This is a schematic diagram of an implementation environment provided by an embodiment of the present application;

[0030] Figure 2 is a schematic diagram of a distributed database system provided in an embodiment of the present application;

[0031] Figure 3 This is a schematic diagram of the hardware structure of a management device provided in an embodiment of the present application;

[0032] Figure 4 This is a schematic diagram of the structure of a device cluster provided in an embodiment of the present application;

[0033] Figure 5 This is a schematic diagram of a device cluster connection method provided in an embodiment of the present application;

[0034] Figure 6 This is a flow chart of a data query method provided by an embodiment of the present application;

[0035] Figure 7 This is a schematic diagram of generating a read request provided by an embodiment of the present application;

[0036] Figure 8 This is a schematic diagram of a file reading process provided by an embodiment of the present application;

[0037] Figure 9 is a schematic diagram of a data query method provided in an embodiment of the present application;

[0038] Figure 10 This is a schematic diagram of a storage solution determination process provided by an embodiment of the present application;

[0039] Figure 11 This is a structural diagram of a data query device provided by this embodiment;

[0040] Figure 12 It is a structural diagram of another data query device provided by this embodiment. DETAILED DESCRIPTION

[0041] In order to make the purpose, technical solutions and advantages of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings. It should be noted that the information involved in this application (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.) and signals are all authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data need to comply with the relevant laws, regulations and standards of the relevant countries and regions. For example, the data query requests, files, etc. involved in this application are all obtained with full authorization.

[0042] For ease of understanding, the key terms and key concepts involved in this application are explained below.

[0043] A database is a digital filing cabinet, a place where electronic files are stored, allowing users to add, query, update, and delete data. A database is a collection of data stored in a specific way, shared by multiple users, with minimal redundancy, and independent of applications.

[0044] A data table is a collection of two-dimensional arrays, consisting of columns and rows. Columns contain information of the same type, and each column is called a field. The column header is called a field name. Rows contain several columns of information, and a row of data is called a record.

[0045] In this application, a file refers to an electronic file obtained by encoding and compressing the original data of structured data or semi-structured data. The logical structure of the file determines the organization and storage method of the data within the file. Generally, the file format of a file can be divided into column storage format and row storage format. Examples of files in row storage format are TextFile files, CSV files, Sequence Files, Map Files, and Avro files. Examples of files in column storage format are RCFile, ORC files, parquet files, and carbon data files.

[0046] Data unit, in this application, refers to the logical division unit of data in a file. In an embodiment of the present application, a file includes multiple data units, that is, includes at least two data units, each data unit can be encoded and compressed separately, and any two data units in the same file can be compressed using different compression algorithms according to the data type, and this application does not limit this. In addition, in this application, the original data in the file is logically split into multiple data units under the premise of ensuring its semantics, and the data unit can be defined according to business requirements and file format, and this application does not limit this. For example, a data unit can be the smallest encoding compression unit of a file, or a data unit is the smallest reading unit of a file, so that redundant data can be minimized. Schematically, taking an ORC file as an example, an ORC file includes multiple groups of data rows, a group of data rows is called a stripe, a stripe includes a row group, and each row group includes multiple streams. Then in an ORC file, a data unit can be a stream. Taking a Parquet file as an example, a table is persisted by partition in a Parquet file. Within a partition, data is divided by row groups, and within a row group, data is stored by columns. A row group includes multiple column chunks, and each column is divided into multiple data pages. When reading a Parquet file, data can be read in pages. Therefore, in a Parquet file, the data unit can be a page.

[0047] Structured Query Language (SQL) is a specialized programming language used to manage data stored in relational databases. SQL can refer to various data-related languages, including data definition languages ​​and data manipulation languages. SQL covers data insertion, querying, updating, deleting, schema creation and modification, and data access control.

[0048] An operator (OP) is a computing unit or function running on a computing device or storage device. In the field of database or big data technology, the database system parses the SQL query initiated by the user and generates an execution plan. After the optimizer optimizes the execution plan, the corresponding operators are called for sequential execution according to the optimized execution plan. For example, operators include scan operators, filter operators, project operators, aggregate operators, and join operators, etc., but the present application is not limited to these.

[0049] The following is an introduction to the application scenarios and implementation environment of this application.

[0050] The technical solution provided by the embodiment of the present application can be applied to data analysis scenarios such as databases and big data. Schematically, taking a distributed database system as an example, a distributed database system generally includes a management node and multiple working nodes. These working nodes can accept the control of the management node and execute multiple query tasks created according to data query requests (such as SQL queries) in parallel to improve data query efficiency. For example, a user initiates an SQL query to a distributed database system, and the management node in the system parses the SQL and generates an execution plan. After the execution plan is optimized by the optimizer, it enters the execution phase, that is, multiple query tasks are created according to the optimized execution plan, and the multiple query tasks are assigned to each working node in the system to execute. Each query task corresponds to the processing of a portion of data, wherein each query task can include multiple operators. For any working node, when executing the received query task, the working node can call multiple operators in sequence to execute the query task. In this process, since each working node independently executes its own query task, different working nodes may redundantly read the data queried by the data query request (for example, multiple query tasks all indicate that the data in the same data table is queried), resulting in a large data reading overhead and affecting data query efficiency. Based on this, the present application provides a data query method that can be applied to a distributed database system. By fusing some operators in multiple query tasks involved in a data query request, redundant reading of data queried for the data query request is avoided, which can effectively reduce data reading overhead and improve data query efficiency.

[0051] Reference below Figure 1 , introduce the implementation environment of this application.

[0052] Figure 1 This is a schematic diagram of an implementation environment provided by an embodiment of the present application. Figure 1As shown, the implementation environment includes a client 100 and a distributed database system 200. The client 100 is directly or indirectly connected to the distributed database system 200 via a wireless network or a wired network.

[0053] Client 100 is used to provide users with services such as data storage, data query, and data analysis. Illustratively, a user operates client 100, triggering client 100 to send a data query request (e.g., an SQL query) to distributed database system 200. For example, the data query request indicates a query for data that meets a filter condition in at least one data table. Client 100 can be deployed on a user's device, such as an application server, desktop computer, laptop computer, mobile phone, etc., and this application does not limit this.

[0054] The distributed database system 200 is a database system with storage capacity and computing capacity, and supports structured query statements SQL queries. Among them, a file system is deployed on the distributed database system 200. Usually, the file corresponding to a data table will be split into multiple sub-files and stored on one node or multiple nodes. Among them, each sub-file includes multiple records of the same data table, and the records included in each sub-file are different. In an embodiment of the present application, the file format of the file in the distributed database system 200 can be an ORC file or a parquet file, etc., but is not limited to this. Schematically, the distributed database system 200 receives a data query request (such as an SQL query) from the client 100, and based on the data query request, reads the data queried by the corresponding data query request from the file system, processes the read data, obtains the query result of the data query request, and feeds the query result back to the client 100.

[0055] In an embodiment of the present application, a distributed database system 200 includes a plurality of nodes 201, and each node 201 is connected in communication. In some embodiments, the distributed database system 200 adopts a storage-computing integrated architecture, and a plurality of nodes 201 are also referred to as a plurality of storage devices, or a plurality of devices, to which the present application is not limited. Wherein, for any one node 201, the node 201 is a device having both computing power and storage capacity, such as a server, a desktop computer, etc., to which the present application is not limited. Node 201 can be an independent physical server, or a server cluster or a distributed system composed of a plurality of physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content distribution networks (content delivery networks, CDNs), and big data and artificial intelligence platforms. Node 201 can also be a node in a virtualized form such as a virtual machine or a container, in which case the virtual machine or container can be deployed in at least one computing device (e.g., a server). In other embodiments, the distributed database system 200 adopts a storage-computing separation architecture, where multiple nodes 201 are divided into computing nodes and storage nodes. Computing nodes, also known as computing devices, are devices with computing capabilities, such as servers, desktop computers, or storage array controllers. Storage nodes, also known as storage devices, are devices with storage capabilities, such as controllers, network cards, and multiple hard disks, which provide storage space for the system.

[0056] The distributed database system 200 includes a memory pool, and multiple nodes 201 can share the memory pool. Schematically, all or part of the multiple nodes 201 can provide storage space for the memory pool, that is, not every node in the system needs to provide storage space for the memory pool. The memory pool is established based on the storage space provided by at least two nodes, and this application does not limit the number of nodes occupied by the memory pool. For example, if the distributed database system 200 adopts a storage and computing integrated architecture, at least two nodes in the system provide storage space for the memory pool; if the distributed database 200 adopts a storage and computing separation architecture, at least two storage nodes in the system provide storage space for the memory pool. In addition, in an embodiment of the present application, the memory pool is used to store data read by the distributed database system 200 in the process of executing a data query request, as well as the location information of the data in the memory pool, etc. This process will be introduced in subsequent method embodiments.

[0057] In the examples of this application, refer to Figure 2 , Figure 2 This is a schematic diagram of a distributed database system provided by an embodiment of the present application. Figure 2As shown, the multiple nodes in the distributed database system 200 are divided into a management node 2011 and multiple worker nodes 2012 according to the functions implemented by the nodes. The management node 2011 and the multiple worker nodes 2012 can all share the memory pool of the distributed database system 200. The management node 2011 is used to receive data query requests sent by clients, generate multiple query tasks based on the data query requests, and distribute these query tasks to multiple worker nodes 2012 for execution. The worker nodes 2012 are used to receive query tasks, obtain data related to the query tasks, execute the query tasks, and return the query task execution results to the management node 2011. Schematically, the present application does not limit the distributed computing architecture adopted by the distributed database system 200. For example, taking the Spark distributed computing architecture as an example, multiple nodes 201 can be divided into driver nodes and worker nodes according to the functions implemented by the nodes, wherein the driver node is used to run the application (i.e., the Spark application written by the user, including the function code of the driver node and the process code running on the worker node, etc.), receive data query requests such as SQL sent by the user, parse the data query request, build a calculation flow graph, split the calculation flow graph into distributed computing query tasks, and submit the distributed computing query tasks to multiple worker nodes for execution, that is, for realizing the functions possessed by the aforementioned management node 2011. The worker node is used to receive the computing query task, obtain the data related to the query task and execute the query task, and return the execution result of the query task to the driver node, that is, for realizing the functions possessed by the aforementioned worker node 2012. Of course, the distributed database system 200 can also adopt other distributed computing architectures, such as MapReduce, FineBI, etc., and the present application is not limited thereto. In addition, the above division and naming methods of the multiple nodes 201 are merely examples and do not constitute a limitation to the present application.

[0058] It should be noted that the above Figure 1 and Figure 2 The number of clients 100 , nodes 201 , and memory pools shown is for illustrative purposes only. The number of clients 100 , nodes 201 , and memory pools may be greater or lesser, and this is not limited in the embodiments of the present application.

[0059] In some embodiments, the wireless network or wired network described above uses standard communication technologies and / or protocols. The network is typically a Transmission Control Protocol / Internet Protocol (TCP / IP) network in a data center network and an RDMA network, such as an RDMA over converged Ethernet (RoCE) network or an InfiniBand (IB) network, without limitation. In other embodiments, customized and / or dedicated data communication technologies can be used to replace or supplement the above-mentioned data communication technologies.

[0060] The following is an introduction to the structure of the nodes in the distributed database system 200.

[0061] This application provides a management device that can be configured as the management node 2011 in the above-mentioned distributed database system 200. Figure 3 , Figure 3 This is a hardware structure diagram of a management device provided in an embodiment of the present application. Figure 3 As shown, the management device 300 includes a memory 301, a processor 302, a communication interface 303 and a bus 304. The memory 301, the processor 302 and the communication interface 303 are connected to each other via the bus 304.

[0062] The memory 301 refers to a device for storing data, which can be a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM) or other types of dynamic storage devices that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. Schematically, the memory 301 is used to store at least one program code, and when the program code stored in the memory 301 is executed by the processor 302, the processor 302 is used to execute the steps performed by the management node in the following method embodiment. In some embodiments, the memory 301 is also used to provide storage space for the memory pool of the distributed database system 200.

[0063] The processor 302 may be a network processor (NP), a central processing unit (CPU), a graphics processing unit (GPU), an application-specific integrated circuit (ASIC), or an integrated circuit used to control the execution of the program of the present application. The processor 302 may be a single-core (single-CPU) processor or a multi-core (multi-CPU) processor. The number of processors 302 may be one or more.

[0064] The communication interface 303 uses a transceiver module such as a transceiver to implement communication between the management device 300 and other devices or a communication network. For example, data can be obtained through the communication interface 303.

[0065] The memory 301 and the processor 302 may be separately provided or integrated together.

[0066] The bus 304 may include a path for transmitting information between various components of the management device 300 (eg, the memory 301 , the processor 302 , and the communication interface 303 ).

[0067] The embodiment of the present application also provides a device cluster, which includes a management device and multiple computing devices. Figure 4 , Figure 4 This is a schematic diagram of the structure of a device cluster provided in an embodiment of the present application. Figure 4 As shown, the device cluster includes a management device and multiple computing devices. Each computing device includes memory, a processor, a communication interface, and a bus. The computing device is used to implement the steps performed by the working nodes in the following method embodiments. In some embodiments, the devices in the device cluster can be connected via a network. The network can be a wide area network, a local area network, or the like. Figure 5 This is a schematic diagram of a device cluster connection method provided by an embodiment of the present application. Figure 5 As shown, the management device and the computing device are connected via a network. Specifically, they are connected to the network via a communication interface in each device.

[0068] Based on the above Figures 1 to 5 The content shown here introduces the distributed database system involved in this application. The following introduces the data query method applied to this system through a method embodiment.

[0069] Figure 6 This is a flow chart of a data query method provided by an embodiment of the present application. Figure 6 As shown, the method is applied to a distributed database system, which includes a management node, multiple working nodes and a memory pool. Schematically, the method includes the following steps 601 to 605.

[0070] 601. A management node receives a data query request and generates multiple query tasks based on the data query request, where the data query request includes multiple filtering conditions.

[0071] In an embodiment of the present application, a distributed database system includes a management node and multiple working nodes, which can execute multiple query tasks created according to data query requests (such as SQL queries) in parallel to improve data query efficiency. The data query request includes multiple filter conditions, wherein one filter condition can correspond to one query task, for example, based on the filter condition "data id < 10", a query task is generated to query data with data id < 10, and one filter condition can also correspond to multiple query tasks, for example, based on the filter condition "data id < 10", query task A is generated to query data with 0 < id < 5 and query task B is generated to query data with 6 ≤ id < 10, and this application does not limit this. In some embodiments, the multiple query tasks generated based on the data query request all indicate queries for the same data object, wherein the data object is, for example, at least one data table, or at least one file, etc., and this application does not limit this. Based on the above introduction to the distributed database system, it can be seen that a file system is deployed on the distributed database system for storing data. Typically, the file corresponding to a data table is split into multiple sub-files and stored on one or more nodes. Each sub-file includes multiple records from the same data table, and each sub-file includes different records. For example, the data queried by the data query request is the data in data table t1, and the file corresponding to data table t1 is stored on multiple nodes of the system (for example, the file corresponding to data table t1 is split into multiple sub-files, and these sub-files are stored on multiple nodes respectively). The multiple query tasks all indicate that data table t1 should be queried.

[0072] In addition, for any query task, the query task includes multiple operators, such as a scan operator, a filter operator, a hash aggregation operator, and a project operator, etc., but the present application is not limited thereto. Taking any filter condition as an example, executing the filter operator corresponding to the filter condition means reading data that meets the filter condition from the file system. For example, if the filter condition is data id < 10, then executing the filter operator corresponding to the filter condition means reading data with data id < 10 from the file system.

[0073] Schematically, the distributed database system is connected to the client for communication. The user triggers the client to send a data query request (such as an SQL query) to the distributed database system by operating the client. The data query request includes multiple filtering conditions. The management node in the system receives the data query request, parses the data query request and generates an execution plan (for example, taking the Spark distributed computing engine as an example, the driver node receives the SQL query, parses the SQL and generates an execution plan). After the execution plan is optimized by the optimizer, it enters the execution stage, that is, multiple query tasks are generated according to the optimized execution plan. Each query task corresponds to the processing of a part of the data. Each query task includes multiple operators, and multiple query tasks are executed in parallel by multiple working nodes.

[0074] 602. The management node generates a read request based on multiple filter conditions, where the read request includes the multiple filter conditions.

[0075] In an embodiment of the present application, each query task includes a read operator for the data required to execute the corresponding query task. Based on this, the management node fuses multiple filter conditions and multiple read operators to generate a read request, which indicates that the data required to execute multiple query tasks will be read into the memory pool at one time based on the multiple filter conditions. It should be understood that since multiple working nodes in a distributed database system can share a memory pool, it is possible to fuse multiple filter conditions and multiple read operators for the required data in multiple query tasks to generate a read request, which is then sent to any one of the multiple working nodes for execution, so that the data required to execute multiple query tasks will be read into the memory pool at one time based on the multiple filter conditions. This process can also be understood as the management node creating a fusion query task based on multiple filter conditions and multiple read operators (that is, fusing some operators involved in multiple query tasks to reduce data transfer between operators). This fusion query task is executed by issuing a read request (or a new operator). The function of this read request is to read the data required to execute multiple query tasks once and store the read data in the memory pool. As a result, multiple working nodes that execute multiple query tasks in the system can quickly obtain data from the memory pool based on the filter conditions corresponding to the query tasks being executed when executing their respective query tasks (for example, when executing other operators other than the read operator in the query task), avoiding redundant reading of data queried for data query requests, and effectively improving data query efficiency.

[0076] For example, a data query request includes multiple filter conditions (c1, c2, c3), and the data queried by the data query request is the data in the data table t1. The management node generates multiple query tasks (query task 1, query task 2, query task 3) based on the data query request, wherein query task 1, query task 2, and query task 3 each include a read operator for reading the data in the data table t1. Based on this, the management node merges the three read operators and three filter conditions of the three query tasks to generate a read request. The function of the read request is to read the data table t1 once and store it in the memory pool. Since the working nodes that execute multiple query tasks in the system can share this memory pool, these working nodes can obtain the data required to execute the corresponding query tasks from the memory pool when executing their respective query tasks.

[0077] Reference below Figure 7 , the process of generating a read request by a management node is illustrated with an example. Figure 7 This is a schematic diagram of generating a read request provided by an embodiment of the present application. Figure 7 As shown, a data query request includes multiple filter conditions (c1, c2, c3). The data queried by this data query request is the data in data table t1. Based on this data query request, the management node generates query tasks corresponding to each filter condition (query task 1, query task 2, query task 3). Query task 1 includes the read operator scan(t1), the projection operator project(c1), the hash aggregation operator HashAgg1, and the filter operator filter(c1); query task 2 includes the read operator scan(t1), the projection operator project(c2), the hash aggregation operator HashAgg2, and the filter operator filter(c2); and query task 3 includes the read operator scan(t1), the projection operator project(c3), the hash operator HashAgg3, and the filter operator filter(c3). Clearly, query tasks 1, 2, and 3 all include the read operator scan(t1) for data table t1. Based on this, the management node generates a read request based on the multiple filter conditions and the read operators in the multiple query tasks. For example, RelationCacheScan is used to represent read requests. Figure 7 The multiple query tasks shown include a projection operator, and the projection operator corresponds to the filter condition. Therefore, the read operator and the projection operator can also be fused. It should be noted that this is only an example and does not constitute a limitation of the present application. In some embodiments, a read request can be generated according to business needs. It should be understood that other methods of fusing the filter condition, the read operator and other operators related to the filter condition in order to avoid redundant reading of the data queried by the data query request can be applied to the present application.

[0078] 603. The management node sends the read request to the first working node among the multiple working nodes.

[0079] In the embodiment of the present application, the first working node refers to any working node among multiple working nodes.

[0080] 604. The first working node executes the read request and reads the data that meets the multiple filtering conditions into the memory pool.

[0081] In an embodiment of the present application, the first working node receives and executes a read request, and reads data that meets multiple filter conditions into a memory pool. Schematically, the first working node executes a read request, and reads data that meets multiple filter conditions in the file system into a memory pool. In some embodiments, after the first working node reads the data that meets multiple filter conditions into the memory pool, the first working node stores the location information of the data corresponding to each filter condition in the memory pool in the memory pool. Schematically, the read request can be used to instruct the first working node to perform the action of storing location information, that is, the read request instructs the data that meets multiple filter conditions to be read into the memory pool based on multiple filter conditions and stores the location information of the data corresponding to each filter condition in the memory pool in the memory pool. Alternatively, after the management node sends the read request, the management node may send a location information storage request to the first working node, and the location information storage request instructs the location information of the data corresponding to each filter condition in the memory pool to be stored in the memory pool. This application does not limit this. The first working node stores the location information of the data corresponding to each filter condition in the memory pool based on the storage address of the read data in the memory pool. This location information is used by the working node that executes multiple query tasks to obtain the data required to execute the corresponding query tasks from the memory pool. This location information can also be understood as data location information of a kind of affinity filter condition. Storing the location information of the data corresponding to each filter condition in the memory pool in the memory pool means storing each filter condition in correspondence with the location information of the data corresponding to each filter condition in the memory pool. For example, taking the case where the multiple filter conditions of a data query request include filter condition c1, filter condition c2, and filter condition c3, the first working node stores the filter condition c1 and the location information of the data corresponding to the filter condition c1 in the memory pool in the form of a data table, stores the filter condition c2 and the location information of the data corresponding to the filter condition c2 in the memory pool in the memory pool in the form of a data table, and stores the filter condition c3 and the location information of the data corresponding to the filter condition c3 in the memory pool in the form of a data table. This application does not limit the storage form of the location information and can be set according to business needs. For example, a key-value pair can be used, with the filter condition as the key and the location information as the value, to store the filter condition and the location information of the data corresponding to the filter condition in the memory pool in the form of a key-value pair. In addition, this application does not limit the storage location of the location information in the memory pool. For example, the location information can be stored in the storage space provided by the default node in the memory pool. The default node is, for example, the central node closest to other nodes in the system. In this way, the communication cost consumed by each node to access the location information can be reduced, thereby improving the efficiency of data query.It should be understood that by storing the location information of the data corresponding to each filter condition in the memory pool, when multiple working nodes in the system execute related query tasks of data query requests, they can obtain the location information of the data corresponding to the filter condition in the memory pool based on the filter condition corresponding to the executed query task, and quickly obtain data from the memory pool based on the obtained location information, thereby avoiding redundant reading of the data queried for the data query request and effectively improving the data query efficiency.

[0082] In addition, based on the foregoing description, a file refers to an electronic file used to store the data queried by a data query request, and a file may include multiple data units. For example, a data unit may be the smallest encoding compression unit of a file, or a data unit may be the smallest read unit of a file. This minimizes redundant data. Schematically, taking an ORC file as an example, the data unit of an ORC file may be a stream. Taking a parquet file as an example, the data unit of a parquet file may be a page. Accordingly, the data queried by a data query request may also be understood to include multiple data units. In some embodiments, the first working node executes a read request to read data that meets multiple filter conditions into a memory pool, including: the first working node executes a read request, based on the multiple filter conditions and statistical information of the file containing the data queried by the data query request, reading target data units that meet the multiple filter conditions from the multiple data units into the memory pool. The statistical information of the file indicates the range of data stored in each data unit in the file. For example, taking a parquet file as an example, the statistical information of the file indicates the range of data IDs stored in each page in the file, such as the maximum and minimum data IDs. This process is that when the first working node reads the data in the file, it filters (or performs preliminary screening) the data in the file based on multiple filtering conditions, and reads the data that meets the multiple filtering conditions into the memory pool, thereby reducing the storage space occupied in the memory pool and reducing the subsequent data processing volume.

[0083] For example, reference Figure 8 , Figure 8 This is a schematic diagram of a file reading process provided by an embodiment of the present application. Figure 8As shown, taking a parquet file and a page as an example, the file's statistical information indicates the data ID range of the data stored in each page of the file. The multiple filter conditions of the data query request include filter condition c1 (data ID < 10), filter condition c2 (data ID > 100), and filter condition c3 (data ID < 0.5). Based on the multiple filter conditions and the statistical information of the file containing the data queried by the data query request, the first working node reads the data in the file and stores the pages (i.e., target data units) in the file that meet filter conditions c1, c2, and c3 in the memory pool. For example, since the statistical information indicates that the maximum data ID of the data stored in page 1 is 3, which meets filter condition c1, page 1 is the target data unit and is stored in the memory pool.

[0084] In some embodiments, if the first working node stores the location information of the data corresponding to each filter condition in the memory pool in the memory pool, then after the first working node reads the data that meets the multiple filter conditions into the memory pool, each filter condition is stored in the memory pool in correspondence with the location information of the data unit corresponding to each filter condition in the memory pool. For example, taking the multiple filter conditions of the data query request as including filter conditions c1, c2, and c3, and the file where the data queried by the data query request is located as a parquet file, the first working node stores the filter condition c1 in the memory pool in correspondence with the data units page1, page2, and page corresponding to the filter condition c1, stores the filter condition c2 in the memory pool in correspondence with the data units page4 and page5 corresponding to the filter condition c2, and stores the filter condition c3 in the memory pool in correspondence with the data units page6, page7, and page8 corresponding to the filter condition c3. Schematically, the first working node can use the form of key-value pairs in the memory pool to store each filter condition and the position information of the data unit corresponding to each filter condition in the memory pool. For example, taking the file as an ORC file as an example, the first working node uses the filter condition as the key and the position information of the stream corresponding to each filter condition in the memory pool as the value for corresponding storage; or, taking the file as a parquet file as an example, the first working node uses the filter condition as the key and the position information of the page corresponding to each filter condition in the memory pool as the value for corresponding storage. That is, different precisions can be used for different types of files.

[0085] It should be understood that when the first working node stores each filter condition in correspondence with the location information of the data unit corresponding to each filter condition in the memory pool, the location information is correspondingly used by multiple working nodes to obtain the data unit corresponding to the filter condition from the memory pool based on the filter condition corresponding to the executed query task when executing operators other than the read operator in their respective query tasks. That is, when multiple working nodes in the system execute query tasks related to data query requests, they can obtain the location information of the data unit corresponding to the filter condition in the memory pool based on the filter condition corresponding to the executed query task, and based on this fine-grained location information obtained, quickly and accurately obtain the data unit from the memory pool, thereby reducing the amount of data processing when executing subsequent operators, thereby improving data query efficiency.

[0086] In other embodiments, the first working node can also store each filter condition and the statistical information of the data unit corresponding to each filter condition in the memory pool in correspondence with each filter condition, wherein the statistical information of the data unit is, for example, the maximum value, minimum value, and value of the data id in the data unit, etc., but the present application is not limited to this. For example, a key-value pair is used, with the filter condition as the key and the statistical information of the data unit corresponding to each filter condition and the location information of the data unit in the memory pool as the value for corresponding storage. In this way, it is convenient for multiple nodes to execute subsequent operators based on the statistical information of the data unit, thereby improving data query efficiency.

[0087] After steps 601 to 604, for a data query request involving multiple filter conditions, the management node in the distributed database system generates multiple query tasks corresponding to the multiple filter conditions, generates a read request based on the multiple filter conditions, and sends the read request to the first working node for execution. After receiving the read request, the first working node reads the relevant data required to execute the multiple query tasks into the memory pool at one time. Based on this, when executing their respective query tasks, the multiple working nodes in the system can obtain the required data from the memory pool based on the filter conditions of the executed query tasks. Moreover, the first working node can also store the location information of the data corresponding to each filter condition in the memory pool in the memory pool. Thus, when executing their respective query tasks, the multiple working nodes can obtain the location information of the data corresponding to the filter conditions from the memory pool based on the filter conditions of the executed query tasks, and obtain the required data from the memory pool based on the obtained location information. The process of the working node executing the query task will be described below with reference to step 605.

[0088] 605. The working node executing multiple query tasks obtains data required to execute the corresponding query tasks from the memory pool.

[0089] In an embodiment of the present application, the management node assigns multiple query tasks to multiple working nodes, which are executed in parallel by the multiple working nodes. Since the data corresponding to all the filter conditions are already stored in the memory pool, for any working node that executes a query task, the working node can quickly obtain the required data from the memory pool based on the filter conditions of the query task being executed. In some embodiments, the memory pool stores the location information of the data corresponding to each filter condition in the memory pool. Based on this, the working node that executes multiple query tasks obtains the data required to execute the corresponding query task from the memory pool based on the location information. The following takes the second working node in the system as an example to introduce the process of the working node executing the query task, which schematically includes the following steps:

[0090] Step 6051: The second working node obtains first position information of data corresponding to the first filter condition in the memory pool from the memory pool based on the first operator in the first query task, and the first query task corresponds to the first filter condition.

[0091] In an embodiment of the present application, the second working node refers to any one of the multiple working nodes other than the first working node, and is used to execute the first query task. The first query task refers to any one of the multiple query tasks of the data query request, corresponding to the first filter condition, that is, any one of the multiple filter conditions of the data query request. The first query task includes multiple operators, and the multiple operators include a read operator for the data required to execute the first query task. The first operator is an operator other than the read operator in the first query task. For example, the first operator is a filter operator, a hash aggregation operator, or a projection operator, etc. The present application is not limited to this. Based on the aforementioned steps 601 to 604, it can be seen that the data query request corresponds to multiple query tasks. After receiving the data query request, the management node generates a read request to instruct the first working node to read the data that meets the multiple filter conditions into the memory pool. Then, the management node sends the corresponding query task to each working node respectively, so that each working node executes the operator other than the read operator in its respective query task. For example, taking the case where multiple query tasks in a data query request include query task 1, query task 2, and query task 3, the second working node executes the operators other than the read operator in query task 1, the third working node executes the operators other than the read operator in query task 2, and the fourth working node executes the operators other than the read operator in query task 3. It should be noted that the first working node may also execute the operators other than the read operator in the aforementioned query task 1, query task 2, or query task 3, and this application does not limit this.

[0092] Schematically, the process of the second working node obtaining the first position information from the memory pool based on the first operator is that the second working node obtains the first position information that is compatible with the first filter condition from the memory pool based on the first filter condition of the query task it performs, so as to obtain data from the memory pool through the first position information. Based on the aforementioned step 604, it can be seen that the memory pool stores data that meets multiple filter conditions and the position information of the data corresponding to each filter condition in the memory pool. In some embodiments, the data queried by the data query request includes multiple data units, and the memory pool stores data that meets multiple filter conditions and the position information of the data unit corresponding to each filter condition in the memory pool. Accordingly, in this step, the second working node can obtain the first position information of the data unit corresponding to the first filter condition in the memory pool from the memory pool based on the first operator, providing technical support for accurate data reading.

[0093] Step 6052: The second working node obtains data corresponding to the first filtering condition from the memory pool based on the first location information.

[0094] In an embodiment of the present application, the second worker node retrieves the data corresponding to the first filter condition from the memory pool to execute the first operator. Illustratively, if the memory pool stores the location information of the data unit corresponding to each filter condition in the memory pool, the second worker node retrieves the data unit corresponding to the first filter condition from the memory pool based on the first location information. Furthermore, if the first query task also includes other operators in addition to the read operator and the first operator, the second worker node continues to execute the other operators after executing the first operator, which is not further described here.

[0095] In some embodiments, the second working node includes a processor and a computing unit, and the processor is used to offload the first operator to the computing unit for execution. Wherein, the processor is, for example, a central processing unit (CPU), and the computing unit is, for example, a field programmable gate array (FPGA), a data processing unit (DPU), etc., and this application is not limited to this. By offloading the first operator to the computing unit for execution, the computing power burden of the CPU can be reduced. In some embodiments, after executing the first operator, the computing unit stores the execution result of the first operator in a memory pool, so that the second working node can obtain the execution result of the first operator from the memory pool when executing subsequent operators of the first operator. Schematically, the first query task also includes a second operator other than the read operator and the first operator. The execution of the second operator depends on the execution result of the first operator. The computing unit is used to store the execution result of the first operator in the memory pool, and the processor is used to obtain the execution result of the first operator from the memory pool to execute the second operator. In this way, data transfer between the computing unit and the processor can be avoided, saving communication resources.

[0096] Reference below Figure 9 , taking a parquet file and a page as an example, the data query method shown in steps 601 to 605 above is illustrated. Figure 9 This is a schematic diagram of a data query method provided by an embodiment of the present application. Figure 9 As shown, the data query method includes the following steps:

[0097] Step 1: The management node receives a data query request and generates multiple query tasks based on the request. The query request includes multiple filter conditions (c1, c2, c3). The data to be queried is data in table t1. The multiple query tasks (query task 1, query task 2, query task 3) correspond to the multiple filter conditions. The management node generates a read request based on the multiple filter conditions.

[0098] Step 2. The management node sends a read request to the first working node. The first working node executes the read request and reads the data in the file based on multiple filter conditions (c1, c2, c3) and the statistical information of the file where the data table t1 is located. The page (i.e., the target data unit) in the file that meets the filter conditions c1, c2, and c3 is stored in the memory pool. In the memory pool, each filter condition is stored in correspondence with the location information of the data unit corresponding to each filter condition in the memory pool.

[0099] In step 3, taking the example of multiple query tasks of a data query request being executed by the first, second, and third worker nodes, the first worker node obtains all operators except the read operator in query task 1, the second worker node obtains all operators except the read operator in query task 2, and the third worker node obtains all operators except the read operator in query task 3. These worker nodes obtain corresponding location information from the memory pool based on the filter conditions of their respective query tasks, and based on the obtained location information, obtain corresponding data from the memory pool to execute the query tasks.

[0100] In step 4, the processors in the first, second, and third worker nodes offload some operators (such as filtering operators) to a computing unit, such as an FPGA, to save CPU computing power. After executing the operator, the computing unit stores the operator's execution results in a memory pool, allowing the processor to retrieve the operator's execution results from the memory pool to execute other operators (such as hash aggregation operators) and obtain the query results for the data query request.

[0101] In summary, in the data query method provided in the embodiment of the present application, for data query requests involving multiple filtering conditions, the management node in the distributed database system generates a query task corresponding to each filtering condition, and generates a read request based on the multiple filtering conditions. The first working node executes the read request, and reads the data that meets the multiple filtering conditions into the system's memory pool at one time, that is, the data required to execute multiple query tasks is read into the memory pool at one time. Based on this, the working nodes that execute multiple query tasks in the system can obtain the data required to execute the corresponding query tasks from the memory pool when executing their respective query tasks, avoiding redundant reading of data queried for data query requests, reducing data reading overhead, and improving data query efficiency.

[0102] After the above Figure 6 The method embodiment shown in the figure introduces the data query method provided by this application. Based on the above introduction to the distributed database system, it can be seen that the system can adopt a storage-computing integrated architecture or a storage-computing separated architecture. If the system adopts a storage-computing integrated architecture, the above Figure 6The management node, the first working node, and the second working node in the illustrated embodiment are all devices with computing and storage capabilities. If the system adopts a storage-computing separation architecture, the management node, the first working node, and the second working node in the aforementioned method embodiment are all devices with computing capabilities. In some embodiments, if the distributed database system adopts a storage-computing integration architecture, when the first working node reads data that meets multiple filtering conditions into the memory pool, it can classify and store the data according to the frequency of use of the data to reduce the communication delay during the data query process. This optional implementation method is introduced below. Schematically, the above-mentioned step 604 may include the following steps A to B.

[0103] Step A: The first working node executes a read request and classifies data that meets multiple filter conditions based on the data heat information of the data query request to obtain data of multiple categories. The data heat information indicates the frequency of data use during the execution of the data query request.

[0104] Among them, the frequency of use of data is, for example, the number of joins, the number of filters, the number of broadcasts, and so on, and the present application is not limited thereto. Schematically, the data heat information is determined by the management node in the system. For example, the management node obtains the data heat information of the data query request based on the frequency of use of the data queried by the request during the execution of the data query request, and sends the data heat information to the first working node. For example, the data heat information is carried in the read request. Of course, the management node can also send the read request and the data heat information to the first working node in sequence, and the present application does not limit this. In some embodiments, the data heat information can also be determined by the first working node, and the present application does not limit this. In addition, the present application does not limit the number of data categories. For example, data that meets multiple filtering conditions can be classified to obtain data in three categories: cold, hot, and warm. Data that meets multiple filtering conditions can also be classified to obtain more categories of data, such as category 1 to category 5, and so on.

[0105] Step B: The first working node stores the data of multiple categories in the storage spaces provided by different working nodes in the memory pool.

[0106] Among them, the first working node determines multiple storage schemes based on multiple categories of data, and determines the communication cost of each storage scheme based on the data heat information and the memory reading time between each working node. According to the communication cost of each storage scheme, the storage scheme with the minimum communication cost is determined from multiple storage schemes, and the multiple categories of data are stored in the storage space provided by different working nodes in the memory pool according to the storage scheme. Among them, for any storage scheme, the communication cost of the storage scheme indicates the total communication cost consumed by multiple query tasks of multiple working nodes executing data query requests when storing data according to the storage scheme. This process is to store data with a higher frequency of use in working nodes with a shorter memory reading time, while taking into account the use of other working nodes, so that the total communication cost of multiple working nodes executing multiple query tasks is minimized.

[0107] The following uses an example of data of multiple categories including data of a first category and data of a second category to illustrate the process of the first working node executing step B. It should be understood that since the number of data categories is 2, the number of storage schemes is correspondingly 2. Schematically, step B includes:

[0108] Step B1. Based on the first storage scheme, data heat information and memory reading time between each working node, determine the first communication cost of the first storage scheme. The first storage scheme indicates that the first category of data is stored in the storage space provided by the first target node in the memory pool, and the second category of data is stored in the storage space provided by the second target node in the memory pool. The first communication cost refers to the total communication cost consumed by multiple working nodes to execute multiple query tasks when storing data according to the first storage scheme.

[0109] The number of the first target nodes may be one or more, and similarly, the number of the second target nodes may be one or more. Schematically, the first working node determines the first communication cost of the first storage solution based on the following formula (1).

[0110]

[0111] In the above formula (1), sum1 refers to the first communication cost of the first storage solution; i represents any working node; n represents the number of working nodes in the system, and n is a positive integer; cost[i] represents the communication cost consumed by the i-th working node in executing its own query task. In cost[i], v i，k represents the memory read time between the i-th worker node and the k-th worker node, f i，k represents the data access frequency from the i-th worker node to the k-th worker node (determined by data heat information), v i，irepresents the local memory read time of the i-th worker node, f i，i In some embodiments, v i，i Approximately 0, for example, referring to Table 1, Table 1 lists the memory read time between various working nodes in the system, in milliseconds. It should be noted that Table 1 is only for illustration and does not constitute a limitation of the present application.

[0112] Table 1

[0113] <![CDATA[v1]]> <![CDATA[v2]]> …… <![CDATA[v n ]]> <![CDATA[v1]]> 0 5 …… 10 <![CDATA[v2]]> 5 0 …… 15 …… …… …… …… …… <![CDATA[v n ]]> 10 15 …… ……

[0114] Step B2: Based on the second storage scheme, data heat information, and memory reading time between each working node, determine the second communication cost of the second storage scheme. The second storage scheme indicates that the first category of data is stored in the storage space provided by the second target node in the memory pool, and the second category of data is stored in the storage space provided by the first target node in the memory pool. The second communication cost refers to the total communication cost consumed by multiple working nodes to execute multiple query tasks when storing data according to the second storage scheme.

[0115] The method for determining the second communication cost is referred to the aforementioned formula (1) and will not be repeated here.

[0116] Step B3: If the first communication cost is less than the second communication cost, store the data of multiple categories according to the first storage scheme; if the first communication cost is greater than the second communication cost, store the data of multiple categories according to the second storage scheme.

[0117] If the first communication cost is less than the second communication cost, this indicates that when storing multiple categories of data according to the first storage scheme, the total communication cost consumed by multiple worker nodes executing multiple query tasks is relatively low. Therefore, storing multiple categories of data according to the first storage scheme can achieve a reasonable allocation of storage space in the memory pool and reduce communication latency during data queries, thereby improving data query efficiency. Conversely, if the first communication cost is greater than the second communication cost, this indicates that when storing multiple categories of data according to the second storage scheme, the total communication cost consumed by multiple worker nodes executing multiple query tasks is relatively low, and this will not be further described.

[0118] In some embodiments, taking the case where multiple query tasks of a data query request include a first query task and a second query task, if the data processed by the first query task and the second query task are both first-category data, and multiple categories of data are stored according to the aforementioned first storage scheme, the first target node is used to execute the first query task, and the second target node is used to obtain data from the first target node to execute the second query task. It should be understood that since the first-category data is stored in the storage space provided by the first target node in the memory pool, the communication cost consumed by the first target node when executing the first query task is relatively small. Moreover, since the total communication cost consumed by multiple nodes when executing multiple query tasks is taken into account when determining the storage scheme, even if the second target node needs to obtain data from the first target node to execute the second query task, it will not incur excessive communication costs.

[0119] It should be noted that the above steps B1 to B3 are introduced based on two categories of data as an example. In some embodiments, the number of data categories may be greater. Figure 10 , taking the data categories including cold, hot and warm as an example, the process of determining the final storage solution based on the communication costs of different storage solutions is illustrated. Figure 10 This is a schematic diagram of the process of determining a storage solution provided by an embodiment of the present application. Figure 10 As shown in the figure, after classifying the data that meets multiple filtering conditions, three categories of data are obtained: cold, hot, and warm. The memory pool includes the storage space provided by the first, second, and third target nodes. For example, each target node has 20 cores and can process 20 query tasks in parallel. The memory read time of each target node is 1. The cold data is used 10 times, which means that the hot data is used 30 times, and the warm data is used 20 times. Each data usage corresponds to one query task, which means that there are currently 60 query tasks.

[0120] like Figure 10As shown in Figure (a), storage plan A indicates that cold data is stored in the first target node, hot data is stored in the second target node, and warm data is stored in the third target node. Since the second target node can process 20 query tasks in parallel, the remaining 10 query tasks need to be assigned to the first target node (the first target node has idle computing power). Then, combined with the above formula (1), we can know that cost[first target node] = 10×1 + 10×30, cost[second target node] = 20×1, cost[third target node] = 20×1; based on this, the communication cost of storage plan A = 10×1 + 10×30 + 20×1 + 20×1 = 350. Among them, 10×30 is the additional communication cost, that is, the communication cost consumed after the query tasks corresponding to some hot data are assigned to the first target node for execution.

[0121] like Figure 10 As shown in Figure (b), storage solution B indicates that warm data is stored in the first target node, hot data is stored in the second target node, and cold data is stored in the third target node. Since the second target node can process 20 query tasks in parallel, the remaining 10 query tasks need to be assigned to the third target node (the third target node has idle computing power). Therefore, combined with the above formula (1), we know that cost[first target node] = 20×1, cost[second target node] = 20×1, cost[third target node] = 10×1 + 10×5; based on this, the communication cost of storage solution B = 20×1 + 20×1 + 10×1 + 10×5 = 100. Among them, 10×5 is the additional communication cost, that is, the communication cost consumed after the query tasks corresponding to some hot data are assigned to the third target node for execution.

[0122] It can be seen that the communication cost of storage plan B is lower than that of storage plan A. Storing data according to storage plan B can effectively reduce the communication cost during data query, thereby improving data query efficiency.

[0123] Through the above steps A to B, the process of the first working node classifying and storing data according to the frequency of data use is introduced. Through this classified storage method, on the one hand, it can achieve the effect of expanding memory and data grading. On the other hand, it can reasonably allocate storage space in the memory pool and reduce communication latency.

[0124] In addition, reference Figure 11 , Figure 11 This is a structural diagram of a data query device provided by this embodiment, which can be configured in the management node involved in the above method embodiment. The device can realize part or all of the functions of the above management node through software, hardware or a combination of both. Figure 11As shown, the apparatus includes a receiving module 1101 , a first generating module 1102 , a second generating module 1103 and a sending module 1104 .

[0125] The receiving module 1101 is configured to receive a data query request, where the data query request includes multiple filtering conditions.

[0126] A first generating module 1102 is configured to generate multiple query tasks based on a data query request;

[0127] The second generating module 1103 is configured to generate a read request based on multiple filtering conditions, where the read request includes the multiple filtering conditions.

[0128] The sending module 1104 is configured to send the read request to a first working node among the multiple working nodes.

[0129] refer to Figure 12 , Figure 12 This is a structural diagram of another data query device provided by this embodiment, which can be configured on the working node involved in the above method embodiment. The device can realize part or all of the functions of the above first working node or second working node through software, hardware or a combination of both. Figure 12 As shown, the device includes an execution module 1201.

[0130] The execution module 1201 is used to execute the read request sent by the management node and read the data that meets multiple filtering conditions into the memory pool.

[0131] In some embodiments, the device further includes an acquisition module for acquiring data required to execute a corresponding query task from a memory pool.

[0132] In some embodiments, the device further includes a storage module for storing location information of the data corresponding to each filtering condition in the memory pool after reading the data that meets the multiple filtering conditions into the memory pool.

[0133] In some embodiments, the acquisition module is used to acquire data required to execute the corresponding query task from the memory pool based on the location information.

[0134] In some embodiments, the data queried by the data query request includes multiple data units; the storage module is used to store each filter condition and the location information of the data unit corresponding to each filter condition in the memory pool in correspondence with each other.

[0135] In some embodiments, the acquisition module is used to acquire data units required to execute the corresponding query task from the memory pool based on the location information.

[0136] In some embodiments, the execution module 1201 is used to execute a read request, classify data that meets multiple filtering conditions based on the data heat information of the data query request, and obtain multiple categories of data, where the data heat information indicates the frequency of data use during the execution of the data query request; and store the multiple categories of data in the storage space provided by different working nodes in the memory pool respectively.

[0137] Through the above-mentioned device, for data query requests involving multiple filter conditions, the management node in the distributed database system generates a query task corresponding to each filter condition, and generates a read request based on the multiple filter conditions. The first working node executes the read request and reads the data that meets the multiple filter conditions into the system's memory pool at one time. Based on this, the working nodes that execute multiple query tasks in the system can obtain the data required to execute the corresponding query tasks from the memory pool when executing their respective query tasks, avoiding redundant reading of data queried for data query requests, reducing data reading overhead, and improving data query efficiency.

[0138] It should be noted that the data query device provided in the above embodiment only uses the division of the above functional modules as an example to illustrate data query. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the data query device provided in the above embodiment and the data query method embodiment are based on the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0139] In this application, the terms "first", "second", etc. are used to distinguish between identical or similar items having substantially the same effects and functions. It should be understood that there is no logical or temporal dependency between "first", "second", and "nth", nor is there a limit on quantity and execution order. It should also be understood that although the following description uses the terms first, second, etc. to describe various elements, these elements should not be limited by the terms. These terms are only used to distinguish one element from another. For example, without departing from the scope of the various described examples, the first working node may be referred to as the second working node, and similarly, the second working node may be referred to as the first working node. Both the first working node and the second working node may be nodes, and in some cases, may be separate and different nodes.

[0140] In this application, the term "at least one" means one or more, and the term "plurality" means two or more. For example, a plurality of nodes means two or more nodes.

[0141] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and such modifications or substitutions should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

[0142] In the above embodiments, all or part of the embodiments may be implemented using software, hardware, firmware, or any combination thereof. When implemented using software, all or part of the embodiments may be implemented in the form of program structure information. The program structure information includes one or more program instructions. When the program instructions are loaded and executed on a computing device, all or part of the processes or functions described in the embodiments of the present application are generated.

[0143] Those skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware, or may be accomplished by a program to instruct the relevant hardware, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0144] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A data query method, characterized in that: Applied to a distributed database system, the distributed database system includes a management node, multiple working nodes, and a memory pool, the method includes: The management node receives a data query request and generates a plurality of query tasks based on the data query request, wherein the data query request includes a plurality of filtering conditions; The management node generates a read request based on the multiple filtering conditions, and sends the read request to a first working node among the multiple working nodes, where the read request includes the multiple filtering conditions; The first working node executes the read request and reads the data that meets the multiple filtering conditions into the memory pool; The working nodes executing the multiple query tasks obtain the data required to execute the corresponding query tasks from the memory pool.

2. The method according to claim 1, characterized in that The method further comprises: After reading the data that meets the multiple filtering conditions into the memory pool, the first working node stores the location information of the data corresponding to each filtering condition in the memory pool in the memory pool; The worker nodes executing the multiple query tasks obtain the data required to execute the corresponding query tasks from the memory pool, including: the worker nodes executing the multiple query tasks obtain the data required to execute the corresponding query tasks from the memory pool based on the location information.

3. The method according to claim 2, characterized in that The data queried by the data query request includes multiple data units; The first working node storing, in the memory pool, location information of data corresponding to each filtering condition in the memory pool, comprising: the first working node storing, in the memory pool, each filtering condition and location information of data units corresponding to each filtering condition in the memory pool in correspondence with each other; The working node that executes the multiple query tasks obtains the data required to execute the corresponding query tasks from the memory pool based on the location information, including: the working node that executes the multiple query tasks obtains the data units required to execute the corresponding query tasks from the memory pool based on the location information.

4. The method according to any one of claims 1 to 3, characterized in that If the distributed database system adopts a storage-computing integrated architecture, the first working node executes the read request and reads the data that meets the multiple filtering conditions into the memory pool, including: executing the read request, and classifying the data that meets the multiple filter conditions based on data popularity information of the data query request to obtain multiple categories of data, wherein the data popularity information indicates a frequency of use of the data during the execution of the data query request; The data of the multiple categories are respectively stored in the storage space provided by different working nodes in the memory pool.

5. The method according to any one of claims 1 to 4, characterized in that The second working node executing the first query task includes a processor and a computing unit, wherein the processor is used to offload the first operator in the first query task to the computing unit for execution, and the first operator is an operator in the first query task other than the read operator.

6. The method according to claim 5, characterized in that The first query task also includes a second operator in addition to the read operator and the first operator. The execution of the second operator depends on the execution result of the first operator. The computing unit is used to store the execution result of the first operator to the memory pool, and the processor is used to obtain the execution result of the first operator from the memory pool to execute the second operator.

7. A distributed database system, characterized in that: The system includes a management node, multiple working nodes and a memory pool; The management node is used to: receiving a data query request, and generating a plurality of query tasks based on the data query request, wherein the data query request includes a plurality of filtering conditions; Generate a read request based on the multiple filter conditions, and send the read request to a first working node among the multiple working nodes, wherein the read request includes the multiple filter conditions; The first working node is configured to execute the read request and read the data that meets the multiple filtering conditions into the memory pool; The working nodes that execute the multiple query tasks are used to obtain data required to execute the corresponding query tasks from the memory pool.

8. The system according to claim 7, characterized in that The first working node is further configured to, after reading the data that meets the multiple filtering conditions into the memory pool, store in the memory pool the location information of the data corresponding to each filtering condition in the memory pool; The working node that executes the multiple query tasks is used to obtain data required to execute the corresponding query tasks from the memory pool based on the location information.

9. The system according to claim 7 or 8, characterized in that The data queried by the data query request includes multiple data units; The first working node is configured to store, in the memory pool, each filtering condition and location information of a data unit corresponding to each filtering condition in the memory pool in correspondence with each other; The working node that executes the multiple query tasks is used to obtain the data units required to execute the corresponding query tasks from the memory pool based on the location information.

10. The system according to any one of claims 7 to 9, characterized in that If the distributed database system adopts a storage-computing integrated architecture, the first working node is used to: executing the read request, and classifying the data that meets the multiple filter conditions based on data popularity information of the data query request to obtain multiple categories of data, wherein the data popularity information indicates a frequency of use of the data during the execution of the data query request; The data of the multiple categories are respectively stored in the storage space provided by different working nodes in the memory pool.

11. The system according to any one of claims 7 to 10, characterized in that The second working node executing the first query task includes a processor and a computing unit, wherein the processor is used to offload the first operator in the first query task to the computing unit for execution, and the first operator is an operator in the first query task other than the read operator.

12. The system according to claim 11, wherein: The first query task also includes a second operator in addition to the read operator and the first operator. The execution of the second operator depends on the execution result of the first operator. The computing unit is used to store the execution result of the first operator to the memory pool, and the processor is used to obtain the execution result of the first operator from the memory pool to execute the second operator.

13. A management device, characterized in that: The management device includes a processor and a memory, and the processor is used to execute at least one program code stored in the memory, so that the management device can realize the function of the management node in the data query method as described in any one of claims 1 to 6.

14. A computer program product, characterized in that When the computer program product is run on a management device, the management device is enabled to implement the function of a management node in the data query method according to any one of claims 1 to 6.

Images