Data query method, device and distributed data management system

Abstract

This application discloses a data query method, apparatus, and distributed data management system. The method includes: receiving a data query job sent by a current user client; determining a resource pool corresponding to the data query job; determining a process corresponding to the data query job in the computing nodes corresponding to the resource pool; and sending the data query job to the process, so that the process sends the data query result corresponding to the data query job to the current user client. The distributed data management system includes multiple resource pools; each resource pool manages resources for each process within it based on multiple resource control groups. The method and apparatus provided in this application can rationally allocate resources in the distributed data management system without causing memory overflow, thereby improving the efficiency of data querying.

Description

Technical Field

[0001] This application relates to the field of database technology, and more specifically, to a data query method, apparatus, and distributed data management system. Background Technology

[0002] ClickHouse is a highly regarded open-source columnar database primarily used in the field of Online Analytic Processing (OLAP). Based on the needs of OLAP scenarios, ClickHouse has custom-developed a new, high-efficiency columnar storage engine and implements rich features such as ordered data storage, primary key indexes, sparse indexes, data sharding, data partitioning, and master-slave replication.

[0003] ClickHouse's execution framework is a simple two-phase (Scatter / Gather) approach, where scattered tasks are completed by a single node, resulting in poor support for multi-table joins. It lacks a distributed execution plan and can only handle multi-table joins through recursive subqueries and broadcast tables, making it unable to execute complex queries and prone to causing Out of Memory (OOM) errors. Summary of the Invention

[0004] Firstly, this application provides a data query method applied to a distributed data management system, the method comprising:

[0005] Receive data query jobs sent by the current user's client;

[0006] Determine the resource pool corresponding to the data query job;

[0007] Determine the process corresponding to the data query job in the computing node corresponding to the resource pool;

[0008] The data query job is sent to the process, so that the process sends the data query result corresponding to the data query job to the current user client;

[0009] The distributed data management system includes multiple resource pools; each resource pool manages resources for each process within it based on multiple resource control groups.

[0010] In some embodiments, determining the process corresponding to the data query job in the computing node corresponding to the resource pool includes:

[0011] Based on all jobs currently being executed in the distributed data management system, determine global job execution information;

[0012] If the number of job executions in the global job execution information is less than a first preset number, obtain the resource usage information of the resource pool;

[0013] If the resource requirement of the data query job is less than the available resource quantity in the resource usage information, the process corresponding to the data query job is determined in the computing node corresponding to the resource pool.

[0014] In some embodiments, after obtaining the resource usage information of the resource pool, the method further includes:

[0015] Determine the memory requirement within the resource requirements of the data query job;

[0016] If the memory requirement is greater than the available memory in the resource usage information and the memory requirement is less than the total memory in the resource usage information, the data query job will be added to the job waiting queue corresponding to the resource pool.

[0017] If the memory requirement is greater than the total memory in the resource usage information, the first data and the second data corresponding to the data query job are determined, and the first data is loaded into the memory space corresponding to the resource pool, and the second data is loaded into the storage node in the distributed data management system.

[0018] In some embodiments, determining the resource pool corresponding to the data query job includes:

[0019] Obtain the various user clients corresponding to the distributed data management system;

[0020] Each user client is grouped to determine multiple user groups corresponding to the distributed data management system;

[0021] Based on the number of the multiple user groups and the resource requirements of each user group, the resources in the distributed data management system are divided to determine the resource pool corresponding to each user group.

[0022] Based on the user group corresponding to the current user client and the resource pool corresponding to each user group, the resource pool corresponding to the data query job is determined.

[0023] In some embodiments, the resource pool is managed based on the following steps:

[0024] Based on the Cgroups mechanism, multiple resource control groups corresponding to the resource pool are determined; the resource control groups include at least one of processor usage control group, memory usage control group, and input / output read / write control group.

[0025] Resource management is performed on each process in the resource pool based on multiple resource control groups.

[0026] In some embodiments, the distributed data management system is built on the Clickhouse system.

[0027] Secondly, this application provides a data query device for use in a distributed data management system, the device comprising:

[0028] The receiving unit is used to receive data query jobs sent by the current user client;

[0029] The resource pool determination unit is used to determine the resource pool corresponding to the data query job;

[0030] A process determination unit is used to determine the process corresponding to the data query job in the computing node corresponding to the resource pool.

[0031] An execution unit is configured to send the data query job to the process, so that the process sends the data query result corresponding to the data query job to the current user client;

[0032] The distributed data management system includes multiple resource pools; each resource pool manages resources for each process within it based on multiple resource control groups.

[0033] Thirdly, this application provides a distributed data management system, including a control node, multiple computing nodes, and multiple storage nodes;

[0034] The storage node is connected to the computing node and is used to store data;

[0035] The computing node is connected to the control node and is used to execute data query jobs sent by the user client.

[0036] The control node is used to execute any of the data query methods described above.

[0037] Fourthly, this application provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method described in any of the above descriptions.

[0038] Fifthly, this application provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in any of the above descriptions.

[0039] The data query method, apparatus, and distributed data management system provided in this application, by determining the process corresponding to the data query job in the computing node corresponding to the resource pool and sending the data query job to the process, enables the process to send the data query result corresponding to the data query job to the current user client; by setting up multiple resource pools in the distributed data management system, the resources in the distributed data management system can be divided to realize the distributed execution of each process, and complex queries can be executed; by managing the resources of each process in the resource pool through multiple resource control groups, the resources in the distributed data management system can be reasonably allocated to each process, preventing memory overflow, improving the resource utilization efficiency of the distributed data management system, and improving the efficiency of data query. Attached Figure Description

[0040] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0041] To more clearly illustrate the technical solutions in this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0042] Figure 1 A flowchart illustrating a data query method provided in one embodiment of this application;

[0043] Figure 2 This is a flowchart illustrating a data query method provided in another embodiment of this application;

[0044] Figure 3 This is a schematic diagram of the structure of a data query device provided in one embodiment of this application;

[0045] Figure 4 This is a schematic diagram of the structure of a distributed data management system provided in one embodiment of this application;

[0046] Figure 5 This is a schematic diagram of the structure of an electronic device provided in one embodiment of this application. Detailed Implementation

[0047] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0048] It should be noted that the terms "first," "second," etc., used in this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0049] The data query method provided in this application is applicable to a distributed data management system, which may include a server or a server cluster.

[0050] Figure 1 This is a flowchart illustrating a data query method provided in one embodiment of this application, as shown below. Figure 1 As shown, the method includes steps 110, 120, 130, and 140. These method steps are merely one possible implementation of this application.

[0051] Step 110: Receive the data query job sent by the current user client.

[0052] Specifically, the data query method provided in this application embodiment is executed by a data query device, which can be a hardware device set in a distributed data management system or a software program running in a distributed data management system.

[0053] The current user client refers to the client currently being used by the user, which can be various electronic devices, including but not limited to smartphones, tablets, laptops, desktop computers, and servers.

[0054] The current user inputs query text or sends a voice signal to the current user client. The current user client parses the query text or voice signal input by the current user, obtains the data query job, and sends it to the data query device. The data query device receives the data query job sent by the current user client.

[0055] Step 120: Determine the resource pool corresponding to the data query job.

[0056] Specifically, a resource pool is a resource allocation mechanism for distributed database systems, used to divide the resources of a distributed data management system.

[0057] Different current user clients may correspond to different resource pools. Therefore, it is necessary to determine the current user client corresponding to the data query job. The resource pool corresponding to the current user client is the resource pool corresponding to the data query job.

[0058] Step 130: Determine the process corresponding to the data query job in the computing node corresponding to the resource pool.

[0059] Specifically, the resources in the resource pool include computing resources and storage resources, and the resource pool includes compute nodes and storage nodes. Compute nodes are used for reading and writing data; storage nodes are used for storing data. Compute nodes or storage nodes can be specific computers or servers.

[0060] Different processes run within a compute node. A process is the basic unit for resource allocation and scheduling in the system. Each process is responsible for executing different data query jobs. The process corresponding to a data query job can be determined within the compute nodes of the resource pool corresponding to that data query job. For example, any process can be created on an idle compute node in the resource pool to execute a data query job.

[0061] Step 140: Send the data query job to the process so that the process sends the data query results corresponding to the data query job to the current user client.

[0062] The distributed data management system includes multiple resource pools; each resource pool manages resources for its processes based on multiple resource control groups.

[0063] Specifically, the data query device sends the data query job to the process corresponding to the data query job, the process executes the data query job and obtains the data query result, and the process sends the data query result to the current user client.

[0064] The distributed data management system comprises multiple resource pools, each containing multiple compute nodes and multiple storage nodes. One resource pool can correspond to multiple clients.

[0065] The compute nodes in a resource pool may contain multiple data query jobs. To improve data query efficiency, multiple resource control groups are set up to manage the resources of each process in the resource pool. Resource control groups (Cgroups) are mainly used to isolate, restrict, and audit system resources.

[0066] The data query method provided in this application, by determining the process corresponding to the data query job in the computing node corresponding to the resource pool and sending the data query job to the process, enables the process to send the data query result corresponding to the data query job to the current user client; by setting up multiple resource pools in the distributed data management system, the resources in the distributed data management system can be divided to realize the distributed execution of each process, and complex queries can be executed; by managing the resources of each process in the resource pool through multiple resource control groups, the resources in the distributed data management system can be reasonably allocated to each process, preventing memory overflow, improving the resource utilization efficiency in the distributed data management system, and improving the efficiency of data query.

[0067] It should be noted that each implementation method of this application can be freely combined, rearranged, or executed individually, and does not need to rely on or depend on a fixed execution order.

[0068] In some embodiments, step 130 includes:

[0069] Determine global job execution information based on all jobs currently executing in the distributed data management system;

[0070] If the number of jobs executed in the global job execution information is less than the first preset number, obtain the resource usage information of the resource pool;

[0071] If the resource requirement of a data query job is less than the available resources in the resource usage information, the process corresponding to the data query job is determined in the computing node corresponding to the resource pool.

[0072] Specifically, global job execution information refers to the execution information of all jobs currently being executed by the distributed data management system, including the number of jobs being executed and their execution progress. Global job execution information is determined based on all jobs currently being executed in the distributed data management system.

[0073] The first preset quantity is the maximum number of data query jobs that the distributed data management system can process simultaneously.

[0074] Resource usage information refers to the computing and storage resources that the resource pool is currently using and that are still available. Resource requirements are the amount of resources needed to execute a data query job, including the amount of memory and computing resources required to execute the data query job; that is, resource requirements include both memory and computing requirements.

[0075] If the number of jobs executed in the global job execution information is equal to the first preset number, it indicates that the number of data query jobs currently being executed by the distributed data management system has reached the maximum number of data query jobs that the distributed data management system can process simultaneously. At this time, the distributed data management system can no longer process new data query jobs. In this case, it obtains the resource pool corresponding to the data query job sent by the current user client and adds the data query job sent by the current user client to the job waiting queue of the corresponding resource pool.

[0076] If the number of jobs executed in the global job execution information is less than the first preset number, it indicates that the number of data query jobs currently being executed by the distributed data management system has not yet reached the maximum number of data query jobs that the distributed data management system can process simultaneously. At this time, the distributed data management system can still process new data query jobs. Then, it obtains the resource pool corresponding to the data query job sent by the current user client, as well as the resource usage information of the resource pool.

[0077] Determine the resource requirements of the data query job. If the resource requirements of the data query job are greater than or equal to the available resources in the resource usage information, that is, the available resources of the current resource pool cannot support the compute node to execute the data query job, then add the data query job sent by the current user client to the job waiting queue of the resource pool, or use resources from other resource pools to process the data query job according to the actual situation.

[0078] If the resource requirement of a data query job is less than the available resources in the resource usage information, meaning that the available resources in the current resource pool can support the compute nodes to execute the data query job, then the process corresponding to the data query job is determined in the compute nodes of the resource pool.

[0079] The data query method provided in this application can determine whether the resources of the current distributed data management system support the execution of the data query job by obtaining global job execution information and a first preset quantity; it can also determine whether the resources of the resource pool support the execution of the data query job by obtaining resource usage information. If not, the data query job sent by the current user client is added to the job waiting queue of the corresponding resource pool. This can prevent memory overflow, prevent data query jobs from being blocked in the resource pool, and prevent system alarms, thereby improving the efficiency of data query.

[0080] In some embodiments, after obtaining resource usage information of the resource pool, the data query method further includes:

[0081] Determine the memory requirement within the resource requirements of the data query job;

[0082] If the memory requirement is greater than the available memory in the resource usage information but less than the total memory in the resource usage information, the data query job will be added to the job waiting queue corresponding to the resource pool.

[0083] If the memory requirement exceeds the total memory in the resource usage information, determine the first and second data corresponding to the data query job, load the first data into the memory space corresponding to the resource pool, and load the second data into the storage node in the distributed data management system.

[0084] Specifically, the job waiting queue is used to place data query jobs that the data query device has received but not yet processed. Once the available resources in the distributed data management system and the resource pool corresponding to the data query job are sufficient to support the processing of the data query job, the data query device retrieves the data query job from the job waiting queue and executes the data query job through the process corresponding to the data query job.

[0085] Determine the memory requirement in the resource requirements of the data query job. If the memory requirement is greater than the available memory in the resource usage information but less than the total memory in the resource usage information, add the data query job to the job waiting queue corresponding to the resource pool and wait.

[0086] If the memory requirement exceeds the total memory in the resource usage information, in order to execute the data query job, this embodiment of the application can execute the data query job by using an operator-based disk approach.

[0087] The first and second data for the data query job are determined. The first data is the data required during the execution of the data query job, and this required data is loaded into the memory space corresponding to the resource pool. The second data is the data that is not required during the execution of the data query job. In order to prevent this unused data from occupying the memory space of the resource pool, the unused data is loaded into the storage node of the distributed data management system.

[0088] The distributed data management system is built on the Clickhouse system.

[0089] Clickhouse is a column-oriented database management system (DBMS) with an MPP (Online Analytical Processing) architecture for Online Analytical Processing (OLAP) queries, capable of generating analytical data reports in real time using SQL queries.

[0090] The data query method provided in this application determines the first and second data of the data query job by using an operator-based disk approach. The first data is loaded into the memory space corresponding to the resource pool, and the second data is loaded into the storage node in the distributed data management system. This method can occupy less memory space during the execution of the data query job and provide more memory space for other data query jobs, enabling the memory space to support the processing of more data query jobs and improving the efficiency of data query.

[0091] In some embodiments, step 120 includes:

[0092] Obtain the various user clients corresponding to the distributed data management system;

[0093] Group the various user clients to determine the multiple user groups corresponding to the distributed data management system;

[0094] Based on the number of multiple user groups and the resource requirements of each user group, the resources in the distributed data management system are divided to determine the resource pool corresponding to each user group.

[0095] Based on the user group corresponding to the current user client and the resource pool corresponding to each user group, determine the resource pool corresponding to the data query job.

[0096] Specifically, a distributed data management system may correspond to multiple user clients. By grouping these multiple user clients, multiple user groups can be obtained.

[0097] There are several ways to group users, such as obtaining the Internet Protocol Address (IP) of each user client and grouping them according to their IP addresses.

[0098] The number of resource pools is determined by the number of user groups. Each user group corresponds to at least one resource pool, and the number of user groups is less than or equal to the number of resource pools.

[0099] For example, one user group corresponds to one resource pool. If the various user clients are divided into 6 user groups, then the resources in the distributed data management system need to be divided into 6 parts to obtain 6 resource pools.

[0100] Based on the resource requirements of each user group, determine how to allocate resources in the distributed data management system. For example, user groups with high resource requirements need a resource pool with more resources, so more resources should be allocated to the resource pool corresponding to user groups with high resource requirements.

[0101] Based on the user group corresponding to the current user client and the resource pool corresponding to each user group, the resource pool corresponding to the data query job sent by the current user client can be obtained.

[0102] Among them, idle resources in the resource pool can be provided to other resource pools with less available resources; the data query job can be represented by Structured Query Language (SQL).

[0103] After dividing the resource pools, it is necessary to set the maximum concurrency of dynamic SQL queries, disk access, and memory space for each resource pool. The specific setting method includes: determining the estimated memory for executing SQL queries, setting the maximum concurrency of SQL queries, multiplying the estimated memory for executing SQL queries by the maximum concurrency of SQL queries to obtain the estimated total memory for all SQL queries, ensuring that the estimated total memory for all SQL queries is less than or equal to the total memory of the corresponding resource pool, dynamically monitoring the memory during SQL execution, and performing operator disk access control on the first and second data corresponding to the data query job when the memory exceeds the threshold. If the available resources after operator disk access control are still insufficient to support the execution of the data query job, the data query job is added to the job waiting queue corresponding to the resource pool for waiting.

[0104] The data query method provided in this application involves grouping various user clients into multiple user groups. Based on the number of user groups and the resource requirements of each user group, the resources in the distributed data management system are divided, allowing for reasonable allocation of resources. Different user groups use different resource pools to process data query jobs, enabling rapid processing of data query jobs and achieving high data query efficiency.

[0105] In some embodiments, resource pools are managed based on the following steps:

[0106] Based on the Cgroups mechanism, multiple resource control groups are determined for the resource pool; the resource control groups include at least one of the processor usage control group, memory usage control group, and input / output read / write control group.

[0107] Resource management is performed on each process in the resource pool using multiple resource control groups.

[0108] Specifically, Cgroups can define a subsystem for each type of resource that can be controlled.

[0109] For example, Cgroups can define a CPU subsystem, primarily limiting the CPU usage of processes; the cpuacct subsystem, which can report CPU usage of processes within the Cgroups; the cpuset subsystem, which can allocate individual CPU or memory nodes to processes within the Cgroups; a memory subsystem, which can limit the amount of memory used by processes; a blkio subsystem, which can restrict data read and write operations on block devices by processes; a devices subsystem, which can control process access to devices; a net_cls subsystem, which can mark network packets of processes within the Cgroups, allowing control of these packets using the tc module; a freezer subsystem, which can suspend or resume processes within the Cgroups; and an ns subsystem, which allows processes in different Cgroups to use different namespaces.

[0110] According to the Cgroups mechanism, this application defines multiple resource control groups for the resource pool; the resource control groups include at least one of the following: processor usage control group, memory usage control group, and input / output read / write control group.

[0111] Resource management is performed on each process in the resource pool using multiple resource control groups.

[0112] For example, the processor usage control group can collect CPU usage reports for each process; the memory usage control group can limit the memory usage of each process; and the input / output read / write control group can limit the data read and write operations of each process on block devices.

[0113] The data query method provided in this application embodiment can determine multiple resource control groups corresponding to the resource pool through the Cgroups mechanism; through multiple resource control groups, resource management can be performed on each process in the resource pool, providing protection for the execution process of each process and improving the stability of the process executing the data query job.

[0114] In some embodiments, Figure 2 This is a flowchart illustrating a data query method provided in another embodiment of this application, such as... Figure 2 As shown, the method includes:

[0115] Step 210: The current user client sends a data query job to the control node of the distributed data management system.

[0116] Step 220: Determine whether the number of jobs executed in the global job execution information is less than the first preset number.

[0117] Step 230: If the number of job executions in the global job execution information is less than the first preset number, then determine whether the resource requirement of the data query job is less than the resource availability of the resource pool.

[0118] Step 240: If the number of jobs executed in the global job execution information is greater than or equal to the first preset number, the data query job is added to the job waiting queue of the distributed data management system until the number of jobs executed in the global job execution information is less than the first preset number, and then step 230 is executed.

[0119] Step 250: If the resource requirement of the data query job is less than the available resources in the resource pool, then the processes in the resource pool are dynamically managed, and the data query results are sent to the control node.

[0120] Step 260: If the resource requirement of the data query job is greater than or equal to the available resources of the resource pool, add the data query job to the job waiting queue of the resource pool until the resource requirement of the data query job is less than the available resources of the resource pool, and then proceed to step 250.

[0121] The data query method provided in this application sends the data query job to the control node of the distributed data management system, determines whether the number of jobs executed in the global job execution information is less than a first preset number, and determines whether the resource requirement of the data query job is less than the resource availability of the resource pool. This method can execute complex queries and prevent memory overflow, thereby improving the efficiency of data query.

[0122] The data query device provided in the embodiments of this application is described below. The data query device described below and the data query method described above can be referred to and correspond to each other.

[0123] Figure 3 This is a schematic diagram of the structure of a data query device provided in one embodiment of this application, as shown below. Figure 3 As shown, the data query device is applied to a distributed data management system. The device includes a receiving unit 310, a resource pool determination unit 320, a process determination unit 330, and an execution unit 340.

[0124] The receiving unit 310 is used to receive data query jobs sent by the current user client.

[0125] The resource pool determination unit 320 is used to determine the resource pool corresponding to the data query job.

[0126] The process determination unit 330 is used to determine the process corresponding to the data query job in the computing node corresponding to the resource pool.

[0127] Execution unit 340 is used to send data query jobs to the process, so that the process sends the data query results corresponding to the data query job to the current user client;

[0128] The distributed data management system includes multiple resource pools; each resource pool manages resources for its processes based on multiple resource control groups.

[0129] Specifically, according to the embodiments of this application, any multiple units among the receiving 310, resource pool determining unit 320, process determining unit 330 and execution unit 340 can be merged into one unit, or any one of them can be split into multiple units.

[0130] Alternatively, at least some of the functions of one or more of these units may be combined with at least some of the functions of other units and implemented in one unit.

[0131] According to embodiments of this application, at least one of the receiving unit 310, the resource pool determination unit 320, the process determination unit 330, and the execution unit 340 can be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or any other reasonable means of integrating or packaging the circuitry, or implemented in any one of the three methods of software, hardware, and firmware, or in a suitable combination of any of these.

[0132] Alternatively, at least one of the receiving unit 310, the resource pool determining unit 320, the process determining unit 330, and the execution unit 340 may be at least partially implemented as a computer program unit, which can perform corresponding functions when the computer program unit is run.

[0133] The data query device provided in this application provides a method to send data query results to the current user client by determining the process corresponding to the data query job in the computing node corresponding to the resource pool and sending the data query job to the process. By setting up multiple resource pools in the distributed data management system, the resources in the distributed data management system can be divided to achieve distributed execution of various processes, enabling the execution of complex queries. By managing the resources of each process in the resource pool through multiple resource control groups, the resources in the distributed data management system can be reasonably allocated to each process, preventing memory overflow and improving the resource utilization efficiency and data query efficiency of the distributed data management system.

[0134] In some embodiments, the process determination unit is specifically used for:

[0135] Determine global job execution information based on all jobs currently executing in the distributed data management system;

[0136] If the number of jobs executed in the global job execution information is less than the first preset number, obtain the resource usage information of the resource pool;

[0137] If the resource requirement of a data query job is less than the available resources in the resource usage information, the process corresponding to the data query job is determined in the computing node corresponding to the resource pool.

[0138] In some embodiments, the process determination unit is further configured to:

[0139] Determine the memory requirement within the resource requirements of the data query job;

[0140] If the memory requirement is greater than the available memory in the resource usage information but less than the total memory in the resource usage information, the data query job will be added to the job waiting queue corresponding to the resource pool.

[0141] If the memory requirement exceeds the total memory in the resource usage information, determine the first and second data corresponding to the data query job, load the first data into the memory space corresponding to the resource pool, and load the second data into the storage node in the distributed data management system.

[0142] In some embodiments, the resource pool determination unit is specifically used for:

[0143] Obtain the various user clients corresponding to the distributed data management system;

[0144] Group the various user clients to determine the multiple user groups corresponding to the distributed data management system;

[0145] Based on the number of multiple user groups and the resource requirements of each user group, the resources in the distributed data management system are divided to determine the resource pool corresponding to each user group.

[0146] Based on the user group corresponding to the current user client and the resource pool corresponding to each user group, determine the resource pool corresponding to the data query job.

[0147] In some embodiments, the data query device further includes a management unit, which is used for:

[0148] Based on the Cgroups mechanism, multiple resource control groups are determined for the resource pool; the resource control groups include at least one of the processor usage control group, memory usage control group, and input / output read / write control group.

[0149] Resource management is performed on each process in the resource pool using multiple resource control groups.

[0150] In some embodiments, the distributed data management system is built on the Clickhouse system.

[0151] It should be noted that the data query device provided in this application embodiment can implement all the method steps implemented in the above data query method embodiment and can achieve the same technical effect. Here, the parts that are the same as those in the method embodiment and the beneficial effects will not be described in detail.

[0152] Figure 4 This is a schematic diagram of the structure of a distributed data management system provided in one embodiment of this application, as shown below. Figure 4 As shown, the distributed data management system may include:

[0153] Control node 410, multiple compute nodes 420 and multiple storage nodes 430;

[0154] Storage node 430, connected to compute node 420, is used to store data;

[0155] Compute node 420, connected to control node 410, is used to execute data query jobs sent by user clients;

[0156] Control node 410 is used to execute the data query method described above.

[0157] Specifically, when a user client sends a data query job to the distributed data management system, load balancing can be achieved through LVS (Linux Virtual Server). LVS is responsible for load balancing of high-traffic services, ensuring that the user client's data query jobs are evenly distributed across all computing nodes.

[0158] The control node can be implemented using Etcd technology, thereby ensuring the high availability of the control node.

[0159] The control node of the distributed data management system provided in this application embodiment can execute the above-described data query method. Its specific implementation method is consistent with the aforementioned method implementation method and can achieve the same beneficial effects, which will not be repeated here.

[0160] Figure 5 This is a schematic diagram of the structure of an electronic device provided in one embodiment of this application, as shown below. Figure 5As shown, the electronic device may include: a processor 510, a communications interface 520, a memory 530, and a communications bus 540, wherein the processor 510, the communications interface 520, and the memory 530 communicate with each other via the communications bus 540. The processor 510 can invoke logical operations in the memory 530 to execute a data query method, which includes:

[0161] Receive data query jobs sent by the current user's client;

[0162] Determine the resource pool corresponding to the data query job;

[0163] Determine the process corresponding to the data query job in the compute node corresponding to the resource pool;

[0164] Send the data query job to the process so that the process sends the data query results corresponding to the data query job to the current user client;

[0165] The distributed data management system includes multiple resource pools; each resource pool manages resources for its processes based on multiple resource control groups.

[0166] Furthermore, when the logical operations in the aforementioned memory can be implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several operations to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0167] The processor in the electronic device provided in this application embodiment can call logical instructions in the memory to implement the above method. Its specific implementation method is the same as the aforementioned method implementation method and can achieve the same beneficial effect, which will not be repeated here.

[0168] This application also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, is implemented to perform the methods provided in the above embodiments.

[0169] The specific implementation method is the same as the aforementioned method implementation method and can achieve the same beneficial effects, so it will not be repeated here.

[0170] This application provides a computer program product, including a computer program that, when executed by a processor, implements the method described above.

[0171] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0172] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0173] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A data query method, characterized by, Applied to a distributed data management system, the method includes: Receive data query jobs sent by the current user client; the data query jobs are in the form of Structured Query Language (SQL). Determine the resource pool corresponding to the data query job; Determine the process corresponding to the data query job in the computing node corresponding to the resource pool; The data query job is sent to the process, so that the process sends the data query result corresponding to the data query job to the current user client; The distributed data management system includes multiple resource pools; each resource pool manages resources for each process within it based on multiple resource control groups. Determining the process corresponding to the data query job in the computing node corresponding to the resource pool includes: Based on all jobs currently being executed in the distributed data management system, determine global job execution information; If the number of job executions in the global job execution information is less than a first preset number, the resource usage information of the resource pool is obtained; the first preset number is the maximum number of data query jobs that the distributed data management system can process simultaneously. After obtaining the resource usage information of the resource pool, the method further includes: Determine the memory requirement within the resource requirements of the data query job; Determining the memory requirement in the resource requirements of the data query job includes: determining the estimated memory required to execute the SQL, and determining the memory requirement based on the estimated memory. If the memory requirement is greater than the available memory in the resource usage information and the memory requirement is less than the total memory in the resource usage information, the data query job will be added to the job waiting queue corresponding to the resource pool. If the memory requirement is greater than the total memory in the resource usage information, determine the first data and the second data corresponding to the data query job, load the first data into the memory space corresponding to the resource pool, and load the second data into the storage node in the distributed data management system. The first data is the data required during the execution of the data query job; the second data is the data not required during the execution of the data query job.

2. The data query method according to claim 1, characterized in that, Determining the process corresponding to the data query job in the computing node corresponding to the resource pool includes: If the resource requirement of the data query job is less than the available resource quantity in the resource usage information, the process corresponding to the data query job is determined in the computing node corresponding to the resource pool.

3. The data query method according to claim 1, characterized in that, Determining the resource pool corresponding to the data query job includes: Obtain the various user clients corresponding to the distributed data management system; Each user client is grouped to determine multiple user groups corresponding to the distributed data management system; Based on the number of the multiple user groups and the resource requirements of each user group, the resources in the distributed data management system are divided to determine the resource pool corresponding to each user group. Based on the user group corresponding to the current user client and the resource pool corresponding to each user group, the resource pool corresponding to the data query job is determined.

4. The data query method according to any one of claims 1 to 3, characterized in that, The resource pool is managed based on the following steps: Based on the Cgroups mechanism, multiple resource control groups corresponding to the resource pool are determined; the resource control groups include at least one of processor usage control group, memory usage control group, and input / output read / write control group. Resource management is performed on each process in the resource pool based on multiple resource control groups.

5. The data query method according to any one of claims 1 to 3, characterized in that, The distributed data management system is built on the Clickhouse system.

6. A data query device, characterized in that, The device, used in a distributed data management system, includes: The receiving unit is used to receive data query jobs sent by the current user client; the data query job is in the form of Structured Query Language (SQL). The resource pool determination unit is used to determine the resource pool corresponding to the data query job; A process determination unit is used to determine the process corresponding to the data query job in the computing node corresponding to the resource pool. An execution unit is configured to send the data query job to the process, so that the process sends the data query result corresponding to the data query job to the current user client; The distributed data management system includes multiple resource pools; each resource pool manages resources for each process within it based on multiple resource control groups. Determining the process corresponding to the data query job in the computing node corresponding to the resource pool includes: Based on all jobs currently being executed in the distributed data management system, determine global job execution information; If the number of job executions in the global job execution information is less than a first preset number, the resource usage information of the resource pool is obtained; the first preset number is the maximum number of data query jobs that the distributed data management system can process simultaneously. After obtaining the resource usage information of the resource pool, the device further includes: Determine the memory requirement within the resource requirements of the data query job; Determining the memory requirement in the resource requirements of the data query job includes: determining the estimated memory required to execute the SQL, and determining the memory requirement based on the estimated memory. If the memory requirement is greater than the available memory in the resource usage information and the memory requirement is less than the total memory in the resource usage information, the data query job will be added to the job waiting queue corresponding to the resource pool. If the memory requirement is greater than the total memory in the resource usage information, determine the first data and the second data corresponding to the data query job, load the first data into the memory space corresponding to the resource pool, and load the second data into the storage node in the distributed data management system. The first data is the data required during the execution of the data query job; the second data is the data not required during the execution of the data query job.

7. A distributed data management system, characterized in that, It includes a control node, multiple compute nodes, and multiple storage nodes; The storage node is connected to the computing node and is used to store data; The computing node is connected to the control node and is used to execute data query jobs sent by the user client. The control node is used to execute the data query method according to any one of claims 1 to 5.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the data query method as described in any one of claims 1 to 5.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the data query method as described in any one of claims 1 to 5.

Images