Query method based on distributed computing and related apparatus

Abstract

The application discloses a query method and related device based on distributed computing, and relates to the technical field of data processing. Based on the load balancing between computing nodes, a subtask related to a query request is distributed to a thread in a computing node, so that the computing node can be fully utilized to improve the utilization rate of hardware. The thread of the subtask is independently bound to a CPU core, which is conducive to fully utilizing hardware resources. In the case that the execution data required by any subtask cannot be obtained, the thread of the subtask is suspended, and after the data is obtained, the thread of the subtask is woken up, and the machine code of the subtask is executed, which is conducive to improving the running efficiency to reduce the delay. Based on the query results of each subtask, the query result of the query request is obtained. In summary, not only can the query delay be reduced, but also the utilization rate of multi-core and the optimization of scheduling can be improved, and the adaptation degree of the query process and hardware can be improved, so that the computing performance in the query scene can be improved.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a query method and related apparatus based on distributed computing. Background Technology

[0002] With the surge in data volume, the performance of current computing architectures needs improvement, especially when the computing architecture is applied to multi-user, high-concurrency query scenarios. The performance cannot meet the query requirements and needs to be improved. For example, query latency needs to be reduced and the compatibility with hardware resources needs to be improved. Summary of the Invention

[0003] In view of the above problems, this application provides a query method and related apparatus based on distributed computing to reduce query latency and improve adaptability to hardware resources. The specific solution is as follows:

[0004] The first aspect of this application provides a query method based on distributed computing, including:

[0005] Obtain the native machine code of the subtasks related to the query request;

[0006] Based on load balancing among computing nodes, the first subtask in the subtask is assigned to the first thread in the computing node, and the first thread is independently bound to the first central processing unit CPU core.

[0007] If the data required to execute the first subtask cannot be obtained, the first thread is suspended. After the data is obtained, the first thread is woken up, and the native machine code of the first subtask is executed based on the first thread running on the CPU core to obtain the execution result of the first subtask. The CPU core includes the first CPU core or the second CPU core.

[0008] Based on the query results of the subtask, the query result of the query request is obtained.

[0009] In one possible implementation, prior to acquiring the data, the following is also included:

[0010] Unbind the first thread from the first CPU core to release the first CPU core;

[0011] Before waking up the first thread, the following is also included:

[0012] Bind the first thread to the second CPU core.

[0013] In one possible implementation, the first thread is independently bound to the first CPU core, including:

[0014] Based on load balancing across CPU cores, the first thread is independently bound to the first CPU core that is in an idle state.

[0015] In one possible implementation, assigning the first subtask in the subtasks to the first thread in the compute node includes:

[0016] The first subtask in the subtask is assigned to the first thread in the computation-independent thread pool of the compute node, where the threads are dedicated to CPU computation and not to input / output I / O operations.

[0017] In one possible implementation, acquiring the data includes:

[0018] Determine the first route based on the load of the CPU cores bound to the first subtask;

[0019] Based on the first route, the data is transmitted to the CPU core bound to the first subtask, and the data is data after invalid data removal.

[0020] In one possible implementation, the native machine code executing the first subtask includes:

[0021] Read the data blocks that match the target filtering conditions and skip the data blocks that do not match the target filtering conditions.

[0022] A second aspect of this application provides a query device based on distributed computing, comprising: a module for implementing the method of the first aspect or any implementation thereof.

[0023] A third aspect of this application provides a computer program product including computer-readable instructions that, when executed on an electronic device, cause the electronic device to implement the method described in the first aspect or any implementation thereof.

[0024] A fourth aspect of this application provides an electronic device, including at least one processor and a memory connected to the processor, wherein:

[0025] The memory is used to store computer programs;

[0026] The processor is used to execute the computer program to enable the electronic device to implement the method of the first aspect or any implementation thereof.

[0027] The fifth aspect of this application provides a computer storage medium carrying one or more computer programs, which, when executed by an electronic device, enable the electronic device to perform the method described in the first aspect or any implementation thereof.

[0028] By employing the above technical solutions, the query method and related apparatus based on distributed computing provided in this application, based on load balancing among computing nodes, allocate subtasks related to query requests to threads within computing nodes, thereby fully utilizing computing nodes to improve hardware utilization. Each subtask thread is independently bound to a CPU core, which is beneficial for fully utilizing hardware resources. Furthermore, if the execution data required by any subtask cannot be obtained, the thread of that subtask is suspended, and after obtaining the data, the thread of that subtask is awakened and its machine code is executed, which helps improve operating efficiency and reduce latency. Based on the query results of each subtask, the query result of the query request is obtained. In summary, this not only reduces query latency but also improves the adaptability of the query process to the hardware by increasing multi-core utilization and optimizing scheduling, thereby improving computational performance in query scenarios. Attached Figure Description

[0029] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the originals and elements are not necessarily drawn to scale.

[0030] Figure 1 A structural example diagram of a query device based on distributed computing provided in this application;

[0031] Figure 2 A flowchart illustrating a query method based on distributed computing provided in this application;

[0032] Figure 3 A flowchart illustrating another query method based on distributed computing provided in this application;

[0033] Figure 4 A flowchart illustrating another query method based on distributed computing provided in this application;

[0034] Figure 5 A structural example diagram of an electronic device provided in this application. Detailed Implementation

[0035] The embodiments of this application are described below with reference to the accompanying drawings. The terminology used in the implementation section of this application is for explaining specific embodiments only and is not intended to limit the scope of this application.

[0036] The embodiments of this application will now be described with reference to the accompanying drawings. Those skilled in the art will recognize that, with technological advancements and the emergence of new scenarios, the technical solutions provided in the embodiments of this application are equally applicable to similar technical problems.

[0037] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms are interchangeable where appropriate; this is merely a way of distinguishing objects with the same attributes in the embodiments of this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a series of elements is not necessarily limited to those elements, but may include other elements not explicitly listed or inherent to those processes, methods, products, or apparatuses.

[0038] The inventors' research revealed that current computing architectures suffer from the following core pain points in scenarios involving querying massive amounts of data:

[0039] 1. Significant latency: For example, a dashboard preview of 10 million rows of data takes tens of seconds, and a query of 100 million rows of data is prone to timeout, which cannot meet users' timeliness requirements for decision-making.

[0040] 2. Insufficient compatibility with hardware resources: Currently, server CPUs generally have more than 32 cores, providing the hardware foundation to support large-scale parallel computing. However, the current computing architecture cannot adequately adapt to hardware resources, mainly in the following aspects:

[0041] (1) Insufficient multi-core utilization: Traditional computing architectures (such as StarRocks or Trino) use thread pool scheduling and iterator models. The former allocates tasks through a fixed thread pool, resulting in a mismatch between the number of threads and the number of cores and easy thread contention. The latter requires operators to pull data row by row, and frequent function calls will clear CPU registers, resulting in poor data locality. It is this core contradiction of "mismatch between hardware characteristics and scheduling and execution models" that makes the CPU utilization of traditional architectures only about 30%, with a large number of cores in an idle state, directly causing a serious waste of hardware potential and failing to take advantage of the parallel computing of multi-core CPUs.

[0042] (2) Defects in scheduling mechanism: The thread pool scheduling mode of traditional architecture takes "task retrieval" as the core logic. Task execution depends on the active allocation of the thread pool, and the thread and task are strongly bound. However, in distributed computing, the Exchange operator has a natural data dependency relationship (such as downstream nodes needing to wait for the Shuffle data of upstream nodes). When multiple nodes form a circular dependency, the bound threads will be blocked. This "passive retrieval scheduling logic" conflicts with the "data dependency characteristics of the Exchange operator", which will not only cause true deadlock, but also cause a large number of threads to wait for resources and not release the CPU, resulting in CPU idle time, which seriously reduces the scheduling efficiency and stability of the system.

[0043] Based on this, embodiments of this application provide a query device and method based on distributed computing, with the aim of constructing a three-in-one parallel computing scheduling architecture of "multi-core adaptation-event-machine code execution". This architecture can significantly solve the above-mentioned pain points when used for querying.

[0044] Figure 1 A schematic diagram of a query device based on distributed computing is shown. For example... Figure 1 As shown, it includes: a front-end access layer, an optimizer module, a compilation engine module, a coordinator module, compute nodes (Workers), a storage access module, and a scheduling framework module. Optionally, the device may also include: an initialization module, a monitoring module, a cleanup module, and an archiving module. It is understood that... Figure 1 The lines in the diagram are just examples to represent interactions between modules. Modules without lines do not necessarily mean there is no interaction, and consecutive modules are not necessarily meant to have interaction.

[0045] The front-end access layer communicates with the Business Intelligence (BI) system. The front-end access layer receives query requests from the BI system and performs syntax parsing and preliminary validation. In other words, the front-end access layer serves as the user's interaction point with the system, shielding the underlying technical details.

[0046] The optimizer module is used to generate the optimal execution plan, which includes a logical execution plan and a physical execution plan.

[0047] The compilation engine module is used to convert the physical execution plan into an LLVM intermediate representation (IR), which is then optimized and compiled into native machine code to reduce the interpretation overhead during execution and improve execution efficiency.

[0048] The coordinator module is used for task splitting, node scheduling, status monitoring and result aggregation to achieve distributed node collaboration and ensure the orderly execution of tasks.

[0049] A compute node consists of multiple compute nodes, each of which runs a compute engine (such as a C++ compute engine) and a multi-threaded executor to execute compiled machine code. In other words, compute nodes undertake core computing tasks and use multi-core CPUs to process data in parallel.

[0050] The storage access module is compatible with distributed storage such as S3 or HDFS, providing a data read interface and supporting a data read mechanism that pushes back the filter conditions, so as to achieve efficient data access under the storage-compute separation architecture.

[0051] The scheduling framework module is used to manage the task lifecycle based on an event-driven mechanism, including at least one of the following operations: execution suspension, wake-up, and cancellation of scheduling, in order to avoid deadlock and CPU idling and improve the resource utilization of the application.

[0052] The following section provides a detailed explanation of how the above modules implement a query method based on distributed computing.

[0053] Figure 2 This is a flowchart illustrating a query method based on distributed computing provided in an embodiment of this application. Figure 2 The process includes the following steps:

[0054] S11. Obtain the native machine code of the subtasks related to the query request.

[0055] In this embodiment, the tasks related to the query request are divided into subtasks, and the native machine code of the subtasks is directly obtained. For details of the implementation, please refer to S21-S22.

[0056] Dividing tasks into subtasks facilitates the concurrent execution of multiple subtasks.

[0057] S12. Based on load balancing among computing nodes, the first subtask in the subtask is assigned to the first thread in the computing node, and the first thread is independently bound to the first CPU core.

[0058] The first thread can be any thread. The first CPU core can be any CPU core.

[0059] The first thread being independently bound to the first CPU core means that the first CPU core is not bound to any other threads besides the first thread. In other words, the first CPU core is used and only used to run the first thread. It can be understood that other threads are also independently bound to a CPU core. It is clear that dividing tasks into subtasks and binding each to a CPU core is beneficial for making full use of CPU cores.

[0060] Load balancing subtask allocation among compute nodes refers to distributing subtasks to each compute node, such as a worker node, based on the load of each compute node to achieve load balancing among the compute nodes. Combined with the binding relationship between threads and CPU cores, load balancing among CPU cores can be achieved, further improving the utilization of CPU cores.

[0061] For details on the implementation of S12, please refer to S23.

[0062] S13. If the data required to execute the first subtask cannot be obtained, suspend the first thread. After obtaining the data, wake up the first thread and execute the native machine code of the first subtask based on the first thread running on the CPU core to obtain the execution result of the first subtask.

[0063] In one implementation, if a thread executing any subtask fails to acquire the required data for that subtask, the thread is suspended. Once the data is acquired, the thread is then reawakened to execute the subtask. It's understandable that this implementation does not change the CPU core to which the subtask is bound; that is, the awakened thread still runs on the first CPU core.

[0064] In another implementation, for a thread executing any subtask, if the data required by that subtask is not obtained (i.e., unobtainable), the thread is suspended, and the first CPU core bound to that thread is released. After the data is obtained, a new CPU core, called the second CPU core, is bound to the thread, and the thread is awakened to execute the subtask on the second CPU core. It can be understood that in this implementation, the first CPU core bound to the unrunnable thread is released, allowing the first CPU core to be used for other operations, thereby further improving the utilization of hardware resources and avoiding CPU idling.

[0065] In some implementations, CPU cores are bound to any thread based on the CPU core load. For example, threads are preferentially bound to idle CPU cores to achieve load balancing among CPU cores.

[0066] As can be seen from the above implementation method, this step adopts an event-driven scheduling mechanism that dynamically allocates threads based on CPU core load, suspends threads when data cannot be obtained, and wakes them up again after data is obtained. This mechanism can not only make full use of CPU cores, but also resolve the coupling conflict between thread pool scheduling and data. For example, in existing technologies, the dependency conflict between thread pool scheduling and data can lead to a false deadlock rate of ≥15% and a CPU idle rate of ≥8%. However, by adopting the implementation method in this step, the core conflict in the scheduling process is resolved, making the task scheduling process stable and smooth, and avoiding subsequent calculation stage stuttering caused by scheduling blocking.

[0067] Understandably, this step executes raw machine code, which reduces the number of code interpretation steps compared to existing interpreted execution methods, thus reducing execution latency.

[0068] For example, this step is performed by the scheduling framework module.

[0069] S14. Based on the query results of each subtask, obtain the query results of the query request.

[0070] The method provided in this embodiment has the following beneficial effects:

[0071] 1. The native machine code of the subtask is directly obtained, and the native machine code of the first subtask is directly executed by the first thread running on the CPU core. It can be seen that the native machine code running on the CPU core reduces the execution latency compared with the interpreted execution method.

[0072] 2. If the required data cannot be obtained when executing any subtask, the thread used to execute that subtask is suspended. Only after the required data is obtained is the thread awakened and run on the CPU core, which reduces the CPU idle rate and improves the CPU utilization.

[0073] 3. Breaking the query request into subtasks and binding the threads executing the subtasks to CPU cores independently facilitates the concurrent execution of each subtask, thereby further improving CPU utilization.

[0074] In summary, the method provided in this embodiment can not only reduce query latency, but also improve the adaptability of the query process to the hardware by improving the utilization of multi-cores and optimizing scheduling, thereby improving the computing performance in query scenarios.

[0075] The following will combine Figure 3 ,right Figure 2 The process shown will be explained in more detail.

[0076] Figure 3 This is a flowchart of another query method based on distributed computing provided in the embodiments of this application. Figure 3 The process includes the following steps:

[0077] S21. Based on the query request from the user, generate a logical execution plan and a physical execution plan.

[0078] In other words, this step parses and optimizes the query requests from users, specifically including the following two aspects:

[0079] 1. After receiving the request, the front-end access layer generates a structured query abstract syntax tree (AST).

[0080] For example, a request is an SQL query initiated by a user through a BI tool, such as "SELECT A, SUM (D) FROM sales WHERE B='b1' GROUP BY A".

[0081] For example, after receiving a request, the front-end access layer performs syntax parsing and validity verification, and generates an AST by extracting query fields, filtering conditions, grouping keys and aggregation functions.

[0082] 2. The optimizer module generates logical execution plans and physical execution plans based on the AST.

[0083] For example, a logical optimization of the AST is performed based on cost optimization (CBO) to generate a logical execution plan.

[0084] For example, a logical execution plan includes pushdown filtering conditions, task splitting strategies, and other optimization information. A physical execution plan includes task splitting rules and data sharding strategies. For instance, a task is split into subtasks, and the task splitting rules indicate the rules for splitting the task into subtasks. Correspondingly, based on each subtask, the data required for task execution is divided into data shards required by each subtask, and the data sharding strategy indicates the rules for dividing the data into data shards.

[0085] S11 is a front-end preparation step, and the generated logical execution plan and physical execution plan are used to lay the foundation for subsequent steps.

[0086] S22. Obtain the native machine code based on the logical execution plan.

[0087] For example, the process of obtaining native machine code includes: the compilation engine module converts the operator logic (such as filtering and aggregation) in the logical execution plan into LLVM intermediate representation (IR) through custom mapping rules, optimizes the IR (such as constant folding and dead code elimination), and then calls the LLVM JIT compiler to compile the IR into native machine code.

[0088] By using the LLVM JIT compiler, the locality of instruction execution can be optimized, allowing data to reside in CPU registers for longer periods and reducing the frequent migration of data between memory and registers.

[0089] In this step, the logical execution plan is directly converted into native machine code. In subsequent execution steps, there is no need to parse instructions, which reduces the overhead of the execution process, thereby reducing latency and improving execution efficiency.

[0090] S23. Based on load balancing among computing nodes, subtasks related to query requests are assigned to threads (i.e., the first thread) in the computing node's independent thread pool, and the first thread is independently bound to the first CPU core.

[0091] For example, the subtasks related to the query request are obtained by the coordinator module based on the physical execution plan.

[0092] A compute-independent thread pool refers to a thread pool dedicated to CPU computation. Threads in this pool are not used for I / O operations (such as data reading) (memory-computation separation) to avoid the blocking of computation threads caused by the "mixing of computation and I / O threads" in existing technologies. Therefore, the first thread is dedicated to executing the subtasks assigned to it, and not for I / O operations.

[0093] For example, assigning a subtask to a computing node means associating the information of the subtask and the information of the data required to execute the subtask with the information of the computing node. The data information includes the aforementioned data shard information and may also include cached data information. In other words, the data required to execute the subtask may come from data shards of the actual data source or from cached data in the caching system, with the aim of improving the performance of data retrieval.

[0094] For example, this step is performed by the coordinator module.

[0095] S24. Based on the data required to execute the subtask, concurrently execute the machine code of the subtask to obtain the processing result of the subtask.

[0096] For example, the machine code executing a subtask includes executing machine code to perform computational operations such as filtering, aggregation, and joining of data, generating local intermediate results (i.e., the processing results of the subtask). As mentioned earlier, the object of execution is machine code, which can improve execution efficiency and reduce execution latency compared to interpreted execution.

[0097] For example, before transmitting the data required for subtask execution to the CPU core, invalid data is filtered out. Furthermore, based on the load of the CPU core receiving the data, the data transmission route is optimized to reduce the consumption of core computing resources by invalid transmissions and ensure multi-core computing efficiency. For the first subtask, i.e., based on the load of the CPU core bound to the first subtask, a first route is determined. Based on the first route, data is transmitted to the CPU core bound to the first subtask. This data is the data after invalid data has been removed.

[0098] For example, caching hot data on compute nodes can reduce the frequency of repeated access to the data storage layer, further reducing the latency of storage-compute interaction.

[0099] For example, in the process of executing subtasks, the storage access module reads data blocks that match the pushdown filter conditions (also known as target filter conditions) and skips (i.e., does not read) data blocks that do not match the pushdown filter conditions. This improves data reading efficiency and reduces latency. The pushdown filter conditions refer to filter conditions pushed from the upper layer to the storage access module. These filter conditions are obtained based on query requests. The step of skipping data blocks that do not match the pushdown filter conditions, combined with the feature of pre-sorting data blocks according to certain rules, achieves better results, namely, skipping more data blocks and thus obtaining greater latency reduction benefits.

[0100] For example, in the process of executing subtasks, after the upstream operator generates data, it actively sends it to the downstream operator. The operator only caches a very small batch of intermediate states (instead of materializing the entire data). The data forms a continuous streaming flow between operators. Compared with the existing method of downstream operators pulling data from upstream operators in the process of "upstream materialization → downstream pull → downstream materialization → downstream pull", this method only maintains a very small batch of incremental states within the operator and does not perform any storage operations on the entire intermediate results. This completely avoids the problems of large memory and disk usage, high data flow latency, and large I / O overhead caused by "intermediate result materialization" in the existing technology.

[0101] S25. If the data required to execute the subtask cannot be obtained, suspend the first thread. After obtaining the data, wake up and execute the first thread.

[0102] For details on the implementation of S24-S25, please refer to S13.

[0103] It is understandable that during the process of waking up and executing the first thread, other threads executing other subtasks may also be executing, that is, the first thread and other threads are executing concurrently.

[0104] S26. Based on the processing results of each subtask, obtain the query results of the query request.

[0105] For example, the coordinator module collects the local intermediate results of all worker nodes, performs global aggregation, performs data consistency verification and format conversion on the aggregation results, and transmits the final result to the front-end access layer.

[0106] For example, the coordinator caches the query results of high-frequency query requests, as well as the processing results of subtasks, laying the foundation for subsequent identical query requests to directly retrieve query results from the cache, thus reducing the consumption of system resources.

[0107] The method provided in this embodiment has the following beneficial effects:

[0108] 1. Obtaining native machine code based on the logical execution plan allows for direct execution of the native machine code, avoiding the instruction parsing overhead of interpreted execution from the source. Furthermore, data is actively pushed to downstream operators directly after being generated by upstream operators, and operators only cache a very small batch of computation states (rather than materializing the entire dataset), avoiding I / O read / write latency caused by intermediate result materialization. All of these factors contribute to reducing execution latency. Research has found that the average latency for queries involving billions of data points is reduced by 30%-60%, the performance of high-selectivity queries is improved by over 90%, the latency for dashboard previews of 10 million rows of data is compressed to the second level, and queries of 100 million rows of data are completed stably without timeout, meeting the timeliness requirements of enterprise decision-making.

[0109] 2. Separate the CPU computation thread pool from the I / O thread pool. I / O operations (such as data reading) are assigned to independent thread pools for execution, avoiding the computation thread blocking caused by the "mixed programming of computation and I / O threads" in the traditional process. After breaking down the query task into fine-grained subtasks, they are precisely allocated to idle CPU cores to achieve load balancing. The one-to-one binding between threads and CPU cores avoids the waste of core resources caused by thread switching, allowing the potential of multi-cores to be released. Research has found that the utilization rate of CPUs with more than 32 cores has increased from below 10% to more than 85%, fully releasing the potential of multi-core hardware and completely solving the problem of hardware resource waste.

[0110] 3. Whether it's assigning subtasks to computing nodes or binding threads to CPU cores, the goal is load balancing, thus achieving global load balancing. Furthermore, suspending threads when data is not being retrieved and caching query results improve query concurrency. Research shows that with more than 200 users accessing the site simultaneously, throughput in simple query scenarios increases by 100%. There are no downtimes and resource recovery is normal under high concurrency, making it suitable for enterprise-wide analysis and peak business access scenarios.

[0111] 4. Removing invalid data before transmitting it to threads helps reduce cross-node data transfer volume. Compute nodes cache hot data, reducing cross-node transfer volume and connection loss from the source, thereby improving the efficiency of storage-compute collaboration. Research has found that cross-node data transfer latency is reduced by 50% and computing-storage connection loss is reduced by 35% under a storage-compute separation architecture. Compute nodes can be independently and elastically expanded to adapt to the dynamic resource needs of enterprises.

[0112] Figure 4 This is a flowchart of another query method based on distributed computing provided in this application embodiment. Compared with the above embodiments, this embodiment adds some auxiliary steps. Figure 4 The following new steps are included:

[0113] S31. Perform metadata initialization when the system starts up.

[0114] The specific steps for performing metadata initialization are as follows:

[0115] a. Load distributed storage metadata. Examples of distributed storage metadata include: file paths for data storage in S23, data sharding information, table sort keys, and block-level statistics of data blocks.

[0116] b. Read the node's configuration information, such as the number of CPU cores, memory threshold, and communication address of the Worker node.

[0117] c. Initialize the caching strategy, including the memory capacity of the multi-level cache, the disk cache path, and the hot data statistics rules.

[0118] d. Establish communication connections between the coordinator module and the worker nodes, and between the storage access module and the distributed storage.

[0119] This step is performed by the initialization module and lays the foundation for S23.

[0120] S32, Monitor task status.

[0121] Specifically, it includes the following steps:

[0122] a. Monitor the IR generation and machine code compilation time of the compilation engine, and record the query plan for compilation failures.

[0123] b. Track the progress of subtask allocation and data reading latency, and mark worker nodes with excessive load.

[0124] c. Monitor the CPU utilization, memory usage, and local intermediate result generation status of the Worker nodes.

[0125] d. Record the data consistency verification results and result transmission delay of the global aggregation.

[0126] This step is executed by the monitoring module and can provide data support for troubleshooting query anomalies (such as memory overflow or aggregation failure).

[0127] S33. Clean up intermediate results.

[0128] For example, specifically including:

[0129] a. After the multi-threaded executor of the Worker node sends the local intermediate results to the coordinator module, it clears the intermediate data in the memory buffer.

[0130] b. Release the bound CPU cores and temporary thread resources, and reset the operator execution state.

[0131] c. If intermediate results are temporarily written to disk, delete the corresponding temporary files.

[0132] This step is performed by the cleanup module and aims to save storage space on the Worker nodes.

[0133] S34, Archived Results Log.

[0134] For example, specifically including:

[0135] a. After receiving the final query results from the coordinator, the front-end access layer synchronizes and archives the result data (including query SQL, execution time, and result set snapshot).

[0136] b. Associate task monitoring logs to generate a complete query chain report.

[0137] c. Clean up expired result logs according to a preset cycle to release storage resources.

[0138] This step is performed by the archiving module and provides a basis for data backtracking.

[0139] The process described in this embodiment can monitor the query process and provide a backtracking function, thereby improving the availability of the query system.

[0140] This application also provides an electronic device in its embodiments. (See reference...) Figure 5 The diagram illustrates a structural schematic suitable for implementing the electronic device in the embodiments of this application. The electronic device in the embodiments of this application may include, but is not limited to, fixed terminals such as laptops, PDAs (Personal Digital Assistants), PADs (Tablet PCs), desktop computers, etc. Figure 5 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0141] like Figure 5 As shown, the electronic device may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage device 608 into a random access memory (RAM) 603. When the electronic device is powered on, the RAM 603 also stores various programs and data required for the operation of the electronic device. The processing unit 601, ROM 602, and RAM 603 are interconnected via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0142] Typically, the following devices can be connected to I / O interface 605: input devices 606 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 607 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 608 including, for example, memory cards, hard drives, etc.; and communication devices 609. Communication device 609 allows electronic devices to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 5 Electronic devices with various devices are shown, but it should be understood that it is not required to implement or have all of the devices shown. More or fewer devices may be implemented or have alternatively.

[0143] This application also provides a computer program product including computer-readable instructions, which, when executed on an electronic device, cause the electronic device to implement any of the distributed computing-based query methods provided in this application.

[0144] This application also provides a computer-readable storage medium that carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement any of the distributed computing-based query methods provided in this application.

[0145] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the device embodiment drawings provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0146] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk, or optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, training equipment, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0147] In the above embodiments, the implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, in the form of a computer program product.

[0148] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, training device, or data center to another website, computer, training device, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives (SSDs)).

Claims

1. A query method based on distributed computing, characterized in that, include: Obtain the native machine code of the subtasks related to the query request; Based on load balancing among computing nodes, the first subtask in the subtask is assigned to the first thread in the computing node, and the first thread is independently bound to the first central processing unit CPU core. If the data required to execute the first subtask cannot be obtained, the first thread is suspended. After the data is obtained, the first thread is woken up, and the native machine code of the first subtask is executed based on the first thread running on the CPU core to obtain the execution result of the first subtask. The CPU core includes the first CPU core or the second CPU core. Based on the query results of the subtask, the query result of the query request is obtained.

2. The method according to claim 1, characterized in that, Before acquiring the data, the method further includes: Unbind the first thread from the first CPU core to release the first CPU core; Before waking up the first thread, the following is also included: Bind the first thread to the second CPU core.

3. The method according to claim 1, characterized in that, The first thread is independently bound to the first CPU core, including: Based on load balancing across CPU cores, the first thread is independently bound to the first CPU core that is in an idle state.

4. The method according to any one of claims 1-3, characterized in that, Assigning the first subtask in the subtasks to the first thread in the computing node includes: The first subtask in the subtask is assigned to the first thread in the computation-independent thread pool of the compute node, where the threads are dedicated to CPU computation and not to input / output I / O operations.

5. The method according to any one of claims 1-3, characterized in that, The acquisition of the data includes: Determine the first route based on the load of the CPU cores bound to the first subtask; Based on the first route, the data is transmitted to the CPU core bound to the first subtask, and the data is data after invalid data removal.

6. The method according to any one of claims 1-3, characterized in that, The native machine code that executes the first subtask includes: Read the data blocks that match the target filtering conditions and skip the data blocks that do not match the target filtering conditions.

7. A query device based on distributed computing, characterized in that, include: A module for implementing the method according to any one of claims 1 to 6.

8. A computer program product, characterized in that, Includes computer-readable instructions that, when executed on an electronic device, cause the electronic device to perform the method as described in any one of claims 1 to 6.

9. An electronic device, characterized in that, It includes at least one processor and a memory connected to the processor, wherein: The memory is used to store computer programs; The processor is used to execute the computer program to enable the electronic device to implement the method as described in any one of claims 1 to 6.

10. A computer storage medium, characterized in that, The storage medium carries one or more computer programs that, when executed by an electronic device, enable the electronic device to perform the method as described in any one of claims 1 to 6.

Images