A database heterogeneous resource management and scheduling method, device, equipment and medium

Abstract

The application discloses a database heterogeneous resource management and scheduling method and device, equipment and medium, relates to the technical field of database and heterogeneous computing, and comprises the following steps: receiving a query instruction sent by a user, and performing a preset preprocessing operation on the query instruction to obtain a target operator; reading idle communication bandwidth monitored by a data acceleration card, and performing an acceleration kernel deployment operation based on the target operator and the idle communication bandwidth to obtain a target acceleration kernel; sending a data storage address to the target acceleration kernel, so that the target acceleration kernel reads to-be-operated data based on the data storage address and performs operation; receiving a completion signal returned by the target acceleration kernel after obtaining an operation result, and reading the operation result. When the acceleration kernel is called, a corresponding number of acceleration kernels are adaptively deployed by monitoring the idle communication bandwidth on the data acceleration card, so that the communication bandwidth is maximally utilized, the data exchange efficiency is improved, and the waste of heterogeneous resources is reduced.

Description

Technical Field

[0001] This invention relates to the field of database and heterogeneous computing technology, and in particular to a method, apparatus, device and medium for managing and scheduling heterogeneous database resources. Background Technology

[0002] As an integrated system for recording and storing data, a database employs various models to organize data, storing it in different formats such as rows, columns, and tables. Databases support comprehensive data storage, searching, and parsing, and are now widely used in commerce, industry, smart homes, and healthcare. With the advent of the artificial intelligence era and the continuous advancement of big data and information technology, the installed capacity of databases is increasing exponentially, and the demand for faster database data query speeds is growing daily.

[0003] Currently, most databases rely on the CPU (Central Processing Unit / Processor) on servers to process user data queries. However, CPUs can place a heavy burden on processes and memory when handling computationally intensive tasks, affecting the task scheduling and processing speed of other processes on the server. Therefore, using FPGA (Field-Programmable Gate Array) and GPU (Graphics Processing Unit) as coprocessors to share the computationally intensive tasks of databases has become an emerging technological direction. The scheduling of heterogeneous resources and the deployment of acceleration kernels (operating system kernels) in CPU+X heterogeneous systems are usually determined by monitoring the resource requirements of acceleration cards and the kernel's idle status. However, this often overlooks the limitation of communication bandwidth on parallel acceleration kernels, causing multiple parallel acceleration kernels to execute sequentially due to communication bandwidth constraints. This not only fails to achieve parallel acceleration but also wastes additional heterogeneous resources. Summary of the Invention

[0004] In view of this, the purpose of this invention is to provide a method, apparatus, device, and medium for managing and scheduling heterogeneous database resources, which can maximize the utilization of communication bandwidth, improve data exchange efficiency, and reduce the waste of heterogeneous resources. The specific solution is as follows:

[0005] In a first aspect, this application discloses a method for managing and scheduling heterogeneous database resources, applied to a database server, comprising:

[0006] Receive a query command sent by a user and perform a preset preprocessing operation on the query command to obtain the target operator;

[0007] Read the idle communication bandwidth detected by the data acceleration card, and perform the acceleration kernel deployment operation based on the target operator and the idle communication bandwidth to obtain the target acceleration kernel;

[0008] Send the data storage address to the target accelerated kernel so that the target accelerated kernel can read the data to be processed based on the data storage address and perform the processing;

[0009] After receiving the calculation result, the target acceleration kernel returns a completion signal and reads the calculation result.

[0010] Optionally, performing a preset preprocessing operation on the query instruction to obtain the target operator includes:

[0011] The query instruction is decomposed into a target number of operators;

[0012] Each of the operators is processed by a preset quality improvement algorithm to obtain the corresponding target operator; wherein the preset quality improvement algorithm includes an unrelated algorithm.

[0013] Optionally, after processing each operator using a preset quality improvement algorithm to obtain the corresponding target operator, the method further includes:

[0014] The target operator is parsed through a preset instruction parsing operation to translate the target operator into target code; wherein the target code is low-level code that can be processed by the central processing unit.

[0015] Optionally, before the data acceleration card detects the idle communication bandwidth, it further includes:

[0016] The idle communication bandwidth is obtained by real-time monitoring of the idle information of the heterogeneous resources of the data acceleration card through the bandwidth and resource monitoring area on the data acceleration card.

[0017] Optionally, before the data acceleration card detects the idle communication bandwidth, it further includes:

[0018] The data to be processed, corresponding to the query instruction, is transferred from the host memory to the on-chip memory via direct memory access to obtain the data storage address corresponding to the data to be processed.

[0019] Optionally, the step of performing the accelerated kernel deployment operation based on the target operator and the idle communication bandwidth to obtain the target accelerated kernel includes:

[0020] The corresponding API function is invoked based on the target operator, and the bandwidth utilization rate of the current data bus is obtained based on the idle communication bandwidth.

[0021] Based on the API function and the bandwidth utilization, the accelerated kernel deployment operation is performed to obtain the target accelerated kernel.

[0022] Optionally, the step of performing the accelerated kernel deployment operation based on the API function and the bandwidth utilization to obtain the target accelerated kernel includes:

[0023] The current accelerated kernel set is determined based on the API function;

[0024] Based on the bandwidth utilization, a set of acceleration kernels to be deployed that meet the current requirements is matched from the current set of acceleration kernels;

[0025] An acceleration kernel deployment operation is performed on the set of acceleration kernels to be deployed to obtain the corresponding target acceleration kernel.

[0026] Secondly, this application discloses a database heterogeneous resource management and scheduling device, applied to a database server, comprising:

[0027] The operator acquisition module is used to receive query instructions sent by users and perform preset preprocessing operations on the query instructions to obtain target operators;

[0028] The accelerated kernel deployment module is used to read the idle communication bandwidth detected by the data acceleration card, and perform the accelerated kernel deployment operation based on the target operator and the idle communication bandwidth to obtain the target accelerated kernel;

[0029] The data processing module is used to send a data storage address to the target acceleration kernel, so that the target acceleration kernel can read the data to be processed based on the data storage address and perform the processing.

[0030] The calculation result reading module is used to receive the completion signal returned by the target acceleration kernel after obtaining the calculation result, and to read the calculation result.

[0031] Thirdly, this application discloses an electronic device, comprising:

[0032] Memory, used to store computer programs;

[0033] A processor is configured to execute the computer program to implement the steps of the database heterogeneous resource management and scheduling method disclosed above.

[0034] Fourthly, this application discloses a computer-readable medium for storing a computer program; wherein, when the computer program is executed by a processor, it implements the database heterogeneous resource management and scheduling method disclosed above.

[0035] As can be seen, this application provides a method for managing and scheduling heterogeneous database resources, including: receiving a query command sent by a user and performing a preset preprocessing operation on the query command to obtain a target operator; reading the idle communication bandwidth monitored by the data acceleration card and performing an acceleration kernel deployment operation based on the target operator and the idle communication bandwidth to obtain a target acceleration kernel; sending a data storage address to the target acceleration kernel so that the target acceleration kernel can read the data to be processed based on the data storage address and perform the processing; receiving a completion signal returned by the target acceleration kernel after obtaining the processing result, and reading the processing result. Therefore, this application, when retrieving acceleration kernels, adaptively deploys a corresponding number of acceleration kernels by monitoring the idle communication bandwidth on the data acceleration card to maximize the utilization of communication bandwidth, improve data exchange efficiency, and reduce the waste of heterogeneous resources. Attached Figure Description

[0036] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0037] Figure 1 This is a flowchart of a database heterogeneous resource management and scheduling method disclosed in this application;

[0038] Figure 2 This is an overall architecture diagram of the database heterogeneous resource management method disclosed in this application;

[0039] Figure 3 This is a schematic diagram illustrating the implementation of the database heterogeneous resource management and scheduling method based on bandwidth monitoring disclosed in this application;

[0040] Figure 4 This application discloses a specific method for managing and scheduling heterogeneous database resources;

[0041] Figure 5 This is a schematic diagram of communication bandwidth allocation disclosed in this application;

[0042] Figure 6 A schematic diagram of the database heterogeneous resource management and scheduling device provided in this application;

[0043] Figure 7 This application provides a structural diagram of an electronic device. Detailed Implementation

[0044] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0045] Currently, most databases rely on the CPU on servers to process user data queries. However, CPUs place a significant burden on processes and memory when handling computationally intensive tasks, impacting the task scheduling and processing speed of other server processes. Therefore, using FPGAs and GPUs as coprocessors to share the computationally intensive tasks of databases has become an emerging technological trend. In CPU+X heterogeneous architectures, the scheduling of heterogeneous resources and the deployment of acceleration kernels are typically determined by monitoring the resource requirements of acceleration cards and kernel idle status. However, this often overlooks the limitation of communication bandwidth on parallel acceleration kernels, causing multiple parallel acceleration kernels to execute sequentially due to communication bandwidth constraints. This fails to achieve parallel acceleration and wastes additional heterogeneous resources. To address this, this application provides a database heterogeneous resource management and scheduling method that maximizes the utilization of communication bandwidth, improves data exchange efficiency, and reduces the waste of heterogeneous resources.

[0046] This invention discloses a method for managing and scheduling heterogeneous database resources. See [link to relevant documentation]. Figure 1 As shown, applied to a database server, the method includes:

[0047] Step S11: Receive the query command sent by the user and perform a preset preprocessing operation on the query command to obtain the target operator.

[0048] In this embodiment, a query command sent by a user is received, and a preset preprocessing operation is performed on the query command to obtain a target operator. Specifically, the query command is decomposed into a target number of operators; each operator is processed by a preset quality improvement algorithm to obtain the corresponding target operator; wherein, the preset quality improvement algorithm includes an irrelevant algorithm. It is understood that, as Figure 2As shown, the database heterogeneous resource management method mainly includes a user application layer, an instruction parsing layer, an acceleration interface layer, a task management layer, a device driver layer, and a query acceleration layer. The user application layer provides database management applications for the database server, and users set instructions for database behavior at the application layer to receive query instructions sent by the application layer. The instruction parsing layer includes instruction parsing, instruction decomposition, and instruction optimization. The acceleration interface layer includes sorting APIs (Application Program Interface), connection APIs, aggregation APIs, and filtering APIs. The task management layer includes resource management, data transmission, and task scheduling. The device driver layer includes the Xilinx Runtime Library, XDMA, and PCIe (PCI-Express, a bus and interface standard). The query acceleration layer includes bandwidth and resource monitoring, data loading, a sort kernel, a join kernel, an aggregation kernel, and a filter kernel.

[0049] After receiving the query command sent by the user, the original query command is first decomposed into several operators through the command parsing layer. These operators work together to complete the function of the command. Then, the decomposed operators are optimized by the command optimization through the algorithm (i.e., the preset quality improvement algorithm, such as incoherent parallelization, algorithm optimization, etc.) to obtain the target operator, thereby maximizing the improvement of the command's running efficiency and acceleration efficiency.

[0050] It is understandable that the device driver layer includes the device drivers required for the normal operation of the data acceleration card, as well as the communication protocols and drivers required for data transmission between the database CPU and the data acceleration card, such as XRT, PCIe, XDMA, etc.

[0051] Step S12: Read the idle communication bandwidth detected by the data acceleration card, and perform the acceleration kernel deployment operation based on the target operator and the idle communication bandwidth to obtain the target acceleration kernel.

[0052] In this embodiment, a query command sent by a user is received, and a preset preprocessing operation is performed on the query command to obtain a target operator. Then, the idle communication bandwidth monitored by the data acceleration card is read, and an acceleration kernel deployment operation is performed based on the target operator and the idle communication bandwidth to obtain a target acceleration kernel. Specifically, the corresponding API function is invoked based on the target operator, and the bandwidth utilization rate corresponding to the current data bus is obtained based on the idle communication bandwidth. An acceleration kernel deployment operation is performed based on the API function and the bandwidth utilization rate to obtain the target acceleration kernel. The step of performing the acceleration kernel deployment operation based on the API function and the bandwidth utilization rate to obtain the target acceleration kernel includes: determining the current acceleration kernel set based on the API function; matching a set of acceleration kernels to be deployed that meets the current requirements from the current acceleration kernel set based on the bandwidth utilization rate; and performing an acceleration kernel deployment operation on the set of acceleration kernels to be deployed to obtain the corresponding target acceleration kernel.

[0053] Understandably, the acceleration interface layer contains API functions for corresponding query operations. Based on the required operators, the corresponding APIs in the acceleration interface layer are called to deploy the acceleration kernel. Users dynamically reconstruct the logical functions of the data acceleration card by calling these API functions. In the task management layer, task scheduling comprehensively manages the processes of operator execution, data encryption, compression, and transmission, ensuring the orderly execution of instructions and data processing. Resource management, based on the bandwidth and resource monitoring results from the acceleration layer, rationally allocates heterogeneous resources and deploys the acceleration kernel according to bandwidth utilization, maximizing the use of communication bandwidth and heterogeneous resources. Data transmission ensures data transfer between the CPU and the acceleration card.

[0054] Step S13: Send the data storage address to the target accelerated kernel so that the target accelerated kernel can read the data to be processed and perform the processing based on the data storage address.

[0055] In this embodiment, an accelerated kernel deployment operation is performed based on the target operator and the idle communication bandwidth to obtain the target accelerated kernel. Then, a data storage address is sent to the target accelerated kernel so that the target accelerated kernel can read the data to be processed based on the data storage address and perform the processing. It is understood that, as... Figure 3As shown, the specific implementation process of the database heterogeneous resource management and scheduling method based on bandwidth monitoring is as follows: After receiving the query operation instruction, the database server CPU parses and optimizes the instruction; the database CPU transfers the data to be processed from the host memory to the on-chip memory through DMA (Direct Memory Access); the database CPU reads the idle communication bandwidth monitored by the data acceleration card at this time, and completes the dynamic deployment of the coprocessor acceleration kernel according to the communication bandwidth required by the kernel corresponding to the query operation; after the kernel deployment is completed, the CPU sends the storage address of the data to be processed in the on-chip memory to the kernel and starts the kernel; after the kernel starts, it reads the data to be processed from the corresponding storage address, performs the operation, and then rewrites the operation result back to the memory and returns a completion signal.

[0056] Step S14: Receive the completion signal returned by the target acceleration kernel after obtaining the calculation result, and read the calculation result.

[0057] In this embodiment, a data storage address is sent to the target acceleration kernel so that the target acceleration kernel can read the data to be processed based on the data storage address, perform the processing, receive a completion signal returned by the target acceleration kernel after obtaining the processing result, and then read the processing result. It can be understood that after the database CPU receives the completion signal, it reads the processing result back to the host memory from the corresponding memory address, completing the query operation.

[0058] The database server architecture of this invention is a CPU+X (FPGA, GPU, DPU, etc.) heterogeneous structure. The CPU is responsible for overall database task scheduling, low-intensity computing tasks, and logical management processes, while computationally intensive tasks are handled by X, thus fully utilizing the advantages of both CPU and X to improve database performance. This invention monitors the bandwidth utilization of the coprocessor and on-chip memory on the data acceleration card. When the CPU calls the acceleration kernel, it adaptively deploys an appropriate number of acceleration kernels based on the current available bandwidth and the bandwidth required by the kernel, maximizing communication bandwidth utilization, improving data exchange efficiency, and saving heterogeneous resource waste caused by communication bandwidth overflow.

[0059] As can be seen, this application provides a method for managing and scheduling heterogeneous database resources, including: receiving a query command sent by a user and performing a preset preprocessing operation on the query command to obtain a target operator; reading the idle communication bandwidth monitored by the data acceleration card and performing an acceleration kernel deployment operation based on the target operator and the idle communication bandwidth to obtain a target acceleration kernel; sending a data storage address to the target acceleration kernel so that the target acceleration kernel can read the data to be processed based on the data storage address and perform the processing; receiving a completion signal returned by the target acceleration kernel after obtaining the processing result, and reading the processing result. Therefore, this application, when retrieving acceleration kernels, adaptively deploys a corresponding number of acceleration kernels by monitoring the idle communication bandwidth on the data acceleration card to maximize the utilization of communication bandwidth, improve data exchange efficiency, and reduce the waste of heterogeneous resources.

[0060] See Figure 4 As shown, this embodiment of the invention discloses a method for managing and scheduling heterogeneous database resources. Compared with the previous embodiment, this embodiment further explains and optimizes the technical solution.

[0061] Step S21: Receive the query command sent by the user and perform a preset preprocessing operation on the query command to obtain the target operator.

[0062] Step S22: The target operator is parsed through a preset instruction parsing operation to translate the target operator into target code.

[0063] In this embodiment, a query command sent by a user is received, and a preset preprocessing operation is performed on the query command to obtain a target operator. Then, a preset instruction parsing operation is used to parse the target operator, so as to translate the target operator into target code; the target code is low-level code that the central processing unit can process. It can be understood that after instruction decomposition and instruction optimization operations at the instruction parsing layer, the target operator is parsed through instruction parsing at the instruction parsing layer, so as to translate the query command, which has undergone instruction decomposition and instruction optimization, into low-level code that the CPU can understand.

[0064] Step S23: Monitor the idle information of heterogeneous resources of the data acceleration card in real time through the bandwidth and resource monitoring area on the data acceleration card to obtain the idle communication bandwidth.

[0065] In this embodiment, after the target operator is translated into target code, the idle information of the heterogeneous resources of the data acceleration card is monitored in real time through the bandwidth and resource monitoring area on the data acceleration card to obtain the idle communication bandwidth. It can be understood that the query acceleration layer is the data acceleration card, which includes bandwidth and resource monitoring, data loading, and an acceleration kernel. The data loading module receives the data to be processed sent by the database CPU, and the bandwidth and resource monitoring module monitors the idle status of the heterogeneous resources of the data acceleration card.

[0066] Step S24: Transfer the data to be processed corresponding to the query instruction from the host memory to the on-chip memory via direct memory access to obtain the data storage address corresponding to the data to be processed.

[0067] In this embodiment, after obtaining the available communication bandwidth, the data to be processed corresponding to the query instruction is transferred from the host memory to the on-chip memory via direct memory access to obtain the data storage address corresponding to the data to be processed. It is understood that after the database CPU transfers the data to be processed to the on-chip memory, it analyzes the current available bandwidth of the memory and dynamically deploys it according to the communication bandwidth required to accelerate the kernel, maximizing the utilization of the communication bandwidth.

[0068] Step S25: Read the idle communication bandwidth detected by the data acceleration card, and perform the acceleration kernel deployment operation based on the target operator and the idle communication bandwidth to obtain the target acceleration kernel.

[0069] In this embodiment, the bandwidth utilization of the current data bus is fed back during kernel deployment. Based on the bandwidth utilization, the accelerated kernel is matched with the functions of the API interface layer, and the corresponding accelerated kernel is deployed according to the different API functions called by the user application layer. Specifically, the usage of heterogeneous resources of the coprocessor on the data acceleration card and the bandwidth utilization of the data bus are monitored in real time. When the CPU executes computationally intensive tasks, the kernel is dynamically deployed on the data acceleration card by calling the corresponding operation operator API. The task management layer dynamically deploys the number of accelerated kernels based on the bandwidth availability, the bandwidth required for the currently deployed kernel, and the memory availability on the heterogeneous acceleration card, so as to make full use of the communication bandwidth of the on-chip memory and save unnecessary heterogeneous resources.

[0070] In one specific implementation, after the database CPU transmits the data to be processed to the on-chip memory, it analyzes the current free bandwidth of the memory and dynamically deploys kernels based on the communication bandwidth required for acceleration, maximizing the utilization of communication bandwidth. Let the free bandwidth of memory 1 be B1, and the required bandwidth of the kernel be B1; therefore, only one kernel is deployed to communicate with memory 1. The free bandwidth of memory 3 is 2B3 / 3, and the required bandwidth of the kernel is B3 / 3; therefore, two kernels can be deployed to communicate with memory 3. If the communication bandwidth is exceeded, such as deploying three kernels to communicate with memory 3, since the communication bandwidth has reached its maximum value, only two of the three kernels can run simultaneously. The third kernel must wait for the first two kernels to finish running and release bandwidth before it can continue running. This does not achieve the ideal goal of three acceleration kernels running in parallel and wastes additional heterogeneous resources by deploying unnecessary kernels.

[0071] For example Figure 5 As shown, the data acceleration card has three on-chip memories, such as DDR4 (a new generation memory specification) or HBM, with total communication bandwidths of B1, B2, and B3 respectively. If the database CPU needs to perform a sorting operation, the communication bandwidth required by the sorting kernel, quantized by memory bandwidth, is B1, B2 / 2, and B3 / 3. The idle communication bandwidths of the on-chip memories are B1, B2 / 2, and 2B3 / 3 respectively. Memory 1 has an idle bandwidth of B1, and the kernel's required bandwidth is also B1; therefore, one kernel is deployed to communicate with memory 1. Memory 2 has an idle bandwidth of B2 / 2, and the kernel's required bandwidth is also B2 / 2; therefore, one kernel is deployed to communicate with memory 2. Memory 3 has an idle bandwidth of 2B3 / 3, and the kernel's required bandwidth is also B3 / 3; therefore, two kernels can be deployed to communicate with memory 3. At this point, if the communication bandwidth is exceeded, such as deploying 3 kernels to communicate with memory 3, since the communication bandwidth has reached its maximum value, only two of kernels 1, 2 and 5 can run at the same time. The third one will have to wait for kernel 1 or kernel 2 to finish running and release the bandwidth before it can continue to run. This cannot achieve the ideal goal of running 3 accelerated kernels in parallel, and it also wastes additional heterogeneous resources.

[0072] Step S26: Send the data storage address to the target accelerated kernel so that the target accelerated kernel can read the data to be processed based on the data storage address and perform the processing.

[0073] Step S27: Receive the completion signal returned by the target acceleration kernel after obtaining the calculation result, and read the calculation result.

[0074] For details regarding steps S21, S26, and S27, please refer to the corresponding content disclosed in the foregoing embodiments, which will not be repeated here.

[0075] As can be seen, this application embodiment receives a query command sent by a user and performs a preset preprocessing operation on the query command to obtain a target operator; it parses the target operator through a preset command parsing operation to translate the target operator into target code; it monitors the idle information of heterogeneous resources of the data acceleration card in real time through the bandwidth and resource monitoring area on the data acceleration card to obtain the idle communication bandwidth; it transfers the data to be processed corresponding to the query command from the host memory to the on-chip memory through direct memory access to obtain the data storage address corresponding to the data to be processed; it reads the idle communication bandwidth monitored by the data acceleration card and performs an acceleration kernel deployment operation based on the target operator and the idle communication bandwidth to obtain a target acceleration kernel; it sends the data storage address to the target acceleration kernel so that the target acceleration kernel reads the data to be processed based on the data storage address and performs the operation; it receives the completion signal returned by the target acceleration kernel after obtaining the operation result and reads the operation result, thereby maximizing the utilization of communication bandwidth, improving data exchange efficiency, and reducing the waste of heterogeneous resources.

[0076] See Figure 6 As shown in the embodiments of this application, a database heterogeneous resource management and scheduling device is also disclosed, applied to a database server, including:

[0077] The operator acquisition module 11 is used to receive a query command sent by the user and perform a preset preprocessing operation on the query command to obtain the target operator;

[0078] The accelerated kernel deployment module 12 is used to read the idle communication bandwidth detected by the data acceleration card, and perform the accelerated kernel deployment operation based on the target operator and the idle communication bandwidth to obtain the target accelerated kernel;

[0079] Data processing module 13 is used to send a data storage address to the target acceleration kernel, so that the target acceleration kernel can read the data to be processed based on the data storage address and perform the processing.

[0080] The calculation result reading module 14 is used to receive the completion signal returned by the target acceleration kernel after obtaining the calculation result, and to read the calculation result.

[0081] As can be seen, this application includes: receiving a query command sent by a user and performing a preset preprocessing operation on the query command to obtain a target operator; reading the idle communication bandwidth monitored by the data acceleration card and performing an acceleration kernel deployment operation based on the target operator and the idle communication bandwidth to obtain a target acceleration kernel; sending a data storage address to the target acceleration kernel so that the target acceleration kernel can read the data to be processed based on the data storage address and perform the operation; receiving a completion signal returned by the target acceleration kernel after obtaining the operation result, and reading the operation result. Therefore, this application, when retrieving the acceleration kernel, adaptively deploys a corresponding number of acceleration kernels by monitoring the idle communication bandwidth on the data acceleration card to maximize the utilization of communication bandwidth, improve data exchange efficiency, and reduce the waste of heterogeneous resources.

[0082] In some specific embodiments, the operator acquisition module 11 specifically includes:

[0083] The instruction receiving unit is used to receive query instructions sent by the user.

[0084] An instruction decomposition unit is used to decompose the query instruction into a target number of operators;

[0085] An operator processing unit is configured to process each of the operators using a preset quality improvement algorithm to obtain the corresponding target operator; wherein the preset quality improvement algorithm includes an irrelevant algorithm;

[0086] The instruction parsing unit is used to parse the target operator through a preset instruction parsing operation so as to translate the target operator into target code; wherein the target code is low-level code that can be processed by the central processing unit.

[0087] In some specific embodiments, the accelerated kernel deployment module 12 specifically includes:

[0088] The idle communication bandwidth acquisition unit is used to monitor the idle information of the heterogeneous resources of the data acceleration card in real time through the bandwidth and resource monitoring area on the data acceleration card, so as to obtain the idle communication bandwidth.

[0089] The data storage address acquisition unit is used to transfer the data to be processed corresponding to the query instruction from the host memory to the on-chip memory through direct memory access, so as to obtain the data storage address corresponding to the data to be processed;

[0090] The idle communication bandwidth reading unit is used to read the idle communication bandwidth detected by the data acceleration card.

[0091] API function retrieval unit, used to retrieve the corresponding API function based on the target operator;

[0092] A bandwidth utilization acquisition unit is used to acquire the bandwidth utilization rate corresponding to the current data bus based on the idle communication bandwidth.

[0093] The current accelerated kernel set determination unit is used to determine the current accelerated kernel set based on the API function;

[0094] The unit for determining the set of acceleration kernels to be deployed is used to match a set of acceleration kernels to be deployed that meets the current requirements from the current set of acceleration kernels based on the bandwidth utilization.

[0095] The target acceleration kernel acquisition unit is used to perform acceleration kernel deployment operations on the set of acceleration kernels to be deployed in order to obtain the corresponding target acceleration kernel.

[0096] In some specific embodiments, the data processing module 13 specifically includes:

[0097] The data processing unit is used to send a data storage address to the target acceleration kernel, so that the target acceleration kernel can read the data to be processed based on the data storage address and perform the processing.

[0098] In some specific embodiments, the calculation result reading module 14 specifically includes:

[0099] The calculation result reading unit is used to receive the completion signal returned by the target acceleration kernel after obtaining the calculation result, and to read the calculation result.

[0100] Furthermore, embodiments of this application also provide an electronic device. Figure 7 This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content of the diagram should not be construed as limiting the scope of this application.

[0101] Figure 7 This is a schematic diagram of the structure of an electronic device 20 provided in an embodiment of this application. Specifically, the electronic device 20 may include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 stores a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the database heterogeneous resource management and scheduling method disclosed in any of the foregoing embodiments. Furthermore, the electronic device 20 in this embodiment may specifically be an electronic computer.

[0102] In this embodiment, the power supply 23 is used to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and external devices, and the communication protocol it follows can be any communication protocol applicable to the technical solution of this application, and is not specifically limited here; the input / output interface 25 is used to acquire external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs, and is not specifically limited here.

[0103] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk or optical disk, etc. The resources stored thereon can include operating system 221, computer program 222, etc., and the storage method can be temporary storage or permanent storage.

[0104] The operating system 221 is used to manage and control the various hardware devices on the electronic device 20 and the computer program 222, which may be Windows Server, Netware, Unix, Linux, etc. In addition to including a computer program capable of performing the database heterogeneous resource management and scheduling method executed by the electronic device 20 as disclosed in any of the foregoing embodiments, the computer program 222 may further include a computer program capable of performing other specific tasks.

[0105] Furthermore, this application also discloses a medium storing a computer program, which, when loaded and executed by a processor, implements the database heterogeneous resource management and scheduling method steps disclosed in any of the foregoing embodiments.

[0106] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.

[0107] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0108] The above provides a detailed description of a database heterogeneous resource management and scheduling method, apparatus, device, and medium provided by the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A method for managing and scheduling heterogeneous database resources, characterized in that, Applied to database servers, including: Receive a query command sent by a user and perform a preset preprocessing operation on the query command to obtain the target operator; Read the idle communication bandwidth detected by the data acceleration card, and perform the acceleration kernel deployment operation based on the target operator and the idle communication bandwidth to obtain the target acceleration kernel; Send the data storage address to the target accelerated kernel so that the target accelerated kernel can read the data to be processed based on the data storage address and perform the processing; Receive the completion signal returned by the target acceleration kernel after obtaining the calculation result, and read the calculation result; The step of performing the accelerated kernel deployment operation based on the target operator and the idle communication bandwidth to obtain the target accelerated kernel includes: The corresponding API function is invoked based on the target operator, and the bandwidth utilization rate of the current data bus is obtained based on the idle communication bandwidth. Based on the API function and the bandwidth utilization, perform the accelerated kernel deployment operation to obtain the target accelerated kernel; The step of performing the accelerated kernel deployment operation based on the API function and the bandwidth utilization to obtain the target accelerated kernel includes: The current accelerated kernel set is determined based on the API function; Based on the bandwidth utilization, a set of acceleration kernels to be deployed that meet the current requirements is matched from the current set of acceleration kernels; An acceleration kernel deployment operation is performed on the set of acceleration kernels to be deployed to obtain the corresponding target acceleration kernel.

2. The database heterogeneous resource management and scheduling method according to claim 1, characterized in that, The step of performing a preset preprocessing operation on the query instruction to obtain the target operator includes: The query instruction is decomposed into a target number of operators; Each of the operators is processed by a preset quality improvement algorithm to obtain the corresponding target operator; wherein the preset quality improvement algorithm includes an unrelated algorithm.

3. The database heterogeneous resource management and scheduling method according to claim 2, characterized in that, After processing each operator using a preset quality improvement algorithm to obtain the corresponding target operator, the method further includes: The target operator is parsed through a preset instruction parsing operation to translate the target operator into target code; wherein the target code is low-level code that can be processed by the central processing unit.

4. The database heterogeneous resource management and scheduling method according to claim 1, characterized in that, Before the idle communication bandwidth detected by the data acceleration card, it also includes: The idle communication bandwidth is obtained by real-time monitoring of the idle information of the heterogeneous resources of the data acceleration card through the bandwidth and resource monitoring area on the data acceleration card.

5. The database heterogeneous resource management and scheduling method according to claim 1, characterized in that, Before the idle communication bandwidth detected by the data acceleration card, it also includes: The data to be processed, corresponding to the query instruction, is transferred from the host memory to the on-chip memory via direct memory access to obtain the data storage address corresponding to the data to be processed.

6. A database heterogeneous resource management and scheduling device, characterized in that, Applied to database servers, including: The operator acquisition module is used to receive query instructions sent by users and perform preset preprocessing operations on the query instructions to obtain target operators; The accelerated kernel deployment module is used to read the idle communication bandwidth detected by the data acceleration card, and perform the accelerated kernel deployment operation based on the target operator and the idle communication bandwidth to obtain the target accelerated kernel; The data processing module is used to send a data storage address to the target acceleration kernel, so that the target acceleration kernel can read the data to be processed based on the data storage address and perform the processing. The calculation result reading module is used to receive the completion signal returned by the target acceleration kernel after obtaining the calculation result, and to read the calculation result; Specifically, the accelerated kernel deployment module is used to call the corresponding API function based on the target operator, and obtain the bandwidth utilization rate of the current data bus based on the idle communication bandwidth; and to perform the accelerated kernel deployment operation based on the API function and the bandwidth utilization rate to obtain the target accelerated kernel. The accelerated kernel deployment module includes: The current accelerated kernel set determination unit is used to determine the current accelerated kernel set based on the API function; The unit for determining the set of acceleration kernels to be deployed is used to match a set of acceleration kernels to be deployed that meets the current requirements from the current set of acceleration kernels based on the bandwidth utilization. The target acceleration kernel acquisition unit is used to perform acceleration kernel deployment operations on the set of acceleration kernels to be deployed in order to obtain the corresponding target acceleration kernel.

7. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor is configured to execute the computer program to implement the steps of the database heterogeneous resource management and scheduling method as described in any one of claims 1 to 5.

8. A computer-readable medium, characterized in that, Used to store computer programs; wherein, when the computer programs are executed by a processor, they implement the database heterogeneous resource management and scheduling method as described in any one of claims 1 to 5.

Images