Monitoring Method, Device, Equipment and Storage Medium of DPU Hybrid Cluster

Abstract

Embodiments of the present disclosure relate to a monitoring method, device, equipment, and storage medium for a DPU hybrid cluster. The method includes monitoring at least one CPU node and at least one DPU node. Specifically, the driver of the DPU node is called to obtain the resource usage information of the DPU node; the DPU executable plan tree of the DPU node is obtained, where the DPU executable plan tree is obtained by the DPU node according to the SQL statement it executes; the resource usage information and the DPU executable plan tree are output. It can be seen that according to the embodiments of the present disclosure, it is possible to monitor both the CPU nodes and the DPU nodes in the cluster, and the monitoring results of the DPU nodes can be output, so that users can timely understand information such as the running status and job situation of the DPU nodes. In this way, the comprehensiveness of the monitoring of the DPU hybrid cluster can be improved.

Description

Technical Field

[0001] Embodiments of the present disclosure relate to the field of computer technologies, and in particular, to a monitoring method, apparatus, device, and storage medium for a DPU hybrid cluster. Background Art

[0002] Spark (i.e., Apache Spark) is a fast and general-purpose computing engine designed for large-scale data processing, using an open-source cluster computing environment similar to Hadoop. Spark SQL is one of the computing modules of Spark, specifically for processing structured data. Spark SQL allows users to execute SQL queries and read / writes using standard SQL statements, or use Hive SQL to perform queries and read / writes on a Hive warehouse. Currently, Spark SQL is based on the Spark framework, and all its computations are based on the CPU. In addition to maintaining the scheduling of the entire computation, the CPU also requires additional computing power for data-intensive computations. However, as a general-purpose processing chip, the CPU has no obvious advantage in high-density data computations in big data, which makes the computing power of Spark SQL based on CPU operations the main bottleneck of performance. However, as a dedicated data processing chip based on the KPU architecture, the DPU has a significant performance improvement compared to the CPU when processing complex data computations. Therefore, offloading the data computations of SparkSQL from the CPU to the DPU can greatly improve the performance of Spark SQL, accelerate the computing power of Spark SQL in big data scenarios, enable the CPU to focus on the computation scheduling of Spark, and the DPU to focus on the data computations in Spark SQL.

[0003] As a popular framework for processing big data, Spark is usually deployed on multiple nodes. During operation, all the machines in a cluster jointly process data in parallel to achieve an acceleration effect. Since the application scenario of Spark is cluster jobs, Spark itself provides a complete cluster monitoring interface called Spark Web UI. However, this interface is designed for native Spark and can only monitor the CPUs in the cluster, unable to monitor the DPU. Summary of the Invention

[0004] To solve the above technical problems or at least partially solve the above technical problems, embodiments of the present disclosure provide a monitoring method, apparatus, device, and storage medium for a DPU hybrid cluster.

[0005] In a first aspect of an embodiment of the present disclosure, a method for monitoring a DPU hybrid cluster is provided. The DPU hybrid cluster includes at least one CPU node and at least one DPU node. The method includes monitoring the at least one CPU node, and the method further includes monitoring the at least one DPU node. Wherein, the monitoring of the at least one DPU node includes:

[0006] Invoking the driver of the DPU node to obtain the resource usage information of the DPU node;

[0007] Obtaining the DPU executable plan tree of the DPU node, where the DPU executable plan tree is obtained by the DPU node according to the SQL statement it executes;

[0008] Outputting the resource usage information and the DPU executable plan tree.

[0009] In a second aspect of an embodiment of the present disclosure, a monitoring device for a DPU hybrid cluster is provided. The DPU hybrid cluster includes at least one CPU node and at least one DPU node. The device includes a first monitoring module for monitoring the at least one CPU node, and the device further includes a second monitoring module for monitoring the at least one DPU node. Wherein, the second monitoring module includes:

[0010] A first acquisition module for invoking the driver of the DPU node to obtain the resource usage information of the DPU node;

[0011] A second acquisition module for obtaining the DPU executable plan tree of the DPU node, where the DPU executable plan tree is obtained by the DPU node according to the SQL statement it executes;

[0012] A first output module for outputting the resource usage information and the DPU executable plan tree.

[0013] In a third aspect of an embodiment of the present disclosure, an electronic device is provided. The server includes: a processor and a memory. Wherein, a computer program is stored in the memory, and when the computer program is executed by the processor, the processor executes the method of the first aspect above.

[0014] In a fourth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided. A computer program is stored in the storage medium, and when the computer program is executed by a processor, the method of the first aspect above can be implemented.

[0015] The technical solution provided by the embodiment of the present disclosure has the following advantages compared with the prior art:

[0016] In the embodiments of the present disclosure, the DPU hybrid cluster includes at least one CPU node and at least one DPU node. According to the embodiments of the present disclosure, it is possible to monitor not only at least one CPU node but also at least one DPU node. Specifically, it is possible to call the driver of the DPU node to obtain the resource usage information of the DPU node; obtain the DPU executable plan tree of the DPU node, where the DPU executable plan tree is obtained by the DPU node according to the SQL statement it executes; output the resource usage information and the DPU executable plan tree. It can be seen that by adopting the above technical solution, it is possible to monitor both the CPU nodes and the DPU nodes in the cluster, and output the monitoring results of the DPU nodes, enabling users to timely understand information such as the running status and job conditions of the DPU nodes. Thus, the comprehensiveness of monitoring the DPU hybrid cluster can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The accompanying drawings herein are incorporated into the specification and form a part of the specification, showing embodiments consistent with the present disclosure and, together with the specification, are used to explain the principles of the present disclosure.

[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure or the prior art, the following will briefly introduce the accompanying drawings required for use in the description of the embodiments or the prior art. Obviously, for those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0019] Figure 1 is a flowchart of a method for monitoring a DPU hybrid cluster provided by an embodiment of the present disclosure;

[0020] Figure 2 is a schematic structural diagram of a monitoring device for a DPU hybrid cluster provided by an embodiment of the present disclosure;

[0021] Figure 3 is a schematic structural diagram of an electronic device in an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0022] In order to more clearly understand the above objects, features, and advantages of the present disclosure, the following will further describe the solutions of the present disclosure. It should be noted that, without conflict, the embodiments of the present disclosure and the features in the embodiments can be combined with each other.

[0023] Many specific details are set forth in the following description to fully understand the present disclosure, but the present disclosure can also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only a part of the embodiments of the present disclosure, rather than all the embodiments.

[0024] Figure 1It is a flowchart of a monitoring method for a DPU hybrid cluster provided by an embodiment of the present disclosure, and this method can be executed by an electronic device. The electronic device can be exemplarily understood as devices such as mobile phones, tablet computers, laptop computers, desktop computers, smart TVs, etc.

[0025] As Figure 1 shown, the method provided in this embodiment includes the following steps:

[0026] S110: Call the driver of the DPU node to obtain the resource usage information of the DPU node.

[0027] In an embodiment of the present disclosure, the DPU hybrid cluster includes multiple nodes. Among them, the multiple nodes include at least one CPU node and at least one Data Processing Unit (DPU) node. The DPU is a hardware accelerator specifically designed for processing data-intensive tasks, and it mainly has the following advantages: offloading the CPU, reducing the burden on the host CPU so that it can focus on running applications and services; low latency, as it is optimized for data processing, lower latency can be achieved; energy saving, compared with the CPU, the DPU is more energy-efficient when performing specific tasks; programmability, supporting software-defined functions and can be customized according to requirements; security, providing the ability to implement encryption and security policies to enhance system security. The DPU hybrid cluster can be exemplarily understood as a DPU hybrid cluster based on Spark. Among them, Spark is an open-source big data processing framework, and the DPU hybrid cluster is a group of interconnected nodes that cooperate with each other to execute computing tasks. In other words, as an efficient distributed computing framework, Spark relies on the nodes provided by the DPU hybrid cluster to execute large-scale data processing tasks, and the DPU hybrid cluster provides the necessary infrastructure and support for Spark, enabling Spark to give full play to its distributed computing advantages. The DPU node can be exemplarily understood as a DPU node based on the KPU architecture. But it is not limited to this.

[0028] In an embodiment of the present disclosure, this method further includes monitoring the CPU nodes in the cluster. Exemplarily, Spark itself provides a complete cluster monitoring interface for CPU nodes, called Spark Web UI, and the CPU nodes in the cluster can be monitored through Spark WebUI, but it is not limited to this.

[0029] Specifically, the resource usage information of the DPU node is used to describe the usage of the computing resources of the DPU node.

[0030] Optionally, the resource usage information of the DPU node includes: the total amount of DPU tasks, the number of completed DPU tasks, the progress of DPU tasks, the memory occupied by the DPU, the DPU temperature, and / or the power consumption of the DPU.

[0031] Specifically, in the process of data processing by the DPU hybrid cluster, the original data is often split into multiple smaller data blocks for parallel processing, and one or more tasks can be assigned to each data block. These tasks corresponding to the multiple data blocks obtained by splitting the original data can be assigned to different nodes in the DPU hybrid cluster for execution, and each node processes the data block assigned to it.

[0032] Among them, the total amount of DPU tasks is the total number of tasks assigned to the DPU node.

[0033] Among them, the number of completed DPU tasks is the number of tasks completed on the DPU node.

[0034] Among them, the progress of DPU tasks is the degree of completion of the tasks in progress on the DPU node. Exemplarily, if the total amount of DPU tasks is n1 and the number of completed DPU tasks is n2, then the progress of DPU tasks is n2 / n1, where both n1 and n2 are positive integers.

[0035] Among them, the memory occupied by the DPU is the total amount of memory being used by the DPU node.

[0036] Among them, the DPU temperature is the current temperature of the DPU node.

[0037] Among them, the power consumption of the DPU is the current power consumption of the DPU node.

[0038] It can be understood that by setting the resource usage information of the DPU node to include: the total amount of DPU tasks, the number of completed DPU tasks, the progress of DPU tasks, the memory occupied by the DPU, the DPU temperature, and / or the power consumption of the DPU, the monitoring items for the DPU node are made richer and more comprehensive, which is conducive to improving the monitoring quality of the DPU node.

[0039] In some embodiments, monitoring at least one CPU node includes monitoring at least one CPU node through the Spark Web UI, and the SparkDPUInfo class is integrated in the Spark Web UI;

[0040] Calling the driver of the DPU node to obtain the resource usage information of the DPU node, including:

[0041] Calling the driver of the DPU node through the SparkDPUInfo class to obtain the resource usage information of the DPU node.

[0042] Specifically, through the plug-in function provided by Spark, a SparkDPUInfo class can be created. The code in this SparkDPUInfo class is responsible for calling the drivers on each DPU node in the DPU hybrid cluster to obtain the resource usage information of the DPU nodes.

[0043] Exemplarily, the SparkDPUInfo class can be regarded as a class that encapsulates the configurations and information related to DPU nodes. It can include the following functions: (1) managing the information of DPU nodes, such as the model, version, etc. of DPU nodes; (2) providing interfaces to call the drivers of DPU nodes to obtain the resource usage information of DPU nodes. In a cluster environment, each DPU node has its own driver, and these drivers provide interfaces for interacting with DPU hardware. By calling these drivers, the resource usage information of DPU nodes can be obtained. Since the SparkDPUInfo class encapsulates the basic information of DPU and the interaction with DPU drivers, the resource usage information can be obtained from the DPU drivers by calling the SparkDPUInfo class.

[0044] It can be understood that by extending the SparkDPUInfo class on the basis of the original Spark WEB UI to call the drivers of DPU nodes to obtain the resource usage information of DPU nodes, it is convenient to manage and monitor the resource usage of DPU nodes in the cluster, which is very important for optimizing the cluster performance and resource management. And there is no need to develop an additional monitoring software from scratch to monitor DPU nodes. In this way, the R & D cycle and cost can be reduced.

[0045] Of course, in some other embodiments, the resource usage information of DPU nodes can also be obtained by using specific monitoring tools or command-line tools provided by DPU node vendors.

[0046] S120. Obtain the DPU executable plan tree of the DPU node, where the DPU executable plan tree is obtained by the DPU node according to the SQL statement it executes.

[0047] Specifically, Spark SQL is a module for processing structured data based on Spark. SparkSQL supports querying and loading of multiple data sources, is compatible with Hive, and can use the JDBC / ODBC connection method to execute SQL statements. It provides important technical support for the Spark framework in structured data analysis. Converting an SQL statement into an executable plan tree means parsing and converting the SQL query statement into an internal representation form. The executable plan tree is a data structure that describes the specific execution steps of the query, including data sources, operators, connection methods, etc.

[0048] S130. Output resource usage information and the DPU executable plan tree.

[0049] Specifically, the resource usage information of the DPU node can be displayed on the Spark Web UI, and the Spark Web UI can print the DPU executable plan tree on the web page by calling the printing function of the executable plan tree. However, it is not limited to this.

[0050] It can be understood that the DPU node is mainly used for Spark SQL jobs. Printing both the DPU executable plan tree of the DPU node and the CPU executable physical plan tree of the CPU node on the Spark Web UI enables users to understand how the operators in the SQL statement are unloaded and computationally optimized based on the DPU executable plan tree.

[0051] In the embodiments of the present disclosure, the DPU hybrid cluster includes at least one CPU node and at least one DPU node. According to the embodiments of the present disclosure, it is possible to monitor not only at least one CPU node but also at least one DPU node. Specifically, it is possible to call the driver of the DPU node to obtain the resource usage information of the DPU node; obtain the DPU executable plan tree of the DPU node, where the DPU executable plan tree is obtained by the DPU node according to the SQL statement it executes; output the resource usage information and the DPU executable plan tree. It can be seen that by adopting the above technical solution, it is possible to monitor both the CPU nodes and the DPU nodes in the cluster and output the monitoring results of the DPU nodes, enabling users to timely understand information such as the running status and job situation of the DPU nodes. In this way, the comprehensiveness of monitoring the DPU hybrid cluster can be improved.

[0052] In another implementation manner of the present disclosure, the method further includes: when the data block corresponding to the original data is transferred to the DPU, determining the data processing progress information of the original data according to the resource usage information;

[0053] Output the data processing progress information.

[0054] Specifically, the data processing progress information refers to information about the data segmentation status and task execution progress of the original data during the data processing process.

[0055] Optionally, the data processing progress information includes the number of data segments, the number of completed data blocks, and / or the number of data blocks to be processed.

[0056] Among them, the number of data segments is the total number of data blocks into which the original data is segmented.

[0057] Among them, the number of completed data blocks is the number of data blocks that have been processed.

[0058] Among them, the number of data blocks to be processed is the number of data blocks that have not started processing yet.

[0059] Specifically, the data processing progress information can be displayed on the Spark Web UI, but is not limited thereto.

[0060] It can be understood that each time the data block corresponding to the original data is transferred from the host memory to the DPU node memory, according to the resource usage information, it is counted how many data blocks the original data is divided into (data segmentation quantity), how many data blocks have been calculated (i.e., the number of completed data blocks), and how many data blocks are left waiting for calculation (the number of data blocks to be processed), and are displayed on the Spark Web UI, enabling users to clearly see the data processing progress of the original data, enhancing transparency and trust, and the real-time data processing progress information helps users make better decisions.

[0061] In another implementation manner of the present disclosure, the method further includes: comparing the DPU executable plan tree with the corresponding DPU executable plan tree to obtain comparison result information, where the DPU executable plan tree and the corresponding CPU executable plan tree are converted from the same SQL statement, and the CPU executable plan tree is converted from the CPU node;

[0062] Outputting the comparison result information.

[0063] Specifically, since the executable plan tree converted from the SQL statement using the CPU node (i.e., the CPU executable plan tree) is different from the executable plan tree converted from the SQL statement using the DPU node (i.e., the DPU executable plan tree), therefore, when monitoring the DPU node, the CPU executable plan tree and the DPU executable plan tree converted from the same SQL statement can be compared to obtain the comparison result information. In this way, it is convenient for users to intuitively obtain the comparison result information.

[0064] Optionally, comparing the DPU executable plan tree with the corresponding DPU executable plan tree to obtain comparison result information includes:

[0065] Comparing the DPU executable plan tree with the corresponding CPU executable plan tree to determine the operators unloaded to the DPU node for calculation, the operators to be optimized and their corresponding optimization methods, and / or whether there is a mixed calculation of the DPU node and the CPU node.

[0066] Specifically, by comparing the DPU executable plan tree with the corresponding CPU executable plan tree, it can be determined which operators are offloaded to the DPU node for calculation, whether optimization processing is performed between the calculations of the operators and what specific optimizations are performed (such as merging adjacent operators), and / or whether there is hybrid calculation between the DPU node and the CPU node.

[0067] It can be understood that by determining the operators offloaded to the DPU node for calculation, the operators undergoing optimization processing and their corresponding optimization methods, and / or whether there is hybrid calculation between the DPU node and the CPU node, the user can understand the execution situation of the SQL statement in detail. Furthermore, based on this, it is possible to deeply analyze whether the operators that are more suitable for execution on the DPU node are actually executed on the DPU node, whether the optimization processing between the calculations of the operators is reasonable, etc., so as to deeply understand the resource utilization of the DPU hybrid cluster.

[0068] In another implementation manner of the present disclosure, the monitoring results of at least one CPU node are displayed in the first display area, and the monitoring results of at least one DPU node are displayed in the second display area, where the monitoring results of at least one DPU node include: resource usage information and the DPU executable plan tree, including resource usage information, the DPU executable plan tree, data processing progress information, and comparison result information.

[0069] Specifically, the first display area and the second display area can be different areas of the same page or different pages, and this is not limited.

[0070] It can be understood that by separately displaying the monitoring results of the CPU node and the monitoring results of the DPU node, the respective display interfaces are made more concise and clear, and the user can find the information they care about more quickly. For example, when the user wants to view the monitoring results of the DPU node, they can quickly locate the monitoring results of the DPU node, and the same is true for the monitoring results of the CPU node. In this way, the user experience can be improved.

[0071] Exemplarily, a specific example is used below to elaborate in detail on the method provided by the embodiments of the present disclosure. Based on the original Spark WEB UI, an extension is made to monitor and count the DPU nodes in the DPU hybrid cluster as computing resources. Specifically, the SparkDPUInfo class is used to call the driver of the DPU node to obtain the resource usage information of each DPU node, and it is displayed as an additional independent page on the Spark WEB UI. Therefore, the number of DPU nodes in the entire DPU hybrid cluster, the current status of each DPU node, including resource usage information such as memory usage, the number of completed tasks, the total amount of assigned tasks, and the current temperature, will be displayed on the Spark WEB UI. For example, each worker node will call the driver of the DPU node to request this resource usage information, and at the same time, transmit this resource usage information back to the master node in the DPU hybrid cluster and display it on the Spark WEB UI. In this way, users can monitor the current status of the DPU nodes in the entire DPU hybrid cluster through the extended SparkWEB UI of Spark, without the need to start an additional dedicated DPU node monitoring software for monitoring. And the DPU executable plan tree will be printed on the Spark WEB UI just like the CPU executable physical plan tree, and the comparison result information between the two can also be printed on the Spark WEB UI. In this way, users can understand how the operators in the SQL statement are unloaded and computationally optimized based on the DPU executable physical plan tree. In summary, according to the embodiments of the present disclosure, the DPU node can be presented as an additional computing resource on the Spark WEB UI, making up for the defect that the Spark WEB UI can only display the host memory, CPU working conditions, and CPU executable plan tree in the cluster, and being able to display more information about the DPU node on the Spark WEB UI to help users monitor the scenario where the DPU node is used as a computing resource for computing.

[0072] Figure 2 FIG. is a schematic structural diagram of a monitoring device for a DPU hybrid cluster provided by an embodiment of the present disclosure. The monitoring device for the DPU hybrid cluster can be understood as the above-mentioned electronic device or a partial functional module in the above-mentioned electronic device. The DPU hybrid cluster includes at least one CPU node and at least one DPU node. The monitoring device for the DPU hybrid cluster includes a first monitoring module for monitoring the at least one CPU node. The monitoring device for the DPU hybrid cluster further includes: a second monitoring module for monitoring the at least one DPU node. Among them, as Figure 2 shown, the second monitoring module includes:

[0073] The first acquisition module 210 is configured to call the driver of the DPU node to obtain the resource usage information of the DPU node;

[0074] The second acquisition module 220 is configured to obtain the DPU executable plan tree of the DPU node, where the DPU executable plan tree is obtained by the DPU node according to the SQL statement it executes;

[0075] The first output module 230 is configured to output the resource usage information and the DPU executable plan tree.

[0076] Optionally, the first monitoring module is specifically configured to monitor the at least one CPU node through the Spark Web UI, and the SparkDPUInfo class is integrated in the Spark Web UI;

[0077] The first acquisition module 210 is specifically configured to call the driver of the DPU node through the SparkDPUInfo class to obtain the resource usage information of the DPU node.

[0078] Optionally, the device further includes: a first determination module, configured to determine the data processing progress information of the original data according to the resource usage information when the data block corresponding to the original data is transferred to the DPU;

[0079] The second output module is configured to output the data processing progress information.

[0080] Optionally, the resource usage information includes: the total amount of DPU tasks, the number of completed DPU tasks, the DPU task progress, the DPU occupied memory, the DPU temperature, and / or the DPU power consumption;

[0081] The data processing progress information includes the number of data segments, the number of completed data blocks, and / or the number of data blocks to be processed.

[0082] Optionally, the device further includes: a first comparison module, configured to compare the DPU executable plan tree with the corresponding DPU executable plan tree to obtain comparison result information, where the DPU executable plan tree and the corresponding CPU executable plan tree are obtained by converting the same SQL statement, and the CPU executable plan tree is obtained by the CPU node;

[0083] The third data module is configured to output the comparison result information.

[0084] Optionally, the first comparison module is specifically configured to compare the DPU executable plan tree with the corresponding CPU executable plan tree, to determine the operators unloaded to the DPU node for calculation, the operators to be optimized and their corresponding optimization methods, and / or whether there is hybrid calculation between the DPU node and the CPU node.

[0085] Optionally, the monitoring results of the at least one CPU node are displayed in a first display area, and the monitoring results of the at least one DPU node are displayed in a second display area, where the monitoring results of the at least one DPU node include: resource usage information, and the DPU executable plan tree includes the resource usage information, the DPU executable plan tree, data processing progress information, and comparison result information.

[0086] The device provided in this embodiment can execute the method of any of the above embodiments, and its execution manner and beneficial effects are similar, which will not be elaborated here.

[0087] The embodiments of the present disclosure also provide an electronic device, which includes: a memory, in which a computer program is stored; a processor, configured to execute the computer program, and when the computer program is executed by the processor, the method of any of the above embodiments can be implemented.

[0088] Exemplarily, Figure 3 is a schematic structural diagram of an electronic device in the embodiments of the present disclosure. Specifically refer to the following Figure 3 , which shows a schematic structural diagram of an electronic device 300 suitable for implementing the embodiments of the present disclosure. The electronic device 300 in the embodiments of the present disclosure may include, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Tablet Computers), PMPs (Portable Multimedia Players), vehicle terminals (such as vehicle navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 3 The electronic device shown is only an example, and should not bring any limitation to the functions and usage scope of the embodiments of the present disclosure.

[0089] As Figure 3 shown, the electronic device 300 may include a processing device (such as a central processing unit, a graphics processing unit, etc.) 301, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 302 or a program loaded from a storage device 308 into a random access memory (RAM) 303. In the RAM 303, various programs and data required for the operation of the electronic device 300 are also stored. The processing device 301, the ROM 302, and the RAM 303 are connected to each other through a bus 304. An input / output (I / O) interface 305 is also connected to the bus 304.

[0090] Typically, the following devices can be connected to the I / O interface 305: an input device 306 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 307 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 308 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 309. The communication device 309 can allow the electronic device 300 to communicate with other devices wirelessly or wiredly to exchange data. Although Figure 3 the electronic device 300 with various devices is shown, it should be understood that it is not required to implement or have all the shown devices. Instead, more or fewer devices can be implemented or had.

[0091] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program includes program codes for performing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through the communication device 309, or installed from the storage device 308, or installed from the ROM 302. When the computer program is executed by the processing device 301, the above functions defined in the method of the embodiment of the present disclosure are performed.

[0092] It should be noted that the computer-readable medium described above can be a computer-readable signal medium, a computer-readable storage medium, or any combination of the two. A computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the above. More specific examples of the computer-readable storage medium can include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, the computer-readable storage medium can be any tangible medium that contains or stores a program, and this program can be used by or in combination with an instruction execution system, apparatus, or device. In the present disclosure, the computer-readable signal medium can include a data signal propagated in a baseband or as part of a carrier wave, which carries computer-readable program code. Such a propagated data signal can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer-readable signal medium can also be any computer-readable medium other than the computer-readable storage medium, and this computer-readable signal medium can send, propagate, or transmit a program for use by or in combination with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted by any appropriate medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0093] In some embodiments, the client and the server can communicate using any currently known or future-developed network protocol such as HTTP (HyperText Transfer Protocol), and can be interconnected with digital data communication in any form or medium (e.g., a communication network). Examples of communication networks include local area networks ("LAN"), wide area networks ("WAN"), the Internet (e.g., the Internet), and end-to-end networks (e.g., ad hoc end-to-end networks), as well as any currently known or future-developed networks.

[0094] The above computer-readable medium can be included in the above electronic device; or it can exist separately without being assembled into the electronic device.

[0095] The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by the electronic device, the electronic device is caused to: call the driver of the DPU node to obtain the resource usage information of the DPU node;

[0096] Obtain the DPU executable plan tree of the DPU node, where the DPU executable plan tree is obtained by the DPU node according to the SQL statement it executes;

[0097] Output the resource usage information and the DPU executable plan tree.

[0098] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages or combinations thereof. The above-mentioned programming languages include, but are not limited to, object-oriented programming languages such as Java, Smalltalk, C++, and also include conventional procedural programming languages such as the "C" language or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, executed as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (for example, by connecting through the Internet using an Internet service provider).

[0099] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architectures, functions, and operations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagram may represent a module, a program segment, or a part of code that contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than marked in the accompanying drawings. For example, two consecutive blocks shown may actually be executed substantially in parallel, and they may sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and / or flowchart, and the combinations of blocks in the block diagram and / or flowchart, may be implemented by a dedicated hardware-based system for performing the specified functions or operations, or may be implemented by a combination of dedicated hardware and computer instructions.

[0100] The units involved in the embodiments of the present disclosure may be implemented in software or in hardware. Wherein, the name of the unit does not constitute a limitation to the unit itself in some cases.

[0101] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that can be used include: Field Programmable Gate Arrays (FPGAs), Application Specific Integrated Circuits (ASICs), Application Specific Standard Products (ASSPs), Systems on Chip (SOCs), Complex Programmable Logic Devices (CPLDs), and the like.

[0102] In the context of this disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in connection with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a Read-Only Memory (ROM), an Erasable Programmable Read-Only Memory (EPROM or Flash memory), an optical fiber, a portable Compact Disc Read-Only Memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0103] Embodiments of this disclosure also provide a computer-readable storage medium storing a computer program which, when executed by a processor, can implement the method of any of the foregoing embodiments. The execution manner and beneficial effects are similar and will not be elaborated here.

[0104] It should be noted that in this document, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the term "comprising", "including" or any other variation thereof is intended to cover a non-exclusive inclusion, such that a process, method, article or device comprising a series of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article or device. Without further limitation, an element defined by the phrase "comprising a..." does not exclude the presence of additional identical elements in the process, method, article or device comprising the element.

[0105] The above are only specific embodiments of the present disclosure, enabling those skilled in the art to understand or implement the present disclosure. Various modifications to these embodiments will be obvious to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure will not be limited to these embodiments described herein, but rather to the broadest scope consistent with the principles and novel features disclosed herein.

Claims

1. A monitoring method for a DPU hybrid cluster, the DPU hybrid cluster including at least one CPU node and at least one DPU node, the method including monitoring the at least one CPU node, characterized in that, The method further includes monitoring the at least one DPU node, where monitoring the at least one DPU node includes: Invoking the driver of the DPU node to obtain the resource usage information of the DPU node; Obtaining the DPU executable plan tree of the DPU node, where the DPU executable plan tree is obtained by the DPU node according to the SQL statement it executes; Outputting the resource usage information and the DPU executable plan tree; The method further includes: comparing the DPU executable plan tree with the corresponding CPU executable plan tree to determine the operators unloaded to the DPU node for calculation, the operators to be optimized and their corresponding optimization methods, and / or whether there is hybrid calculation between the DPU node and the CPU node, where the DPU executable plan tree and the corresponding CPU executable plan tree are obtained by converting the same SQL statement, and the CPU executable plan tree is obtained by the CPU node.

2. The method according to claim 1, wherein the monitoring of the at least one CPU node includes monitoring the at least one CPU node through the Spark Web UI, characterized in that The Spark Web UI integrates a SparkDPUInfo class; The invoking the driver of the DPU node to obtain the resource usage information of the DPU node includes: Invoking the driver of the DPU node through the SparkDPUInfo class to obtain the resource usage information of the DPU node.

3. The method according to claim 1, wherein It further includes: When the data block corresponding to the original data is transferred to the DPU, determining the data processing progress information of the original data according to the resource usage information; Outputting the data processing progress information.

4. The method according to claim 3, wherein The resource usage information includes: the total amount of DPU tasks, the number of completed DPU tasks, the DPU task progress, the DPU memory occupancy, the DPU temperature, and / or the DPU power consumption; The data processing progress information includes the number of data partitions, the number of completed data blocks, and / or the number of data blocks to be processed.

5. The method according to claim 1, characterized in that, It further includes: Outputting the comparison result information.

6. According to the method described in any one of claims 1-5, the monitoring results of the at least one CPU node are displayed in the first display area, and the monitoring results of the at least one DPU node are displayed in the second display area, wherein, The monitoring result of the at least one DPU node includes: the resource usage information and the DPU executable plan tree includes the resource usage information, the DPU executable plan tree, the data processing progress information, and the comparison result information.

7. A monitoring device for a DPU hybrid cluster, the DPU hybrid cluster including at least one CPU node and at least one DPU node, the device including a first monitoring module for monitoring the at least one CPU node, characterized in that, The device further includes a second monitoring module for monitoring the at least one DPU node, where the second monitoring module includes: A first obtaining module for invoking the driver of the DPU node to obtain the resource usage information of the DPU node; A second obtaining module for obtaining the DPU executable plan tree of the DPU node, where the DPU executable plan tree is obtained by the DPU node according to the SQL statement it executes; A first output module for outputting the resource usage information and the DPU executable plan tree; The device further includes: comparing the DPU executable plan tree and its corresponding CPU executable plan tree to determine the operators unloaded to the DPU node for calculation, the operators to be optimized and their corresponding optimization methods, and / or whether there is hybrid calculation between the DPU node and the CPU node, where the DPU executable plan tree and its corresponding CPU executable plan tree are obtained by converting the same SQL statement, and the CPU executable plan tree is obtained by converting the CPU node.

8. An electronic device, characterized in that, including: a processor and a memory, where a computer program is stored in the memory, and when the computer program is executed by the processor, the processor executes the method according to any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, A computer program is stored in the storage medium, and when the computer program is executed by a processor, the method according to any one of claims 1-6 is implemented.

Images