Cluster performance analysis system, method and device, storage medium and electronic equipment

Abstract

The invention relates to the technical field of computers, in particular to a cluster performance analysis system, method and device, a storage medium and electronic equipment.The system comprises a server side and a front end, and the front end is used for sending a monitoring data obtaining request to the server side to request monitoring data; the monitoring data comprises hardware monitoring data and module performance data, the hardware monitoring data represents the use condition of hardware resources in a cluster, and the module performance data represents the performance during module execution; the server side is used for responding to a monitoring data acquisition request and sending monitoring data to the front end; the front end is used for receiving the monitoring data and displaying the monitoring data in multiple levels according to multiple display levels, and the multiple levels comprise overall data of a cluster level and refined data of a node level. According to the embodiment of the invention, the user can identify the performance bottleneck more accurately.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to a cluster performance analysis system, method, apparatus, storage medium, and electronic device. Background Technology

[0002] In the field of cluster monitoring for high-performance computing and module cluster performance analysis, although existing performance monitoring technologies can provide basic system resource status, such as total cluster-level data on CPU utilization, memory usage, network traffic, and Remote Direct Memory Access (RDMA) throughput, existing technologies mainly rely on simple resource monitoring methods, displaying the overall resource utilization of the training process through line graphs on the monitoring page.

[0003] However, this approach typically only provides overall or average data at the cluster level. This makes it difficult to identify the usage and bottlenecks of specific nodes or hardware resources in large-scale cluster environments, thus limiting the depth and accuracy of performance analysis. Summary of the Invention

[0004] This disclosure proposes a cluster performance analysis technique.

[0005] According to one aspect of this disclosure, a cluster performance analysis system is provided for use in a model training cluster, comprising a server and a front-end, wherein:

[0006] The front end is used to send a monitoring data acquisition request to the server to request monitoring data; the monitoring data includes: hardware monitoring data and module performance data, wherein the hardware monitoring data represents the usage of hardware resources in the cluster, and the module performance data represents the performance of the module during execution;

[0007] The server is used to send monitoring data to the front end in response to a monitoring data acquisition request;

[0008] The front end is used to receive the monitoring data and display the monitoring data in multiple display levels, including: overall data at the cluster level and refined data at the node level.

[0009] In one possible implementation, the plurality of levels includes:

[0010] The Model Computing Power Utilization (MFU) level is used to demonstrate the computing power utilization in the cluster.

[0011] The control plane node hierarchy is used to display the utilization and memory usage of the control plane nodes;

[0012] The worker node level displays monitoring data from the worker nodes that actually perform the computational tasks.

[0013] In one possible implementation, the module performance data includes call information between modules during task execution in the cluster, communication information between GPU kernel functions, and the execution time of each module.

[0014] In one possible implementation, the front end is used to render the length of the corresponding graphical controls of each module in the interface according to the execution time of each module.

[0015] The graphical control is used to respond to user commands and display the module performance data of the corresponding module.

[0016] In one possible implementation, the server is configured to delete data whose execution time is less than a set threshold based on the execution time of each module, thereby obtaining filtered module performance data.

[0017] In one possible implementation, the front end includes a time period control component and a monitoring data display component;

[0018] The time period control component at the front end is used to receive the time period information of the monitoring data input by the user, and after adding the time period information to the monitoring data acquisition request, send the monitoring data acquisition request to the server.

[0019] The server, in response to the monitoring data acquisition request, filters the pre-stored monitoring data based on the time period information and returns the filtered monitoring data to the front-end monitoring data display component.

[0020] The front-end monitoring data display component displays the received monitoring data according to multiple display levels.

[0021] In one possible implementation, the front end constructs a tree structure of nodes with multiple levels based on the hierarchy in the received monitoring data.

[0022] The front end maps the received monitoring data to nodes at each level of the tree structure and renders the tree structure.

[0023] In one possible implementation, the front-end and the server communicate via a full-duplex, bidirectional WebSocket connection.

[0024] The server obtains monitoring data from the data acquisition module via a WebSocket connection and preprocesses the obtained monitoring data. The preprocessing includes converting the raw monitoring data into data in a standardized format adapted to the front-end components.

[0025] According to another aspect of this disclosure, a cluster performance analysis method is provided, applied to a model training cluster, comprising:

[0026] Send a monitoring data retrieval request to the server to request monitoring data; the monitoring data includes: hardware monitoring data and module performance data, wherein the hardware monitoring data represents the usage of hardware resources in the cluster, and the module performance data represents the performance of the module during execution;

[0027] The system receives the monitoring data sent by the server and displays the monitoring data at multiple levels, including: overall data at the cluster level and detailed data at the node level.

[0028] In one possible implementation, the plurality of levels includes:

[0029] The Model Computing Power Utilization (MFU) level is used to demonstrate the computing power utilization in the cluster.

[0030] The control plane node hierarchy is used to display the utilization and memory usage of the control plane nodes;

[0031] The worker node level displays monitoring data from the worker nodes that actually perform the computational tasks.

[0032] In one possible implementation, the module performance data includes call information between modules during task execution in the cluster, communication information between GPU kernel functions, and the execution time of each module.

[0033] In one possible implementation, the method further includes: rendering the length of the corresponding graphical control in the interface according to the execution time of each module; the graphical control is used to display the module performance data of the corresponding module in response to user instructions.

[0034] In one possible implementation, the method further includes: receiving time period information of monitoring data input by the user, adding the time period information to the monitoring data acquisition request, and sending the monitoring data acquisition request to the server; so that the server, in response to the monitoring data acquisition request, filters the pre-stored monitoring data based on the time period information, and returns the filtered monitoring data to the front-end monitoring data display component.

[0035] The received monitoring data is displayed in multiple display levels.

[0036] In one possible implementation, the method further includes:

[0037] Based on the hierarchy in the received monitoring data, construct a tree structure with nodes of multiple levels;

[0038] The received monitoring data is mapped to nodes at each level of the tree structure, and the tree structure is then rendered.

[0039] In one possible implementation, the front-end and the server are connected via a full-duplex bidirectional WebSocket connection. The server obtains monitoring data from the data acquisition module through the WebSocket connection and preprocesses the obtained monitoring data. The preprocessing includes converting the original monitoring data into data in a standardized format adapted to the front-end components.

[0040] According to another aspect of this disclosure, a cluster performance analysis apparatus is provided for use in a model training cluster, comprising:

[0041] The request sending module is used to send a monitoring data acquisition request to the server to request monitoring data; the monitoring data includes: hardware monitoring data and module performance data, wherein the hardware monitoring data represents the usage of hardware resources in the cluster, and the module performance data represents the performance of the module during execution;

[0042] The data receiving module is used to receive the monitoring data sent by the server and display the monitoring data in multiple display levels, including: overall data at the cluster level and detailed data at the node level.

[0043] According to one aspect of this disclosure, an electronic device is provided, comprising: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to invoke the instructions stored in the memory to perform the method described above.

[0044] According to one aspect of this disclosure, a computer-readable storage medium is provided that stores computer program instructions thereon, which, when executed by a processor, implement the above-described method.

[0045] In this embodiment, a multi-level, fine-grained data display method allows users to clearly see the hardware resource usage and performance data of each node and its subordinate resources in the model training cluster. This not only improves data readability but also helps users more accurately identify the usage and performance bottlenecks of specific nodes or hardware resources. By combining the display of hardware resource usage and module performance, users can optimize resource allocation in the model training cluster and improve the overall performance of the cluster.

[0046] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Other features and aspects of this disclosure will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0047] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the specification, serve to illustrate the technical solutions of this disclosure.

[0048] Figure 1 A block diagram of a cluster performance analysis system according to an embodiment of the present disclosure is shown.

[0049] Figure 2 A structural diagram of the cluster performance analysis system provided in an embodiment of this disclosure is shown.

[0050] Figure 3 A flowchart illustrating a cluster performance analysis method according to an embodiment of the present disclosure is shown.

[0051] Figure 4 A block diagram of a cluster performance analysis apparatus according to an embodiment of the present disclosure is shown.

[0052] Figure 5 A block diagram of an electronic device according to an embodiment of the present disclosure is shown. Detailed Implementation

[0053] Various exemplary embodiments, features, and aspects of this disclosure will now be described in detail with reference to the accompanying drawings. The same reference numerals in the drawings denote elements that have the same or similar functions. Although various aspects of the embodiments are shown in the drawings, they are not necessarily drawn to scale unless specifically indicated otherwise.

[0054] The term “exemplary” as used herein means “serving as an example, embodiment, or illustration.” Any embodiment illustrated herein as “exemplary” is not necessarily to be construed as superior to or better than other embodiments.

[0055] In this document, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent three cases: A alone, A and B simultaneously, and B alone. Furthermore, the term "at least one" in this document means any combination of at least two of any one or more elements. For example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C.

[0056] Furthermore, to better illustrate this disclosure, numerous specific details are set forth in the following detailed description. Those skilled in the art will understand that this disclosure can be practiced without certain specific details. In some instances, methods, means, components, and circuits well known to those skilled in the art have not been described in detail in order to highlight the main points of this disclosure.

[0057] Figure 1 A block diagram of a cluster performance analysis system according to an embodiment of the present disclosure is shown, such as Figure 1 As shown, the system is applied to a model training cluster and includes a server 101 and a front-end 102, wherein:

[0058] The front end 102 is used to send a monitoring data acquisition request to the server to request monitoring data; the monitoring data includes: hardware monitoring data and module performance data, wherein the hardware monitoring data represents the usage of hardware resources in the cluster, and the module performance data represents the performance of the module during execution;

[0059] The server 101 is used to send monitoring data to the front end in response to a monitoring data acquisition request;

[0060] The front end 102 is used to receive the monitoring data and display the monitoring data in multiple display levels, including: overall data at the cluster level and refined data at the node level.

[0061] The cluster performance analysis system disclosed herein is used to analyze the performance of large-scale clusters (especially in the scenario of large model training clusters). Model training clusters typically involve a large number of computing resources such as GPUs and CPUs, as well as many software modules in the model. This makes it difficult to identify the usage and bottlenecks of specific nodes or hardware resources in a large-scale cluster environment.

[0062] The server-side component receives monitoring data retrieval requests from the front-end and processes these requests to return the corresponding monitoring data. This data includes hardware monitoring data and module performance data.

[0063] The front end provides an intuitive and easy-to-use user interface. This interface is the window through which users interact with the system. Users can use it to send monitoring data retrieval requests and view the returned monitoring data. The monitoring data retrieval request can include the time period for which the monitoring data is to be viewed.

[0064] When a user needs to view cluster performance data, the frontend generates a monitoring data retrieval request. This request can include key information such as the data type the user wants to obtain (e.g., hardware monitoring data or module performance data) and the time period. The frontend sends this request to the server via the network (e.g., a full-duplex WebSocket connection).

[0065] After receiving a request from the front end, the server will obtain the corresponding monitoring data from the data acquisition module based on the information in the request.

[0066] The data acquisition module can include a hardware monitoring module (such as Prometheus) and a module performance monitoring module (such as Profiler Server). These two modules are responsible for collecting hardware monitoring data and module performance data from the cluster, respectively. The hardware monitoring module collects hardware monitoring data from each hardware node in the cluster. This data characterizes the usage of hardware resources (such as CPU, GPU, memory, network, etc.) in the cluster. This data typically includes key indicators such as resource utilization, power consumption, and frequency, allowing users to understand the allocation and usage of hardware resources in the cluster.

[0067] The module performance monitoring module is specifically designed for training large models, collecting performance data of software modules such as large models during the training process. Module performance data characterizes the performance of a module during execution, such as function call patterns within the module, and communication and computation data from GPU kernel functions. This data helps users deeply analyze module execution efficiency, identify performance bottlenecks, and optimize them, thereby improving training efficiency.

[0068] The server processes the collected data and returns it to the front end via the network. Specific processing steps can be found in the possible implementations provided in this disclosure, and will not be elaborated upon here.

[0069] After receiving the monitoring data, the front end will display the monitoring data in multiple levels and render the monitoring data on the user interface to facilitate users to view and analyze the cluster's performance.

[0070] The multi-level data includes overall data at the cluster level and granular data at the node level. The overall cluster-level data represents the cluster's overall performance, such as the overall Model FLOPs Utilization (MFU), which helps users quickly understand the cluster's overall operational status. The granular node-level data includes detailed performance data for each node, such as CPU and GPU usage, network traffic, and RDMA throughput. This fine-grained visualization allows users to gain a deeper understanding of each node's resource utilization, thereby identifying potential resource bottlenecks or performance issues.

[0071] Furthermore, this disclosure provides comprehensive monitoring data. Through hardware monitoring data, it can provide users with a complete view of cluster hardware resources. Through module performance data, it can help users gain a deeper understanding of the details of module execution, thereby optimizing module performance.

[0072] In this embodiment, a multi-level, fine-grained data display method allows users to clearly see the hardware resource usage and performance data of each node and its subordinate resources in the model training cluster. This not only improves data readability but also helps users more accurately identify the usage and performance bottlenecks of specific nodes or hardware resources. By combining the display of hardware resource usage and module performance, users can optimize resource allocation in the model training cluster and improve the overall performance of the cluster.

[0073] In one possible implementation, the multiple levels include: a Model Utilization (MFU) level, used to display the computing power utilization in the cluster; a control plane node level, used to display the utilization rate and memory usage of the control plane nodes; and a worker node level, used to display monitoring data of the worker nodes that actually perform computing tasks.

[0074] The MFU (Model-Future-Function) level displays the model computing power utilization within the cluster. This utilization provides a macro-level monitoring perspective, allowing users to quickly understand the overall computing power usage of the cluster during model training. Through this level of display, users can intuitively see whether the cluster's computing power is being fully utilized, and whether there are any instances of idle or overloaded computing power.

[0075] The control plane node (Master) is responsible for cluster management and control functions, including receiving user requests, scheduling and allocating resources, and monitoring cluster status. At the control plane node level, the primary data displayed is the node's utilization and memory usage. This data is crucial for evaluating cluster management efficiency, identifying potential performance bottlenecks, and optimizing resource allocation strategies. Users can gain a deeper understanding of the control plane node's resource usage through this level of display, enabling them to make corresponding adjustments and optimizations.

[0076] Worker nodes are the nodes that actually perform computational tasks. Each worker node contains resources such as CPU and GPU, which are used for tasks like model training. At the worker node level, detailed monitoring data for the worker nodes that actually perform computational tasks is displayed. This monitoring data includes key performance indicators such as CPU utilization, memory usage, network traffic, and RDMA throughput.

[0077] Users can delve into detailed monitoring data for individual worker nodes or even individual GPUs by expanding the data layer by layer. This fine-grained data visualization helps users accurately identify resource allocation issues, performance bottlenecks, and potential optimization opportunities.

[0078] In this embodiment of the disclosure, by displaying data at multiple levels such as MFU level, control plane node level and worker node level, a more comprehensive, in-depth and detailed monitoring perspective is provided to users, which makes it easier for users to better monitor the large model training process and optimize the performance of the cluster during the large model training process.

[0079] In one possible implementation, the module performance data includes call information between modules during task execution in the cluster, communication information between GPU kernel functions, and the execution time of each module.

[0080] In this implementation, the module performance data can cover multiple key nodes of the cluster when executing training tasks, providing users with detailed performance observation dimensions.

[0081] Specifically, module performance data can include call information between modules. When training large module models in a cluster, multiple modules or components work together. Therefore, the call relationships between these modules can be recorded and displayed, including the order and frequency of calls, as well as their dependencies. This helps users understand the module execution flow and identify potential bottlenecks or redundant calls.

[0082] Module performance data can also include communication information between GPU kernel functions. In large-scale cluster training, the GPU serves as the primary computing resource, and communication between its kernel functions is crucial to overall performance. Therefore, it is possible to capture and display the communication details between GPU kernel functions, including communication time, data volume, and communication mode (such as synchronous or asynchronous). This allows users to analyze GPU resource utilization, optimize communication strategies between kernel functions, and reduce communication overhead.

[0083] Module performance data can also include the execution time of each module, that is, the time spent by each module from the start to the end of execution. This makes it easier for users to understand the distribution of the execution time of each module during the entire training process. Users can intuitively understand the execution efficiency of each part of the module, identify modules or steps that take a long time, and thus optimize them in a targeted manner.

[0084] In this embodiment, the module performance data includes call information between modules during task execution within the cluster, communication information between GPU kernel functions, and the execution time of each module, providing users with fine-grained performance analysis data. In scenarios involving large-scale cluster and large model training, this data not only helps users understand the cluster's behavior patterns during task execution but also assists them in performance tuning, resource allocation, and module optimization.

[0085] In one possible implementation, the front end is used to render the length of the corresponding graphical control in the interface according to the execution time of each module; the graphical control is used to display the module performance data of the corresponding module in response to user instructions.

[0086] The front-end interface can dynamically render the length of the corresponding graphical controls for each module based on the execution time of each module in the module performance data. Through intuitive visual elements, it displays the performance of each module during task execution to the user.

[0087] Specifically, the front-end receives module performance data from the back-end server. Based on this data, it dynamically generates or adjusts graphical controls on the interface, such as bar charts or timelines, to represent the execution time of each module.

[0088] In addition, graphical controls can respond to user commands. When a user clicks or hovers over a graphical control, the front end displays detailed performance data for the corresponding module. This includes information such as specific function calls, communication details of GPU kernel functions, and execution time of computational tasks.

[0089] In this embodiment, the front-end can render the length of the corresponding graphical controls in the interface according to the execution time of each module. The graphical controls, responding to user commands, display the module performance data of the corresponding module. This interactive design allows users to gain a deeper understanding of performance bottlenecks and resource allocation during task execution. Users can quickly locate performance problems and take corresponding optimization measures by observing the length of the graphical controls and the detailed response data.

[0090] In one possible implementation, the server is configured to delete data whose execution time is less than a set threshold based on the execution time of each module, thereby obtaining filtered module performance data.

[0091] The server can filter module performance data based on the execution duration of each module. Specifically, after obtaining the raw module performance data, the server analyzes the execution duration of each module within this data. Then, based on a pre-set time threshold (e.g., 2 seconds), the server considers data with an execution duration less than this threshold as non-critical events or invalid data and removes this data from the dataset. This reduces the burden of data transmission and rendering while ensuring that users can focus on critical events that have a significant impact on performance analysis.

[0092] After filtering, the server sends the filtered module performance data to the front end for display. Upon receiving this data, the front end dynamically renders the length of the corresponding graphical controls for each module in the interface based on the module's execution time. These graphical controls not only visually display the execution time of each module but also allow users to further view detailed module performance data through interactive commands (such as clicking and hovering).

[0093] In this embodiment, the server deletes data whose execution time is less than a set threshold based on the execution time of each module, thus obtaining filtered module performance data. This effectively reduces the amount of data that users need to process during performance analysis, improves the overall system efficiency, and ensures that users can quickly locate performance bottlenecks and problems, thereby accelerating problem resolution.

[0094] In one possible implementation, the front end includes a time period control component and a monitoring data display component. The time period control component receives time period information of the monitoring data input by the user, adds the time period information to the monitoring data acquisition request, and sends the monitoring data acquisition request to the server. The server, in response to the monitoring data acquisition request, filters the pre-stored monitoring data based on the time period information and returns the filtered monitoring data to the monitoring data display component of the front end. The monitoring data display component of the front end displays the received monitoring data according to multiple display levels.

[0095] When training large models in ultra-large-scale clusters, monitoring performance data is crucial for resource allocation, task scheduling, and performance optimization. However, existing technologies lack flexibility in viewing data during long-term training tasks, failing to meet users' needs for selecting monitoring time ranges according to their requirements. In this embodiment, users can flexibly select the time period for monitoring data through a front-end time period control component.

[0096] The time period control component can be located in a prominent position on the front-end interface (such as the upper right corner) to receive monitoring data time period information input by the user. Users can set the time range of monitoring data to be viewed by selecting preset time periods (such as the most recent 10 minutes, 20 minutes, 30 minutes, 1 hour, 3 hours, 1 day, 3 days, etc.) or by entering a custom time period.

[0097] The monitoring data display component is used to display filtered monitoring data retrieved from the server. The data is organized according to multiple display levels and presented to the user in a tree structure or other easy-to-understand manner.

[0098] Users set the time period for the monitoring data they want to view through the time period control component. The front end will add this time period information to the monitoring data retrieval request and then send it to the server. This request includes the user-selected time period information and possible other filtering conditions (such as specific nodes, resource types, etc.).

[0099] After receiving the request, the server filters the pre-stored monitoring data based on the time period and possible filtering criteria. The filtered data can include detailed information such as CPU and GPU resource utilization, module execution details, etc.

[0100] The server returns the filtered monitoring data to the front-end monitoring data display component. This component then presents the data to the user in an appropriate manner based on its hierarchical structure and type.

[0101] The monitoring data display component shows the monitoring data of each node in the cluster and its subordinate resources (such as GPUs and CPUs) layer by layer, based on the user's selection. Users can click or expand nodes to view more detailed information.

[0102] In this embodiment, by introducing a time period control component and a monitoring data display component, a flexible time period selection function for monitoring data is achieved. Users can select specific time periods to view monitoring data according to their needs, thereby optimizing the accuracy and efficiency of analysis. Simultaneously, the use of layered data display and graphical controls improves the data visualization effect and the convenience of user interaction, making it more practical and applicable in performance analysis of large-scale cluster and large model training.

[0103] In one possible implementation, the front end constructs a tree structure with multiple levels of nodes based on the hierarchy in the received monitoring data; the front end maps the received monitoring data to the nodes at each level of the tree structure and renders the tree structure.

[0104] In this implementation, the front end can construct a tree structure with multiple levels of nodes based on the hierarchical information in the received monitoring data. This structure can clearly reflect the hierarchical relationship between each node in the cluster and its subordinate resources (such as CPU, GPU, etc.).

[0105] A tree structure can contain three root nodes: the MFU level, the control plane node level, and the worker node level.

[0106] in:

[0107] MFU level: No child nodes. The MFU level is mainly used to monitor the overall computing power utilization of cluster resources. It does not contain other child nodes, but is displayed as an overall performance indicator.

[0108] The control plane node hierarchy includes two child nodes: "Utilization" and "Memory Usage." The control plane node is responsible for cluster management and control functions. Its child nodes display key resource data for the control plane node, such as CPU utilization and memory usage, so that users can understand the resource consumption of the control plane.

[0109] Worker Node Hierarchy: Worker nodes are the nodes that actually execute computing tasks, including information on CPU, GPU, and other resources, as well as module performance data. The child nodes under the worker node hierarchy are the individual worker nodes. Expanding a single worker node can display child nodes for its hardware monitoring data and module performance data.

[0110] The sub-nodes of hardware monitoring data include hardware nodes such as GPUs and CPUs within the worker node. Each hardware node can be further expanded to display various monitoring data for that hardware. For example, when a single GPU node is expanded, it can display detailed metrics such as GPU utilization, memory usage, power consumption, and frequency.

[0111] The sub-nodes of the module performance data are displayed as module nodes. Each module node can be further expanded. When a single module node is expanded, it will display the call information between modules within that module, the communication information between GPU kernel functions, and the execution time of each module.

[0112] During the construction of the tree structure, the front-end can parse the hierarchical information in the monitoring data, correctly placing each node in the corresponding level of the tree structure. Simultaneously, the front-end can assign a unique identifier to each node for accurate mapping during subsequent data mapping and rendering. Then, the front-end maps the received monitoring data to nodes at each level of the tree structure, ensuring that the performance data of each node matches its position within the tree structure.

[0113] After the data mapping is complete, the front-end can render the tree structure. During the rendering process, the front-end dynamically generates and displays corresponding graphical controls based on the hierarchical relationship of the nodes, performance data, and user selections (such as time period, node type, etc.). The front-end adopts a hierarchical rendering method, that is, it renders the parent node first, and then expands the child nodes level by level as needed, so that when users view monitoring data, they can clearly see the relationship and differences between nodes at different levels.

[0114] In this embodiment, a tree structure is used to display monitoring data, providing users with an intuitive and clear view of performance monitoring. Users can expand the tree structure level by level to gain a deeper understanding of the performance of each node in the cluster and its subordinate resources. Simultaneously, the tree structure allows users to easily locate performance bottlenecks or anomalies, enabling targeted analysis and optimization.

[0115] In one possible implementation, the front-end and the server are connected via a full-duplex bidirectional WebSocket connection. The server obtains monitoring data from the data acquisition module through the WebSocket connection and preprocesses the obtained monitoring data. The preprocessing includes converting the original monitoring data into data in a standardized format adapted to the front-end components.

[0116] A WebSocket connection based on full-duplex bidirectional communication is established between the front-end and the server. Compared with the traditional Hypertext Transfer Protocol (HTTP) request / response model, this connection method has lower latency and higher real-time performance, allowing bidirectional data transmission over a single connection without needing to establish a new connection each time as with HTTP.

[0117] Once a WebSocket connection is established, the server can proactively push data to the front end, and the front end can also send requests or data to the server at any time. This mechanism is well-suited for real-time performance analysis systems, ensuring that the front-end interface reflects the performance data of each module in the cluster in real time.

[0118] The server retrieves monitoring data from the data acquisition module via a WebSocket connection. This data is raw and unprocessed, containing information such as hardware resource usage of each node in the cluster and module execution details.

[0119] To adapt this raw data to the front-end components, the server can preprocess the data to transform it into a standardized format that matches the requirements of the front-end components.

[0120] Specifically, the preprocessing process can include the following steps:

[0121] Data cleaning: Remove invalid or redundant data to ensure data accuracy and consistency.

[0122] Data conversion: Converting the format of raw data into a standardized format that can be recognized by front-end components.

[0123] Data aggregation: Based on the needs of front-end components, raw data is aggregated to generate a higher-level data view. For example, data from multiple nodes can be aggregated to generate a performance overview of the entire cluster.

[0124] In this embodiment, the WebSocket connection enables the front-end to receive data pushed by the server within milliseconds, ensuring the timeliness of performance data on the interface. Furthermore, unlike monitoring data from Perfetto and similar platforms that can only receive data in a fixed JavaScript Object Notation (JSON) file format, this disclosure allows the front-end to send requests or data to the server at any time as needed via WebSocket. Through the server-side data preprocessing, the raw data can be converted into a standardized format and adapted to front-end components, thereby improving the efficiency and accuracy of data processing.

[0125] Figure 2 The diagram shows the structure of the cluster performance analysis system provided in this embodiment of the disclosure, such as... Figure 2 As shown, the system includes a front-end and a server. The front-end includes a time period control component and a tree structure monitoring component, while the server includes a data processing module. The server obtains monitoring data from the hardware monitoring module and the module performance monitoring module.

[0126] The front-end time period control component can receive monitoring data time period information input by the user, such as the last 10 minutes, 20 minutes, 1 hour, etc., and add this information to the monitoring data retrieval request. After the user selects the time period through the front-end interface, the front-end sends the monitoring data retrieval request containing the time period to the server.

[0127] The server receives monitoring data retrieval requests from the frontend and, based on the time period and filtering criteria in the request, retrieves the corresponding monitoring data from the hardware monitoring module and the module performance monitoring module. The data processing module on the server preprocesses the monitoring data, performing operations such as cleaning, transformation, and aggregation on the raw monitoring data to generate data in a standardized format adapted to the frontend components. The preprocessed data is then pushed in real-time to the tree-structured monitoring component on the frontend via a WebSocket connection.

[0128] The tree-structured monitoring component receives monitoring data returned from the server and renders and displays the data according to multiple display levels (such as cluster level and node level). The data is presented in a tree structure, and users can click or expand nodes to view more detailed information, such as hardware monitoring data and module performance data for each node.

[0129] In one possible implementation, the cluster performance analysis system can be executed by electronic devices such as terminal devices or servers. The terminal devices can be user equipment (UE), mobile devices, user terminals, terminals, cellular phones, cordless phones, personal digital assistants (PDAs), handheld devices, computing devices, in-vehicle devices, wearable devices, etc. The system can be implemented by a processor calling computer-readable instructions stored in memory. Alternatively, the system can be implemented via a server.

[0130] In addition, this disclosure also provides cluster performance analysis methods, apparatus, electronic devices, computer-readable storage media, and programs, all of which can be used to implement any of the cluster performance analysis systems provided in this disclosure. The corresponding technical solutions and descriptions can be found in the relevant records in the method section, and will not be repeated here.

[0131] Figure 3 A flowchart illustrating a cluster performance analysis method according to an embodiment of this disclosure is shown, such as... Figure 3 As shown, the method is applied to a model training cluster, and the method includes:

[0132] In step S21, a monitoring data acquisition request is sent to the server to request monitoring data; the monitoring data includes: hardware monitoring data and module performance data, wherein the hardware monitoring data represents the usage of hardware resources in the cluster, and the module performance data represents the performance of the module during execution;

[0133] In step S22, the monitoring data sent by the server is received, and the monitoring data is displayed in multiple levels according to multiple display levels, including: overall data at the cluster level and detailed data at the node level.

[0134] In one possible implementation, the plurality of levels includes:

[0135] The Model Computing Power Utilization (MFU) level is used to demonstrate the computing power utilization in the cluster.

[0136] The control plane node hierarchy is used to display the utilization and memory usage of the control plane nodes;

[0137] The worker node level displays monitoring data from the worker nodes that actually perform the computational tasks.

[0138] In one possible implementation, the module performance data includes call information between modules during task execution in the cluster, communication information between GPU kernel functions, and the execution time of each module.

[0139] In one possible implementation, the method further includes: rendering the length of the corresponding graphical control in the interface according to the execution time of each module; the graphical control is used to display the module performance data of the corresponding module in response to user instructions.

[0140] In one possible implementation, the method further includes: receiving time period information of monitoring data input by the user, adding the time period information to the monitoring data acquisition request, and sending the monitoring data acquisition request to the server; so that the server, in response to the monitoring data acquisition request, filters the pre-stored monitoring data based on the time period information, and returns the filtered monitoring data to the front-end monitoring data display component.

[0141] The received monitoring data is displayed in multiple display levels.

[0142] In one possible implementation, the method further includes:

[0143] Based on the hierarchy in the received monitoring data, construct a tree structure with nodes of multiple levels;

[0144] The received monitoring data is mapped to nodes at each level of the tree structure, and the tree structure is then rendered.

[0145] In one possible implementation, the front-end and the server are connected via a full-duplex bidirectional WebSocket connection. The server obtains monitoring data from the data acquisition module through the WebSocket connection and preprocesses the obtained monitoring data. The preprocessing includes converting the original monitoring data into data in a standardized format adapted to the front-end components.

[0146] Figure 4 A block diagram of a cluster performance analysis apparatus according to an embodiment of the present disclosure is shown, such as Figure 4 As shown, the device 30, applied to a model training cluster, includes:

[0147] The request sending module 31 is used to send a monitoring data acquisition request to the server to request monitoring data; the monitoring data includes: hardware monitoring data and module performance data, wherein the hardware monitoring data represents the usage of hardware resources in the cluster, and the module performance data represents the performance of the module during execution;

[0148] The data receiving module 32 is used to receive the monitoring data sent by the server and display the monitoring data in multiple display levels, including: overall data at the cluster level and detailed data at the node level.

[0149] In one possible implementation, the plurality of levels includes:

[0150] The Model Computing Power Utilization (MFU) level is used to demonstrate the computing power utilization in the cluster.

[0151] The control plane node hierarchy is used to display the utilization and memory usage of the control plane nodes;

[0152] The worker node level displays monitoring data from the worker nodes that actually perform the computational tasks.

[0153] In one possible implementation, the module performance data includes call information between modules during task execution in the cluster, communication information between GPU kernel functions, and the execution time of each module.

[0154] In one possible implementation, the device further includes:

[0155] The first rendering module is used to render the length of the corresponding graphical controls in the interface according to the execution time of each module; the graphical controls are used to respond to user commands and display the module performance data of the corresponding module.

[0156] In one possible implementation, the device further includes:

[0157] The input receiving module is used to receive the time period information of the monitoring data input by the user, and after adding the time period information to the monitoring data acquisition request, send the monitoring data acquisition request to the server; so that the server can respond to the monitoring data acquisition request, filter the pre-stored monitoring data based on the time period information, and return the filtered monitoring data to the front-end monitoring data display component.

[0158] The hierarchical display module is used to display the received monitoring data according to multiple display levels.

[0159] In one possible implementation, the device further includes:

[0160] The tree structure building module is used to construct a tree structure with multiple levels of nodes based on the hierarchy in the received monitoring data.

[0161] The second rendering module is used to map the received monitoring data to nodes at each level of the tree structure and to render the tree structure.

[0162] In one possible implementation, the front-end and the server are connected via a full-duplex bidirectional WebSocket connection. The server obtains monitoring data from the data acquisition module through the WebSocket connection and preprocesses the obtained monitoring data. The preprocessing includes converting the original monitoring data into data in a standardized format adapted to the front-end components.

[0163] In some embodiments, the functions or modules of the apparatus provided in this disclosure can be used to execute the system described in the above method embodiments. The specific implementation can be referred to the description of the above system embodiments, which will not be repeated here for the sake of brevity.

[0164] This disclosure also proposes a computer-readable storage medium storing computer program instructions that, when executed by a processor, implement the above-described method. The computer-readable storage medium can be volatile or non-volatile.

[0165] This disclosure also proposes an electronic device, including: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to invoke the instructions stored in the memory to execute the above-described method.

[0166] This disclosure also provides a computer program product, including computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code, wherein when the computer-readable code is run in a processor of an electronic device, the processor in the electronic device performs the above-described method.

[0167] Electronic devices can be provided as terminals, servers, or other forms of devices.

[0168] Figure 5 A block diagram of an electronic device according to an embodiment of the present disclosure is shown. For example, electronic device 1900 may be provided as a server or terminal device. (Refer to...) Figure 5 The electronic device 1900 includes a processing component 1922, which further includes one or more processors, and memory resources represented by memory 1932 for storing instructions, such as application programs, that can be executed by the processing component 1922. The application programs stored in memory 1932 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processing component 1922 is configured to execute instructions to perform the methods described above.

[0169] Electronic device 1900 may also include a power supply component 1926 configured to perform power management of electronic device 1900, a wired or wireless network interface 1950 configured to connect electronic device 1900 to a network, and an input / output (I / O) interface 1958. Electronic device 1900 can operate on an operating system stored in memory 1932, such as Microsoft Server operating system (Windows Server). TM Apple's graphical user interface-based operating system (Mac OSX) TM ), a multi-user, multi-process computer operating system (Unix) TM Linux is a free and open-source Unix-like operating system. TM ), the open-source Unix-like operating system (FreeBSD) TM (or similar.)

[0170] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 1932 including computer program instructions that can be executed by a processing component 1922 of an electronic device 1900 to perform the above-described method.

[0171] This disclosure can be a system, method, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of this disclosure.

[0172] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example, (but not limited to) electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination of the foregoing. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0173] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0174] Computer program instructions used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing the status information of the computer-readable program instructions to implement various aspects of this disclosure.

[0175] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0176] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0177] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0178] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0179] The computer program product can be implemented specifically through hardware, software, or a combination thereof. In one alternative embodiment, the computer program product is specifically embodied in a computer storage medium; in another alternative embodiment, the computer program product is specifically embodied in a software product, such as a software development kit (SDK), etc.

[0180] The description of the various embodiments above tends to emphasize the differences between the various embodiments. The similarities or similarities between them can be referred to, and for the sake of brevity, they will not be repeated here.

[0181] Those skilled in the art will understand that, in the above-described method of the specific implementation, the order in which each step is written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.

[0182] If the technical solution of this application involves personal information, the product using this technical solution has clearly informed the user of the personal information processing rules and obtained the user's voluntary consent before processing the personal information. If the technical solution of this application involves sensitive personal information, the product using this technical solution has obtained the user's separate consent before processing the sensitive personal information, and also meets the requirement of "express consent". For example, at personal information collection devices such as cameras, clear and prominent signs are set up to inform users that they have entered the scope of personal information collection and that personal information will be collected. If an individual voluntarily enters the collection scope, it is deemed that they have agreed to the collection of their personal information; or on the personal information processing device, with clear signs / information informing users of the personal information processing rules, authorization is obtained from the individual through pop-up information or by asking the individual to upload their personal information; wherein, the personal information processing rules may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the types of personal information processed.

[0183] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or improvement of the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A cluster performance analysis system, characterized in that, It is applied to model training clusters, including server-side and front-end, where: The front end is used to send a monitoring data acquisition request to the server to request monitoring data; the monitoring data includes: hardware monitoring data and module performance data, wherein the hardware monitoring data represents the usage of hardware resources in the cluster, and the module performance data represents the performance of the module during execution; The server is used to send monitoring data to the front end in response to a monitoring data acquisition request; The front end is used to receive the monitoring data and display the monitoring data in multiple display levels, including: overall data at the cluster level and refined data at the node level.

2. The system according to claim 1, characterized in that, The multiple levels include: The Model Computing Power Utilization (MFU) level is used to demonstrate the computing power utilization in the cluster. The control plane node hierarchy is used to display the utilization and memory usage of the control plane nodes; The worker node level displays monitoring data from the worker nodes that actually perform the computational tasks.

3. The system according to claim 1, characterized in that, The module performance data includes the call information between modules during task execution in the cluster, the communication information between GPU kernel functions, and the execution time of each module.

4. The system according to claim 3, characterized in that, The front end is used to render the length of the corresponding graphical controls of each module in the interface according to the execution time of each module. The graphical control is used to respond to user commands and display the module performance data of the corresponding module.

5. The system according to claim 4, characterized in that, The server is used to delete data whose execution time is less than a set threshold based on the execution time of each module, thereby obtaining filtered module performance data.

6. The system according to claim 1, characterized in that, The front end includes a time period control component and a monitoring data display component; The time period control component at the front end is used to receive the time period information of the monitoring data input by the user, and after adding the time period information to the monitoring data acquisition request, send the monitoring data acquisition request to the server. The server, in response to the monitoring data acquisition request, filters the pre-stored monitoring data based on the time period information and returns the filtered monitoring data to the front-end monitoring data display component. The front-end monitoring data display component displays the received monitoring data according to multiple display levels.

7. The system according to claim 6, characterized in that, The front end constructs a tree structure with multiple levels of nodes based on the hierarchy in the received monitoring data. The front end maps the received monitoring data to nodes at each level of the tree structure and renders the tree structure.

8. The system according to claim 6, characterized in that, The front-end and the server are connected via a full-duplex, bidirectional WebSocket connection. The server obtains monitoring data from the data acquisition module via a WebSocket connection and preprocesses the obtained monitoring data. The preprocessing includes converting the raw monitoring data into data in a standardized format adapted to the front-end components.

9. A cluster performance analysis method, characterized in that, Applied to model training clusters, including: Send a monitoring data retrieval request to the server to request monitoring data; the monitoring data includes: hardware monitoring data and module performance data, wherein the hardware monitoring data represents the usage of hardware resources in the cluster, and the module performance data represents the performance of the module during execution; The system receives the monitoring data sent by the server and displays the monitoring data at multiple levels, including: overall data at the cluster level and detailed data at the node level.

10. A cluster performance analysis device, characterized in that, Applied to model training clusters, including: The request sending module is used to send a monitoring data acquisition request to the server to request monitoring data; the monitoring data includes: hardware monitoring data and module performance data, wherein the hardware monitoring data represents the usage of hardware resources in the cluster, and the module performance data represents the performance of the module during execution; The data receiving module is used to receive the monitoring data sent by the server and display the monitoring data in multiple display levels, including: overall data at the cluster level and detailed data at the node level.

11. An electronic device, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to invoke instructions stored in the memory to execute the method of claim 9.

12. A computer-readable storage medium having computer program instructions stored thereon, characterized in that, When the computer program instructions are executed by the processor, they implement the method of claim 9.

Images