Large language model training-oriented cluster monitoring method and related device

Abstract

The invention relates to the field of data processing, in particular to a large language model training-oriented cluster monitoring method and a related device. The method comprises the following steps: acquiring original operation data of a target object; the target object is one or more TPU nodes in a special integrated circuit TPU cluster; the TPU node is used for accelerating a training task and a reasoning task of a large language model; converting the original operation data into to-be-processed hardware information of the target object; the conversion mode of the to-be-processed hardware information is obtained based on a plurality of to-be-monitored index dimension configurations in the TPU cluster; generating real-time display information according to the to-be-processed hardware information, and updating the real-time display information to a monitoring panel in real time for display; the real-time display information at least comprises hardware monitoring information and calculation state information corresponding to the TPU cluster. According to the method, real-time monitoring of the TPU cluster can be realized, the cluster monitoring efficiency is improved, and the expansibility and flexibility of a cluster monitoring system are improved.

Description

Technical Field

[0001] The present application relates to the field of data processing, and in particular to a cluster monitoring method and related devices for large language model training. Background Art

[0002] With the rapid development of artificial intelligence technology, large language models (LLMs) have shown excellent performance in the field of natural language processing. The training of LLMs requires a lot of computing resources and data, and usually relies on high-performance computing clusters. In order to meet this high computing demand, computing centers can be built. These computing centers are composed of application-specific integrated circuits (Tensor Processing Unit, TPU) for accelerating machine learning tasks, forming a large computing cluster to support LLM training tasks. TPU is designed to efficiently process tensor (i.e. multi-dimensional array) operations, which is a core component of deep learning and neural network computing. TPU is mainly used to accelerate machine learning and deep learning tasks, especially those applications that require a large number of matrix or tensor operations, such as image recognition, natural language processing, etc.

[0003] In the LLM training process, the performance and stability of the computing cluster are crucial. Since the training process requires long-term high-load operation, any problem in any link may cause the training task to be interrupted or the results to be inaccurate. Therefore, how to monitor the TPU computing center cluster in real time has become one of the technical problems that need to be solved in the large model training process. Summary of the invention

[0004] In response to the technical problems existing in the prior art, the present application provides a cluster monitoring method and related devices for large language model training, so as to realize real-time monitoring of TPU clusters, improve cluster monitoring efficiency, as well as the scalability and flexibility of cluster monitoring systems, and reduce the cost of setting up cluster monitoring systems.

[0005] In a first aspect, an embodiment of the present application provides a cluster monitoring method for large language model training, the method comprising: Obtaining original running data of a target object; the target object is one or more TPU nodes in a TPU cluster of a dedicated integrated circuit; the TPU nodes are used to accelerate training tasks and reasoning tasks of a large language model; The raw operating data is converted into hardware information to be processed of the target object; the conversion method of the hardware information to be processed is obtained based on the configuration of multiple indicator dimensions to be monitored in the TPU cluster; the multiple indicator dimensions include at least one of the following: temperature, real-time power consumption, and memory usage of the target object in the TPU cluster; Real-time display information is generated according to the hardware information to be processed, and updated in real time to the monitoring panel for display; the real-time display information at least includes: hardware monitoring information and computing status information corresponding to the TPU cluster.

[0006] In a second aspect, an embodiment of the present application provides a cluster monitoring device for large language model training, the device comprising at least the following units: An acquisition unit is configured to acquire original operation data of a target object; the target object is one or more TPU nodes in a TPU cluster of a dedicated integrated circuit; the TPU nodes are used to accelerate training tasks and reasoning tasks of a large language model; A conversion unit is configured to convert the original operation data into hardware information to be processed of a target object; the conversion method of the hardware information to be processed is obtained based on a plurality of indicator dimensions to be monitored in the TPU cluster; the plurality of indicator dimensions include at least one of the following: temperature, real-time power consumption, and memory usage of the target object in the TPU cluster; The display unit is configured to generate real-time display information according to the hardware information to be processed, and update it in real time to the monitoring panel for display; the real-time display information at least includes: hardware monitoring information and computing status information corresponding to the TPU cluster.

[0007] In a third aspect, an embodiment of the present application provides an electronic device, the electronic device comprising: at least one processor, memory, and input-output unit; The memory is used to store a computer program, and the processor is used to call the computer program stored in the memory to execute the cluster monitoring method for large language model training of the first aspect.

[0008] In a fourth aspect, a computer-readable storage medium is provided, comprising instructions, which, when executed on a computer, cause the computer to execute the cluster monitoring method for large language model training of the first aspect.

[0009] The beneficial effect of the present application is that it provides a cluster monitoring method and related devices for large language model training. In this technical solution, first, the original operation data of the target object is obtained. Among them, the target object is one or more TPU nodes in a dedicated integrated circuit TPU cluster; the TPU node is used to accelerate the training task and reasoning task of the large language model. Then, the original operation data is converted into the hardware information to be processed of the target object. Among them, the conversion method of the hardware information to be processed is obtained based on the configuration of multiple indicator dimensions to be monitored in the TPU cluster; the multiple indicator dimensions include at least one of the following: temperature, real-time power consumption, and memory usage of the target object in the TPU cluster. Finally, real-time display information is generated according to the hardware information to be processed, and updated in real time to the monitoring panel for display. The real-time display information includes at least: hardware monitoring information and computing status information corresponding to the TPU cluster. The technical solution of this application realizes real-time monitoring of the TPU cluster, improves the cluster monitoring efficiency, as well as the scalability and flexibility of the cluster monitoring system, and reduces the construction cost of the cluster monitoring system. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 It is a flow chart of a cluster monitoring method for large language model training according to an embodiment of the present application; Figure 2 It is a structural diagram of a cluster monitoring device for large language model training according to an embodiment of the present application; Figure 3 is a schematic diagram of the structure of an electronic device according to an embodiment of the present application; Figure 4 It is a structural schematic diagram of a medium device according to an embodiment of the present application. DETAILED DESCRIPTION

[0011] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.

[0012] With the rapid development of artificial intelligence technology, large language models (LLMs) have shown excellent performance in the field of natural language processing. The training of LLMs requires a lot of computing resources and data, and usually relies on high-performance computing clusters. In order to meet this high computing demand, many companies and research institutions have built computing centers. These computing centers are composed of application-specific integrated circuits (Tensor Processing Unit, TPU) used to accelerate machine learning tasks, forming a large computing cluster to support LLM training tasks. The main purpose of TPU is to accelerate machine learning and deep learning tasks, especially those applications that require a large number of matrix or tensor operations, such as image recognition, natural language processing, etc. TPU is designed to efficiently process tensor (i.e. multi-dimensional array) operations, which is a core component of deep learning and neural network computing. TPU is mainly used to accelerate machine learning and deep learning tasks, especially those applications that require a large number of matrix or tensor operations, such as image recognition, natural language processing, etc.

[0013] During the LLM training process, the performance and stability of the computing cluster are crucial. Since the training process requires long-term high-load operation, any problem in any link may cause the training task to be interrupted or the results to be inaccurate.

[0014] Therefore, how to monitor the TPU computing center cluster in real time has become one of the technical problems that need to be solved in the process of large model training.

[0015] In order to solve at least one technical problem in the related art, the embodiment of the present application provides a cluster monitoring method and related devices for large language model training. In this technical solution, first, the original operation data of the target object is obtained. Among them, the target object is one or more TPU nodes in a dedicated integrated circuit TPU cluster; the TPU node is used to accelerate the training task and reasoning task of the large language model. Then, the original operation data is converted into the hardware information to be processed of the target object. Among them, the conversion method of the hardware information to be processed is obtained based on the configuration of multiple indicator dimensions to be monitored in the TPU cluster; the multiple indicator dimensions include at least one of the following: temperature, real-time power consumption, and memory usage of the target object in the TPU cluster. Finally, real-time display information is generated according to the hardware information to be processed, and updated to the monitoring panel in real time for display. The real-time display information includes at least: hardware monitoring information and computing status information corresponding to the TPU cluster.

[0016] The following analyzes the improvements of the technical solution of this application compared with the related art from multiple perspectives: Regarding the problem that the monitoring system in the related technology often relies on a general hardware information reading interface and is difficult to meet the high computing density requirements of the TPU cluster, the technical solution provided by this application obtains the hardware information of the TPU through independent research and development, including the number of TPUs, memory block usage, temperature, power consumption, etc., to ensure the accuracy and real-time nature of the data. This application can monitor the TPU cluster more finely, improve the efficiency and accuracy of data collection, and provide a reliable data foundation for subsequent real-time monitoring and alarms.

[0017] In order to solve the problem that the data update and processing speed of the monitoring system in the related technology is slow and it is difficult to achieve real-time monitoring, the technical solution provided by this application ensures that the hardware information of the TPU cluster can be displayed in time through the real-time data update function, which helps to quickly discover and deal with potential problems. As a result, the cluster monitoring solution has higher real-time performance, can quickly respond to changes in hardware status, and improve the real-time monitoring capability of the system.

[0018] In order to solve the problems in related technologies, such as unfriendly interfaces of monitoring systems, unintuitive data display, and difficulty in quickly locating problems, the technical solution provided by this application uses a custom-configured monitoring panel, reasonable arrangement of various monitoring charts, clear data display, and easy and fast problem location. This helps to improve user experience, lower the threshold for system use, and enable non-professionals to quickly understand and locate problems.

[0019] Some monitoring systems in related technologies have insufficient expansion and adaptability when facing the expansion of cluster scale or the introduction of new technologies. The technical solution provided by this application supports multiple data sources and allows users to customize monitoring panels and charts, so that the system can quickly adapt and expand when facing the expansion of cluster scale or the introduction of new technologies. The technical solution provided by this application has higher flexibility and scalability, can adapt to different monitoring needs and environmental changes, and reduce the cost and time of system upgrades.

[0020] The alarm function of the monitoring system in the related technology is often not timely enough and cannot effectively notify relevant personnel to deal with the problem. The technical solution provided by this application can detect abnormal situations in real time through the configured alarm rules, and notify relevant personnel through emails, text messages, etc., to ensure that the problem is handled in a timely manner, which helps to improve the system's response speed and problem handling efficiency, reduce downtime caused by hardware failures or anomalies, and improve the stability and reliability of the system.

[0021] The data generated by different monitoring tools and systems in related technologies are scattered and difficult to integrate and analyze uniformly. The technical solution provided by this application integrates data from different monitoring tools and systems, provides a unified analysis platform, solves the problems of data silos and integration difficulties, improves resource utilization and data analysis efficiency, and thus provides more comprehensive data analysis capabilities, helping users better understand and manage TPU clusters, and improve the accuracy and efficiency of decision-making.

[0022] In summary, the technical solution provided by this application significantly improves the resource utilization of TPU clusters in the training and reasoning process of large language models through independently developed data collection methods, real-time data processing and storage, flexible and intuitive monitoring interface, powerful scalability and flexible configuration, real-time alarm function, and data island solution, timely discovers and solves potential problems, reduces downtime, and improves system stability and reliability. Compared with related technologies, this application provides a more real-time, more beautiful, and more scalable monitoring solution that can better meet the monitoring needs of large-scale TPU clusters.

[0023] The technical solution of the present application and the cluster monitoring solution for large language model training provided in the embodiment of the present application can also be executed by an electronic device, which can be a server, a server cluster, or a cloud server. The electronic device can also be a terminal device such as a mobile phone, a computer, a tablet computer, a wearable device, or a dedicated device (such as a dedicated terminal device with a cluster monitoring method system for large language model training, etc.). These electronic devices can also be equipped with the chips introduced in the above embodiments. Alternatively, these electronic devices can also be installed with a service program for executing a cluster monitoring solution for large language model training.

[0024] Figure 1 A flow chart of a cluster monitoring method for large language model training provided in an embodiment of the present application is as follows: Figure 1 As shown, the method comprises the following steps: 101, obtaining original operation data of the target object; 102, converting the original operation data into hardware information to be processed of a target object; 103 , generating real-time display information according to the hardware information to be processed, and updating the real-time display information to the monitoring panel for display.

[0025] Among them, TPU is a custom hardware chip designed to accelerate machine learning (especially deep learning) tasks. The main purpose of TPU is to provide higher computing efficiency and lower power consumption than traditional CPUs and GPUs, thereby significantly improving performance when training and reasoning large-scale neural network models. TPU is deeply optimized for matrix operations, which are the core operations of deep learning algorithms such as convolutional neural networks and recurrent neural networks. TPU is designed to quickly process large-scale data and reduce data transmission delays within the chip, thereby increasing overall computing speed. TPU performs well in deep learning training and reasoning tasks, can significantly improve computing efficiency and reduce power consumption, and is widely used in cloud services and edge computing scenarios. By providing efficient hardware acceleration, TPU has promoted the development and application of deep learning technology.

[0026] In the embodiment of the present application, the target object is one or more TPU nodes in a TPU cluster. Further optionally, the TPU node is used to accelerate the training and reasoning tasks of a large language model.

[0027] The TPU cluster is a high-performance computing cluster composed of multiple TPUs, specifically designed to accelerate machine learning and deep learning tasks. These clusters excel in processing large-scale data sets, training complex neural network models, and performing efficient parallel computing. By connecting multiple TPU nodes together, the TPU cluster can process multiple computing tasks simultaneously, greatly improving computing throughput and efficiency. Each TPU node is equipped with high-bandwidth memory (HBM) to ensure that data can be transmitted quickly during processing and reduce waiting time. The TPU has an instruction set optimized for deep learning, which can perform common neural network operations more efficiently.

[0028] The TPU cluster adopts a modular design, which can easily expand or reduce the number of nodes according to needs to adapt to different computing needs. Through the cloud computing platform, users can dynamically adjust the usage of TPU resources as needed to achieve elastic scaling. The TPU cluster connects each node through a high-speed Internet network (such as NVLink or a customized high-speed network) to ensure efficient data transmission between nodes. By optimizing the communication protocol and architecture, the TPU cluster can achieve low-latency data exchange and is suitable for parallel computing tasks that require frequent communication. The TPU cluster supports a variety of deep learning frameworks, such as TensorFlow, PyTorch, etc. Users can easily migrate existing models to TPU for acceleration. Through automated tools and APIs, users can manage and optimize the use of TPU clusters and simplify deployment and maintenance processes.

[0029] In practical applications, TPU clusters are suitable for training large-scale deep learning models, such as image classification, speech recognition, natural language processing, etc., which can significantly shorten the training time. Through distributed training technology, TPU clusters can train multiple models at the same time, further improving training efficiency. In a production environment, TPU clusters can efficiently perform model reasoning tasks and provide low-latency and high-throughput services. It is suitable for application scenarios that require real-time reasoning, such as real-time speech recognition, image processing, etc. TPU clusters perform well in the fields of scientific computing and data analysis, and are suitable for large-scale data processing and complex computing tasks. Through powerful computing power, TPU clusters can efficiently perform data mining and analysis to discover potential patterns in the data.

[0030] Compared with traditional CPUs and GPUs, TPUs have higher computing efficiency in deep learning tasks and can significantly improve training and reasoning speeds. Through specially optimized hardware design, TPU clusters can maintain high computing performance while reducing energy consumption. Users can dynamically adjust the usage of TPU resources according to actual needs, pay on demand, and reduce resource waste. With the support of cloud service providers, users can easily deploy and manage TPU clusters, reducing operation and maintenance costs and complexity. TPU clusters support mainstream deep learning frameworks, and users can seamlessly migrate existing models and enjoy high-performance acceleration. Provides a wealth of tools and libraries to help users optimize model and cluster configurations and improve overall performance.

[0031] The TPU cluster provides powerful computing power for deep learning and scientific computing through high-performance hardware design, flexible scalability, efficient communication architecture and powerful software support. Whether in large-scale model training, real-time reasoning or high-performance computing, the TPU cluster has shown significant advantages and potential, promoting the development and application of related technologies. With the support of cloud service providers, users can easily access and use TPU clusters to improve their computing power and efficiency.

[0032] As an optional embodiment, it is assumed that each TPU node in the TPU cluster is deployed with a corresponding register, and the register of each TPU node is used to store the original operation data of the corresponding node. Based on the above assumption, obtaining the original operation data of the target object in 101 can be implemented as follows: reading the original operation data of the target object from the register corresponding to the target object through a pre-configured hardware data interface.

[0033] Wherein, the hardware data interface at least includes: i 2c and / or PCIE. In the TPU cluster, each TPU node is deployed with a corresponding register to store the original operation data of the corresponding node. These registers can be accessed and read through pre-configured hardware data interfaces such as I²C and PCIe. Among them, I²C (Inter-Integrated Circuit) is a serial communication interface used to transmit data between devices. PCIe (Peripheral Component Interconnect Express) is a high-speed serial computer expansion bus standard used to connect peripherals and host systems.

[0034] The registers on each TPU node are used to store the original operating data of the node, including computing status, memory usage, temperature, power consumption, etc. By reading the data in the registers, the operating status of the TPU node can be monitored in real time, helping with performance analysis and fault diagnosis.

[0035] In an optional example, taking the reading of temperature data of a TPU node through I²C as an example, TPU node A is connected to the host system through the I²C bus. Assume that the I²C address of TPU node A is 0x5A. The host system initializes the I²C controller and sets the I²C bus speed and communication mode. The host system sends a read command to address 0x5A, specifying the address of the temperature register to be read (assuming it is 0x01). TPU node A responds to the read request and sends the temperature data (assuming it is 45 degrees Celsius) back to the host system. The host system stores the received data in a cache and further processes or displays the temperature information.

[0036] In another optional example, taking reading the memory usage data of the TPU node through PCIe as an example, the TPU node B is connected to the host system through the PCIe bus. Assume that the PCIe device ID of the TPU node B is 0x1234. Based on this, the host system initializes the PCIe controller, identifies and connects to the TPU node B with a device ID of 0x1234. The host system sends a request to read data to the TPU node B, specifying the address of the memory usage register to read (assumed to be 0x100) and the data length (assumed to be 4 bytes). TPU node B responds to the read request and sends the memory usage data (assumed to be 512MB) back to the host system. The host system stores the received data in the cache and further processes or displays the memory usage information.

[0037] In summary, through the pre-configured hardware data interfaces (such as I²C and PCIe), the original operation data can be read from the registers of each TPU node in the TPU cluster. This design not only improves the efficiency and accuracy of data collection, but also enhances the real-time monitoring and performance analysis capabilities of the system.

[0038] In 102, the original operation data is converted into hardware information to be processed of the target object.

[0039] In an embodiment of the present application, the conversion method of the hardware information to be processed is obtained based on the configuration of multiple indicator dimensions to be monitored in the TPU cluster. The multiple indicator dimensions include at least one of the following: temperature, real-time power consumption, and memory usage of the target object in the TPU cluster.

[0040] In a TPU cluster, raw operating data (such as temperature, real-time power consumption, memory usage, etc.) needs to be converted into the target object's hardware information to be processed. This hardware information is configured based on multiple indicator dimensions to provide support for subsequent analysis and decision-making. The following is a specific conversion method and example introduction.

[0041] Temperature, such as the temperature of a TPU node, indicates the current heating condition of the chip. Monitoring temperature can help prevent performance degradation or hardware damage caused by overheating. During operation, temperature sensors are used to monitor the temperature readings of TPU nodes in real time, and the measured values ​​are sent to the central monitoring system through a dedicated data transmission channel. The central monitoring system analyzes the temperature data of multiple TPU nodes through a pre-designed data processing algorithm. If it is detected that the temperature of a node exceeds the preset safety threshold, the system will immediately initiate a predetermined response mechanism, such as reducing the task load of the node, starting the cooling system to enhance the cooling effect, or in extreme cases, in order to protect the hardware safety, the system will automatically shut down the node to avoid overheating damage.

[0042] Real-time power consumption, such as that of a TPU node, indicates the current energy consumption level of the chip. Monitoring power consumption can help optimize energy use, reduce costs, and prevent damage to hardware due to excessive power consumption. Real-time power consumption monitoring is a key part of TPU cluster management, aiming to monitor the energy consumption level of each TPU node.

[0043] Through high-performance power consumption sensors, the power consumption data of each node can be captured in real time and transmitted to the central monitoring system through the I²C or PCIe interface. Optionally, these data are processed using a preset analysis algorithm to calculate key indicators such as average power consumption, maximum power consumption, and minimum power consumption, and these indicators are mapped to different energy consumption levels (such as low, medium, and high). When it is detected that the power consumption of a node exceeds the preset threshold, the system will immediately take optimization measures, such as adjusting the allocation of computing tasks, reducing the load of high-power nodes, or optimizing the execution strategy of computing tasks to reduce overall energy consumption. This not only helps to reduce energy costs, but also effectively prevents hardware overheating and damage caused by excessive power consumption, thereby ensuring the efficient and stable operation of the TPU cluster. Through real-time power consumption monitoring and optimization, the TPU cluster can achieve effective use of energy, extend hardware life, and reduce operation and maintenance costs while maintaining high performance.

[0044] Memory usage, such as the memory usage of TPU nodes, indicates the current utilization of memory resources. Monitoring memory usage can help identify memory bottlenecks and optimize resource allocation for computing tasks. Memory usage monitoring is an important part of TPU cluster management, which aims to monitor the memory resource utilization of each node in real time. Through the memory monitoring module, the system can obtain the current memory data used by each TPU node and transmit it to the central monitoring system through the network interface. The monitoring system uses built-in algorithms to calculate the average, peak, and minimum values ​​of memory usage, and maps these data to predefined memory usage levels (such as low, medium, and high).

[0045] When it is detected that the memory usage of a node is close to or exceeds the preset threshold, the system will automatically trigger optimization measures. For example, adjust the task allocation strategy, transfer some computing tasks to nodes with more abundant memory resources, or optimize the data processing algorithm to reduce memory usage. This real-time monitoring and optimization mechanism helps to identify and solve memory bottlenecks, ensure the efficient execution of computing tasks, and maximize the use of the computing resources of the TPU cluster. By accurately monitoring and dynamically adjusting memory usage, the TPU cluster can maintain stable performance under high load conditions, improve resource utilization, and provide reliable support for complex computing tasks.

[0046] In TPU cluster management, the original operating data (such as temperature, real-time power consumption, and memory usage) are first converted into standardized numerical forms for subsequent processing and analysis. For example, temperature data is converted to degrees Celsius or Kelvin, power consumption data is converted to watts (W), and memory usage data is converted to percentages (%) or bytes (Byte). Next, by calculating the average, maximum, and minimum values ​​of multiple TPU nodes on specific indicators, these raw data are aggregated into an overall indicator to represent the status of the entire TPU cluster. Finally, the standardized and aggregated data is mapped to predefined states or levels to facilitate system decision-making. For example, temperature data is mapped to "normal", "warning", and "dangerous" levels, power consumption data is mapped to "low", "medium", and "high" levels, and memory usage data is mapped to "low", "medium", and "high" levels, thereby providing strong support for optimizing energy use, preventing hardware damage, and identifying memory bottlenecks.

[0047] In the TPU cluster, by converting the original running data into the hardware information to be processed by the target object, multiple indicator dimensions (such as temperature, real-time power consumption, and memory usage) can be effectively monitored and managed. This conversion method not only improves the availability and understandability of the data, but also provides a solid foundation for subsequent performance analysis and decision-making.

[0048] For example, the original operating data is TPU node A -50°C, node B -55°C, node C -60°C. Keep the original data unit (°C). The average temperature is 55°C, the maximum temperature is 60°C, and the minimum temperature is 50°C. The average temperature of 55°C is mapped to the "warning" level (assuming the threshold is 50°C normal and 60°C warning), the maximum temperature of 60°C is mapped to the "warning" level, and the minimum temperature of 50°C is mapped to the "normal" level.

[0049] Exemplarily, the original operation data is TPU node A-150W, node B-160W, node C-170W. Keep the original data unit (W). Average power consumption 160W, maximum power consumption 170W, minimum power consumption 150W. Average power consumption 160W is mapped to the "medium" level (assuming the threshold is 150W low, 160W medium, 170W high), maximum power consumption 170W is mapped to the "high" level, and minimum power consumption 150W is mapped to the "low" level.

[0050] For example, the original running data is TPU node A - 80%, node B - 85%, node C - 90%. Keep the original data unit (%). Average memory usage is 85%, maximum memory usage is 90%, and minimum memory usage is 80%. Average memory usage of 85% is mapped to the "medium" level (assuming the threshold is 80% low, 85% medium, and 90% high), maximum memory usage of 90% is mapped to the "high" level, and minimum memory usage of 80% is mapped to the "low" level.

[0051] Further optionally, in 102, converting the original operation data into the hardware information to be processed of the target object can be implemented as follows: cleaning the read original operation data; performing data conversion calculation on the original operation data according to the data calculation method corresponding to each data type to obtain the corresponding data endpoint; constructing the hardware information to be processed with the data endpoint corresponding to the original operation data. In the embodiment of the present application, the data endpoint is the key indicator data to be extracted from the original operation data.

[0052] In TPU cluster management, the original operation data needs to be cleaned, converted, and built to be converted into the target object's hardware information to be processed. Assume that the original operation data read from TPU nodes A, B, and C is in TPU cluster management. The original operation data needs to be cleaned, converted, and built to be converted into the target object's hardware information to be processed.

[0053] Assume that the original operation data read from TPU nodes A, B, and C are as follows: the temperature of TPU node A is 50°C, the power consumption is 150W, and the memory usage is 80%; the temperature of TPU node B is 55°C, the power consumption is 160W, and the memory usage is 85%; the temperature of TPU node C is 60°C, the power consumption is 170W, and the memory usage is 90%. Clean the original operation data to remove abnormal data points and noise, such as checking whether the data is within a reasonable range and removing obviously unreasonable data (such as data with negative temperature or zero power consumption). According to the data calculation method corresponding to each data type, the original operation data is converted and calculated to obtain the corresponding data endpoints: the temperature index remains in the original data unit (°C), and the data endpoints are 50°C, 55°C, and 60°C; the power consumption index remains in the original data unit (W), and the data endpoints are 150W, 160W, and 170W; the memory usage index remains in the original data unit (%), and the data endpoints are 80%, 85%, and 90%.

[0054] The hardware information to be processed is constructed using the data endpoints corresponding to the original running data: in the temperature data endpoint, the average temperature is 55°C, the maximum temperature is 60°C, the minimum temperature is 50°C, and the temperature status is mapped to an average temperature of 55°C warning (assuming the threshold is 50°C normal, 60°C warning), a maximum temperature of 60°C warning, and a minimum temperature of 50°C normal; in the power consumption data endpoint, the average power consumption is 160W, the maximum power consumption is 170W, and the minimum power consumption is 150W. The power consumption status is mapped to an average power consumption of 160W medium (assuming the threshold is 150W low, 160W medium, and 170W high), a maximum power consumption of 170W high, and a minimum power consumption of 150W low; in the memory usage data endpoint, the average memory usage is 85%, the maximum memory usage is 90%, and the minimum memory usage is 80%. The memory usage status is mapped to an average memory usage of 85% medium (assuming the threshold is 80% low, 85% medium, and 90% high), a maximum memory usage of 90% high, and a minimum memory usage of 80% low. The final hardware information to be processed includes: in temperature information, node A is 50°C normal, node B is 55°C warning, node C is 60°C warning, average temperature is 55°C warning, maximum temperature is 60°C warning, and minimum temperature is 50°C normal; in power consumption information, node A is 150W low, node B is 160W medium, node C is 170W high, average power consumption is 160W medium, maximum power consumption is 170W high, and minimum power consumption is 150W low. In memory usage information, node A is 80% low, node B is 85% medium, node C is 90% high, average memory usage is 85% medium, maximum memory usage is 90% high, and minimum memory usage is 80% low.

[0055] Through the above data conversion and construction process, the processed hardware information including temperature, power consumption and memory usage is obtained, which provides basic data support for subsequent system decision-making and management.

[0056] As an optional embodiment, in 102, after the original operation data is converted into the hardware information to be processed of the target object, the hardware information to be processed can also be converted into a corresponding multidimensional hardware information sequence in the multidimensional data model and stored in the storage space corresponding to the multidimensional tag. The multidimensional hardware information sequence is associated with multiple indicator dimension tags.

[0057] In TPU cluster management, the original running data is first converted into hardware information to be processed, such as temperature, power consumption, and memory usage data. Then, the hardware information to be processed is converted into a multidimensional hardware information sequence in a multidimensional data model and stored in the storage space corresponding to the multidimensional label. Specifically, the dimension labels (such as node labels, indicator labels, statistical labels, and status labels) are determined, and the data points of each dimension are constructed into a multidimensional data sequence. For example, for the temperature indicator, the constructed multidimensional hardware information sequence includes the temperature data of node A, node B, and node C, as well as the average, maximum, and minimum temperature data, and is associated with a status label (such as normal, warning). Subsequently, these multidimensional hardware information sequences are stored in the corresponding storage space. For example, the temperature information sequence is stored in "temperature_storage", the power consumption information sequence is stored in "power_storage", and the memory usage information sequence is stored in "memory_usage_storage". In this way, the hardware information to be processed is converted into a multidimensional hardware information sequence and stored in the storage space corresponding to the multidimensional label, providing support for subsequent query, analysis, and decision-making.

[0058] As an optional embodiment, in 102, after converting the hardware information to be processed into the corresponding multidimensional hardware information sequence in the multidimensional data model, it is also possible to detect in real time whether the multidimensional hardware information sequence contains target hardware information that meets the alarm conditions; the detection index of the multidimensional hardware information sequence is composed of alarm thresholds and / or alarm conditions of different types of hardware information; if the multidimensional hardware information sequence contains target hardware information that meets the alarm conditions, alarm information of the target object is generated based on the target hardware information; the alarm information is displayed in real time on the monitoring panel to prompt the user to promptly deal with the operating risks existing in the TPU cluster.

[0059] Exemplarily, after converting the hardware information to be processed into a multidimensional hardware information sequence in a multidimensional data model, the system can detect in real time whether the multidimensional hardware information sequence contains target hardware information that meets the alarm condition. Assume that the multidimensional hardware information sequence includes temperature, power consumption, and memory usage information, such as the temperatures of nodes A, B, and C are 50°C, 55°C, and 60°C, respectively, the power consumption is 150 W, 160 W, and 170 W, respectively, and the memory usage is 80%, 85%, and 90%, respectively. The alarm thresholds are set as follows: temperature warning 55°C and above, severe warning 60°C and above; power consumption warning 160 W and above, severe warning 170 W and above; memory usage warning 85% and above, severe warning 90% and above. The system detects that the temperature, power consumption, and memory usage of nodes B and C meet the warning or severe warning conditions, generates alarm information, and displays it in real time on the monitoring panel, for example: Node B: temperature 55°C, warning; Node C: temperature 60°C, severe warning; Node B: power consumption 160W, warning; Node C: power consumption 170W, severe warning; Node B: memory usage 85%, warning; Node C: memory usage 90%, severe warning. Through these alarm prompts, users can promptly identify and handle operational risks in the TPU cluster.

[0060] Further optionally, in the above steps, before real-time detection of whether the multi-dimensional hardware information sequence contains target hardware information that meets the alarm condition, the alarm threshold and / or alarm condition of different types of hardware information can be configured. Specifically, a monitoring target for collecting raw operation data is set in a pre-set configuration file; an alarm rule is defined in a configuration management file; the configuration management file is applied in the configuration file, and the alarm rule is loaded using the configuration file.

[0061] In the embodiment of the present application, the alarm rules at least include: alarm thresholds and / or alarm conditions for different types of hardware information. Further optionally, the alarm thresholds and / or alarm conditions for different types of hardware information include at least one of the following: memory usage exceeds a preset memory capacity upper limit, temperature is lower than a preset temperature lower limit, temperature is higher than a preset temperature upper limit, and node load is greater than a preset load value.

[0062] Before real-time detection of whether the multi-dimensional hardware information sequence contains target hardware information that meets the alarm conditions, it is a very important step to configure the alarm thresholds and / or alarm conditions of different types of hardware information. First, the monitoring targets for collecting raw operation data are clearly defined in the pre-set configuration file, such as the temperature, power consumption and memory usage of nodes A, B, and C. Next, the alarm rules are defined in the configuration management file, including the alarm thresholds and / or alarm conditions of different types of hardware information, such as temperature warnings of 55°C and above and severe warnings of 60°C and above, power consumption warnings of 160W and above and severe warnings of 170W and above, memory usage warnings of 85% and above and severe warnings of 90% and above. In addition, the alarm rules can also include specific conditions such as memory usage exceeding the preset memory capacity upper limit (e.g., 80%), temperature below the preset temperature lower limit (e.g., 0°C), temperature above the preset temperature upper limit (e.g., 70°C), and node load greater than the preset load value (e.g., 85%). Apply the configuration management file in the configuration file and load the alarm rules to ensure that the system can identify and handle these alarm conditions.

[0063] For example, a sample configuration file may contain detailed configurations of monitoring targets and alarm rules to ensure that when target hardware information that meets the conditions is detected, alarm information is generated and displayed to prompt users to deal with operational risks in a timely manner.

[0064] To ensure that Prometheus can correctly collect relevant metrics and trigger alerts, first ensure that the monitoring targets for collecting memory usage, temperature, and power consumption are correctly configured in the "prometheus.yml" configuration file. Next, define the alert rules in a YAML file, such as "alerts.yml", and import the file in "prometheus.yml". The alert rules should include trigger conditions such as memory usage exceeding 64GB, temperature below 5°C or above 90°C, and power consumption greater than 500W. In the "prometheus.yml" file, add or modify the "rule_files" section to load "alerts.yml". Finally, verify that the alerts are generated normally by checking whether the rules are activated on the Rules page of the Prometheus Web UI and simulating the corresponding conditions (such as high memory usage, abnormal temperature, or high power consumption). Through these steps, you can ensure that Prometheus can detect and handle potential operational risks in a timely manner.

[0065] Further optionally, in 102, before converting the original operating data into the target object's hardware information to be processed, the application scenario of the large language model can also be obtained; based on the real-time environmental data and / or scenario attributes in the application scenario, the operating status of the large language model is predicted to obtain a real-time operating status prediction value of the large language model; adaptive configuration is performed according to the real-time operating status prediction value to obtain multiple indicator dimensions for TPU cluster matching.

[0066] For example, when using Prometheus and Grafana to monitor hardware information, combined with the application scenarios of large language models, the accuracy and adaptability of monitoring and alarming can be further improved. When the large language model is running, its current application scenario is first obtained. The application scenario may include different computing tasks, load types, real-time environmental factors (such as network latency, temperature, humidity, etc.) and scenario attributes (such as task priority, data sensitivity, etc.). Based on the real-time environmental data and scenario attributes in the acquired application scenarios, machine learning and prediction models are used to predict the operation of the large language model. These predictions may include, but are not limited to, memory usage trends, CPU load predictions, temperature change trends, etc.

[0067] Based on the predicted values ​​of the running status, dynamically adjust the monitoring indicators of Prometheus and the dashboard configuration of Grafana to match the current running status. For example, if it is predicted that memory usage will increase significantly, the alarm threshold of memory usage can be adjusted in advance. If the temperature is predicted to rise significantly, the sensitivity of the temperature alarm can be increased or the cooling strategy can be adjusted. Prometheus continues to be responsible for collecting and storing hardware information such as memory usage, temperature, and power consumption. At the same time, Prometheus can also collect data related to the application scenario and input it into the prediction model. Create a dynamic dashboard in Grafana to display customized monitoring data based on the current application scenario and predicted values. For example, when the memory usage is predicted to exceed 64GB, the dashboard can display a highlighted prompt. Use Grafana's alarm function to set alarm rules based on predicted values ​​so that relevant personnel can be notified in advance when potential problems are predicted.

[0068] Prometheus and Alertmanager can continue to be responsible for the triggering and notification of rules, while Grafana's alarm function can serve as a supplement to provide more intuitive alarm information. Combined with the prediction model, the alarm can be configured to be adaptive, and the sensitivity and threshold of the alarm can be adjusted according to the predicted value. Suppose a scenario in which a large language model is used for a high-priority natural language processing task. First, obtain the real-time network latency and priority information of the task. Based on this information, predict the operation of the model, especially memory usage and CPU load. If it is predicted that memory usage will increase rapidly, Prometheus can adjust the alarm rules for memory usage in advance, and Grafana can display the relevant memory usage trend chart and set a highlight prompt. If it is predicted that the CPU load will increase significantly, Grafana can display a time series chart of CPU usage and issue an alarm in advance.

[0069] In this way, by combining the real-time data and prediction models of the application scenarios, more accurate and adaptive monitoring and alarm configuration can be achieved, improving the stability and reliability of large language models during runtime.

[0070] In 103, real-time display information is generated according to the hardware information to be processed, and updated in real time to the monitoring panel for display. In the embodiment of the present application, the real-time display information at least includes: hardware monitoring information and computing status information corresponding to the TPU cluster.

[0071] In an embodiment, the real-time display information includes hardware monitoring information and computing status information of the TPU cluster. The hardware monitoring information covers key performance indicators such as memory usage, CPU load, network traffic, temperature, and power consumption. This information is collected and stored by Prometheus and displayed in a visual form in Grafana.

[0072] For example, memory usage is updated in real time in the form of a line chart through the "node_memory_MemTotal_bytes - node_memory_MemFree_bytes" metric, in GB, showing current memory usage and historical trends; CPU load is updated in real time in the form of a line chart through the "rate(node_cpu_seconds_total{mode= The "system"}[5m])" metric is displayed in the form of a bar graph in seconds, showing the CPU load in system mode in real time; the network traffic is displayed in the form of a line graph through the "rate(node_network_receive_bytes_total[5m])" metric in MB / s, showing the network receive traffic in real time; the temperature is displayed in the form of a single value through the "node_hwmon_temp_celsius" metric in °C, showing the current temperature in real time and providing color warnings (such as red for too high and green for normal); the power consumption is displayed in the form of a line graph through the "node_power_usage_watts" metric in watts, showing the power consumption and historical trends in real time. The computing status information includes task execution status, data processing speed, and error rate.

[0073] For example, the task execution status is updated in real time through the "task_status{state="running"}" indicator in a single value display format, showing the number of tasks currently running; the data processing speed is displayed in the form of a line graph through the "rate(data_processing_bytes_total[5m])" indicator in MB / s, showing the data processing speed in real time; the error rate is displayed in the form of a line graph through the "rate(error_count_total[5m]) / rate(total_count_total[5m])" indicator, showing the error rate in real time.

[0074] In Grafana, you can create a dedicated dashboard that displays the task execution status and node temperature at the top, memory usage trends and data processing speed on the left, CPU load and network traffic on the right, and power consumption and error rate at the bottom. Through these steps and examples, you can create a dynamic real-time monitoring dashboard in Grafana to help the operation and maintenance team monitor and manage the operation status of the TPU cluster in real time.

[0075] As an optional embodiment, in 103, generating real-time display information according to the hardware information to be processed and updating it in real time to the monitoring panel for display can be implemented as follows: According to different types of hardware information in the hardware information to be processed, dynamically configure the corresponding data display method; based on the configured data display method, construct the information arrangement method in the monitoring panel; based on the hardware information to be processed, determine the hardware monitoring information and computing status information corresponding to the TPU cluster; and load the determined hardware monitoring information and computing status information into the constructed monitoring panel respectively.

[0076] In an optional embodiment, the corresponding data display mode is dynamically configured according to different types of hardware information. For example, information that changes over time, such as memory usage and network traffic, is suitable for display using a line graph, while instantaneous values ​​such as CPU load and task execution status are suitable for display using a bar graph or a single value.

[0077] Example configuration: Memory Usage: Line Chart CPU Load: Histogram Network Traffic: Line Chart Temperature: Single value display Power consumption: Line graph Task execution status: single value display Data processing speed: line chart Error Rate: Line Chart Based on the configured data display method, construct the information layout in the monitoring panel. Make sure that key information is easy to view and understand. For example, important indicators (such as task execution status and temperature) can be placed at the top, time series data (such as memory usage and data processing speed) can be placed on the left and right, and power consumption and error rate can be placed at the bottom.

[0078] Top: Task execution status (single value display), TPU node temperature (single value display).

[0079] Left: Memory usage trend (line graph), data processing speed (line graph).

[0080] Right: CPU load (bar graph), network traffic (line graph).

[0081] Bottom: Power consumption (line graph), error rate (line graph).

[0082] Based on the hardware information to be processed, determine the hardware monitoring information and computing status information corresponding to the TPU cluster. This information is collected through Prometheus and stored in the time series database.

[0083] Taking hardware monitoring information as an example, memory usage: `node_memory_MemTotal_bytes - node_memory_MemFree_bytes`; CPU load: `rate(node_cpu_seconds_total{mode= "system"}[5m])`; Network traffic: `rate(node_network_receive_bytes_total[5m])`; Temperature: `node_hwmon_temp_celsius`; Power consumption: `node_power_usage_watts`.

[0084] The calculation status information is: task execution status: `task_status{state="running"}`; data processing speed: `rate(data_processing_bytes_total[5m])`; error rate: `rate(error_count_total[5m]) / rate(total_count_total[5m])`.

[0085] Load the determined hardware monitoring information and computing status information into the built monitoring panels. Ensure that this information can be updated in real time and displayed on the panel. Configure Prometheus as a data source for Grafana to ensure that Grafana can query and obtain data from Prometheus. Create new line charts, bar charts, and single value display panels in Grafana. Configure Prometheus queries for each panel, using the previously determined metrics. For example, for the memory usage trend panel, configure the query to be `node_memory_MemTotal_bytes - node_memory_MemFree_bytes`, in GB, to display time series data. Arrange the created panels on the dashboard according to the previously designed layout. Adjust the style of each panel, such as color, font size, chart title, etc., to ensure that the information is clear and easy to read. Ensure that the data of each panel can be automatically refreshed and updated in real time. Regularly check and adjust the panel configuration to adapt to changes in hardware information and updates to metrics.

[0086] Through the above steps and examples, you can dynamically configure the data display method in Grafana, build the information layout method in the monitoring panel, determine the hardware monitoring information and computing status information of the TPU cluster, and load this information into the monitoring panel for real-time display. This method can help the operation and maintenance team monitor and manage the operation status of the TPU cluster in real time to ensure the efficient and stable operation of the system.

[0087] In an embodiment of the present application, first, the original operation data of the target object is obtained. Wherein, the target object is one or more TPU nodes in a dedicated integrated circuit TPU cluster; the TPU node is used to accelerate the training task and reasoning task of a large language model. Then, the original operation data is converted into the hardware information to be processed of the target object. Wherein, the conversion method of the hardware information to be processed is obtained based on the configuration of multiple indicator dimensions to be monitored in the TPU cluster; the multiple indicator dimensions include at least one of the following: temperature, real-time power consumption, and memory usage of the target object in the TPU cluster. Finally, real-time display information is generated according to the hardware information to be processed, and updated in real time to the monitoring panel for display. The real-time display information includes at least: hardware monitoring information and computing status information corresponding to the TPU cluster. The technical solution of the present application realizes real-time monitoring of the TPU cluster, improves the cluster monitoring efficiency, as well as the scalability and flexibility of the cluster monitoring system, and reduces the cost of building the cluster monitoring system.

[0088] The technical solution of this application has significant innovation and practicality in the monitoring and management of TPU clusters. Through the following key technical points, real-time monitoring and efficient management of TPU clusters are achieved: First, this application uses an independently developed method to obtain TPU hardware information. This method not only ensures the accuracy and completeness of the data, but also can customize specific hardware information according to specific needs, thereby providing a solid foundation for subsequent data processing and display. Secondly, this application pushes the processed hardware information to a Prometheus-based data processing application for storage in real time. This real-time push mechanism ensures the timely update and traceability of data. By using the powerful data processing capabilities of Prometheus, it can efficiently process massive amounts of hardware information and provide data support for subsequent analysis and display. Third, this application uses Grafana to configure a beautiful and elegant monitoring panel to display the hardware information and computing status of the TPU cluster processed in P in real time. Through the carefully designed monitoring panel, users can intuitively view key hardware indicators and computing status, discover and solve problems in a timely manner, thereby improving the stability and reliability of the system. Especially in the environment of large-scale TPU clusters, this real-time and visual monitoring method is particularly important. In addition, this application also configures Prometheus alarm rules to achieve real-time alarms and processing. This real-time alarm mechanism can promptly notify operation and maintenance personnel of possible abnormal situations in the system, so that they can take prompt measures to avoid the expansion of problems and system downtime, further improving the stability and reliability of the system.

[0089] In the technical solution of this application, first of all, the resource utilization rate of the TPU cluster in the training and reasoning process of large language models is significantly improved. Through real-time monitoring and accurate data analysis, resources can be better allocated and scheduled to improve resource utilization efficiency. Secondly, potential problems are discovered and solved in a timely manner to reduce downtime and ensure the continuous and stable operation of the system. Finally, this application provides a more real-time, more beautiful, and more scalable monitoring solution, which can better meet the monitoring needs of large-scale TPU clusters and has obvious advantages over the prior art. In summary, the technical solution of this application significantly improves the monitoring efficiency and system stability of the TPU cluster through independently developed hardware information acquisition methods, real-time data push and processing, beautiful monitoring panel design, and real-time alarm processing, providing strong support for the management and operation of large-scale TPU clusters.

[0090] In another embodiment of the present application, a cluster monitoring device for large language model training is also provided. Figure 2 The device comprises the following units: An acquisition unit is configured to acquire original operation data of a target object; the target object is one or more TPU nodes in a TPU cluster of a dedicated integrated circuit; the TPU nodes are used to accelerate training tasks and reasoning tasks of a large language model; A conversion unit is configured to convert the original operation data into hardware information to be processed of a target object; the conversion method of the hardware information to be processed is obtained based on a plurality of indicator dimensions to be monitored in the TPU cluster; the plurality of indicator dimensions include at least one of the following: temperature, real-time power consumption, and memory usage of the target object in the TPU cluster; The display unit is configured to generate real-time display information according to the hardware information to be processed, and update it in real time to the monitoring panel for display; the real-time display information at least includes: hardware monitoring information and computing status information corresponding to the TPU cluster.

[0091] Further optionally, each TPU node in the TPU cluster is deployed with a corresponding register, and the register of each TPU node is used to store the original operation data of the corresponding node; the acquisition unit acquires the original operation data of the target object and is configured as follows: The original operation data of the target object is read from the register corresponding to the target object through a pre-configured hardware data interface; the hardware data interface at least includes: i2c and / or PCIE.

[0092] Further optionally, the conversion unit converts the original operation data into the to-be-processed hardware information of the target object, and is configured to: Cleaning the read original operation data; According to the data calculation method corresponding to each data type, the original operation data is converted and calculated to obtain the corresponding data endpoint; wherein the data endpoint is the key indicator data to be extracted from the original operation data; The hardware information to be processed is constructed using the data endpoint corresponding to the original operation data.

[0093] Further optionally, it further includes a storage unit configured to: after the conversion unit converts the original operation data into the to-be-processed hardware information of the target object, convert the to-be-processed hardware information into a corresponding multidimensional hardware information sequence in the multidimensional data model, and store it in a storage space corresponding to the multidimensional tag; Wherein, the multi-dimensional hardware information sequence is associated with multiple indicator dimension tags.

[0094] Further optionally, an alarm unit is further included, which is configured to: after the conversion unit converts the hardware information to be processed into a corresponding multidimensional hardware information sequence in the multidimensional data model, detect in real time whether the multidimensional hardware information sequence contains target hardware information that meets the alarm condition; the detection index of the multidimensional hardware information sequence is composed of alarm thresholds and / or alarm conditions of different types of hardware information; If the multi-dimensional hardware information sequence includes target hardware information that meets the alarm condition, generating alarm information of the target object based on the target hardware information; The alarm information is displayed in real time on the monitoring panel to prompt the user to promptly handle the operational risks existing in the TPU cluster.

[0095] Further optionally, a configuration unit is further included, which is configured to: Before the detection unit detects in real time whether the multi-dimensional hardware information sequence contains target hardware information that meets the alarm condition, a monitoring target for collecting raw operation data is set in a pre-set configuration file; Alarm rules are defined in the configuration management file; the alarm rules include at least: alarm thresholds and / or alarm conditions for different types of hardware information, and the alarm thresholds and / or alarm conditions for different types of hardware information include at least one of the following: memory usage exceeds a preset memory capacity upper limit, temperature is lower than a preset temperature lower limit, temperature is higher than a preset temperature upper limit, and node load is greater than a preset load value; The configuration management file is applied in the configuration file, and the alarm rule is loaded using the configuration file.

[0096] Further optionally, it further includes a prediction unit configured to: before the conversion unit converts the original operation data into the to-be-processed hardware information of the target object, obtain the application scenario of the large language model; Based on the real-time environment data and / or scene attributes in the application scene, the operation status of the large language model is predicted to obtain a real-time operation status prediction value of the large language model; Adaptive configuration is performed according to the real-time running status prediction value to obtain multiple indicator dimensions matching the TPU cluster.

[0097] Further optionally, the display unit generates real-time display information according to the hardware information to be processed, and updates the real-time display information to the monitoring panel for display, and is configured to: Dynamically configure corresponding data display modes according to different types of hardware information in the hardware information to be processed; Based on the configured data display method, construct the information arrangement method in the monitoring panel; Determine hardware monitoring information and computing status information corresponding to the TPU cluster based on the hardware information to be processed; The determined hardware monitoring information and computing status information are respectively loaded into the monitoring panel that has been constructed.

[0098] The system can implement various steps in the above method embodiment, which will not be expanded here.

[0099] In the embodiment of the present application, a cluster monitoring device for large language model training is used to achieve real-time monitoring of the TPU cluster, improve the cluster monitoring efficiency, the scalability and flexibility of the cluster monitoring system, and reduce the cost of building the cluster monitoring system.

[0100] See also Figure 3 , Figure 3 This is a schematic diagram of an electronic device provided in an embodiment of the present application. Figure 3 As shown, an embodiment of the present application provides an electronic device 500, including a memory 510, a processor 520 and a computer program 511 stored in the memory 510 and executable on the processor 520. When the processor 520 executes the computer program 511, the following steps are implemented: obtaining the original operation data of the target object; the target object is one or more TPU nodes in a dedicated integrated circuit TPU cluster; the TPU node is used to accelerate the training tasks and reasoning tasks of a large language model; converting the original operation data into hardware information to be processed of the target object; the conversion method of the hardware information to be processed is obtained based on the configuration of multiple indicator dimensions to be monitored in the TPU cluster; the multiple indicator dimensions include at least one of the following: temperature, real-time power consumption, and memory usage of the target object in the TPU cluster; generating real-time display information based on the hardware information to be processed, and updating it to the monitoring panel in real time for display; the real-time display information includes at least: hardware monitoring information and computing status information corresponding to the TPU cluster.

[0101] See also Figure 4 , Figure 4 A schematic diagram of an embodiment of a computer-readable storage medium provided in an embodiment of the present application. Figure 4 As shown, this embodiment provides a computer-readable storage medium 600, on which a computer program 611 is stored. When the computer program 611 is executed by a processor, the following steps are implemented: obtaining the original operation data of the target object; the target object is one or more TPU nodes in a dedicated integrated circuit TPU cluster; the TPU node is used to accelerate the training task and reasoning task of a large language model; converting the original operation data into hardware information to be processed of the target object; the conversion method of the hardware information to be processed is obtained based on the configuration of multiple indicator dimensions to be monitored in the TPU cluster; the multiple indicator dimensions include at least one of the following: temperature, real-time power consumption, and memory usage of the target object in the TPU cluster; generating real-time display information based on the hardware information to be processed, and updating it in real time to the monitoring panel for display; the real-time display information includes at least: hardware monitoring information and computing status information corresponding to the TPU cluster.

[0102] It should be noted that in the above embodiments, the description of each embodiment has its own emphasis, and for parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0103] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0104] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present application.

[0105] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A cluster monitoring method for large language model training, characterized in that: The method comprises: Obtaining original running data of a target object; the target object is one or more TPU nodes in a TPU cluster of a dedicated integrated circuit; the TPU nodes are used to accelerate training tasks and reasoning tasks of a large language model; The raw operating data is converted into hardware information to be processed of the target object; the conversion method of the hardware information to be processed is obtained based on the configuration of multiple indicator dimensions to be monitored in the TPU cluster; the multiple indicator dimensions include at least one of the following: temperature, real-time power consumption, and memory usage of the target object in the TPU cluster; Real-time display information is generated according to the hardware information to be processed, and updated in real time to the monitoring panel for display; the real-time display information at least includes: hardware monitoring information and computing status information corresponding to the TPU cluster.

2. The cluster monitoring method for large language model training according to claim 1, characterized in that: Each TPU node in the TPU cluster is deployed with a corresponding register, and the register of each TPU node is used to store the original running data of the corresponding node; The obtaining of original operation data of the target object includes: The original operation data of the target object is read from the register corresponding to the target object through a pre-configured hardware data interface; the hardware data interface at least includes: i 2 c and / or PCIE.

3. The cluster monitoring method for large language model training according to claim 2, characterized in that: The converting the original operation data into hardware information to be processed of the target object includes: Cleaning the read original operation data; According to the data calculation method corresponding to each data type, the original operation data is converted and calculated to obtain the corresponding data endpoint; wherein the data endpoint is the key indicator data to be extracted from the original operation data; The hardware information to be processed is constructed using the data endpoint corresponding to the original operation data.

4. The cluster monitoring method for large language model training according to claim 1, characterized in that: After converting the original operation data into the hardware information to be processed of the target object, the method further includes: Converting the hardware information to be processed into a corresponding multidimensional hardware information sequence in a multidimensional data model, and storing the sequence in a storage space corresponding to the multidimensional tag; Wherein, the multi-dimensional hardware information sequence is associated with multiple indicator dimension tags.

5. The cluster monitoring method for large language model training according to claim 4, characterized in that: After converting the to-be-processed hardware information into a corresponding multidimensional hardware information sequence in a multidimensional data model, the method further includes: Real-time detection of whether the multi-dimensional hardware information sequence contains target hardware information that meets the alarm condition; the detection index of the multi-dimensional hardware information sequence is composed of alarm thresholds and / or alarm conditions of different types of hardware information; If the multi-dimensional hardware information sequence contains target hardware information that meets the alarm condition, generating alarm information of the target object based on the target hardware information; The alarm information is displayed in real time on the monitoring panel to prompt the user to promptly handle the operational risks existing in the TPU cluster.

6. The cluster monitoring method for large language model training according to claim 5, characterized in that: Before the real-time detection of whether the multi-dimensional hardware information sequence contains target hardware information that meets the alarm condition, the step of configuring alarm thresholds and / or alarm conditions for different types of hardware information also includes: Setting a monitoring target for collecting raw operation data in a pre-set configuration file; Alarm rules are defined in the configuration management file; the alarm rules include at least: alarm thresholds and / or alarm conditions for different types of hardware information, and the alarm thresholds and / or alarm conditions for different types of hardware information include at least one of the following: memory usage exceeds a preset memory capacity upper limit, temperature is lower than a preset temperature lower limit, temperature is higher than a preset temperature upper limit, and node load is greater than a preset load value; The configuration management file is applied in the configuration file, and the alarm rule is loaded using the configuration file.

7. The cluster monitoring method for large language model training according to claim 4, characterized in that: Before converting the original operation data into the to-be-processed hardware information of the target object, the method further includes: Obtaining an application scenario of the large language model; Based on the real-time environment data and / or scene attributes in the application scene, the operation status of the large language model is predicted to obtain a real-time operation status prediction value of the large language model; Adaptive configuration is performed according to the real-time running status prediction value to obtain multiple indicator dimensions matching the TPU cluster.

8. The cluster monitoring method for large language model training according to claim 1, characterized in that: Generating real-time display information according to the hardware information to be processed, and updating it in real time to the monitoring panel for display, includes: Dynamically configure corresponding data display modes according to different types of hardware information in the hardware information to be processed; Based on the configured data display method, construct the information arrangement method in the monitoring panel; Determine hardware monitoring information and computing status information corresponding to the TPU cluster based on the hardware information to be processed; The determined hardware monitoring information and computing status information are respectively loaded into the monitoring panel that has been constructed.

9. A cluster monitoring device for large language model training, characterized in that: The device comprises the following units, wherein: An acquisition unit is configured to acquire original operation data of a target object; the target object is one or more TPU nodes in a TPU cluster of a dedicated integrated circuit; the TPU nodes are used to accelerate training tasks and reasoning tasks of a large language model; A conversion unit is configured to convert the original operation data into hardware information to be processed of a target object; the conversion method of the hardware information to be processed is obtained based on a plurality of indicator dimensions to be monitored in the TPU cluster; the plurality of indicator dimensions include at least one of the following: temperature, real-time power consumption, and memory usage of the target object in the TPU cluster; The display unit is configured to generate real-time display information according to the hardware information to be processed, and update it in real time to the monitoring panel for display; the real-time display information at least includes: hardware monitoring information and computing status information corresponding to the TPU cluster.

10. An electronic device, characterized in that: include: Memory for storing computer software programs; A processor is used to read and execute the computer software program, thereby implementing the cluster monitoring method for large language model training as described in any one of claims 1-8.

11. A computer-readable storage medium, characterized in that: The storage medium stores a computer software program, and when the computer software program is executed by the processor, the cluster monitoring method for large language model training according to any one of claims 1 to 8 is implemented.

12. A chip, characterized in that: The chip is loaded with a computer software program and / or a hardware unit, and the computer software program and / or the hardware unit are used to implement the cluster monitoring method for large language model training as described in any one of claims 1-8.

Images