Meteorological supercomputing-based heterogeneous resource index time sequence visualization method and system in artificial intelligence large model training process

Abstract

The invention discloses an artificial intelligence large model training process heterogeneous resource index time sequence visualization method and system based on meteorological supercomputing, and the method comprises the following steps: P1, collecting supercomputing architecture intelligent computing resource indexes, and generating a log file in a standard text format; step P2, analyzing the log and storing the log in a database, and storing the log in a distributed retrieval and analysis engine; p3, acquiring training task operation information, and storing the training task operation information in a distributed retrieval and analysis engine; step P4, performing index information association and aggregation to form resource index time sequence information of the computing nodes during operation of the large-model training task, and issuing the resource index time sequence information as a standard API retrieval interface; and step P5, visually displaying the resource index time sequence. According to the method, the problems that the resource indexes of the large model training task based on supercomputing are difficult to monitor and the resource indexes and the operation are dynamically associated can be solved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence and high-performance cross-technology, and more specifically, to a method and system for visualizing the time series of heterogeneous resource indicators in the training process of a large-scale artificial intelligence model based on a meteorological supercomputer. Background Art

[0002] With the rapid development of large AI models, high-performance distributed training and tuning require time-series monitoring of intelligent computing resource utilization, including CPU utilization, memory usage, video memory usage, and GPU utilization of compute nodes. This monitoring should be associated with training tasks to aid in improving computational efficiency and video memory optimization during model training. Common resource metric monitoring tools include NVIDIA's nvidia-smi, PyTorch's video memory analysis tool Profiler, and third-party open source libraries such as GPUtil and PyTorch Memory Snapshot. Additionally, integrated monitoring systems such as Prometheus and Grafana are available.

[0003] Among them, nvidia-smi is an official NVIDIA tool that monitors basic metrics such as GPU memory usage, utilization, and temperature in real time. It only provides instantaneous snapshots, lacks time-series data storage capabilities, and cannot be dynamically associated with training tasks (such as Slurm jobs), requiring manual log parsing. PyTorch's memory analysis tool, Profiler, provides code-level memory leak detection or local optimization, but only counts memory managed by the framework and lacks global monitoring of resources such as CPU and memory. Third-party libraries require invasive code modifications, lack persistent storage, and require independent integration with databases or logging systems. They also lack global monitoring of resources such as CPU and memory. Cluster-level monitoring systems such as Prometheus + Grafana can collect GPU metrics through Node Exporter or DCGM Exporter, and Grafana can visualize time-series data. However, configuration is complex and requires the deployment of multiple components (such as Exporters and Prometheus Server). They lack deep integration with supercomputing job scheduling systems (such as Slurm), making it difficult to aggregate metrics by task.

[0004] Currently, the meteorological department has built a multi-layered, heterogeneous intelligent computing resource system, encompassing both imported and domestically produced resources. The core intelligent computing resource consists of a high-performance, integrated supercomputing GPU cluster and a domestically produced DCU cluster. Because the computing nodes of these intelligent computing resources are interconnected by internal computing and data networks, and computing tasks are scheduled via queue management software such as Slurm, the node isolation and scheduling mechanisms create isolated nodes, making them difficult to monitor via local area network systems. During large-scale meteorological model training, the massive throughput of meteorological data can easily lead to GPU memory overflow. Traditional cloud-native monitoring methods are unable to capture the memory utilization and memory usage of the intelligent computing resources in the supercomputing architecture in real time, leading to training interruptions and difficulty in locating bottlenecks. Due to the unique nature of the supercomputing architecture and its scheduling mechanism, the compute node resource metrics for distributed training tasks lack dynamic correlation with the training tasks, making it impossible to visually visualize resource time-series changes throughout the training cycle, hindering the optimization of memory and computing efficiency. Existing technologies have yet to integrate these resources with supercomputing job scheduling systems, lacking comprehensive monitoring and visualization of resource metrics related to compute nodes and training tasks.

[0005] Therefore, in order to solve the problems of memory overflow and video memory overflow caused by large-scale meteorological data throughput during the training of large artificial intelligence models, it is necessary to improve the monitoring of resource indicators of large model training tasks based on supercomputing, and realize the dynamic association between resource indicators and jobs. Summary of the Invention

[0006] To this end, the technical problem to be solved by the present invention is to provide a method and system for visualizing the time series of heterogeneous resource indicators in the training process of a large artificial intelligence model based on a meteorological supercomputer. It belongs to the resource monitoring technology for training large artificial intelligence models in an intelligent computing environment oriented to the meteorological supercomputer architecture, so as to solve the problem that resource indicators of large model training tasks based on supercomputers are difficult to monitor and the dynamic association between resource indicators and operations, thereby further solving the problems of memory overflow, video memory overflow and so on caused by large-scale meteorological data throughput in the training process of large artificial intelligence models.

[0007] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0008] A method for visualizing the time series of heterogeneous resource indicators in the training process of a large-scale artificial intelligence model based on a meteorological supercomputer includes the following steps:

[0009] Step P1: Collecting supercomputing architecture intelligent computing resource indicators: Deploy a node resource indicator batch collection program on the management node and an accelerator card resource indicator collection program on the intelligent computing node to collect supercomputing architecture intelligent computing resource indicators and generate a log file in a standard text format.

[0010] Step P2, log parsing and storage: Deploy a log parsing program on the log processing node, parse the log file in the standard text format generated in step P1 to obtain log indicator information and log time information, and store it in the distributed retrieval and analysis engine;

[0011] Step P3, training task job information acquisition: Submit the training task on the system WEB front-end, obtain the training task job JOB_ID through the Slurm RestAPI, and retrieve the computing node name, training task job start time and training task job end time of the training task job, and store them in the distributed retrieval and analysis engine;

[0012] Step P4, indicator information association and aggregation: Obtain the intelligent computing node where the training task job is running through the training task job JOB_ID; Fusion search the intelligent computing node resource data that matches the training task job running time collected and stored in steps P1 and P2 with the obtained intelligent computing node where the training task job is running, to achieve multi-dimensional indicator association retrieval and information aggregation, form the resource indicator time series information of the computing node during the running of the large model training task job and publish it as a standard API retrieval interface;

[0013] Step P5: Visualization of resource indicator time series: Call the standard API retrieval interface released in step P4 on the system WEB front-end interface to visualize the resource indicator time series information of the computing nodes during the operation of the large model training task.

[0014] In the above-mentioned method for visualizing the time series of heterogeneous resource indicators in the training process of the large-scale artificial intelligence model based on meteorological supercomputer, in step P1, the supercomputer architecture intelligent computing resource indicators include CPU utilization, memory usage, memory utilization, total video memory of the accelerator card, accelerator card utilization, accelerator card video memory usage, accelerator card video memory occupancy rate, user process, user video memory usage, accelerator card memory usage and sampling time, etc.

[0015] The above-mentioned method for visualizing the time series of heterogeneous resource indicators in the training process of the large-scale artificial intelligence model based on meteorological supercomputer, step P1 includes the following steps:

[0016] Step P101: Deploy a batch collection program based on SAR (System Activity Reporter) on the management node to obtain real-time resource usage of the intelligent computing node, including CPU utilization, memory usage, and memory utilization, and generate a log file in a standard text format and save it in the shared storage system;

[0017] Step P102: Deploy the accelerator card resource collection program (a custom code module) on the intelligent computing node, and regularly obtain the total video memory of the accelerator card, accelerator card utilization, accelerator card video memory usage, accelerator card video memory occupancy, user process, user video memory usage, accelerator card memory usage and sampling time of each intelligent computing node, and generate a log file in a standard text format and save it in the shared storage system.

[0018] The standard text format of the log file for the above-mentioned method for visualizing the time series of heterogeneous resource indicators in the training process of the large-scale artificial intelligence model based on meteorological supercomputers is as follows:

[0019] The CPU utilization log format of the intelligent computing node is: time / CPU utilization;

[0020] The format of the intelligent computing node memory usage log is: time / memory usage / memory utilization;

[0021] The format of the intelligent computing node accelerator card usage log is: time / node name / accelerator card ID / video memory usage / total video memory / accelerator card utilization / accelerator card video memory occupancy / user name / user process / user video memory usage;

[0022] If average CPU utilization and average memory utilization are also used as resource indicators, the log format for average CPU utilization of intelligent computing nodes is: time / average CPU utilization; the log format for average memory utilization of intelligent computing nodes is: time / average memory utilization.

[0023] In the above-mentioned method for visualizing the time series of heterogeneous resource indicators in the training process of a large-scale artificial intelligence model based on a meteorological supercomputer, in step P2, the log parsing program is a custom code module and Logstash; the data stored in the database after parsing includes: intelligent computing node CPU and memory data: time / node name / CPU utilization / memory usage / memory utilization; intelligent computing node accelerator card data: time / node name / accelerator card ID / video memory usage / total video memory / accelerator card utilization / accelerator card video memory occupancy / user name / user process / user video memory usage.

[0024] In the above-mentioned method for visualizing the time series of heterogeneous resource indicators in the training process of a large artificial intelligence model based on meteorological supercomputers, in step P4, the resource indicator time series information of the computing nodes during the running of the large model training task is published as a standard API retrieval interface based on Flask.

[0025] The above-mentioned method for visualizing the time series of heterogeneous resource indicators in the training process of the large-scale artificial intelligence model based on meteorological supercomputers has a standard API retrieval interface that is an HTTP GET request interface in the Restful format, including a JOB_ID query interface and a training task resource information query interface.

[0026] The above-mentioned method for visualizing the time series of heterogeneous resource indicators in the training process of the large-scale artificial intelligence model based on meteorological supercomputers has the following characteristics: for the JOB_ID query interface, the request parameters include user authorization and system name, and the return information includes training task job information and node information;

[0027] For the training task resource information query interface: the request parameters include user authorization, system name, and training task job JOB_ID; the returned information includes system name, JOB_ID, user name, time, job status, number of intelligent computing nodes used by the job, node name, number of cores, CPU usage, memory usage, accelerator card ID, accelerator card utilization, and accelerator card video memory usage.

[0028] In the above-mentioned method for visualizing the time series of heterogeneous resource indicators in the training process of a large artificial intelligence model based on a meteorological supercomputer, in step P5, the resource indicator information of the large model training task is visualized, including the time series visualization of the resource indicator information of the real-time training task operation and the time series visualization of the resource indicator information of the historical operation; the resource indicators visualized include the CPU utilization, memory usage, accelerator card utilization, and accelerator card video memory usage during the training task operation.

[0029] A meteorological supercomputer-based system for visualizing the time series of heterogeneous resource indicators in the training process of large-scale artificial intelligence models is used to implement the above-mentioned method for visualizing the time series of heterogeneous resource indicators in the training process of large-scale artificial intelligence models based on meteorological supercomputers. The system includes a training task submission module, a training task operation resource indicator time series display module, a distributed retrieval and analysis engine module, a standardized indicator retrieval interface published based on Flask, and a lightweight indicator collection and processing module.

[0030] The training task submission module submits the meteorological artificial intelligence large model training task submitted by the user to the specified supercomputing cluster through the SlurmRest API scheduling, obtains the job ID of the training task job, and retrieves the name of the computing node where the training task job runs, the start and end time of the training task job, and stores it along with the user information in the distributed retrieval and analysis engine module;

[0031] A lightweight indicator collection and processing module collects the supercomputing architecture intelligent computing resource indicator information in each supercomputing cluster, generates log files in a standard text format and stores them in a shared file system. The processing program parses the logs and stores them in the distributed retrieval and analysis engine module.

[0032] Distributed retrieval and analysis engine module, used to store training task job information data, CPU and memory resource indicator data, and accelerator card resource indicator data generated by the training task job submission module and the lightweight indicator collection and processing module;

[0033] Based on the standardized indicator retrieval interface module released by Flask, it realizes the multi-dimensional data fusion retrieval of user training task jobs and resource indicator data, and publishes it as a RESTful standard HTTP GET request interface, including the JOB_ID query interface and the training task resource information query interface;

[0034] The training task job running resource indicator time series display module calls the standard API retrieval interface published by the standardized indicator retrieval interface module based on Flask to obtain the time series resource indicator information of the large model training task, and draws it for visual display on the WEB front-end interface.

[0035] The technical solution of the present invention achieves the following beneficial technical effects:

[0036] The present invention is based on a method and system for visualizing the time series of heterogeneous resource indicators in the training process of an artificial intelligence large model based on a meteorological supercomputer. Through the collection of supercomputer architecture intelligent computing resource indicators, log analysis and storage in the warehouse, acquisition of training task operation information, indicator information correlation and aggregation, and WEB time series visualization during the training process of the artificial intelligence meteorological large model, the method and system can realize the time series visualization of multi-dimensional resource indicators in the training process of the artificial intelligence large model based on the heterogeneous intelligent computing resources of the meteorological supercomputer, including the collection and display of indicators of various heterogeneous intelligent computing nodes and accelerator cards. It has easy scalability, provides support for the memory tuning and computing tuning of the artificial intelligence meteorological large model training process, and at the same time provides multi-dimensional resource indicator monitoring for the training tasks in the supercomputing scenario, providing auxiliary support for model training tuning. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 A flow chart of a method for visualizing the time series of heterogeneous resource indicators in the training process of a large-scale artificial intelligence model based on a meteorological supercomputer in an embodiment of the present invention;

[0038] Figure 2 Schematic diagram of the system structure for visualizing the time series of heterogeneous resource indicators in the training process of a large artificial intelligence model based on a meteorological supercomputer in an embodiment of the present invention. DETAILED DESCRIPTION

[0039] like Figure 1 As shown, the method for visualizing the time series of heterogeneous resource indicators in the training process of an artificial intelligence large model based on a meteorological supercomputer in this embodiment includes the following steps:

[0040] Step P1, supercomputing architecture intelligent computing resource indicator collection: deploy node resource indicator batch collection program on the management node, deploy accelerator card resource indicator collection program on the intelligent computing node, collect CPU utilization, memory usage, accelerator card utilization, accelerator card video memory usage, sampling time and other indicators of multiple intelligent computing nodes in real time, and generate log files in standard text format.

[0041] Specifically, a batch collection program developed based on SAR (System Activity Reporter) is deployed on the management node to obtain the resource usage of the intelligent computing node in real time, including the CPU utilization, memory usage, and memory utilization of the intelligent computing node. A log file in standard text format is generated and saved in the shared storage system. An accelerator card resource collection program (custom code module) is deployed on the intelligent computing node to obtain the total video memory of the accelerator card, accelerator card utilization, accelerator card video memory usage, accelerator card video memory occupancy rate, user processes, user video memory usage, accelerator card memory usage, and sampling time of each intelligent computing node through scheduled execution. A log file in standard text format is generated and saved in the shared storage system. The format of each log file is as follows:

[0042] The CPU utilization log format of the intelligent computing node is: time / CPU utilization;

[0043] The format of the intelligent computing node memory usage log is: time / memory usage / memory utilization;

[0044] The format of the intelligent computing node accelerator card usage log is: time / node name / accelerator card ID / video memory usage / total video memory / accelerator card utilization / accelerator card video memory occupancy / user name / user process / user video memory usage. Step P2, log parsing and storage: Deploy a log parser on the log processing node, parse the log indicator information and time information, and store it in the distributed retrieval and analysis engine; use custom code modules and Logstash to parse and process the logs. The engine data structure includes:

[0045] Intelligent computing node CPU and memory data: time / node name / CPU utilization / memory usage / memory utilization;

[0046] Intelligent computing node accelerator card data: time / node name / accelerator card ID / video memory usage / total video memory / accelerator card utilization / accelerator card video memory occupancy / user name / user process / user video memory usage.

[0047] Step P3, obtain training task job information: submit the training task on the system web front-end, obtain the job JOB_ID through the Slurm RestAPI, and retrieve the computing node name where the training task is running, the training task start and end time, and store them in the distributed retrieval and analysis engine.

[0048] Step P4, indicator information association and aggregation: obtain the intelligent computing node where the training task is running through the job JOB_ID of the training task, and perform fusion retrieval on the intelligent computing node resource data that matches the training task job running time collected and stored in steps P1 and P2, and the obtained intelligent computing node where the training task job is running, to achieve multi-dimensional indicator association retrieval and information aggregation, and form the resource indicator time series information of the computing node during the running of the large model training task job, and publish the resource indicator time series information of the computing node during the running of the large model training task job based on Flask as a standard API retrieval interface; the standard API retrieval interface is an HTTP GET request interface in Restful format, including a JOB_ID query interface and a training task resource information query interface;

[0049] Among them, JOB_ID query interface: request parameters include user authorization key and system name sys; return information includes JOB_ID and node information;

[0050] Training task resource information query interface: Request parameters include user authorization, system name, and training task job JOB_ID. Returned information includes system name, JOB_ID, user name, time, job status, number of intelligent computing nodes used by the job, node name, number of cores, CPU usage, memory usage, accelerator card ID, accelerator card utilization, accelerator card video memory usage, and other resource indicator information.

[0051] Step P5: Time Series Visualization: The system's web front-end interface uses the standard API retrieval interface published in Step P4 to visualize the time series information of compute node resource indicators during the execution of the large model training task. This includes both real-time and historical training tasks. The visualized resource indicators include CPU utilization, memory usage, accelerator card utilization, and accelerator card video memory usage during the training task. The time series information of historical jobs supports video memory and compute optimization during the training of the large AI meteorological model.

[0052] This embodiment collects and aggregates key performance indicators of the training process in real time through a lightweight monitoring agent deployed on the intelligent computing node of the supercomputing architecture, realizes the real-time collection of indicators such as accelerator card usage and video memory usage during the training of the artificial intelligence meteorological large model, generates standardized information log files, and stores them in the distributed search and analysis engine, and efficiently stores the collected indicator data. It encapsulates the job-resource retrieval interface based on the Slurm Rest API and Flask, realizes the multi-dimensional association aggregation of the specified job-computing node-node resource data-job running time, realizes the resource indicator time series data retrieval of the large model training process and realizes the visual display through the proxy service forwarding front-end interface, supports the time series visualization display of multi-dimensional indicators on the Web front-end, and provides support for the video memory tuning and computing tuning of the artificial intelligence meteorological large model training process.

[0053] Use Figure 2 The heterogeneous resource indicator time series visualization system for the artificial intelligence large model training process based on meteorological supercomputer shown in the embodiment implements the heterogeneous resource indicator time series visualization method for the artificial intelligence large model training process based on meteorological supercomputer. The system includes a training task job submission module, a training task job operation resource indicator time series display module, a distributed retrieval and analysis engine module, a standardized indicator retrieval interface published based on Flask, and a lightweight indicator collection and processing module.

[0054] Among them, the training task submission module provides users with the ability to create, configure, and submit meteorological artificial intelligence large-scale model training tasks. The system backend schedules and submits the task to the specified supercomputing cluster through the Slurm API based on the resource information configured by the user task, obtains the JOB_ID of the task job, and retrieves the name of the computing node where the job runs, the start and end time of the job, and stores it in the distributed retrieval and analysis engine along with the user information.

[0055] The lightweight indicator collection and processing module collects resource indicator information of intelligent computing nodes in each supercomputing cluster, including resource indicators of the intelligent computing node CPU and memory: intelligent computing node CPU utilization, memory usage, memory utilization, sampling time, etc.; intelligent computing node accelerator card resource indicators: total accelerator card memory, accelerator card utilization, accelerator card memory usage, accelerator card memory occupancy rate, user process, user memory usage, accelerator card memory usage and sampling time, etc., and generates log files in standard text format and stores them in the shared file system. The processing program parses the logs and stores them in the distributed retrieval and analysis engine.

[0056] The distributed retrieval and analysis engine module is used to store the training task job information data, CPU and memory resource indicator data, and accelerator card resource indicator data generated by the above training task job submission module and lightweight indicator collection and processing module.

[0057] Based on Flask, it is published as a standard API retrieval interface module to implement multi-dimensional data fusion retrieval of user training task operations and resource indicator data, and is published as a HTTP GET request interface in the RESTful standard format, including the JOB_ID query interface and the training task resource information query interface.

[0058] The timing visualization display module uses a system call retrieval interface to obtain timing resource indicator information for large-scale model training tasks, and displays it visually on the WEB front-end interface. This includes real-time timing visualization of CPU utilization, memory usage, accelerator card utilization, accelerator card video memory usage, and other information during the training task, as well as a timing display of resource indicators for historical job information, to assist in supporting video memory and computing optimization during the training process of large-scale artificial intelligence meteorological models.

[0059] Existing technologies often focus on single resources or local scenarios, while large-scale model training requires a global, multi-dimensional monitoring system. This embodiment, based on the meteorological supercomputer, provides a time-series visualization system for heterogeneous resource indicators in the training process of large-scale artificial intelligence models. This system addresses these shortcomings through a design that includes lightweight acquisition agents, distributed storage, and task association aggregation. It is particularly optimized for large-scale training scenarios using meteorological supercomputers, enabling global monitoring of resource indicators during the supercomputer training process. Specifically, this embodiment provides a time-series visualization system for heterogeneous resource indicators in the training process of large-scale artificial intelligence models based on meteorological supercomputers, which has the following advantages:

[0060] (1) Multi-dimensional collaborative monitoring of training resource indicators of supercomputing architecture intelligent computing resources: Dynamic aggregation of job ID-computing node-resource indicators is achieved based on the Slurm Rest API, breaking through data silos and supporting the analysis of collaborative load characteristics of node CPU, accelerator card (GPU, DCU) utilization and video memory utilization from a single task perspective.

[0061] (2) Full-link visualization and tuning support: Through the low-latency visualization of distributed engine storage + Flask API + Web front-end, the resource curve of the entire life cycle of the training task is intuitively displayed, helping developers to quickly locate computing hotspots (such as low accelerator card utilization) and video memory leakage problems, thereby improving training stability and efficiency.

[0062] (3) Monitoring of heterogeneous accelerator card resource indicators for supercomputing architecture intelligent computing resources: Supports monitoring of accelerator card indicators for supercomputing architectures such as NVIDIA P100 cluster, A800 cluster, and domestic Haiguang DCU cluster.

[0063] (4) Lightweight and highly scalable: It adopts a distributed log parsing and retrieval architecture, is compatible with the heterogeneous intelligent computing environment of the meteorological supercomputing architecture, supports horizontal expansion to monitoring scenarios of larger-scale computing power clusters, and supports rapid expansion of multiple heterogeneous acceleration card clusters.

[0064] (5) Real-time and historical resource indicator information analysis: Supports real-time visualization analysis of resource indicator information for large model training tasks, and also supports visualization analysis of resource indicator information for historical training tasks, facilitating optimization and comparison of multiple sets of training tasks.

[0065] Obviously, the above embodiments are merely examples for clarity of explanation and are not intended to limit the implementation methods. Those skilled in the art will readily appreciate that other variations or modifications based on the above descriptions are possible. It is not necessary and impossible to enumerate all implementation methods here. Obvious variations or modifications arising therefrom remain within the scope of protection of the claims of this patent application.

Claims

1. A method for visualizing the time series of heterogeneous resource indicators in the training process of an artificial intelligence large model based on meteorological supercomputer, characterized by: The steps include: Step P1: Collecting supercomputing architecture intelligent computing resource indicators: Deploy a node resource indicator batch collection program on the management node and an accelerator card resource indicator collection program on the intelligent computing node to collect supercomputing architecture intelligent computing resource indicators and generate a log file in a standard text format. Step P2, log parsing and storage: Deploy a log parsing program on the log processing node, parse the log file in the standard text format generated in step P1 to obtain log indicator information and log time information, and store it in the distributed retrieval and analysis engine; Step P3, training task job information acquisition: Submit the training task on the system web front-end, obtain the training task job JOB_ID through the Slurm Rest API, and retrieve the computing node name, training task job start time and training task job end time of the training task job, and store them in the distributed retrieval and analysis engine; Step P4, indicator information association and aggregation: Obtain the intelligent computing node where the training task job is running through the training task job JOB_ID; Fusion search the intelligent computing node resource data that matches the training task job running time collected and stored in steps P1 and P2 with the obtained intelligent computing node where the training task job is running, to achieve multi-dimensional indicator association retrieval and information aggregation, form the resource indicator time series information of the computing node during the running of the large model training task job and publish it as a standard API retrieval interface; Step P5: Visualization of resource indicator time series: Call the standard API retrieval interface released in step P4 on the system WEB front-end interface to visualize the resource indicator time series information of the computing nodes during the operation of the large model training task.

2. The method for visualizing the time series of heterogeneous resource indicators in the training process of an artificial intelligence large model based on meteorological supercomputer according to claim 1 is characterized in that: In step P1, the supercomputing architecture intelligent computing resource indicators include CPU utilization, memory usage, memory utilization, total video memory of the accelerator card, accelerator card utilization, accelerator card video memory usage, accelerator card video memory occupancy, user process, user video memory usage, accelerator card memory usage and sampling time.

3. The method for visualizing the time series of heterogeneous resource indicators in the training process of an artificial intelligence large model based on meteorological supercomputer according to claim 2 is characterized in that: Step P1 includes the following steps: Step P101: Deploy a batch data collection program developed based on SAR on the management node to obtain the resource usage of the intelligent computing node in real time, including the CPU utilization, memory usage, and memory utilization of the intelligent computing node. Generate a log file in a standard text format and save it in the shared storage system. Step P102: Deploy the accelerator card resource collection program on the intelligent computing node, regularly obtain the total accelerator card memory, accelerator card utilization, accelerator card memory usage, accelerator card memory occupancy, user process, user memory usage, accelerator card memory usage and sampling time of each intelligent computing node, and generate a log file in standard text format and save it in the shared storage system.

4. The method for visualizing the time series of heterogeneous resource indicators in the training process of an artificial intelligence large model based on meteorological supercomputer according to claim 3 is characterized in that: The standard text format of the log file is as follows: The CPU utilization log format of the intelligent computing node is: time / CPU utilization; The format of the intelligent computing node memory usage log is: time / memory usage / memory utilization; The format of the intelligent computing node accelerator card usage log is: time / node name / accelerator card ID / video memory usage / total video memory / accelerator card utilization / accelerator card video memory occupancy / user name / user process / user video memory usage.

5. The method for visualizing the time series of heterogeneous resource indicators in the training process of an artificial intelligence large model based on meteorological supercomputer according to claim 1 is characterized in that: In step P2, the log parsing program is a custom code module and Logstash; the data stored in the database after parsing includes: Intelligent computing node CPU and memory data: time / node name / CPU utilization / memory usage / memory utilization; Intelligent computing node accelerator card data: time / node name / accelerator card ID / video memory usage / total video memory / accelerator card utilization / accelerator card video memory occupancy / user name / user process / user video memory usage.

6. The method for visualizing the time series of heterogeneous resource indicators in the training process of an artificial intelligence large model based on meteorological supercomputer according to claim 1 is characterized in that: In step P4, the resource indicator time series information of the computing nodes during the operation of the large model training task is published as a standard API retrieval interface based on Flask.

7. The method for visualizing the time series of heterogeneous resource indicators in the training process of an artificial intelligence large model based on meteorological supercomputer according to claim 6 is characterized in that: The standard API retrieval interface is a RESTful HTTP GET request interface, including a JOB_ID query interface and a training task resource information query interface.

8. The method for visualizing the time series of heterogeneous resource indicators in the training process of an artificial intelligence large model based on meteorological supercomputer according to claim 7 is characterized in that: For the JOB_ID query interface: the request parameters include user authorization and system name, and the returned information includes training task job information and node information; For the training task resource information query interface: the request parameters include user authorization, system name, and training task job JOB_ID; the returned information includes system name, JOB_ID, user name, time, job status, number of intelligent computing nodes used by the job, node name, number of cores, CPU usage, memory usage, accelerator card ID, accelerator card utilization, and accelerator card video memory usage.

9. The method for visualizing the time series of heterogeneous resource indicators in the training process of an artificial intelligence large model based on meteorological supercomputer according to claim 1 is characterized in that: In step P5, the resource indicator information of the large model training task is visualized, including the time series visualization of the resource indicator information of the real-time training task job and the time series visualization of the resource indicator information of the historical job; The visualized resource indicators include CPU utilization, memory usage, accelerator card utilization, and accelerator card memory usage during the training task.

10. A system for visualizing heterogeneous resource indicators in the training process of an artificial intelligence large-scale model based on meteorological supercomputer, characterized by: The method for realizing the time series visualization of heterogeneous resource indicators in the training process of the large-scale artificial intelligence model based on meteorological supercomputer as claimed in claim 1 comprises a training task job submission module, a training task job operation resource indicator time series display module, a distributed retrieval and analysis engine module, a standardized indicator retrieval interface published based on Flask, and a lightweight indicator collection and processing module: The training task submission module submits the meteorological artificial intelligence large model training task submitted by the user to the specified supercomputing cluster through the Slurm RestAPI scheduling, obtains the job ID of the training task job, and retrieves the name of the computing node where the training task job runs, the start and end time of the training task job, and stores it along with the user information in the distributed retrieval and analysis engine module; A lightweight indicator collection and processing module collects the supercomputing architecture intelligent computing resource indicator information in each supercomputing cluster, generates log files in a standard text format and stores them in a shared file system. The processing program parses the logs and stores them in the distributed retrieval and analysis engine module. Distributed retrieval and analysis engine module, used to store training task job information data, CPU and memory resource indicator data, and accelerator card resource indicator data generated by the training task job submission module and the lightweight indicator collection and processing module; Based on the standardized indicator retrieval interface module released by Flask, it realizes the multi-dimensional data fusion retrieval of user training task jobs and resource indicator data, and publishes it as a RESTful standard HTTP GET request interface, including the JOB_ID query interface and the training task resource information query interface; The training task job running resource indicator time series display module calls the standard API retrieval interface published by the standardized indicator retrieval interface module based on Flask to obtain the time series resource indicator information of the large model training task, and draws it for visual display on the WEB front-end interface.

Images