Cluster diagnosis tool for large-scale artificial intelligence computing service
Through the multi-dimensional diagnosis and automated identification functions of cluster diagnostic tools, the complexity of fault server identification and processing in AI clusters is solved, rapid screening and fault location are achieved, and the operation efficiency and stability of the cluster are improved.
Patent Information
- Application Number
- CN202411403740.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-09
- Publication Date
- 2024-11-05
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
In the existing AI cluster management, the identification and processing of faulty servers is a complex and time-consuming process, which affects the work efficiency of the cluster.
Provide a cluster diagnostic tool for large-scale artificial intelligence computing services, including configuration file analysis module, test sample startup module, log module and volume judgment module. By analyzing configuration files, automatically performing multi-dimensional test cases, and generating diagnostic reports. Quickly identify and isolate the failed server and optimize cluster configuration.
It realizes accurate positioning of failed AI cluster servers within minutes, shortens screening time, improves work efficiency, and improves cluster stability and fault tolerance.
Smart Images

Figure CN118897752A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of server cluster management tools, and in particular to a cluster diagnostic tool for large-scale artificial intelligence computing services. Background Art
[0002] In existing AI cluster management, identifying and handling faulty servers is usually a complex and time-consuming process. Traditional fault handling methods often require checking and repairing servers one by one, which greatly affects the work efficiency of the cluster and leads to a decrease in the operating efficiency of the entire cluster. Summary of the invention
[0003] In order to solve the problems existing in the prior art, the present invention provides the following technical solutions.
[0004] The present invention provides a cluster diagnosis tool for large-scale artificial intelligence computing business, comprising: a configuration file parsing module, configured to parse the configuration file to obtain sample configuration information; a test sample startup module, configured to receive the sample configuration information transmitted by the configuration file parsing module, complete the instruction construction of the test sample executor, instruction distribution and reception of the sample diagnosis result; a test sample execution module, configured to read the sample configuration information, improve the sample startup configuration according to the sample startup information transmitted by the test sample startup module, and start the sample test based on the execution script of the sample itself, obtain the sample diagnosis result, and return the diagnosis result; a log module, configured to generate a sample unified identifier according to the call of the test sample startup module, receive a log storage path specified by a user, manage the log content written under the path, and perform log analysis according to the log content; a paper judging module, configured to generate a judgment result based on the diagnosis result of each sample and preset threshold management information, and obtain a cluster diagnosis report.
[0005] Preferably, the system further comprises an output management module configured to perform local result visualization output and / or local result archiving for the diagnostic report.
[0006] Furthermore, the sample configuration information includes cluster information and sample information.
[0007] Furthermore, the test sample startup module is also configured to be responsible for starting and writing the local storage module.
[0008] Furthermore, the returning the diagnosis result includes: returning the diagnosis result in a character nested manner.
[0009] Furthermore, the log module is also configured to automatically identify and record log entries containing illegal values and their corresponding host information.
[0010] Furthermore, the log module is also configured to summarize and manage bad nodes and information related to bad nodes that are globally diagnosed in each diagnosis scenario, and automatically summarize all output information of the log module.
[0011] Furthermore, the preset threshold management information includes questionnaires, standard answers, thresholds and diagnostic methods required in each diagnosis scenario.
[0012] Furthermore, based on the diagnostic results of each sample and the preset threshold management information, an evaluation result is generated, and obtaining a cluster diagnostic report also includes: managing the diagnostic results of each sample, identifying the bad nodes diagnosed by each sample and information related to the bad nodes, and automatically generating a diagnostic report for each sample.
[0013] Furthermore, archiving the local result of the diagnosis report includes storing the diagnosis report in a local lightweight database sqlite3.
[0014] The beneficial effects of the present invention are: (1) The cluster diagnostic tool for large-scale artificial intelligence computing services provided by the present invention implements multi-dimensional diagnosis of the availability of AI clusters from single-card to multi-machine ranges through preemptive and non-preemptive test cases. The cluster diagnostic tool automatically executes multi-dimensional test case sequences by parsing user configuration files. Based on the calibration values provided by the user before diagnosis, the cluster diagnostic tool diagnoses the health level of the cluster and automatically generates single-sample diagnostic results and global diagnostic reports; (2) The cluster diagnostic tool provided by the present invention can accurately locate faulty servers in an AI cluster within minutes, and optimize cluster configuration in linear time to provide the largest available cluster; (3) The cluster diagnostic tool provided by the present invention can automatically and quickly screen servers and quickly identify servers that may have problems. Compared with the traditional manual screening method, the present invention greatly shortens the screening time and improves work efficiency. It can not only quickly screen out problematic servers, but also further accurately locate the problem. Through minute-level diagnosis, technicians can more accurately identify the specific cause of the fault and take more targeted solutions; (4) For AI clusters, the cluster diagnostic tool provided by the present invention can not only identify faulty servers, but also provide the maximum available cluster in linear time. Bad nodes in the AI cluster system can be isolated and processed in a timely manner, and the maximum available sub-cluster can be given as soon as possible when responding to user tasks. This not only improves the overall stability of the cluster, but also improves the fault tolerance of the AI cluster system. BRIEF DESCRIPTION OF THE DRAWINGS
[0015] Figure 1This is a functional structure diagram of a cluster diagnostic tool for large-scale artificial intelligence computing services described in the present invention. DETAILED DESCRIPTION
[0016] In order to better understand the above technical solution, the above technical solution will be described in detail below in conjunction with the accompanying drawings and specific implementation methods.
[0017] The method provided by the present invention can be implemented in the following terminal environment, and the terminal may include one or more of the following components: a processor, a memory, and a display screen. The memory stores at least one instruction, and the instruction is loaded and executed by the processor to implement the method described in the following embodiment.
[0018] The processor may include one or more processing cores. The processor uses various interfaces and lines to connect various parts of the entire terminal, and executes various functions of the terminal and processes data by running or executing instructions, programs, code sets or instruction sets stored in the memory, and calling data stored in the memory.
[0019] The memory may include random access memory (RAM) or read-only memory (ROM). The memory may be used to store instructions, programs, codes, and commands.
[0020] The display screen is used to display the user interface of each application.
[0021] In addition, those skilled in the art can understand that the structure of the above terminal does not constitute a limitation on the terminal, and the terminal may include more or fewer components, or combine certain components, or arrange the components differently. For example, the terminal also includes components such as a radio frequency circuit, an input unit, a sensor, an audio circuit, and a power supply, which will not be described in detail here.
[0022] like Figure 1 As shown, an embodiment of the present invention provides a cluster diagnostic tool for large-scale artificial intelligence computing services, including: A configuration file parsing module is configured to parse the configuration file to obtain sample configuration information; wherein the sample configuration information includes cluster information and sample information; The test sample startup module is configured to receive the sample configuration information transmitted by the configuration file parsing module, complete the instruction construction of the test sample executor, the instruction distribution and the reception of the sample diagnosis result; in other embodiments, the test sample startup module is also configured to be responsible for the startup and writing of the local storage module; It should be noted that the user input is parsed to obtain a configuration file, and the configuration file parsing module parses the configuration file to obtain sample configuration information including cluster information and sample information; the test sample startup module includes two working modes: single-node startup and cluster startup; single-node startup means that the IP address in the configuration file is one, and cluster startup means that the IP addresses in the configuration file are multiple. According to different configuration requirements, the test sample startup module can select different working modes; the test sample startup module can also receive the diagnostic results of the samples sent by the server under test to the test sample startup module.
[0023] The test sample execution module is configured to read the sample configuration information, improve the sample startup configuration according to the sample startup information transmitted by the test sample startup module, start the sample test based on the sample execution script itself, obtain the sample diagnosis result, and return the diagnosis result; It should be noted that before executing the corresponding sample, the test sample execution module needs to obtain the sample startup information transmitted by the test sample startup module. The test sample startup information includes the determination of the sample and the configuration information required to start the determined sample. After the test sample execution module obtains the test sample startup information transmitted by the test sample startup module, it completes the sample startup configuration and starts the sample test execution based on the execution script of the sample itself. Finally, according to the diagnosis result of the sample, the diagnosis result is returned; preferably, the diagnosis result is returned in a character nested manner to ensure that the sample diagnosis result is not interfered with by other information; The log module is configured to generate a unified test sample identifier according to the call of the test sample startup module, receive a log storage path specified by a user, perform log management on the log content written in the path, and perform log analysis according to the log content; It should be noted that the design of the log module is to achieve efficient and accurate log recording and management. The log module can generate a unified sample identifier based on the call of the test sample startup module, receive the log storage path specified by the user, manage the log content in a unified manner under the specified directory, and manage the log content written under the path to ensure the persistent storage of the log content. It can also call the tool for log analysis based on the log content stored in the log management. At the same time, the log module also has the function of adapting to different diagnostic scenarios based on the normalized sample information passed in. The log module supports multi-level log recording, allowing users to classify and manage logs according to their importance, which is convenient for subsequent log analysis and problem tracking. In addition, the log module can automatically identify and record log entries containing illegal values and their corresponding host information, providing support for rapid location and resolution of problems. The log module can also summarize and manage the globally diagnosed bad nodes and their related information in each diagnostic scenario, and automatically summarize all the output information of the total log module to generate a global summary report.
[0024] The judging module is configured to generate a judging result based on the diagnostic result of each sample and preset threshold management information to obtain a cluster diagnosis report.
[0025] Specifically, the grading module can receive and manage the preset threshold management information such as the questionnaire, standard answers, thresholds and diagnostic methods required in each diagnostic scenario to ensure the standardization and consistency of the diagnostic process; at the same time, according to the log storage path, each sample in the cluster is diagnosed one by one to improve the pertinence and accuracy of the diagnosis; in addition, the grading module is also configured to manage the diagnostic results of each sample, identify the bad nodes diagnosed for each sample and the information related to the bad nodes, and automatically generate a diagnostic report for each sample containing the diagnostic results, problem description and the comparison output of the test value and the standard answer.
[0026] The output management module is configured to perform local result visualization output and / or local result archiving for the diagnostic report.
[0027] Specifically, the output management module is configured to output the diagnostic report through the log module for local result visualization and / or local result archiving, wherein local result archiving of the diagnostic report includes storing the diagnostic report in a local lightweight database sqlite3 for subsequent visualization analysis of historical execution information.
[0028] Although preferred embodiments of the present invention have been described, additional changes and modifications may be made to these embodiments by those skilled in the art once the basic inventive concepts are known. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention. Obviously, those skilled in the art may make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.
Claims
1. A cluster diagnostic tool for large-scale artificial intelligence computing services, characterized in that: include: The configuration file parsing module is configured to parse the configuration file to obtain sample configuration information; A test sample startup module is configured to receive the sample configuration information transmitted by the configuration file parsing module, complete the instruction construction and instruction distribution of the test sample executor and receive the diagnosis result of the sample; The test sample execution module is configured to read the sample configuration information, improve the sample startup configuration according to the sample startup information transmitted by the test sample startup module, start the sample test based on the execution script of the sample itself, obtain the diagnosis result of the sample, and send back the diagnosis result; A log module is configured to generate a sample unified identifier according to the call of the test sample startup module, receive a log storage path specified by a user, manage the log content written in the path, and perform log analysis according to the log content; The judging module is configured to generate a judging result based on the diagnostic result of each sample and preset threshold management information to obtain a cluster diagnosis report.
2. A cluster diagnostic tool for large-scale artificial intelligence computing services as claimed in claim 1, characterized in that: The system also includes an output management module configured to perform local result visualization output and / or local result archiving on the diagnostic report.
3. A cluster diagnostic tool for large-scale artificial intelligence computing services as claimed in claim 1, characterized in that: The sample configuration information includes cluster information and sample information.
4. A cluster diagnostic tool for large-scale artificial intelligence computing services as claimed in claim 1, characterized in that: The test sample startup module is also configured to be responsible for starting and writing the local storage module.
5. A cluster diagnostic tool for large-scale artificial intelligence computing services as claimed in claim 1, characterized in that: The returning the diagnosis result includes: returning the diagnosis result in a character nested manner.
6. A cluster diagnostic tool for large-scale artificial intelligence computing services as claimed in claim 1, characterized in that: The log module is also configured to automatically identify and record log entries containing illegal values and their corresponding host information.
7. A cluster diagnostic tool for large-scale artificial intelligence computing services as claimed in claim 6, characterized in that: The log module is also configured to aggregate and manage bad nodes and information related to bad nodes that are globally diagnosed in each diagnosis scenario, and automatically aggregate all output information of the log module.
8. A cluster diagnostic tool for large-scale artificial intelligence computing services as claimed in claim 1, characterized in that: The preset threshold management information includes the questionnaires, standard answers, thresholds and diagnostic methods required in each diagnosis scenario.
9. A cluster diagnostic tool for large-scale artificial intelligence computing services as claimed in claim 8, characterized in that: The evaluation results are generated based on the diagnostic results of each sample and the preset threshold management information, and obtaining the cluster diagnostic report also includes: managing the diagnostic results of each sample, identifying the bad nodes diagnosed by each sample and the information related to the bad nodes, and automatically generating a diagnostic report for each sample.
10. A cluster diagnostic tool for large-scale artificial intelligence computing services as claimed in claim 2, characterized in that: The local result archiving of the diagnostic report includes storing the diagnostic report in a local lightweight database sqlite3.
Citation Information
Patent Citations
Log based computer system fault diagnosis method and device
CN103761173A
Test and diagnosis integrated development platform based on domestic operating system
CN111782539A
Distributed cluster fault test method and device
CN111953560A
Distributed data monitoring device
US20190138410A1
System and method for automated problem diagnosis
US7379846B1
