Cluster diagnosis tool for large-scale artificial intelligence computing service

Through the multi-dimensional diagnosis and automated identification functions of cluster diagnostic tools, the complexity of fault server identification and processing in AI clusters is solved, rapid screening and fault location are achieved, and the operation efficiency and stability of the cluster are improved.

CN118897752AInactive Publication Date: 2024-11-05BEIJING ACAD OF ARTIFICIAL INTELLLIGENCE
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
CN202411403740.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-10-09
Publication Date
2024-11-05
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

In the existing AI cluster management, the identification and processing of faulty servers is a complex and time-consuming process, which affects the work efficiency of the cluster.

Method used

Provide a cluster diagnostic tool for large-scale artificial intelligence computing services, including configuration file analysis module, test sample startup module, log module and volume judgment module. By analyzing configuration files, automatically performing multi-dimensional test cases, and generating diagnostic reports. Quickly identify and isolate the failed server and optimize cluster configuration.

Benefits of technology

It realizes accurate positioning of failed AI cluster servers within minutes, shortens screening time, improves work efficiency, and improves cluster stability and fault tolerance.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118897752A_ABST
    Figure CN118897752A_ABST
Patent Text Reader

Abstract

The invention discloses a large-scale artificial intelligence computing service-oriented cluster diagnostic tool, which belongs to the technical field of server cluster management tools, and comprises the following steps: a configuration file analysis module analyzes a configuration file to obtain sample configuration information; the test sample starting module receives the sample configuration information and completes instruction construction, instruction distribution and receiving of a test sample actuator; the test sample execution module reads the sample configuration information, starts a sample test based on an execution script of a sample, obtains a diagnosis result of the sample and returns the diagnosis result; the log module receives a log storage path specified by a user, and manages and analyzes log contents under the path; and the paper judgment module generates a judgment result based on the diagnosis result and preset threshold management information, and obtains a cluster diagnosis report. According to the method, the availability of the AI cluster from a single-card range to a multi-machine range can be subjected to multi-dimensional diagnosis, the fault server in the cluster can be quickly and accurately positioned, the screening time is shortened, and the working efficiency is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of server cluster management tools, and in particular to a cluster diagnostic tool for large-scale artificial intelligence computing services. Background Art

[0002] In existing AI cluster management, identifying and handling faulty servers is usually a complex and time-consuming process. Traditional fault handling methods often require checking and repairing servers one by one, which greatly affects the work efficiency of the cluster and leads to a decrease in the operating efficiency of the entire cluster. Summary of the invention

[0003] In order to solve the problems existing in the prior art, the present invention provides the following technical solutions.

[0004] The present invention provides a cluster diagnosis tool for large-scale artificial intelligence computing business, comprising: a configuration file parsing module, configured to parse the configuration file to obtain sample configuration information; a test sample startup module, configured to receive the sample configuration information transmitted by the configuration file parsing module, complete the instruction construction of the test sample executor, instruction distribution and reception of the sample diagnosis result; a test sample execution module, configured to read the sample configuration information, improve the sample startup configuration according to the sample startup information transmitted by the test sample startup module, and start the sample test based on the execution script of the sample itself, obtain the sample diagnosis result, and return the diagnosis result; a log module, configured to generate a sample unified identifier according to the call of the test sample startup module, receive a log storage path specified by a user, manage the log content written under the path, and perform log analysis according to the log content; a paper judging module, configured to generate a judgment result based on the diagnosis result of each sample and preset threshold management information, and obtain a cluster diagnosis report.

[0005] Preferably, the system further comprises an output management module configured to perform local result visualization output and / or local result archiving for the diagnostic report.

[0006] Furthermore, the sample configuration information includes cluster information and sample information.

[0007] Furthermore, the test sample startup module is also configured to be responsible for starting and writing the local storage module.

[0008] Furthermore, the returning the diagnosis result includes: returning the diagnosis result in a character nested manner.

[0009] Furthermore, the log module is also configured to automatically identify and record log entries containing illegal values ​​and their corresponding host information.

[0010] Furthermore, the log module is also configured to summarize and manage bad nodes and information related to bad nodes that are globally diagnosed in each diagnosis scenario, and automatically summarize all output information of the log module.

[0011] Furthermore, the preset threshold management information includes questionnaires, standard answers, thresholds and diagnostic methods required in each diagnosis scenario.

[0012] Furthermore, based on the diagnostic results of each sample and the preset threshold management information, an evaluation result is generated, and obtaining a cluster diagnostic report also includes: managing the diagnostic results of each sample, identifying the bad nodes diagnosed by each sample and information related to the bad nodes, and automatically generating a diagnostic report for each sample.

[0013] Furthermore, archiving the local result of the diagnosis report includes storing the diagnosis report in a local lightweight database sqlite3.

[0014] The beneficial effects of the present invention are: (1) The cluster diagnostic tool for large-scale artificial intelligence computing services provided by the present invention implements multi-dimensional diagnosis of the availability of AI clusters from single-card to multi-machine ranges through preemptive and non-preemptive test cases. The cluster diagnostic tool automatically executes multi-dimensional test case sequences by parsing user configuration files. Based on the calibration values ​​provided by the user before diagnosis, the cluster diagnostic tool diagnoses the health level of the cluster and automatically generates single-sample diagnostic results and global diagnostic reports; (2) The cluster diagnostic tool provided by the present invention can accurately locate faulty servers in an AI cluster within minutes, and optimize cluster configuration in linear time to provide the largest available cluster; (3) The cluster diagnostic tool provided by the present invention can automatically and quickly screen servers and quickly identify servers that may have problems. Compared with the traditional manual screening method, the present invention greatly shortens the screening time and improves work efficiency. It can not only quickly screen out problematic servers, but also further accurately locate the problem. Through minute-level diagnosis, technicians can more accurately identify the specific cause of the fault and take more targeted solutions; (4) For AI clusters, the cluster diagnostic tool provided by the present invention can not only identify faulty servers, but also provide the maximum available cluster in linear time. Bad nodes in the AI ​​cluster system can be isolated and processed in a timely manner, and the maximum available sub-cluster can be given as soon as possible when responding to user tasks. This not only improves the overall stability of the cluster, but also improves the fault tolerance of the AI ​​cluster system. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1This is a functional structure diagram of a cluster diagnostic tool for large-scale artificial intelligence computing services described in the present invention. DETAILED DESCRIPTION

[0016] In order to better understand the above technical solution, the above technical solution will be described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0017] The method provided by the present invention can be implemented in the following terminal environment, and the terminal may include one or more of the following components: a processor, a memory, and a display screen. The memory stores at least one instruction, and the instruction is loaded and executed by the processor to implement the method described in the following embodiment.

[0018] The processor may include one or more processing cores. The processor uses various interfaces and lines to connect various parts of the entire terminal, and executes various functions of the terminal and processes data by running or executing instructions, programs, code sets or instruction sets stored in the memory, and calling data stored in the memory.

[0019] The memory may include random access memory (RAM) or read-only memory (ROM). The memory may be used to store instructions, programs, codes, and commands.

[0020] The display screen is used to display the user interface of each application.

[0021] In addition, those skilled in the art can understand that the structure of the above terminal does not constitute a limitation on the terminal, and the terminal may include more or fewer components, or combine certain components, or arrange the components differently. For example, the terminal also includes components such as a radio frequency circuit, an input unit, a sensor, an audio circuit, and a power supply, which will not be described in detail here.

[0022] like Figure 1 As shown, an embodiment of the present invention provides a cluster diagnostic tool for large-scale artificial intelligence computing services, including: A configuration file parsing module is configured to parse the configuration file to obtain sample configuration information; wherein the sample configuration information includes cluster information and sample information; The test sample startup module is configured to receive the sample configuration information transmitted by the configuration file parsing module, complete the instruction construction of the test sample executor, the instruction distribution and the reception of the sample diagnosis result; in other embodiments, the test sample startup module is also configured to be responsible for the startup and writing of the local storage module; It should be noted that the user input is parsed to obtain a configuration file, and the configuration file parsing module parses the configuration file to obtain sample configuration information including cluster information and sample information; the test sample startup module includes two working modes: single-node startup and cluster startup; single-node startup means that the IP address in the configuration file is one, and cluster startup means that the IP addresses in the configuration file are multiple. According to different configuration requirements, the test sample startup module can select different working modes; the test sample startup module can also receive the diagnostic results of the samples sent by the server under test to the test sample startup module.

[0023] The test sample execution module is configured to read the sample configuration information, improve the sample startup configuration according to the sample startup information transmitted by the test sample startup module, start the sample test based on the sample execution script itself, obtain the sample diagnosis result, and return the diagnosis result; It should be noted that before executing the corresponding sample, the test sample execution module needs to obtain the sample startup information transmitted by the test sample startup module. The test sample startup information includes the determination of the sample and the configuration information required to start the determined sample. After the test sample execution module obtains the test sample startup information transmitted by the test sample startup module, it completes the sample startup configuration and starts the sample test execution based on the execution script of the sample itself. Finally, according to the diagnosis result of the sample, the diagnosis result is returned; preferably, the diagnosis result is returned in a character nested manner to ensure that the sample diagnosis result is not interfered with by other information; The log module is configured to generate a unified test sample identifier according to the call of the test sample startup module, receive a log storage path specified by a user, perform log management on the log content written in the path, and perform log analysis according to the log content; It should be noted that the design of the log module is to achieve efficient and accurate log recording and management. The log module can generate a unified sample identifier based on the call of the test sample startup module, receive the log storage path specified by the user, manage the log content in a unified manner under the specified directory, and manage the log content written under the path to ensure the persistent storage of the log content. It can also call the tool for log analysis based on the log content stored in the log management. At the same time, the log module also has the function of adapting to different diagnostic scenarios based on the normalized sample information passed in. The log module supports multi-level log recording, allowing users to classify and manage logs according to their importance, which is convenient for subsequent log analysis and problem tracking. In addition, the log module can automatically identify and record log entries containing illegal values ​​and their corresponding host information, providing support for rapid location and resolution of problems. The log module can also summarize and manage the globally diagnosed bad nodes and their related information in each diagnostic scenario, and automatically summarize all the output information of the total log module to generate a global summary report.

[0024] The judging module is configured to generate a judging result based on the diagnostic result of each sample and preset threshold management information to obtain a cluster diagnosis report.

[0025] Specifically, the grading module can receive and manage the preset threshold management information such as the questionnaire, standard answers, thresholds and diagnostic methods required in each diagnostic scenario to ensure the standardization and consistency of the diagnostic process; at the same time, according to the log storage path, each sample in the cluster is diagnosed one by one to improve the pertinence and accuracy of the diagnosis; in addition, the grading module is also configured to manage the diagnostic results of each sample, identify the bad nodes diagnosed for each sample and the information related to the bad nodes, and automatically generate a diagnostic report for each sample containing the diagnostic results, problem description and the comparison output of the test value and the standard answer.

[0026] The output management module is configured to perform local result visualization output and / or local result archiving for the diagnostic report.

[0027] Specifically, the output management module is configured to output the diagnostic report through the log module for local result visualization and / or local result archiving, wherein local result archiving of the diagnostic report includes storing the diagnostic report in a local lightweight database sqlite3 for subsequent visualization analysis of historical execution information.

[0028] Although preferred embodiments of the present invention have been described, additional changes and modifications may be made to these embodiments by those skilled in the art once the basic inventive concepts are known. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention. Obviously, those skilled in the art may make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A cluster diagnostic tool for large-scale artificial intelligence computing services, characterized in that: include: The configuration file parsing module is configured to parse the configuration file to obtain sample configuration information; A test sample startup module is configured to receive the sample configuration information transmitted by the configuration file parsing module, complete the instruction construction and instruction distribution of the test sample executor and receive the diagnosis result of the sample; The test sample execution module is configured to read the sample configuration information, improve the sample startup configuration according to the sample startup information transmitted by the test sample startup module, start the sample test based on the execution script of the sample itself, obtain the diagnosis result of the sample, and send back the diagnosis result; A log module is configured to generate a sample unified identifier according to the call of the test sample startup module, receive a log storage path specified by a user, manage the log content written in the path, and perform log analysis according to the log content; The judging module is configured to generate a judging result based on the diagnostic result of each sample and preset threshold management information to obtain a cluster diagnosis report.

2. A cluster diagnostic tool for large-scale artificial intelligence computing services as claimed in claim 1, characterized in that: The system also includes an output management module configured to perform local result visualization output and / or local result archiving on the diagnostic report.

3. A cluster diagnostic tool for large-scale artificial intelligence computing services as claimed in claim 1, characterized in that: The sample configuration information includes cluster information and sample information.

4. A cluster diagnostic tool for large-scale artificial intelligence computing services as claimed in claim 1, characterized in that: The test sample startup module is also configured to be responsible for starting and writing the local storage module.

5. A cluster diagnostic tool for large-scale artificial intelligence computing services as claimed in claim 1, characterized in that: The returning the diagnosis result includes: returning the diagnosis result in a character nested manner.

6. A cluster diagnostic tool for large-scale artificial intelligence computing services as claimed in claim 1, characterized in that: The log module is also configured to automatically identify and record log entries containing illegal values ​​and their corresponding host information.

7. A cluster diagnostic tool for large-scale artificial intelligence computing services as claimed in claim 6, characterized in that: The log module is also configured to aggregate and manage bad nodes and information related to bad nodes that are globally diagnosed in each diagnosis scenario, and automatically aggregate all output information of the log module.

8. A cluster diagnostic tool for large-scale artificial intelligence computing services as claimed in claim 1, characterized in that: The preset threshold management information includes the questionnaires, standard answers, thresholds and diagnostic methods required in each diagnosis scenario.

9. A cluster diagnostic tool for large-scale artificial intelligence computing services as claimed in claim 8, characterized in that: The evaluation results are generated based on the diagnostic results of each sample and the preset threshold management information, and obtaining the cluster diagnostic report also includes: managing the diagnostic results of each sample, identifying the bad nodes diagnosed by each sample and the information related to the bad nodes, and automatically generating a diagnostic report for each sample.

10. A cluster diagnostic tool for large-scale artificial intelligence computing services as claimed in claim 2, characterized in that: The local result archiving of the diagnostic report includes storing the diagnostic report in a local lightweight database sqlite3.

Citation Information

Patent Citations

  • Log based computer system fault diagnosis method and device

    CN103761173A

  • Test and diagnosis integrated development platform based on domestic operating system

    CN111782539A

  • Distributed cluster fault test method and device

    CN111953560A

  • Distributed data monitoring device

    US20190138410A1

  • System and method for automated problem diagnosis

    US7379846B1