Server component quality inspection method, device and equipment and storage medium

Abstract

This invention provides a method, apparatus, device, and storage medium for quality inspection of server components. The method includes: performing fault injection operations on multiple hardware layers of a GPU accelerator card to be tested in an AI server to obtain fault response data for each hardware layer; monitoring the instructions of a test program running on the GPU accelerator card based on the fault response data of each hardware layer to obtain instruction execution anomaly data; performing application-layer verification on the output results of the test program based on the instruction execution anomaly data to obtain output error classification data; and calculating a cross-layer vulnerability assessment index for the GPU accelerator card based on the fault response data, instruction execution anomaly data, and output error classification data to obtain a quality inspection assessment result. This invention accurately assesses the cross-layer propagation impact of faults through correlation analysis across hardware, instruction, and application layers, solving the problem of inaccurate assessment caused by existing methods that only detect at a single layer.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a method, apparatus, device, and storage medium for quality inspection of server components. Background Technology

[0002] With the rapid development of artificial intelligence technology, GPU accelerator cards in AI servers have become core computing components for deep learning training and inference tasks. The quality of GPU accelerator cards directly affects the reliability and computational accuracy of AI systems, therefore, rigorous quality testing is required before leaving the factory or before deployment.

[0003] Existing GPU accelerator card quality inspection methods typically perform fault testing at only a single level, such as electrical performance testing only at the hardware level or functional verification only at the application level. This single-level inspection method cannot comprehensively assess the propagation and impact of faults between the hardware, instruction, and application layers. As a result, although some hardware faults are detected at the hardware level, their actual impact on the final application output cannot be accurately assessed, leading to misjudgments or omissions, and affecting the accuracy and effectiveness of quality inspection. Summary of the Invention

[0004] The main objective of this invention is to solve the technical problem that existing GPU accelerator card quality inspection methods only perform inspections at a single level and cannot accurately assess the impact of fault propagation across levels; This invention provides a server component quality inspection method, wherein the adaptive control method includes: Fault injection operations are performed on multiple hardware levels of the GPU accelerator card to be tested in the AI ​​server to obtain fault response data for each hardware level. Based on the fault response data of each hardware level, the test program running on the GPU accelerator card is monitored for instructions to obtain instruction execution anomaly data. Execute the abnormal data according to the instructions, perform application-layer verification on the output of the test program, and obtain output error classification data. Based on the fault response data, the instruction execution anomaly data, and the output error classification data, the cross-level vulnerability assessment index of the GPU accelerator card is calculated to obtain the quality inspection assessment result.

[0005] The present invention also provides a server component quality inspection device, the server component quality inspection device comprising: The fault injection module is used to perform fault injection operations on multiple hardware levels of the GPU accelerator card to be tested in the AI ​​server, and obtain fault response data for each hardware level. The instruction monitoring module is used to monitor the test program running on the GPU accelerator card based on the fault response data of each hardware level, and obtain instruction execution anomaly data. The application verification module is used to execute abnormal data according to the instructions, perform application-layer verification on the output of the test program, and obtain output error classification data. The evaluation calculation module is used to calculate the cross-level vulnerability assessment index of the GPU accelerator card based on the fault response data, the instruction execution anomaly data and the output error classification data, and obtain the quality inspection assessment result.

[0006] The present invention also provides a server component quality inspection device, comprising: a memory and at least one processor, wherein the memory stores instructions, and the memory and the at least one processor are interconnected via a line; the at least one processor invokes the instructions in the memory to cause the server component quality inspection device to perform the steps of the server component quality inspection method described above.

[0007] The present invention also provides a computer-readable storage medium storing instructions that, when executed on a computer, cause the computer to perform the steps of the server component quality inspection method described above.

[0008] The aforementioned server component quality inspection method, apparatus, equipment, and storage medium perform fault injection operations on multiple hardware layers of the GPU accelerator card to be tested in the AI ​​server to obtain fault response data for each hardware layer. Based on the fault response data of each hardware layer, the test program running on the GPU accelerator card is monitored to obtain instruction execution anomaly data. Based on the instruction execution anomaly data, the output results of the test program are verified at the application layer to obtain output error classification data. Based on the fault response data, instruction execution anomaly data, and output error classification data, the cross-layer vulnerability assessment index of the GPU accelerator card is calculated to obtain the quality inspection assessment result. This invention accurately assesses the cross-layer propagation impact of faults through correlation analysis across the hardware layer, instruction layer, and application layer, solving the problem of inaccurate assessment caused by existing methods that only detect at a single layer.

[0009] Other features and advantages of the invention will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention are realized and obtained in accordance with the structures particularly pointed out in the description, claims and drawings.

[0010] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0011] Figure 1 This is a schematic diagram of the first embodiment of the server component quality inspection method in this invention; Figure 2 This is a schematic diagram of a second embodiment of the server component quality inspection method in this invention; Figure 3 This is a schematic diagram of one embodiment of the server component quality inspection device in this invention; Figure 4 This is a schematic diagram of one embodiment of the server component quality inspection equipment in this invention. Detailed Implementation

[0012] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0013] The terms "comprising" and "having," and any variations thereof, used in the embodiments of this invention are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the steps or units listed, but may optionally include other steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or devices.

[0014] To facilitate understanding of this embodiment, a server component quality inspection method disclosed in this embodiment of the invention will first be described in detail. For example... Figure 1 As shown, this method includes the following steps: 101. Perform fault injection operations on multiple hardware levels of the GPU accelerator card to be tested in the AI ​​server to obtain fault response data for each hardware level. In this embodiment, 102. Based on the fault response data of each hardware level, perform instruction monitoring on the test program running on the GPU accelerator card to obtain instruction execution anomaly data; In this embodiment, 103. Execute the abnormal data according to the instructions, perform application-layer verification on the output of the test program, and obtain output error classification data; In this embodiment, Specifically, 104. Based on the fault response data, the instruction execution anomaly data, and the output error classification data, calculate the cross-level vulnerability assessment index of the GPU accelerator card to obtain the quality inspection assessment result.

[0015] In this embodiment, In this embodiment, fault injection operations are performed on multiple hardware layers of the GPU accelerator card to be tested in the AI ​​server to obtain fault response data for each hardware layer. Based on the fault response data of each hardware layer, instruction monitoring is performed on the test program running on the GPU accelerator card to obtain instruction execution anomaly data. Based on the instruction execution anomaly data, application-layer verification is performed on the output results of the test program to obtain output error classification data. Based on the fault response data, instruction execution anomaly data, and output error classification data, the cross-layer vulnerability assessment index of the GPU accelerator card is calculated to obtain the quality inspection assessment result. This invention accurately assesses the cross-layer propagation impact of faults through correlation analysis across the hardware layer, instruction layer, and application layer, solving the problem of inaccurate assessment caused by existing methods that only detect at a single layer.

[0016] Please see Figure 2 Another embodiment of the server component quality inspection method in this application includes: 201. Perform fault injection operations on multiple hardware levels of the GPU accelerator card to be tested in the AI ​​server to obtain fault response data for each hardware level. In this embodiment, step 201 is similar to step 101, and will not be described again here.

[0017] 202. Based on the fault response data of each hardware level, perform instruction monitoring on the test program running on the GPU accelerator card to obtain instruction execution anomaly data; 203. Execute the abnormal data according to the instructions, perform application-layer verification on the output of the test program, and obtain output error classification data; In this embodiment, 204. Based on the fault response data, count the total number of faults injected and the number of faults that generate responses in each of the multiple hardware layers, and calculate the hardware layer vulnerability coefficient for each hardware layer. In this embodiment, 205. Based on the instruction execution exception data and the fault response data, count the number of faults that caused instruction execution exceptions among the faults that generated responses, and calculate the instruction layer vulnerability coefficient. In this embodiment, 206. Based on the output error classification data and the fault response data, count the number of faults that cause output errors, and calculate the application layer vulnerability coefficient according to the error type. 207. Perform cross-level correlation analysis on the hardware layer vulnerability coefficient, the instruction layer vulnerability coefficient, and the application layer vulnerability coefficient, calculate the propagation rate of hardware layer failure to instruction layer and the propagation rate of instruction layer anomaly to application layer, and obtain cross-level vulnerability assessment index. 208. Compare the cross-level vulnerability assessment index with the preset qualified threshold to obtain the quality inspection assessment result.

[0018] In this embodiment, fault injection operations are performed on multiple hardware layers of the GPU accelerator card to be tested in the AI ​​server to obtain fault response data for each hardware layer. Based on the fault response data of each hardware layer, instruction monitoring is performed on the test program running on the GPU accelerator card to obtain instruction execution anomaly data. Based on the instruction execution anomaly data, application-layer verification is performed on the output results of the test program to obtain output error classification data. Based on the fault response data, instruction execution anomaly data, and output error classification data, the cross-layer vulnerability assessment index of the GPU accelerator card is calculated to obtain the quality inspection assessment result. This invention accurately assesses the cross-layer propagation impact of faults through correlation analysis across the hardware layer, instruction layer, and application layer, solving the problem of inaccurate assessment caused by existing methods that only detect at a single layer.

[0019] The server component quality inspection method in the embodiments of the present invention has been described above. The server component quality inspection device in the embodiments of the present invention is described below. Please refer to [link to relevant documentation] for details. Figure 3 One embodiment of the server component quality inspection device in this invention includes: The fault injection module 301 is used to perform fault injection operations on multiple hardware levels of the GPU accelerator card to be tested in the AI ​​server to obtain fault response data for each hardware level. The instruction monitoring module 302 is used to monitor the test program running on the GPU accelerator card based on the fault response data of each hardware level, and obtain instruction execution abnormal data. Application verification module 303 is used to execute abnormal data according to the instructions, perform application-layer verification on the output results of the test program, and obtain output error classification data; The evaluation calculation module 304 is used to calculate the cross-level vulnerability assessment index of the GPU accelerator card based on the fault response data, the instruction execution anomaly data and the output error classification data, and obtain the quality inspection assessment result.

[0020] In this embodiment of the invention, the server component quality inspection device runs the aforementioned server component quality inspection method. The device performs fault injection operations on multiple hardware layers of the GPU accelerator card to be tested in the AI ​​server, obtaining fault response data for each hardware layer. Based on the fault response data for each hardware layer, it monitors the instructions running on the GPU accelerator card to obtain instruction execution anomaly data. Based on the instruction execution anomaly data, it performs application-layer verification on the output results of the test program to obtain output error classification data. Based on the fault response data, instruction execution anomaly data, and output error classification data, it calculates the cross-layer vulnerability assessment index of the GPU accelerator card to obtain the quality inspection assessment result. This invention accurately assesses the cross-layer propagation impact of faults through correlation analysis across the hardware layer, instruction layer, and application layer, solving the problem of inaccurate assessment caused by existing methods that only detect at a single layer.

[0021] above Figure 3 The server component quality inspection device in this embodiment of the invention will be described in detail from the perspective of unitized functional entities. The server component quality inspection equipment in this embodiment of the invention will be described in detail from the perspective of hardware processing.

[0022] Figure 4 This is a schematic diagram of a server component quality inspection device 400 provided in an embodiment of the present invention. The server component quality inspection device 400 can vary significantly due to different configurations or performance. It may include one or more central processing units (CPUs) 410 (e.g., one or more processors) and a memory 420, and one or more storage media 430 (e.g., one or more mass storage devices) for storing application programs 433 or data 432. The memory 420 and storage media 430 can be temporary or persistent storage. The program stored in the storage media 430 may include one or more units (not shown in the diagram), each unit may include a series of instruction operations on the server component quality inspection device 400. Furthermore, the processor 410 may be configured to communicate with the storage media 430 and execute the series of instruction operations in the storage media 430 on the server component quality inspection device 400 to implement the steps of the above-described server component quality inspection method.

[0023] Server component quality inspection equipment 400 may also include one or more power supplies 440, one or more wired or wireless network interfaces 450, one or more input / output interfaces 460, and / or one or more operating systems 431, such as Windows Server, Mac OS X, Unix, Linux, FreeBSD, etc. Those skilled in the art will understand that... Figure 4The server component quality inspection equipment structure shown does not constitute a limitation on the server component quality inspection equipment provided by the present invention. It may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0024] The present invention also provides a computer-readable storage medium, which may be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium, wherein the computer-readable storage medium stores instructions that, when executed on a computer, cause the computer to perform the steps of the server component quality inspection method.

[0025] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the system, device, or unit described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0026] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0027] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A server component quality inspection method, characterized in that, The server component quality inspection method includes: Fault injection operations are performed on multiple hardware levels of the GPU accelerator card to be tested in the AI ​​server to obtain fault response data for each hardware level. Based on the fault response data of each hardware level, the test program running on the GPU accelerator card is monitored for instructions to obtain instruction execution anomaly data. Execute the abnormal data according to the instructions, perform application-layer verification on the output of the test program, and obtain output error classification data. Based on the fault response data, the instruction execution anomaly data, and the output error classification data, the cross-level vulnerability assessment index of the GPU accelerator card is calculated to obtain the quality inspection assessment result.

2. The method of claim 1, wherein, The fault injection operation is performed on multiple hardware levels of the GPU accelerator card to be tested in the AI ​​server, and the fault response data of each hardware level is obtained, including: Obtain the hardware architecture information of the GPU accelerator card, determine the storage structure location information of each level in the multiple hardware levels based on the hardware architecture information, and obtain a level location mapping table; Based on the hierarchical location mapping table, multiple fault injection target locations are determined for each of the multiple hardware levels. For each hardware level fault injection target location, a bit flipping operation is performed, and after the bit flipping operation is performed, the hardware state change information of the GPU accelerator card is recorded to obtain local fault response data for each target location. The local fault response data of each target location at each hardware level are classified according to the level to obtain the fault response data of each hardware level.

3. The server component quality inspection method according to claim 1, characterized in that, The step of monitoring the instruction execution of the test program running on the GPU accelerator card based on the fault response data of each hardware level, and obtaining instruction execution anomaly data, includes: Based on the fault response data of each hardware level, the location of the hardware affected by the fault is determined, and the fault-affected area is obtained. During the execution of the test program, the instructions that access the affected area of ​​the fault are traced to obtain the set of affected instructions; The execution status of the instructions in the affected instruction set is monitored to identify the instructions that are abnormal and obtain instruction execution abnormality data.

4. The server component quality inspection method according to claim 1, characterized in that, The step of executing abnormal data according to the instructions and performing application-layer verification on the output of the test program to obtain output error classification data includes: Run the test program under fault-free injection conditions and obtain the baseline output results of the test program; Run the test program under fault injection conditions and obtain the actual output results of the test program; The benchmark output result and the actual output result are compared. Based on the comparison result and the termination status of the test program, the error type of the output result is determined. The error type includes silent error type, detectable error type and normal type. Based on the execution of the instruction exception data, determine the subset of exception instructions that cause the error type, establish the association mapping relationship between the exception instructions and the output error type, and obtain the output error classification data.

5. The server component quality inspection method according to claim 1, characterized in that, The step of calculating the cross-level vulnerability assessment index of the GPU accelerator card based on the fault response data, the instruction execution anomaly data, and the output error classification data, and obtaining the quality inspection assessment results, includes: Based on the fault response data, the total number of faults injected and the number of faults that generated responses in each of the multiple hardware layers are counted, and the hardware layer vulnerability coefficient of each hardware layer is calculated. Based on the instruction execution exception data and the fault response data, count the number of faults that caused instruction execution exceptions among the faults that generated responses, and calculate the instruction layer vulnerability coefficient. Based on the output error classification data and the fault response data, the number of faults that caused the output errors was counted, and the application layer vulnerability coefficient was calculated according to the error type. Cross-level correlation analysis is performed on the vulnerability coefficients of the hardware layer, the instruction layer, and the application layer to calculate the propagation rate of hardware layer faults to the instruction layer and the propagation rate of instruction layer anomalies to the application layer, thereby obtaining cross-level vulnerability assessment indicators. The cross-level vulnerability assessment index is compared with a preset pass threshold to obtain the quality inspection assessment result.

6. The server component quality inspection method according to claim 5, characterized in that, The cross-layer correlation analysis of the hardware layer vulnerability coefficient, the instruction layer vulnerability coefficient, and the application layer vulnerability coefficient is performed to calculate the propagation rate of hardware layer faults to the instruction layer and the propagation rate of instruction layer anomalies to the application layer, resulting in cross-layer vulnerability assessment indicators, including: Based on the fault response data, the instruction execution anomaly data, and the output error classification data, the fault masking situation between levels is identified, and the masking data of each level is obtained. Based on the masking data of each layer, calculate the propagation rate of hardware layer faults to the instruction layer and the propagation rate of instruction layer anomalies to the application layer; The hardware layer vulnerability coefficient, the instruction layer vulnerability coefficient, the application layer vulnerability coefficient, and the propagation rate are used as cross-layer vulnerability assessment indicators.

7. The server component quality inspection method according to claim 1, characterized in that, Before performing fault injection operations on multiple hardware levels of the GPU accelerator card to be tested in the AI ​​server, the method further includes: Static analysis is performed on the test program to extract its parallelism, memory access, and computational intensity characteristics, thereby obtaining workload characteristic parameters. Based on the workload characteristic parameters, the fault sensitivity of the test program at each hardware level is predicted to obtain the level sensitivity prediction value. Based on the predicted sensitivity values ​​of the different hardware levels, the fault injection density of each level is adjusted, and the number of fault injections is increased for levels with high predicted sensitivity.

8. A server component quality inspection device, characterized in that, The server component quality inspection device includes: The fault injection module is used to perform fault injection operations on multiple hardware levels of the GPU accelerator card to be tested in the AI ​​server, and obtain fault response data for each hardware level. The instruction monitoring module is used to monitor the test program running on the GPU accelerator card based on the fault response data of each hardware level, and obtain instruction execution anomaly data. The application verification module is used to execute abnormal data according to the instructions, perform application-layer verification on the output of the test program, and obtain output error classification data. The evaluation calculation module is used to calculate the cross-level vulnerability assessment index of the GPU accelerator card based on the fault response data, the instruction execution anomaly data and the output error classification data, and obtain the quality inspection assessment result.

9. A server component quality inspection device, characterized in that, The server component quality inspection equipment includes: a memory and at least one processor, wherein the memory stores instructions; The at least one processor invokes the instructions in the memory to cause the server component quality inspection device to perform the steps of the server component quality inspection method as described in any one of claims 1-7.

10. A computer-readable storage medium storing instructions thereon, characterized in that, When the instruction is executed by the processor, it implements the steps of the server component quality inspection method as described in any one of claims 1-7.

Images