Methods, systems, and computer program products for diagnosing hardware status of a diagnostic accelerator card

Abstract

The present application discloses a method, system and computer program product for diagnosing hardware status of an accelerator card. The method comprises: collecting runtime physical feature data of a tested accelerator card during execution of a test case by a diagnosis thread independent of a test framework; analyzing volatility of the runtime physical feature data to generate a hardware health index associated with the test case; in response to an execution event of the test case issued through a life cycle interface provided by the test framework, obtaining the hardware health index and associating the hardware health index with an identification of the test case; and generating a diagnosis report based on a business execution result of the test case and the associated hardware health index. Such a method realizes early warning of hardware "sub-health" status and can provide clear software / hardware problem direction when a test fails.

Description

Technical Field

[0001] This application generally relates to the field of artificial intelligence, and more specifically to a method, system, and computer program product for diagnosing the hardware status of an accelerator card. Background Technology

[0002] With the widespread application of artificial intelligence (AI) computing, AI accelerator cards have become core hardware in data centers and high-performance computing. Currently, the industry commonly uses automated testing frameworks (such as pytest and shellspec) to verify the functionality and performance of AI accelerator cards.

[0003] However, existing testing frameworks can typically only verify software logic (e.g., only record whether business logic "passes" or "fails"), but cannot monitor the hardware runtime state, which makes it difficult to miss potential hardware defects and trace the root cause of failures. Summary of the Invention

[0004] To address the aforementioned problems in the prior art, this application proposes a method for diagnosing the hardware status of an accelerator card in automated testing, as well as a system, computer program product, and non-transitory computer-readable medium associated with the method.

[0005] According to a first aspect of this application, a method for diagnosing the hardware status of an accelerator card in automated testing is provided. The method includes: collecting runtime physical characteristic data of the accelerator card under test during the execution of test cases via a diagnostic thread independent of the testing framework; analyzing the volatility of the runtime physical characteristic data to generate hardware health indicators associated with the test cases; obtaining the hardware health indicators in response to execution events related to the test cases emitted through lifecycle interfaces provided by the testing framework, and associating the hardware health indicators with the identifiers of the test cases; and generating a diagnostic report based on the business execution results of the test cases and the associated hardware health indicators.

[0006] According to a second aspect of this application, a system for diagnosing the hardware status of an accelerator card during automated testing is provided. The system includes a processor and a memory, the memory storing instructions that, when executed by the processor, perform the methods described in this application.

[0007] In other aspects of this application, a non-transitory computer-readable medium storing instructions and a computer program product including the instructions are provided. These instructions, when executed by one or more processors, cause the processors to perform the methods described in this application. Attached Figure Description

[0008] The operation and function of these and other features disclosed herein, as well as related structural elements and combinations thereof, will become more apparent upon consideration of the following description and appended claims with reference to the accompanying drawings (all of which form a part of this specification, wherein like reference numerals denote corresponding portions in the drawings). However, it should be clearly understood that the drawings are for illustrative and descriptive purposes only and not for limiting purposes. In the drawings:

[0009] Figure 1 A flowchart illustrating a method for diagnosing the hardware status of an accelerator card in automated testing, according to an embodiment of this application, is shown.

[0010] Figure 2 An example computing device is shown that can implement the methods described in this application according to embodiments of the present disclosure. Detailed Implementation

[0011] The following disclosure provides numerous different embodiments or examples for implementing various features of the provided subject matter. Specific examples of components and arrangements are described below to simplify this application. Of course, these are merely examples and not limiting.

[0012] Currently, AI accelerator card testing based on automated testing frameworks (such as pytest and shellspec) is limited to verifying only the functional logic at the software level. Throughout the entire test execution cycle, the system lacks the ability to monitor the physical state of the accelerator card during operation (such as core temperature, voltage fluctuations, and computing frequency). This disconnect between "verifying logic and not checking the state" leads to several drawbacks: Firstly, the hardware may be in an unstable or "sub-healthy" state during testing (e.g., exhibiting abnormal fluctuations or intermittent frequency reduction), but because it does not immediately trigger functional errors, these hidden dangers are completely masked in the test report, posing a risk to long-term operational stability. Secondly, when tests fail, because there is no hardware state data precisely aligned with the test case execution process available for traceability, engineers find it difficult to quickly determine whether the root cause of the failure is a software defect or a hardware / environmental issue, making the troubleshooting process inefficient and inefficient.

[0013] To address this issue, this application proposes an automated testing method that deeply integrates hardware status diagnosis. This method initiates an independent, parallel diagnostic thread during test case execution within the testing framework. This enables the synchronous, non-intrusive acquisition of physical characteristic data of the accelerator card during runtime. Based on the temporal variations of this data, volatility analysis is performed to generate hardware health indicators reflecting hardware stability. Furthermore, to establish accurate fault tracing correlations, this method utilizes the testing framework's own lifecycle interface to proactively acquire hardware health indicators generated within a specific timeframe (such as report generation) and associate them with the current test case's identifier when key execution events of the test case are triggered. Thus, a unified diagnostic report can be generated by comprehensively analyzing the business execution results of the test cases and the hardware health indicators.

[0014] This automated testing method changes the traditional testing model that only verifies software logic, enables early warning of hardware "sub-health" states, and can provide clear software / hardware problem indications when tests fail, thus completing the leap from "functional verification" to "health diagnosis".

[0015] Figure 1 A flowchart of a method 100 for diagnosing the hardware status of an accelerator card in automated testing, according to an embodiment of this application, is shown. The method 100 is executed during the execution of test cases within an automated testing framework and specifically includes the following operations S110-S140.

[0016] Operation S110: Data Acquisition

[0017] When operating S110, the system can launch a diagnostic thread independent of the main execution flow of the automated testing framework (such as pytest, shellspec, etc.) to collect runtime physical characteristic data of the accelerator card under test throughout the entire lifecycle of executing the current test case in a "non-intrusive" manner. In this article, "non-intrusive" means that the execution of the diagnostic thread and the data collection activities will not block, interrupt, or significantly delay the normal execution of the test framework's business logic for the test cases, thereby ensuring that the performance and results of the original test are not affected.

[0018] In some embodiments, the diagnostic thread is configured to run synchronously with the execution of test cases by the automated testing framework, thereby completing data collection within the same time period of test execution (e.g., within the execution lifecycle of pytest or shellspec), ensuring that the acquired data is aligned with the test activities on the timeline.

[0019] In some embodiments, the collected runtime physical characteristic data can be a comprehensive characterization signal reflecting the working state of the accelerator card, including but not limited to one or more of the following: computing core (such as neural network computing core (NNCore)) utilization, memory utilization, core operating frequency, operating voltage, operating current, chip temperature, and cooling fan speed.

[0020] Operation S120: Volatility Analysis

[0021] After acquiring the timing-based physical characteristic data, the system can perform intelligent diagnostic analysis. In operation S120, the volatility of the runtime physical characteristic data collected in operation S110 is analyzed to generate hardware health indicators associated with test cases. For example, abnormal patterns that do not cause immediate functional errors but suggest hardware "suboptimal health" or environmental risks can be identified.

[0022] Specifically, in some embodiments, volatility analysis may include: calculating the volatility ratio of runtime physical characteristic data during test case execution; and comparing this volatility ratio with a predefined "expected envelope" for the runtime physical characteristics. In some embodiments, the expected envelope is a dynamic range based on historical normal operation data statistics, which may be defined by the statistical mean and standard deviation (σ), describing the expected performance boundaries of the hardware in a healthy state.

[0023] Furthermore, the aforementioned volatility analysis may also include generating a hardware health index based on the comparison results. In the embodiments of this application, the hardware health index is not a simple numerical value, but a diagnostic identifier with clear semantics, such as: a "hardware healthy" identifier indicating that the hardware is in a completely normal state, a "hardware sub-healthy" identifier indicating that there is a potential stability risk, or a "hardware failed" identifier indicating that a clear failure has occurred.

[0024] In one example, the system monitors the core voltage of the accelerator card. Assume its historical average voltage during healthy operation is 0.8V, and the preset envelope boundary is ±3σ (σ=0.01V). In one test, although the voltage did not fall below the absolute lower limit of 0.77V, it experienced several rapid drops and recoveries of 0.05V within a short period (abnormal fluctuation ratio). Because this fluctuation exceeded the stable range of 3σ, the system determined it to be in a "sub-healthy" state and generated a hardware health index of "abnormal voltage fluctuation" in the diagnostic report (described below), even though all functional verifications for this test passed. This diagnosis provides an early warning to prevent potential hardware failures or calculation errors caused by voltage instability in the future.

[0025] Operation S130: Event Triggering and Association

[0026] In operation S130, in response to a specific execution event concerning the current test case emitted through the lifecycle interface provided by the test framework, hardware health metrics are obtained. This operation S130 enables precise binding of hardware diagnostic results to the software testing context. In one embodiment, the lifecycle interface is embodied in a hook function of the test framework.

[0027] Taking the widely used pytest framework as an example, the system achieves precise event response and data association by registering and listening to the built-in hook function pytest_runtest_makereport. This hook function is designed so that whenever the testing framework completes the execution of a test case and is about to generate the final result report, the testing framework can automatically call all registered pytest_runtest_makereport hooks.

[0028] When this event is triggered, the hardware health metric generated in operation S120, corresponding to the current test case execution period, can be immediately retrieved, and this hardware health metric can be associated with the identifier of the test case (such as Test CaseID). Thus, the system achieves automated and accurate association between test results and hardware status information, providing crucial data support for subsequent comprehensive evaluation and fault tracing.

[0029] Operation S140: Generate a diagnostic report

[0030] In operation S140, a comprehensive diagnosis is performed based on the business execution results of the test cases (such as Pass or Fail) and the hardware health indicators already associated with the test cases in operation S130, in order to generate an enhanced diagnostic report.

[0031] In the embodiments of this application, the diagnostic report not only includes traditional business execution results and hardware health indicators, but more importantly, it can also include diagnostic conclusions based on the fusion analysis of these two. For example, the report generation logic can output conclusions with direct guiding significance, such as "perfectly passed," "logic passed but hardware is suboptimal," and "logic failed and hardware is malfunctioning," along with corresponding conclusion analysis and prompts, according to defined rules.

[0032] Table 1 below provides example diagnostic reports according to embodiments of this application.

[0033]

[0034] Table 1

[0035] By applying the rules described in Table 1 above, the diagnostic report described in this paper can transform a single "pass / fail" result into a multidimensional diagnostic conclusion with direct operational guidance.

[0036] Furthermore, in some embodiments, the diagnostic report may also include a visualization of the changes in physical characteristic data corresponding to the test cases over time, allowing any fluctuations or anomalies to be traced intuitively. For example, the changes in key physical characteristics (such as temperature, voltage, and utilization) over time during test execution can be displayed as a graph, with the times when abnormal fluctuations occur highlighted on the graph.

[0037] In some embodiments, the diagnostic report may also include a detailed analysis of abnormal parameters. For example, for indicators marked as “warning” or “failure”, the diagnostic report may further list specific abnormal parameters, such as “voltage fluctuations exceed the expected envelope 3σ” or “chip temperature remains above the threshold during t1 to t2”.

[0038] By executing operations S110 to S140 above, the method for diagnosing the hardware status of accelerator cards described in this paper can integrate data acquisition, volatility analysis, event triggering and correlation, and diagnostic report generation. This not only reveals potential "sub-health" issues in the hardware when tests pass, enabling proactive early warning, but also quickly clarifies whether the problem stems from software defects or hardware / environmental failures when tests fail, greatly improving the depth of automated testing and the efficiency of operational troubleshooting.

[0039] The following is a pseudocode example based on the example implementation of the pytest_runtest_makereport hook function as described in this article.

[0040] # Pseudocode: Implementation of the pytest_runtest_makereport hook function

[0041] # Function: When generating a test report, associate hardware health metrics with test cases and inject them into the report.

[0042] # Import necessary modules (example)

[0043] import pytest

[0044] from diagnostic_system import HardwareDiagnosticCache # Hypothetical Diagnostic System Unified caching class

[0045] # Global or configuration-level diagnostic system cache instance

[0046] diagnostic_cache = HardwareDiagnosticCache()

[0047] def pytest_runtest_makereport(item, call):

[0048] """

[0049] The pytest hook function is called when a report is generated for each test case.

[0050] parameter:

[0051] item: Test case object, containing metadata such as unique identifiers (e.g., nodeid).

[0052] call: The calling object, which describes the test execution phase (setup, call, teardown).

[0053] """

[0054] # 1. Only process at the end of the test execution phase (call) to avoid overlapping setup / teardown phases. Reprocessing

[0055] if call.when != "call":

[0056] return None # Do not process non-execution phases

[0057] # 2. Extract the unique identifier of the test case from the item (e.g., test_module.py::test_ function)

[0058] test_case_id = item.nodeid

[0059] # 3. Retrieve the hardware health metrics corresponding to this test case from the diagnostic system cache.

[0060] # Assume the cache is indexed by test_case_id and stores metrics generated by the diagnostic thread.

[0061] hardware_health_metrics = diagnostic_cache.get_metrics(test_case_id)

[0062] # 4. If hardware health metrics exist, inject them into the test report object.

[0063] if hardware_health_metrics is not None:

[0064] # Retrieves the report object for the current test case (managed by the pytest framework)

[0065] report = item.

[0066] # Method A: Add the metric as a custom attribute to the report object

[0067] report.hardware_health = hardware_health_metrics

[0068] # Method B: Or use pytest's built-in list of user attributes (more standard)

[0069] report.user_properties.append(("hardware_health", hardware_health_metrics))

[0070] # 5. (Optional) Generate diagnostic conclusions based on business logic results and hardware metrics.

[0071] # Assume that the hardware health metrics include status tags and detailed data.

[0072] if report.outcome == "passed":

[0073] if hardware_health_metrics.status == "WARNING":

[0074] report.diagnosis = "Logic passes, but hardware is in poor condition"

[0075] elif hardware_health_metrics.status == "FAILURE":

[0076] report.diagnosis = "Logic passed but hardware failed"

[0077] else:

[0078] report.diagnosis = "Passed perfectly"

[0079] else: # Test failed

[0080] if hardware_health_metrics.status in ["WARNING", "FAILURE"]:

[0081] report.diagnosis = "Logical failure and hardware malfunction"

[0082] else:

[0083] report.diagnosis = "Logical failure but hardware is normal"

[0084] # 6. Log the action or trigger other actions (e.g., send an alert).

[0085] log_diagnostic_event(test_case_id, report.outcome, hardware_health_metrics)

[0086] # 7. Return the report object (for the pytest framework to continue processing).

[0087] return report

[0088] # Helper function: Record diagnostic events

[0089] def log_diagnostic_event(test_id, test_outcome, health_metrics):

[0090] Record diagnostic results to a log or diagnostic system.

[0091] print(f"[Diagnostic Log] Test Case: {test_id}")

[0092] print(f"Test result: {test_outcome}")

[0093] print(f"Hardware health: {health_metrics.status}")

[0094] print(f"Detailed metrics: {health_metrics.details}")

[0095] # Example hardware health metric data structure (simple representation using classes)

[0096] class HardwareHealthMetrics:

[0097] def __init__(self, status, details):

[0098] self.status = status # For example: "NORMAL", "WARNING", "FAILURE"

[0099] self.details = details # Dictionary, containing detailed time-series data such as temperature and voltage, and... Analysis results

[0100] Based on the pseudocode above, the workflow of the pytest_runtest_makereport hook function can be specifically described as follows:

[0101] 1. Trigger and Condition Filtering

[0102] Once the testing framework completes the core execution phase of a test case, the `pytest_runtest_makereport` hook function is automatically triggered. This function first determines whether the current phase is the "execution" phase of the test (`call.when == "call")) to ensure that processing only occurs after the test logic has finished running and the result has been determined, avoiding duplicate operations during the initialization or cleanup phases.

[0103] 2. Context Extraction

[0104] The hook function extracts a unique identifier, such as item.nodeid, from the passed-in test case object (e.g., test_module.py::test_function). This identifier is used to associate hardware data with the test case.

[0105] 3. Diagnostic data retrieval

[0106] The system uses the extracted test case identifiers to query the background diagnostic cache system. This cache is pre-populated by an independently running thread and stores hardware health indicators (such as status labels, detailed timing analysis data, etc.) corresponding to each test case during execution.

[0107] 4. Data Binding and Injection

[0108] If the corresponding hardware health metric is retrieved, the hook function dynamically injects it as a custom attribute (e.g., stored in the `report.user_properties` or `report.hardware_health` field) into the report object generated by the testing framework. This step completes the association between the hardware metric and the test report at the memory level.

[0109] 5. Intelligent diagnostic decision-making (fusion analysis)

[0110] Based on the injected data, the hook function integrates business logic results with hardware health status and executes pre-defined decision logic. For example, when the test "PASS" but the hardware status is "WARNING," a diagnostic conclusion of "logic passed but hardware is in poor health" is generated and attached to the report. This achieves an upgrade from simple result reporting to root cause analysis and risk warning.

[0111] 6. Log recording and process continuity

[0112] After all operations are completed, the hook function returns the modified report object, and the test framework continues its original process, ultimately generating an enhanced report that integrates software results and hardware diagnostics.

[0113] Figure 2 An example computing device 200 that can implement the method 100 described in this application according to an embodiment of this disclosure is shown. For example... Figure 2 As shown, computing device 200 may include bus 202 or other communication mechanism for transmitting information, and one or more hardware processors 204 coupled to bus 202 for processing information. The one or more hardware processors 204 may include, for example, one or more general-purpose microprocessors.

[0114] like Figure 2 As shown, in some embodiments, computing device 200 may further include main memory 206 coupled to bus 202. Main memory 206 is used to store information and instructions executed by one or more processors 204, such as random access memory (RAM), cache, and / or other dynamic storage devices. Main memory 206 may also be used to store temporary variables or other intermediate information during the execution of instructions executed by one or more processors 204. When these instructions are stored in storage media accessible to one or more processors 204, they can cause computing device 200 to become a dedicated machine customized to perform the operations specified in the instructions. Storage device 208 may include non-volatile and / or volatile storage media. Non-volatile storage media may include, for example, optical discs or magnetic disks. Volatile storage media may include dynamic memory. Common forms of storage media may include, for example, floppy disks, hard disks, solid-state drives, magnetic tape, or any other magnetic data storage media, CD-ROMs, any other optical data storage media, any physical media with a perforated pattern, RAM, DRAM, PROM, and EPROM, FLASH-EPROM, NVRAM, any other memory chip or cartridge, or their networking versions.

[0115] like Figure 2As shown, in some embodiments, computing device 200 may further include one or more communication interfaces or network interfaces 210 coupled to bus 202. Network interface 210 may provide bidirectional data communication coupling to one or more network links connected to one or more networks. As another example, network interface 210 may be a local area network (LAN) card to provide data communication connectivity to a LAN-compatible (or WAN component communicating with a WAN) network. Wireless links may also be implemented.

[0116] The execution of certain operations can be distributed across processors rather than residing within a single machine, but rather deployed across multiple machines. In some example embodiments, the processor or processor-implemented engine may reside in a single geographic location (e.g., in a home environment, office environment, or server farm). In other example embodiments, the processor or processor-implemented engine may be distributed across multiple geographic locations.

[0117] Each of the processes, methods, and algorithms described in the preceding sections may be embodied in code modules executed by one or more computer systems or computer processors including computer hardware, and may be fully or partially automated by these code modules. The processes and algorithms may be implemented, partially or fully, in dedicated circuit systems.

[0118] When the functions disclosed herein are implemented as software functional units and sold or used as stand-alone products, they may be stored in a processor-executable, non-volatile, computer-readable storage medium. Specific technical solutions (all or part) disclosed herein, or aspects contributing to the prior art, may be embodied in the form of a software product. The software product may be stored in a storage medium and includes several instructions that cause a computing device (which may be a personal computer, server, network device, etc.) to perform all or some steps of the methods of the embodiments of this application. The storage medium may include a flash drive, portable hard disk drive, ROM, RAM, magnetic disk, optical disk, other media operable to store program code, or any combination thereof.

[0119] Specific embodiments further provide a system including a processor and a non-transitory computer-readable storage medium storing instructions executable by the processor to cause the system to perform operations corresponding to steps in any method of the embodiments disclosed above. Specific embodiments further provide a non-transitory computer-readable storage medium storing instructions executable by one or more processors to cause the one or more processors to perform operations corresponding to steps in any method of the embodiments disclosed above.

[0120] The embodiments disclosed herein can be implemented via a cloud platform, server, or server cluster (collectively referred to below as the "Service System") that interacts with a client. The client can be a terminal device or a client registered by a user at the platform, wherein the terminal device can be a mobile terminal, a personal computer (PC), or any device that can have the platform application installed.

[0121] The various features and processes described above can be used independently of each other or combined in various ways. All possible combinations and sub-combinations are intended to fall within the scope of this application. Additionally, certain method or process blocks may be omitted in some embodiments. The methods and processes described herein are not limited to any particular order, and their associated blocks or states may be executed in other suitable orders. For example, described blocks or states may be executed in an order other than that specifically disclosed, or multiple blocks or states may be combined into a single block or state. Example blocks or states may be executed sequentially, in parallel, or in some other manner. Blocks or states may be added to or removed from the disclosed example embodiments. The exemplary systems and components described herein may be configured differently than described. For example, components may be added to, removed from, or rearranged compared to the disclosed example embodiments.

[0122] The various operations of the exemplary methods described herein can be performed at least in part by an algorithm. The algorithm may be included in program code or instructions stored in memory (e.g., the aforementioned non-transitory computer-readable storage medium). This algorithm may include a machine learning algorithm. In some embodiments, the machine learning algorithm may not explicitly refer to the computer as performing the function but may learn from training data to generate a predictive model of the function.

[0123] The various operations of the exemplary methods described herein can be performed, at least in part, by one or more processors that are temporarily configured (e.g., by software) or permanently configured to perform the relevant operations. Whether temporarily or permanently configured, these processors can constitute an engine of processor implementation that operates to perform one or more of the operations or functions described herein.

[0124] Similarly, the methods described herein may be implemented at least in part by a processor, wherein one or more specific processors are instances of hardware. For example, at least some operations of the methods may be performed by one or more processors or an engine implemented by a processor. Furthermore, one or more processors may also be operable to support the execution of relevant operations in a “cloud computing” environment or as the execution of relevant operations in a “Software as a Service” (SaaS) context. For example, at least some operations may be performed by a group of computers (as an example of a machine containing processors), wherein these operations are accessible via a network (e.g., the Internet) and via one or more appropriate interfaces (e.g., application programming interfaces (APIs)).

[0125] The execution of certain operations can be distributed across processors rather than residing within a single machine, and can be deployed across multiple machines. In some example embodiments, the processor or processor-implemented engine may reside in a single geographic location (e.g., in a home environment, office environment, or server farm). In other example embodiments, the processor or processor-implemented engine may be distributed across multiple geographic locations.

[0126] Throughout this specification, multiple instances may be implemented as components, operations, or structures described as a single instance. Although individual operations of one or more methods are illustrated and described as separate operations, one or more of these individual operations may be performed simultaneously, and not necessarily in the order illustrated. Structures and functions presented as separate components in the example configuration may be implemented as composite structures or components. Similarly, structures and functions presented as single components may be implemented as single components. These and other variations, modifications, additions, and improvements fall within the scope of this document.

[0127] As used herein, "or" is inclusive rather than exclusive unless explicitly indicated by the context. Therefore, in this document, "A, B, or C" means "A, B, A and B, A and C, B and C, or A, B, and C" unless explicitly indicated by the context. Furthermore, "and" is combined and separate unless explicitly indicated by the context. Therefore, in this document, "A and B" means "A and B, combined or separate" unless explicitly indicated by the context. Additionally, multiple instances of resources, operations, or structures described herein may be provided as a single instance. Furthermore, the boundaries between various resources, operations, engines, and data storage devices are somewhat arbitrary and specific operations are illustrated within the context of a particular illustrative configuration. Other functional assignments are foreseeable and fall within the scope of various embodiments of this application. Generally, structures and functions presented as individual resources in example configurations may be implemented as combined structures or resources. Similarly, structures and functions presented as single resources may be implemented as single resources. These and other changes, modifications, additions, and improvements fall within the scope of the embodiments of this application as defined by the appended claims. Therefore, this specification and drawings should be considered illustrative rather than restrictive.

[0128] The terms “comprising” or “including” are used to indicate the presence of a subsequently claimed feature, but do not preclude the addition of other features. Unless otherwise specifically stated or otherwise understood in the context in which they are used, conditional language such as “may,” “can,” “may,” and “can” is generally intended to convey that certain embodiments include certain features, components, and / or steps that are not included in other embodiments. Therefore, this conditional language is generally not intended to imply that one or more embodiments require features, components, and / or steps in any way, or that one or more embodiments must include logic for determining whether such features, components, and / or steps are included in or performed in any particular embodiment, with or without user input or prompts.

[0129] Although the general outline of the subject matter has been described with reference to specific exemplary embodiments, various modifications and changes can be made to these embodiments without departing from the broad scope of embodiments of this application. Where more than one embodiment is disclosed, these embodiments of the subject matter may be referred to individually or collectively herein as the term "invention," this is for convenience only and is not intended to automatically limit the scope of this application to any single disclosure or concept.

[0130] The embodiments illustrated herein are described in detail to enable those skilled in the art to practice the disclosed teachings. Other embodiments may be used and derived therefrom, such that structural and logical substitutions and changes may be made without departing from the scope of this application. Therefore, "implementation" is not intended to be limiting, and the scope of the various embodiments is defined only by the appended claims and their equivalents in their full scope.

Claims

1. A method for diagnosing the hardware status of an accelerator card in automated testing, characterized in that, The method includes: A diagnostic thread independent of the testing framework is used to collect runtime physical characteristic data of the accelerator card under test during the execution of test cases in a non-intrusive manner. The diagnostic thread runs in parallel with the thread of the testing framework that executes the test cases, and the non-intrusive collection of runtime physical characteristic data will not block, interrupt or delay the normal execution of the test case business logic by the testing framework. The volatility of the runtime physical characteristic data is analyzed to generate hardware health metrics associated with the test cases, including: Calculate the fluctuation rate of the runtime physical characteristic data during the execution of the test case; Compare the volatility ratio with a predefined expected envelope for the runtime physical characteristics; and The hardware health index is generated based on the comparison results. The expected envelope is a dynamic range obtained based on historical normal operation data statistics, and the dynamic range is defined by the statistical mean and standard deviation. In response to an execution event for the test case emitted through the lifecycle interface provided by the testing framework, the hardware health metric is obtained, and the hardware health metric is associated with the identifier of the test case; and A diagnostic report is generated based on the business execution results of the test cases and the associated hardware health indicators.

2. The method according to claim 1, characterized in that, The runtime physical characteristic data includes at least one of the following: core utilization, memory utilization, core operating frequency, operating voltage, operating current, chip temperature, and cooling fan speed.

3. The method according to claim 1, characterized in that, The lifecycle interface provided by the testing framework includes hook functions, which are configured to be triggered when the test case is executed and the diagnostic report is generated.

4. The method according to claim 1, characterized in that, The diagnostic report includes: The business execution results of the test cases; The hardware health indicators; and The diagnostic conclusion is based on the business execution results and the hardware health indicators.

5. The method according to claim 4, characterized in that, The diagnostic report also includes: A visualization of the physical characteristic data corresponding to the test case changing over time.

6. The method according to claim 1, characterized in that, The hardware health indicators include: an indicator indicating that the hardware is in normal condition, a warning indicator indicating that the hardware is in a sub-healthy state, or an indicator indicating that the hardware is malfunctioning.

7. The method according to claim 6, characterized in that, A diagnostic report is generated based on the business execution results of the test cases and the associated hardware health indicators, including: When the business execution result is passed and the hardware health index indicates that the hardware status is normal, a diagnostic report indicating perfect success is generated. When the business execution result is passed but the hardware health index indicates that the hardware is in a sub-healthy state, a diagnostic report is generated indicating that the logic is passed but the hardware is in a sub-healthy state. When the business execution result is "passed" but the hardware health indicator indicates hardware failure, a diagnostic report is generated indicating that the logic passed but the hardware failed. When the business execution result is failure but the hardware health index indicates that the hardware status is normal, a diagnostic report is generated indicating that the logic failed but the hardware is normal. When the business execution result is failure and the hardware health index indicates that the hardware is in a sub-healthy state or has failed, a diagnostic report indicating logical failure and hardware failure is generated.

8. A system for diagnosing the hardware status of an accelerator card during automated testing, characterized in that, The system includes: processor; The memory stores instructions that, when executed by the processor, perform the following operations: A diagnostic thread independent of the testing framework is used to collect runtime physical characteristic data of the accelerator card under test during the execution of test cases in a non-intrusive manner. The non-intrusive collection of the runtime physical characteristic data does not block, interrupt or delay the normal execution of the test case business logic by the testing framework. The volatility of the runtime physical characteristic data is analyzed to generate hardware health metrics associated with the test cases, including: Calculate the fluctuation rate of the runtime physical characteristic data during the execution of the test case; Compare the volatility ratio with a predefined expected envelope for the runtime physical characteristics; and The hardware health index is generated based on the comparison results. The expected envelope is a dynamic range obtained based on historical normal operation data statistics, and the dynamic range is defined by the statistical mean and standard deviation. In response to an execution event for the test case emitted through the lifecycle interface provided by the testing framework, the hardware health metric is obtained, and the hardware health metric is associated with the identifier of the test case; and A diagnostic report is generated based on the business execution results of the test cases and the associated hardware health indicators.

9. A computer program product, characterized in that, The computer program product includes instructions that, when executed by a processor, perform the method according to any one of claims 1-7.

Images