Automatic testing method and device and storage medium

Abstract

The invention provides an automatic testing method and device and a storage medium. According to the embodiment of the invention, the case log generated by the test case is acquired and analyzed in real time in the process of executing the test task, the type of the exception occurring in the test environment can be identified in time, and the exception can be repaired based on the preset repair strategy and the generated exception context information containing the structured data. And dynamically executing a repair operation aiming at the abnormal type so as to recover the execution process of the influenced test case after successful repair. According to the embodiment of the invention, continuous monitoring and accurate diagnosis of the test environment are realized, and the test environment is dynamically repaired during the operation of the test task, so that test interruption caused by the abnormality of the test environment is reduced, and the continuity of the test process and the reliability of the test result are enhanced.

Description

Technical Field

[0001] This application relates to the field of automated testing technology, and in particular to an automated testing method, apparatus and storage medium. Background Technology

[0002] In the field of server automated testing, the accuracy of test results is highly dependent on the stability of the test environment. Existing automated testing technologies mainly rely on two test environment management methods to improve test environment stability. The first method is to periodically reset the test environment, that is, to restore the entire test environment to a preset baseline state before the start of the test cycle or at a specific time point; the second method is to pre-install all possible dependencies, including tools, library files, and other resources, before the test begins, thereby building a relatively complete initial environment to avoid subsequent problems.

[0003] None of the above methods can detect or handle anomalies caused by dynamic changes in the test environment after the test task has started. When unexpected environmental anomalies occur during testing, the only solution is usually to reset the test environment, which can easily lead to unexpected interruptions in the test process due to environmental issues and reduce the reliability of test results. Summary of the Invention

[0004] In view of this, the present disclosure provides an automated testing method, apparatus and storage medium to address the shortcomings of the related art.

[0005] According to a first aspect of the present invention, an automated testing method is provided, comprising the steps of: Obtain a test task queue, which includes at least one test case; the test case is used to execute in a pre-configured test environment of the server under test, and generates a test case log after execution; During the execution of at least one test case based on the test task queue, obtain the test case log generated by the currently executing test case of interest; When the test case log indicates that there is an anomaly in the test environment where the test case of concern is executed, the type of the anomaly is determined based on the test case log; the type of the anomaly includes firmware version anomaly, network environment anomaly, tool dependency anomaly, or file dependency anomaly. An exception context information is generated based on the type of the exception and the use case log, wherein the exception context information includes structured data required to repair the exception; Based on the exception context information, a repair strategy corresponding to the type of exception is executed to repair the test environment; If the repair is successful, the test cases of concern will be added back to the test task queue so that they can be executed again.

[0006] According to a second aspect of the present invention, an automated testing apparatus is provided, comprising: include: The test case management module is used to obtain a test task queue, which includes at least one test case; the test case is used to execute in a pre-configured test environment of the server under test and generate a test case log after execution; The runtime detection module is used to obtain the test case logs generated by the currently executing test cases of interest during the execution of at least one test case based on the test task queue. When the test case log indicates that there is an anomaly in the test environment where the test case of concern is executed, the type of the anomaly is determined based on the test case log; the type of the anomaly includes firmware version anomaly, network environment anomaly, tool dependency anomaly, or file dependency anomaly. An exception repair module is used to generate exception context information based on the type of the exception and the test case log, and to execute a repair strategy corresponding to the type of the exception based on the exception context information to repair the test environment; wherein, the exception context information includes structured data required to repair the exception; The test case management module is also used to add the test cases of concern back to the test task queue so that they can be executed again if the repair is successful.

[0007] According to a third aspect of the present invention, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described in the first aspect.

[0008] This embodiment of the specification acquires and analyzes test case logs generated during test task execution in real time. This allows for timely identification of anomalies occurring in the test environment. Based on preset repair strategies and generated anomaly context information containing structured data, it dynamically executes repair operations targeting the specific anomaly type. Upon successful repair, the execution flow of the affected test cases is restored. This embodiment achieves continuous monitoring and accurate diagnosis of the test environment, and dynamically repairs the test environment during test task execution. This reduces test interruptions caused by test environment anomalies, enhancing the continuity of the test process and the reliability of test results. Attached Figure Description

[0009] Figure 1 This is a flowchart illustrating an automated testing method as exemplified in an embodiment of this specification; Figure 2 This is a schematic diagram illustrating the active detection and repair process as exemplarily shown in the embodiments of this specification; Figure 3 This is a schematic diagram illustrating an application example of the automated testing method exemplified in the embodiments of this specification; Figure 4 This is a schematic diagram illustrating the detection and repair process for firmware version anomalies, as exemplarily shown in the embodiments of this specification. Figure 5 This is a schematic diagram illustrating the detection and repair process for network environment anomalies, as exemplified in the embodiments of this specification. Figure 6 This is a schematic diagram illustrating the detection and repair process for tool dependency anomalies, as exemplified in the embodiments of this specification. Figure 7 This is a schematic diagram illustrating the detection and repair process for file dependency anomalies, as exemplified in the embodiments of this specification. Figure 8 This is a schematic diagram of an automated testing apparatus exemplified by an embodiment of this specification. Detailed Implementation

[0010] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0011] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0012] It should be understood that although the terms first, second, third, etc., may be used in this application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0013] In the field of server automated testing, automated testing can efficiently schedule test cases and generate visual reports, but the accuracy of its test results depends on the stability of the test environment. Inconsistent test environments are a major factor causing numerous non-product defect failures during testing. These failures are not caused by product defects themselves, but by dynamic changes in the test environment, such as drifting of environment dependency versions, mismatched system library or firmware versions, and dynamic loss of test resources, such as the unexpected loss of critical tools or configuration files. These test environment anomalies can interfere with testers' accurate judgment of product quality, increase the complexity of problem localization, and reduce the continuity and reliability of the testing process.

[0014] To address the issue of environmental consistency, existing technologies primarily employ two approaches. The first approach relies on periodically resetting the environment template. This method restores the entire test environment to a preset baseline state at the beginning of the test cycle or at a specific time point. It focuses on ensuring initial environmental consistency through a one-time, global operation before test execution or during test intervals. However, this approach is a passive and coarse-grained control method. It cannot detect or handle dynamic environmental changes caused by test case execution after the start of the test task, such as the accidental deletion of a critical file or configuration tampering during the test.

[0015] The second approach involves pre-installing all possible dependencies via an initialization script. This method attempts to deploy all necessary tools, libraries, and other resources to the test environment before the test starts by executing a complex installation script. Essentially, it's a preventative strategy designed to configure a sufficiently complete initial state to avoid subsequent problems. However, its drawbacks include the difficulty in exhaustively listing all dependencies, low efficiency of a full installation, and, more importantly, a lack of continuous monitoring and feedback mechanisms for the test runtime environment. When unexpected resource shortages or abnormal states occur during test execution, the system cannot respond.

[0016] Furthermore, containerization technology can be seen as an implementation of the second path of evolution. Containerized testing provides a consistent software runtime environment for applications through image encapsulation and resource isolation. Its core value lies in efficient environment replication and process-level isolation, making it more suitable for testing software products, such as functional testing of basic software like databases, caches, and message queues, or application scenarios such as verifying the compatibility of applications under different system libraries and language runtime versions.

[0017] This solution targets automated testing of the entire server system, a scenario that differs from the pure software testing described above. The core objective of server testing is not to verify a specific software function, but to ensure that server hardware components, firmware, and the interoperability between hardware and software work properly. Examples include firmware flashing and functional verification of BMC, BIOS, and CPLD; BIOS settings, upgrades, and compatibility testing; and hardware resource allocation and stability stress testing.

[0018] Specifically, if containerization is applied to the application scenarios described in this solution, the following drawbacks will occur: Firstly, it inherits the disadvantage of the second path in that it cannot cope with dynamic changes at runtime. It cannot specifically repair the environment drift that occurs during the test. It can usually only be restored by rebuilding the container instance, which is essentially still a "reset" behavior of the container and cannot achieve accurate and dynamic environment repair.

[0019] Secondly, while container isolation is essentially process-level isolation, it still shares the same operating system kernel at its underlying level, making it impossible to directly access or control physical hardware resources. Therefore, containerization technology has limitations in automated server testing scenarios that require verifying software compatibility and performing firmware upgrades. For example, it struggles to directly access or control resources such as the firmware of smart network cards, BMCs, or BIOSes, and cannot simulate a real hardware environment.

[0020] Thirdly, containerization technology provides an isolated runtime environment for applications at the user space level. This isolation mechanism is achieved through technologies such as namespaces and control groups, allowing each container to have an independent file system view, network stack, and process space, enabling lightweight resource isolation and rapid deployment while maintaining a high security boundary. However, in server automated testing scenarios, the testing process often requires direct manipulation of system-level resources. These operations include loading kernel modules, configuring global network parameters, and accessing physical hardware devices. These system-level operations typically exceed the scope of user space isolation and require higher kernel privileges to execute. Containerization technology can overcome these limitations by escalating privileges. For example, specific device nodes on the host machine can be mounted inside the container. For instance, when it is necessary to access a serial port device to interact with the BIOS during testing, device mounting can be used. However, the allocation and permission settings of device nodes in this approach depend on the dynamic environment of the host machine, which may cause stability and compatibility issues. Alternatively, the privileged mode of the container can be enabled to elevate the container's permissions to a level close to that of the host machine's root user, thereby gaining extensive access to system resources. However, this privilege escalation essentially undermines the isolation characteristics of the container, deviates from the container's security principles, and introduces significant security risks in a strictly controlled testing environment.

[0021] Furthermore, containerization technology also has drawbacks in resource scheduling and execution efficiency. Long-running container instances in standby mode will lead to continuous CPU and memory resource consumption, resulting in resource idleness and waste; while dynamically starting and stopping containers on demand introduces overhead such as image pulling, container startup, and context switching, slowing down the response speed of the testing process, which runs counter to the high efficiency pursued by automated testing.

[0022] Based on the foregoing explanation, none of the above methods can detect or handle anomalies caused by dynamic changes in the test environment after the test task has started. When unexpected environmental anomalies occur during testing, the only solution is usually to reset the test environment, which can easily lead to unexpected interruptions in the testing process due to environmental issues and reduce the reliability of test results.

[0023] To address the above issues, this specification provides an automated testing mechanism for server automation testing scenarios. This mechanism can sense the environmental status in real time, accurately diagnose the root causes of anomalies, and dynamically trigger repairs to cope with various anomalies in the testing environment. This helps maintain the consistency and stability of the testing environment and the continuity of the testing process, thereby improving testing efficiency and the reliability of test results.

[0024] Figure 1 This is a flowchart illustrating an automated testing method exemplified in an embodiment of this specification. One embodiment of this automated testing method can be executed via a control machine, also known as a host computer. It is understood that in other embodiments, the code implementing this method can be loaded onto other devices for execution. The following explanation uses a control machine as an example. The control machine does not directly execute specific product functional tests, but is responsible for the scheduling, management, and decision-making of the entire testing process. The control machine's responsibilities include acquiring and managing test case sets, generating and scheduling test task queues, monitoring test execution status, collecting and analyzing test logs, diagnosing types of environmental anomalies, formulating and triggering remediation strategies, and further advancing the testing process.

[0025] During testing, the test cases are executed by the server under test (DUT), which communicates with the control unit. The DUT receives test instructions from the control unit and loads and runs the specified test cases within its pre-configured test environment. During execution, the test cases utilize the hardware resources, operating system, firmware, and various application software on the DUT to verify its functionality, performance, and reliability. Furthermore, the control unit can connect to a central server, enabling it to retrieve any missing tools or files from the DUT.

[0026] This automated testing method includes the following steps: S100: Obtain the test task queue, which includes at least one test case; the test case is used to execute in a pre-configured test environment of the server under test and generate a test case log after execution.

[0027] The test cases define a series of specific operational instructions that need to be executed on the server under test, as well as the expected result criteria used to verify the correctness of the functions. These test cases together constitute a complete verification scheme for specific functions or performance indicators of the server under test.

[0028] The execution of test cases relies on a pre-configured test environment on the server under test. This test environment includes all the necessary elements to ensure correct test execution, such as a specific version of the operating system, firmware, system services, network configuration, files, and the basic toolchain. The design logic and expected behavior of the test cases are all predicated on the default state of this environment.

[0029] When a test case is scheduled for execution in the test environment, it interacts with various components of the environment and automatically generates a test case log corresponding to that execution. The test case log records key information throughout the test lifecycle, including but not limited to the sequence of executed operations, system response information, generated intermediate data, warnings or error messages, and the final execution conclusion status.

[0030] Test cases in the test task queue are executable task units and can be optimized and sorted according to their priority, dependencies, or resource requirements.

[0031] S102: During the execution of at least one test case based on the test task queue, obtain the test case log generated by the currently executing test case of interest.

[0032] The trigger condition for obtaining test case logs is that a test case in the test task queue has completed its execution process on the server under test. In the embodiments of this specification, a test case of interest refers to a test case that has completed execution and generated a test case log during the execution of tests based on the test task queue. Regardless of whether the execution result of the test case of interest is successful or unsuccessful, the control machine can immediately collect the complete test case log obtained from that execution.

[0033] S104: When the test case log indicates that there is an anomaly in the test environment where the test case of interest is executed, determine the type of the anomaly based on the test case log. The type of anomaly includes firmware version anomaly, network environment anomaly, tool dependency anomaly, or file dependency anomaly.

[0034] To parse the acquired test case logs, the control machine can identify the log content using pre-defined analysis rules and a pattern library. When specific information in the test case log matches the characteristics of known exception information, it is determined that the failure of the current test case was caused by an exception in the test environment.

[0035] After confirming that there are anomalies in the test environment, more specific anomaly type diagnosis can be performed. That is, based on the specified information in the test case log, the test environment can be classified into one of several predefined standard anomaly types, including firmware version anomalies, network environment anomalies, tool dependency anomalies, or file dependency anomalies.

[0036] Among them, firmware version anomaly refers to the firmware version of core hardware components such as the baseboard management controller, basic input / output system, or smart network card on the server under test being inconsistent with the standard version required by the test environment; network environment anomaly refers to the connectivity interruption or configuration error of the management network or business network of the server under test; tool dependency anomaly refers to the absence of command-line tools or applications necessary for executing the test cases of concern in the test environment; and file dependency anomaly refers to the absence of critical system configuration files, firmware images, and other data files that the test cases of concern depend on, such as upgrade files required for BIOS upgrades.

[0037] This step transforms ambiguous test failure results into clear identifiers of exception types, providing a basis for decision-making in initiating targeted remediation processes.

[0038] S106: Generate exception context information based on the exception type and the use case log. The exception context information includes structured data required to repair the exception. S108: Execute a repair strategy corresponding to the type of the exception based on the exception context information to repair the test environment. The exception context information includes structured data required to repair the type of the exception.

[0039] Steps S106 and S108 execute specific repair processes based on the types of anomalies obtained in the preceding steps, aiming to dynamically restore the state of the test environment.

[0040] First, based on the identified anomaly type, a corresponding remediation strategy is matched. In this embodiment, a corresponding remediation strategy is pre-associated for each anomaly type. Each remediation strategy further defines the specific steps, judgment conditions, and the final state goal to be achieved in resolving a particular type of environmental problem. For example, for firmware version anomalies, the remediation strategy might involve rolling back or upgrading the firmware version of the target component to a standard version; for tool dependency anomalies, the remediation strategy might be obtaining and installing the missing toolkit from the central repository. In practical applications, the remediation strategy is implemented using a series of pre-programmed executable programs for specific anomaly types; each remediation strategy can also be called a remediation module.

[0041] After defining the repair strategy, it is necessary to provide the required input parameters for its execution, namely the structured data needed to repair the exception, included in the exception context information. For example, when repairing a file missing exception, the repair strategy requires the exact path and filename of the missing file. The corresponding repair module is programmed to receive and parse the structured data packet containing this information and then execute the file transfer process. Therefore, the purpose of generating exception context information is to meet the rigid requirements of these predefined programs for input data, ensuring that the repair strategy can be accurately instantiated and executed by the repair module.

[0042] Accordingly, this embodiment achieves a shift from traditional "overall environment reset" to "specific and precise repair" through a standardized mapping and routing mechanism between exception types and dedicated repair modules. When specific data is identified in the exception context information, the code-based routing engine will automatically direct it to the corresponding dedicated repair module, which will then perform targeted repair based on the exception context information.

[0043] S110: If the repair is successful, the test cases of concern are added back to the test task queue so that they can be executed again.

[0044] To confirm whether the repair operation for the test environment was successful, you can call the preset environment status check component, which will check whether the test environment has returned to a state that can support test execution normally.

[0045] In one specific embodiment, the environment status checking component is configured to detect the firmware version information of at least one target component through a preset out-of-band management protocol, and to detect the network connectivity status of the at least one target component through a preset network detection command. The target component can be a baseboard management controller, a basic input / output system, or a network interface card (NIC). It should be noted that this environment status checking component does not rely on use case logs, but directly obtains its status information from the hardware components and network infrastructure level, ensuring that the environment is in a usable initial state after repair. For example, the out-of-band management protocol can be IPMI or Redfish, which can bypass the host operating system (the operating system of the server under test) and directly obtain the firmware version information of hardware components such as the BMC, BIOS, and smart NIC, and compare it with a preset baseline version. For network connectivity detection, the ping command is used to send probe packets to the BMC management port IP address to verify network layer reachability, and the nc command is used to attempt to connect to the SSH service port (such as port 22) of the operating system's business network to verify transport layer connectivity. After completing the detection, the environment status checking component obtains detection data, which is usually encapsulated in a Python dictionary structure as the basis for determining whether the repair was successful.

[0046] The successful remediation indicates that the root cause of the failure of the test case of concern was the unstable test environment, rather than a functional defect in the test case itself or the object under test. Therefore, after the environment is restored, the test case of concern that failed due to environmental anomalies is added back to the test task queue to retry it and obtain valid test results in a healthy environment. For example, the test case of concern can be inserted at the end of the test task queue, that is, after executing other test cases in the test case set, the previously failed test case is retried.

[0047] Furthermore, after step S108, the automated testing method further includes: S112: If the repair fails, stop the execution of the test task queue and output an alarm message.

[0048] A repair failure means that the repair strategy cannot restore the server under test to a baseline state that can normally support the test, indicating that there may be deep or structural faults in the environment that cannot be automatically resolved. Therefore, if a repair failure is confirmed, the entire queue of currently executing test tasks will be aborted to prevent the test from continuing on a known faulty or unstable environment, avoid generating a large number of meaningless failure results, and eliminate interference from invalid test data on product quality assessment.

[0049] This embodiment of the specification acquires and analyzes test case logs generated during test task execution in real time. This allows for timely identification of anomalies occurring in the test environment. Based on preset repair strategies and generated anomaly context information containing structured data, it dynamically executes repair operations targeting the specific anomaly type. Upon successful repair, the execution flow of the affected test cases is restored. This embodiment achieves continuous monitoring and accurate diagnosis of the test environment, and dynamically repairs the test environment during test task execution. This reduces test interruptions caused by test environment anomalies, enhancing the continuity of the test process and the reliability of test results.

[0050] The aforementioned embodiments provide an environment anomaly detection and repair mechanism based on test case log analysis during test runtime. On this basis, a more complete test environment protection system can be built through proactive environment detection and repair before test execution.

[0051] As one or more embodiments of the present invention, the automated testing method further includes an active detection and repair process before the test task is executed. Figure 2 This is a schematic diagram illustrating an active detection and repair process as exemplified in an embodiment of this specification. The active detection and repair process includes the following steps: S200: Before the test task queue is executed, the firmware version information of at least one target component is detected by a preset out-of-band management protocol, and the network connectivity status of the at least one target component is detected by a preset network detection command.

[0052] It is understood that step S200 can be implemented by the environmental status check component provided in the foregoing embodiments.

[0053] S202: If the firmware version information of the target component is found to be inconsistent with the preset version information in the test environment, determine that the type of the anomaly includes a firmware version anomaly. S204: If an abnormal network connectivity status of the target component is detected, the type of the abnormality is determined to include an abnormal network environment. S206: Generate exception context information based on the type of the exception, and execute a repair strategy corresponding to the type of the exception based on the exception context information to repair the test environment. It should be noted that the proactive detection and repair process does not depend on test cases; the exception context information can be generated specifically through the exception type and the detection data provided by out-of-band management protocols and / or network detection commands.

[0054] When the detection reveals that the firmware version information of the target component is inconsistent with the standard version required by the test environment, the system will preemptively determine that there is a firmware version anomaly in the environment. Similarly, when an abnormal network connectivity is detected, a network environment anomaly will be preemptively determined. Based on this, the corresponding repair process can be initiated immediately, which involves determining a preset repair strategy for the type of anomaly, generating an anomaly context information containing specific repair parameters based on the detection results, and executing the repair strategy to pre-correct the test environment.

[0055] The significance of the proactive detection and repair process in this manual lies in its ability to effectively identify and eliminate baseline deviation issues that may exist after environment initialization. By proactively discovering and repairing basic environmental anomalies such as firmware version mismatches and unavailable network connections before testing begins, it is possible to prevent a large number of test cases from failing consecutively due to the same environmental problem. This significantly improves the success rate of test execution and overall efficiency, and together with the runtime repair mechanism, constitutes an environment consistency guarantee scheme covering the entire testing lifecycle.

[0056] As one or more embodiments of the present invention, determining the type of the exception based on the use case log includes: Based on a preset regular expression, the specified field in the test case log is matched.

[0057] The matched specified field is input into a preset environment anomaly pattern library to obtain the type of the anomaly; the environment anomaly pattern library is used to store the correspondence between several fields and the type of anomaly.

[0058] The embodiments in this specification first systematically scan the use case logs using preset regular expressions, aiming to identify and match specified fields related to potential environmental problems from messy text information. For example, r"Command not found: (\w+)" is used as a regular expression rule to scan the use case logs to capture key information about command line tool call failures.

[0059] After matching the specified fields, these fields are input as query conditions into a pre-built environment exception pattern library. This pattern library is a pre-built, extensible library that stores the mapping relationships between various fields and exception types. By querying this pattern library in real time, the matched specified fields can be transformed into explicit exception type determinations.

[0060] Based on the determination of the anomaly type, this embodiment further performs in-depth detection and analysis on specified fields of the use case logs that match the anomaly type, and then uses these specified fields to generate anomaly context information. For example, while identifying tool dependency anomalies, the specific tool name can be extracted from the logs, such as extracting the tool name "YafuFlash" from the matched specified field "Command not found: YafuFlash"; file path information can be extracted when diagnosing file dependency anomalies, such as extracting the absolute path of the missing file using the regular expression r"No such file or directory: '(.+)'". These specific specified fields, combined with the determined anomaly type, constitute a structured anomaly context information.

[0061] For example, exception context information includes at least two attributes. The first attribute is an environment flag variable, which is a string containing a standardized exception identifier. For instance, if the exception type is determined to be a tool dependency exception, based on the matched specified field "Command not found: YafuFlash", the specific cause of the exception can be determined to be the missing tool "YafuFlash", and the corresponding environment flag variable "TOOL_YAFU_MISSING" will be generated.

[0062] The second attribute is the exception details, which uses a dictionary-type data structure to systematically record the specific parameters necessary for resolving the execution environment, as well as related diagnostic context information. For example, for a tool dependency exception, the dictionary contains a key `tool_name` with the value "YafuFlash" and a key `failed_command` with the value "YafuFlash--update ...", representing the specific command that triggered the exception. These detailed parameters provide the necessary structured data for the execution of subsequent tool dependency exception remediation modules.

[0063] As one or more embodiments of the present invention, the automated testing method further includes: If the type of exception corresponding to the specified field is not stored in the environment exception pattern library, the test case of concern will be removed from the test task queue.

[0064] Building upon the aforementioned embodiments, this embodiment also includes an exception handling mechanism triggered when the type of exception cannot be determined. If a specified field in the test case log is successfully matched using a regular expression, but a query of the preset environment exception pattern library reveals that no exception type corresponding to the specified field is stored, it indicates that the currently encountered exception pattern may belong to an unknown type of environmental problem or a product defect problem. In this situation, the test case that failed to execute is removed from the currently executing test task queue to avoid improper repair operations or process blockages that may result from the inability to identify the exception type, while ensuring that product defect problems can be correctly identified and handled.

[0065] It should be noted that after a test case is removed from the test task queue, subsequent test cases in the test task queue can choose to continue execution.

[0066] Furthermore, before removing the test case of interest from the test task queue, the test environment can be checked again by the environment status check component. If an abnormal test environment is detected, the execution of the test task queue is stopped and an alarm message is output; if the test environment is detected to be normal, the test case of interest is removed from the test task queue, and then the next test case in the test task queue is executed.

[0067] Specifically, before deciding to remove a test case of interest from the test task queue, the aforementioned environment status check component can be invoked to perform a verification check on the overall status of the current test environment. Based on the aforementioned embodiment, this environment status check component will detect the firmware version information of the target component through a preset out-of-band management protocol, and detect the network connectivity status of the target component through a preset network detection command.

[0068] The value of verification testing lies in its ability to effectively identify and differentiate temporary environmental fluctuations. For example, transient network connection interruptions or occasional process blockages during test execution may lead to specific error messages not yet recorded in the environmental anomaly pattern library. Performing this verification test allows us to obtain a snapshot of the test environment's state at the latest moment. If a clear anomaly is detected, it indicates a problem requiring intervention, necessitating the termination of the test task and the output of an alarm. If the test confirms a normal test environment, it increases the confidence in classifying the failure as a product defect, making it more prudent and reliable to remove the test case from the queue. This mechanism, through proactive secondary verification of the environmental state, reduces the possibility of misjudgments due to transient environmental fluctuations and improves the reliability of test results.

[0069] Combination Figure 1The automated testing methods shown in the test runtime and Figure 2 This specification illustrates the proactive detection and repair process before test execution, and the embodiments further provide an application example of an automated testing method. Figure 3 This is a schematic diagram illustrating an application example of an automated testing method exemplified in the embodiments of this specification. The automated testing method includes an anomaly identification stage and an anomaly handling stage.

[0070] The anomaly detection phase can be further divided into a proactive detection process before test execution and a log detection process during test runtime. The proactive detection process before test execution includes: S3100 and S3101 are executed in parallel.

[0071] S3100: Detects firmware version information of target components via out-of-band management protocol.

[0072] S3101: Detects the network connectivity status of the target component using a preset network detection command.

[0073] After S3100 and S3101, execute S3102: determine whether the type of exception is firmware version exception or network environment exception, generate exception context information for the type of exception, and proceed to step S3300.

[0074] The test runtime log detection process includes: S3200: Get the test task queue; S3201: Execute test cases in the test task queue and retrieve the test case logs generated by the test cases; S3202: Match the specified field in the test case log and input the matched specified field into the preset environment exception mode library; If the environment anomaly pattern library identifies the type of anomaly, execute S3203: determine the preset repair strategy for the type of anomaly, generate anomaly context information based on the type of anomaly and the use case log, and then proceed to step S3300.

[0075] If the environment anomaly pattern library does not match the anomaly type, proceed to step S3400.

[0076] After completing proactive checks before test execution or log checks during test runtime, the exception handling phase is executed. The exception handling phase can be further divided into environmental exception handling procedures and product defect handling procedures.

[0077] The environmental anomaly handling process includes: In response to the exception context information determined by S3102 or S3203, S3300: Execute the repair strategy based on the exception context information to repair the test environment; If the repair is successful, execute S3301: add the test case back to the test task queue and execute the next test case in the test task queue.

[0078] If the repair fails, execute S3500: abort the execution of the test task queue and output an alarm message.

[0079] The product defect handling process includes: S3400: The firmware version information of the target component is detected through the out-of-band management protocol, and the network connectivity status of the target component is detected through the preset network detection command; this step is a combination of steps S3100 and S3101, and aims to determine whether the test environment is abnormal.

[0080] If the test environment is normal, execute S3401: remove the test case from the test task queue, and then execute the next test case in the test task queue.

[0081] If the test environment is abnormal, execute S3500: stop the execution of the test task queue and output alarm information.

[0082] Finally, after executing steps S3301, S3401, or S3500, all data records and status information generated throughout the testing process are integrated to construct the Allure test report (S3501).

[0083] The following section provides a detailed explanation of the exception types: firmware version exception, network environment exception, tool dependency exception, and file dependency exception.

[0084] Specifically, a firmware version anomaly indicates that the firmware version information of at least one target component in the server under test is inconsistent with the preset version information in the test environment. The target component is hardware such as a baseboard management controller, a basic input / output system, or a network interface card (NIC).

[0085] As one or more embodiments of the present invention, when the type of the exception includes a firmware version exception, the step of executing a repair strategy corresponding to the type of the exception based on the exception context information to repair the test environment includes: Configure the firmware of the target component to the firmware corresponding to the preset version information in the test environment.

[0086] When the anomaly is confirmed to be a firmware version anomaly, a remediation strategy based on the anomaly context information will be initiated. The key to this remediation strategy is adjusting the firmware version configuration of the target component to the firmware program corresponding to the standard version information preset in the test environment.

[0087] The exception context information provides structured data required for firmware configuration. This data may include the specific identifier of the target component, the currently detected exception version number, the target version number to be restored, and the corresponding firmware program storage path. Based on the exception context information, the remediation process initiates a firmware update or version rollback operation to the target component through the appropriate management interface. This eliminates the inconsistency between the firmware version of the target component on the server under test and the standard requirements of the test environment, thereby restoring the uniformity of the test environment at the firmware version level and establishing the necessary hardware infrastructure for the correct execution of test cases.

[0088] In one embodiment, firmware version anomalies can be further divided into two cases: those caused by environment configuration and those caused by product defects. When the anomaly type includes firmware version anomalies, before executing the repair strategy corresponding to the anomaly type based on the anomaly context information, the method further includes: Determine whether the use case log contains firmware upgrade operation records; If the test case log contains firmware upgrade operation records, the test environment is checked again by the environment status inspection component. If an abnormal test environment is detected, the execution of the test task queue is stopped and an alarm message is output; if the test environment is detected to be normal, the test case is removed from the test task queue, and then the next test case in the test task queue is executed.

[0089] In this embodiment, if the test log contains a clear record of a firmware upgrade operation, the environment repair process for firmware version anomalies will not be initiated. This is because the execution of a firmware upgrade operation indicates that the test has entered the version update verification phase. Version-related issues that occur at this stage usually stem from inherent product defects such as upgrade process execution failure, corrupted firmware image files, or incompatibility of upgrade tools, and fall under the category of product defect anomalies.

[0090] Only when no firmware upgrade operation record is found in the test log will it be confirmed that the current firmware version anomaly is indeed caused by an environment configuration deviation, and only then will the subsequent repair process continue. Therefore, this embodiment can distinguish between product functional defects and firmware version anomalies caused by environment configuration, ensuring that the repair strategy only targets genuine environment anomalies and avoiding mishandling of product defect issues.

[0091] Combination Figure 3 The embodiments of this specification further provide application examples of the automated testing method and the above-mentioned firmware version anomaly handling methods.

[0092] Figure 4This is a schematic diagram illustrating an exemplary embodiment of the detection and repair process for firmware version anomalies in this specification. The detection process includes: In the case of proactive detection process before test execution, S4100 is executed: the firmware version information of the target component is detected through out-of-band management protocol; If the firmware version information of the target component is found to be inconsistent with the preset version information, step S4101 is executed: the identifier of the target component is determined, and an abnormal context information is generated based on the identifier of the target component and the detection data obtained from the out-of-band management protocol, and then step S4300 is executed. The identifier of the target component is used to uniquely identify hardware such as the baseboard management controller, basic input / output system, and network interface card, and serves as the environment flag variable mentioned in the aforementioned embodiments in the abnormal context information.

[0093] In the case of the log detection process during test runtime, execute S4200: identify firmware version anomalies based on specified fields in the test case log and the environment anomaly pattern library; S4201: Determine whether the test case log contains firmware upgrade operation records; If the test case log contains firmware upgrade operation records, execute S4202: re-inspect the test environment using the environment status inspection component.

[0094] If an abnormality is detected in the test environment, execute S4302: stop the execution of the test task queue and output alarm information.

[0095] If the test environment is detected to be normal, execute S4203: remove the test cases from the test task queue.

[0096] If the use case log does not contain firmware upgrade operation records, execute S4204: determine the identifier of the target component, generate exception context information based on the identifier of the target component and the use case log, and then proceed to step S4300. The identifier of the target component is a unique identifier representing hardware such as the baseboard management controller, basic input / output system, and network interface card.

[0097] S4300: Based on the exception context information and the identifier of the target component, configure the firmware of the target component to the firmware corresponding to the preset version information in the test environment.

[0098] If the repair is successful, execute S4301: Re-add the test cases to the test task queue.

[0099] If the repair fails, execute S4302: Abort the execution of the test task queue and output an alarm message.

[0100] Specifically, the network environment anomaly indicates an anomaly in the network environment of at least one target component in the server under test.

[0101] As one or more embodiments of the present invention, when the type of the anomaly includes a network environment anomaly, the step of executing a repair strategy corresponding to the type of the anomaly based on the anomaly context information to repair the test environment includes: Configure the IP address of the target component, and / or restart the network service function of the target component, and obtain the repair results; If the repair result indicates that the repair has failed, the firmware of the target component will be configured to the firmware corresponding to the preset version information in the test environment.

[0102] When the anomaly is confirmed to be a network environment anomaly, the embodiments in this specification will initiate a tiered repair strategy. The repair process first performs lightweight repair operations, including reconfiguring the IP address parameters of the target component and / or restarting the network service functions of the target component. These operations are designed to resolve network connectivity problems caused by configuration errors or temporary service failures.

[0103] After performing a lightweight repair operation, the results will be acquired and analyzed immediately. If the repair results indicate that the network connection has been restored, the repair process ends. If the repair results clearly show that the lightweight repair operation failed to resolve the issue, the system will automatically escalate to a heavyweight repair strategy.

[0104] The heavyweight remediation strategy involves adjusting the firmware version of the target component. Specifically, this involves configuring the target component's firmware to the version corresponding to a pre-defined standard version in the test environment. This remediation strategy is based on the following technical considerations: persistent network connectivity issues may be related to compatibility defects or functional malfunctions in the firmware version; restoring the firmware to a verified standard version can address network environment anomalies at a deeper level.

[0105] Combination Figure 3 The embodiments of this specification further provide application examples of the automated testing methods and the above-described methods for handling network environment anomalies, as well as application examples of the detection and repair process for network environment anomalies.

[0106] Figure 5 This is a schematic diagram illustrating an exemplary embodiment of the detection and repair process for network environment anomalies in this specification. The detection process includes: In the case of the proactive detection process before test execution, step S5100 is executed: the network connectivity status of the target component is detected using a preset network detection command. Then, if an abnormal network environment of the target component is detected, step S5101 is executed: the identifier of the target component is determined, and an abnormal context information is generated based on the identifier of the target component and the detection data obtained from the network detection command, and then step S5300 is executed. The identifier of the target component is used to uniquely identify hardware such as the baseboard management controller, basic input / output system, and network interface card, and is used as the environment flag variable mentioned in the aforementioned embodiment in the abnormal context information. For the detection data obtained from the network detection command, for example, in the case of a timeout when connecting to the BMC using IPMI out-of-band commands, the returned error may be "Unable to establish IPMI v2 / RMCP+ session".

[0107] In the case of the log detection process during test runtime, execute S5200: identify network environment anomalies based on specified fields in the test case log and the environment anomaly pattern library; S5201: Determine the identifier of the target component, generate exception context information based on the identifier of the target component and the use case log, and then proceed to step S5300.

[0108] S5300: Configure the IP address of the target component, restart the network service function of the target component, and obtain a lightweight repair result; If the lightweight repair is successful, proceed to step S5302; If the lightweight repair fails, perform a heavyweight repair S5301: Configure the firmware of the target component to the firmware corresponding to the preset version information in the test environment, and then proceed to step S5302.

[0109] S5302: The test environment is re-tested using the environment status check component to obtain the final repair results.

[0110] If the repair is successful, execute S5303: Re-add the test cases to the test task queue.

[0111] If the repair fails, execute S5304: Abort the execution of the test task queue and output an alarm message.

[0112] Specifically, the tool dependency exception indicates that the test environment lacks the tools required to execute the test case; the file dependency exception indicates that the test environment lacks the files required to execute the test case.

[0113] As one or more embodiments of the present invention, when the type of the exception includes a tool dependency exception or a file dependency exception, the step of executing a repair strategy corresponding to the type of the exception based on the exception context information to repair the test environment includes: Obtain the missing tools or files, and configure the tools or files into the test environment of the server under test.

[0114] When the anomaly is confirmed to be a tool dependency anomaly or a file dependency anomaly, a resource recovery mechanism will be initiated. The core operation of this repair strategy is to retrieve the missing resources from the control machine or a pre-set central server and correctly deploy them to the test environment of the server under test. For example, the missing tools are retrieved by the control machine from the central server, while the files are stored locally on the control machine.

[0115] The exception context information provides crucial location parameters during this process, specifically identifying the missing tool name or file path. Based on these tool names or file paths, the control machine accurately retrieves the specified tool installation package or dependency files by accessing the resource repository stored locally on the control machine or on the central server.

[0116] Once the acquisition is complete, the control machine performs the corresponding configuration operations. For example, for tool dependency anomalies, the toolkit is installed to the predetermined path in the test environment and executable permissions are set; for file dependency anomalies, the file is transferred to the target directory and its integrity is verified. The entire recovery process, through a standardized deployment procedure, ensures that missing tools or files are accurately restored to the required location in the test environment.

[0117] This centralized resource management mechanism effectively solves the problem of missing dependencies caused by dynamic changes in the test environment. By obtaining resources on demand, it ensures timely supply of test resources and avoids the waste of resources caused by full pre-installation, thereby improving the accuracy and efficiency of test environment maintenance.

[0118] Combination Figure 3 The embodiments of this specification further provide application examples of the automated testing methods and the handling methods for network environment anomalies, respectively, for the detection and repair processes of tool dependency anomalies and file dependency anomalies.

[0119] Figure 6 This is a schematic diagram illustrating an exemplary embodiment of the detection and remediation process for tool dependency anomalies in this specification. The detection process includes: S6100: Tool dependency exceptions are identified based on specified fields in the use case log and the environment exception pattern library.

[0120] Taking the test case "Upgrade BMC via YafuFlash tool" as an example, during the execution of the test case, when the YafuFlash tool is used to upgrade the BMC firmware, the system returns a "Command not found" error.

[0121] After the test case is executed, the test case log is parsed and successfully matched with the environment exception pattern library, indicating that it is a tool dependency exception.

[0122] S6101: Determine the identifier of the missing tool based on the use case log, and generate exception context information based on the tool identifier.

[0123] If the known exception type is a tool dependency exception, the specific missing tool is further identified as YafuFlash based on the use case log. The identifier of the missing tool, TOOL_YAFU_MISSING, is generated as the environment flag variable mentioned in the previous embodiment. The tool name, failed command, and other details are recorded in the exception details to generate exception context information.

[0124] S6102: Based on the exception context information, retrieve the missing tools from the central server and automatically install the tools to the server under test.

[0125] The tool installer is triggered based on the environment flag variable TOOL_YAFU_MISSING in the exception context information. This installer then pulls the YafuFlash toolkit from the central file server according to the parameters in the exception details and performs an automated installation onto the server under test.

[0126] S6103: The test environment is re-tested using the environment status check component to obtain the repair results.

[0127] If the repair is successful, execute S6104: re-add the test cases to the test task queue.

[0128] If the repair fails, execute S6105: Abort the execution of the test task queue and output an alarm message.

[0129] Figure 7 This is a schematic diagram illustrating an exemplary embodiment of the detection and repair process for file dependency anomalies in this specification. The detection process includes: S7100: File dependency exceptions were identified based on specified fields in the use case log and the environment exception pattern library.

[0130] Taking the test case "Upgrade BIOS firmware" as an example, during the execution of the test case, the BIOS upgrade tool received the error message "No such file or directory: ' / home / Upgrade_Pack / BIOS / yinshanv2fpga_bios_v1_10_00_final.hpm' After the test case is executed, the test case log is parsed and successfully matched with the environment exception pattern library, indicating a file dependency exception.

[0131] S7101: Determine the identifier of the missing file based on the use case log, and generate exception context information based on the identifier of the file.

[0132] If the known exception type is a file dependency exception, the specific missing tool is further identified as a BIOS upgrade file based on the use case log. The identifier FILE_BIOS_MISSING of the missing file is generated as the environment flag variable mentioned in the previous embodiment, and details such as file path, file type, and target path are recorded in the exception details, thereby encapsulating it into exception context information.

[0133] S7102: Based on the exception context information, transfer the file to the specified directory on the server under test.

[0134] The file transfer procedure is triggered based on the environment flag variable FILE_BIOS_MISSING in the exception context information. This procedure, according to the file path in the exception details, transfers the missing BIOS upgrade file from the controller to the target path on the server under test via the Rsync protocol.

[0135] S7103: The test environment is re-tested using the environment status check component to obtain the repair results.

[0136] If the repair is successful, execute S7104: Re-add the test cases to the test task queue.

[0137] If the repair fails, execute S7105: Abort the execution of the test task queue and output an alarm message.

[0138] Figure 8 This is a schematic diagram illustrating an automated testing apparatus as exemplified in an embodiment of this specification. The automated testing apparatus includes: The test case management module 800 is used to obtain a test task queue, which includes at least one test case; the test case is used to execute in a pre-configured test environment of the server under test and generate a test case log after execution; The runtime detection module 801 is used to obtain the test case log generated by the currently executing test case of interest during the execution of at least one test case based on the test task queue. When the test case log indicates that there is an anomaly in the test environment where the test case of interest is executed, the type of the anomaly is determined based on the test case log; the type of the anomaly includes firmware version anomaly, network environment anomaly, tool dependency anomaly, or file dependency anomaly.

[0139] An exception repair module 802 is used to generate exception context information based on the type of the exception and the use case log, and to execute a repair strategy corresponding to the type of the exception based on the exception context information to repair the test environment; wherein, the exception context information includes structured data required to repair the exception.

[0140] The test case management module is also used to add the test cases of concern back to the test task queue so that they can be executed again if the repair is successful.

[0141] Furthermore, the anomaly repair module 802 includes a version repair unit, a network repair unit, a tool repair unit, and a file repair unit, which are used to execute repair strategies corresponding to firmware version anomalies, network environment anomalies, tool dependency anomalies, and file dependency anomalies, respectively.

[0142] Continue reading Figure 8 The automated testing device also includes an active detection module 803; The active detection module 803 is used to detect the firmware version information of at least one target component through a preset out-of-band management protocol, and to detect the network connectivity status of the at least one target component through a preset network detection command. If the firmware version information of the target component is found to be inconsistent with the preset version information in the test environment, the type of the anomaly is determined to include firmware version anomaly. If an abnormal network connectivity status of the target component is detected, the type of the abnormality is determined to include network environment abnormality; The exception repair module 802 is further configured to generate exception context information based on the type of the exception, execute a repair strategy corresponding to the type of the exception based on the exception context information, and repair the test environment.

[0143] This specification also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method of any embodiment of this specification.

[0144] Computer-readable storage media include both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, disk storage, quantum memory, graphene-based storage media or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0145] The embodiments of the subject matter and functional operation described in this specification can be implemented in the following ways: digital electronic circuits, tangibly embodied computer software or firmware, computer hardware including the structures disclosed in this specification and their structural equivalents, or combinations thereof. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible, non-transitory program carrier for execution by a data processing apparatus or for controlling the operation of a data processing apparatus. Alternatively or additionally, the program instructions may be encoded on artificially generated propagation signals, such as machine-generated electrical, optical, or electromagnetic signals, which are generated to encode information and transmit it to a suitable receiving device for execution by the data processing apparatus. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or combinations thereof.

[0146] The processing and logic flow described in this specification can be executed by one or more programmable computers that execute one or more computer programs to perform corresponding functions by operating on input data and generating output. The processing and logic flow can also be executed by dedicated logic circuitry—such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits), and the device can also be implemented as dedicated logic circuitry.

[0147] Suitable computers for executing computer programs include, for example, general-purpose and / or special-purpose microprocessors, or any other type of central processing unit. Typically, the central processing unit receives instructions and data from read-only memory and / or random access memory. The basic components of a computer include a central processing unit for implementing or executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include one or more mass storage devices for storing data, such as disks, magneto-optical disks, or optical disks, or the computer will be operatively coupled to such mass storage devices to receive data from or transfer data to them, or both. However, a computer is not required to have such devices. Furthermore, a computer can be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device such as a universal serial bus (USB) flash drive, to name a few.

[0148] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, such as semiconductor memory devices (e.g., EPROM, EEPROM, and flash memory devices), magnetic disks (e.g., internal hard disks or removable disks), magneto-optical disks, and CD-ROM and DVD-ROM disks. Processors and memory may be supplemented by or incorporated into dedicated logic circuitry.

[0149] While this specification contains numerous specific implementation details, these should not be construed as limiting the scope of any invention or the scope of the claims, but rather are primarily intended to describe features of specific embodiments of a particular invention. Certain features described in the various embodiments herein may also be implemented in combination in a single embodiment. Conversely, various features described in a single embodiment may also be implemented separately in various embodiments or in any suitable sub-combination. Furthermore, while features may function in certain combinations as described above and even initially claimed in this way, one or more features from a claimed combination may be removed from that combination in some cases, and a claimed combination may refer to a sub-combination or a variation thereof.

[0150] Similarly, although the operations are depicted in a specific order in the accompanying drawings, this should not be construed as requiring these operations to be performed in the specific order shown or sequentially, or requiring all illustrated operations to be performed to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above embodiments should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0151] Thus, specific embodiments of the subject matter have been described. Other embodiments are within the scope of the appended claims. In some cases, the actions recited in the claims may be performed in a different order and still achieve the desired result. Furthermore, the processes depicted in the drawings are not necessarily shown in a specific order or sequence to achieve the desired result. In some implementations, multitasking and parallel processing may be advantageous.

[0152] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. An automated testing method, characterized in that, Including the following steps: Obtain a test task queue, which includes at least one test case; the test case is used to execute in a pre-configured test environment of the server under test, and generates a test case log after execution; During the execution of at least one test case based on the test task queue, obtain the test case log generated by the currently executing test case of interest; When the test case log indicates that there is an anomaly in the test environment where the test case of concern is executed, the type of the anomaly is determined based on the test case log; the type of the anomaly includes firmware version anomaly, network environment anomaly, tool dependency anomaly, or file dependency anomaly. An exception context information is generated based on the type of the exception and the use case log, wherein the exception context information includes structured data required to repair the exception; Based on the exception context information, a repair strategy corresponding to the type of exception is executed to repair the test environment; If the repair is successful, the test cases of concern will be added back to the test task queue so that they can be executed again.

2. The automated testing method according to claim 1, characterized in that, Determining the type of the exception based on the use case log includes: Based on a preset regular expression, match the specified field in the test case log; The matched specified field is input into a preset environment anomaly pattern library to obtain the type of the anomaly; the environment anomaly pattern library is used to store the correspondence between several fields and the type of anomaly.

3. The automated testing method according to claim 2, characterized in that, Also includes: If the type of exception corresponding to the specified field is not stored in the environment exception pattern library, the test case of concern will be removed from the test task queue.

4. The automated testing method according to claim 1, characterized in that, The firmware version anomaly indicates that the firmware version information of at least one target component in the server under test is inconsistent with the preset version information in the test environment; the target component is a baseboard management controller, a basic input / output system, or a network interface card; When the type of the anomaly includes a firmware version anomaly, the step of executing a repair strategy corresponding to the type of the anomaly based on the anomaly context information to repair the test environment includes: Configure the firmware of the target component to the firmware corresponding to the preset version information in the test environment.

5. The automated testing method according to claim 1, characterized in that, The network environment anomaly indicates an abnormality in the network environment of at least one target component in the server under test; the target component is a baseboard management controller, a basic input / output system, or a network interface card (NIC). When the type of anomaly includes network environment anomaly, the step of executing a repair strategy corresponding to the type of anomaly based on the anomaly context information to repair the test environment includes: Configure the IP address of the target component, and / or restart the network service function of the target component, and obtain the repair results; If the repair result indicates that the repair has failed, the firmware of the target component will be configured to the firmware corresponding to the preset version information in the test environment.

6. The automated testing method according to claim 1, characterized in that, The tool dependency anomaly indicates that the test environment lacks the tools required to execute the test cases of interest; the file dependency anomaly indicates that the test environment lacks the files required to execute the test cases of interest. When the type of the exception includes a tool dependency exception or a file dependency exception, the step of executing a repair strategy corresponding to the type of the exception based on the exception context information to repair the test environment includes: Obtain the missing tools or files, and configure the tools or files into the test environment of the server under test.

7. The automated testing method according to claim 1, characterized in that, Also includes: Before the test task queue is executed, the firmware version information of at least one target component is detected by a preset out-of-band management protocol, and the network connectivity status of the at least one target component is detected by a preset network detection command. If the firmware version information of the target component is found to be inconsistent with the preset version information in the test environment, the type of the anomaly is determined to include firmware version anomaly. If an abnormal network connectivity status of the target component is detected, the type of the abnormality is determined to include network environment abnormality; An exception context information is generated based on the type of the exception, and a repair strategy corresponding to the type of the exception is executed based on the exception context information to repair the test environment.

8. An automated testing device, characterized in that, include: The test case management module is used to obtain the test task queue, which includes at least one test case. The test cases are used to execute in a pre-configured test environment of the server under test, and test case logs are generated after execution; The runtime detection module is used to obtain the test case logs generated by the currently executing test cases of interest during the execution of at least one test case based on the test task queue. When the test case log indicates that there is an anomaly in the test environment where the test case of concern is executed, the type of the anomaly is determined based on the test case log; the type of the anomaly includes firmware version anomaly, network environment anomaly, tool dependency anomaly, or file dependency anomaly. An exception repair module is used to generate exception context information based on the type of the exception and the test case log, and to execute a repair strategy corresponding to the type of the exception based on the exception context information to repair the test environment; wherein, the exception context information includes structured data required to repair the exception; The test case management module is also used to add the test cases of concern back to the test task queue so that they can be executed again if the repair is successful.

9. The automated testing device according to claim 8, characterized in that, It also includes an active detection module; The active detection module is used to detect the firmware version information of at least one target component through a preset out-of-band management protocol, and to detect the network connectivity status of the at least one target component through a preset network detection command. If the firmware version information of the target component is found to be inconsistent with the preset version information in the test environment, the type of the anomaly is determined to include firmware version anomaly. If an abnormal network connectivity status of the target component is detected, the type of the abnormality is determined to include network environment abnormality; The anomaly repair module is further configured to generate anomaly context information based on the anomaly type, and execute a repair strategy corresponding to the anomaly type based on the anomaly context information to repair the test environment.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by a processor, it implements the steps of the method according to any one of claims 1-7.

Images