An interface automation testing method and system

Abstract

This disclosure relates to an automated interface testing method and system, comprising: sequentially executing test steps in test cases for a target interface to obtain a first response result for each test step; when executing any target test step in the test cases, independently and in parallel with the execution of the target test step, initiating at least one verification operation to the endpoint involved in the target test step to obtain a second response result including each verification operation; the type of verification operation is dynamically determined based on the characteristics of the target test step; for each target test step, obtaining a test result based on the consistency comparison between the first and second response results of the target test step; wherein, when the test result indicates a test failure, the test result at least identifies the root cause of the failure as an inherent product defect or an occasional environmental problem. This can reduce the false alarm rate and improve the reliability of the test results.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of automation testing, in particular, to an interface automation testing method and system. BACKGROUND

[0002] Hardware management interface is a key way for software to remotely control physical hardware. Taking the Redfish interface widely used in data centers as an example, its implementation runs in the Baseboard Management Controller (BMC) of the server, and is responsible for executing all remote instructions such as power on / off, firmware update, and state monitoring. Therefore, before the deployment of related products, the Redfish interface implementation must be strictly tested to verify whether it meets the standard specifications and fully exposes its potential defects in terms of function, performance, security, and robustness.

[0003] However, in the interface automatic testing method provided by the related art, in the case of interface test failure, it is unable to determine whether the failure reason is an interface itself defect or a test environment problem, resulting in low reliability of the test result. SUMMARY

[0004] To solve the above technical problems, the present disclosure provides an interface automation testing method and system, which can effectively distinguish between product inherent defects and environmental occasional problems to some extent, thereby reducing the false positive rate and improving the reliability of the test result.

[0005] To achieve the above purpose, in a first aspect, the present disclosure provides an interface automation testing method, the method comprising: obtaining at least one test case for a target interface; sequentially executing the test steps in the test case for each test case to obtain a first execution result, the first execution result including a first response result of each test step; when executing any target test step in the test case, independently and in parallel with the execution of the target test step, initiating at least one verification operation to an endpoint involved in the target test step to obtain a second execution result, the second execution result including a second response result of each verification operation; the type of the verification operation is dynamically determined based on the characteristics of the target test step; for each target test step, comparing the consistency of the first response result and the second response result of the target test step to obtain a test result; when the test result indicates a test failure, the test result at least identifies the failure root type as a product inherent defect or an environmental occasional problem.

[0006] In the embodiments of the present disclosure, the single standard execution result of the same operation is compared with the parallel verification result obtained under different dynamically applied external conditions, and the consistency or difference between the two is used to automatically determine whether the root cause of the test failure is a product logic defect or an unstable factor of the external execution environment. For example, when only the second response result is abnormal, it can be identified that the abnormality is caused by an environmental incident or a test logic error; when the first response result and the second response result are both abnormal, it can be determined that the product has an inherent defect. Thus, compared with the existing single-channel assertion method, the embodiments of the present disclosure can effectively distinguish between product inherent defects and environmental incident problems to some extent, thereby reducing the false positive rate and improving the reliability of the test result.

[0007] In some possible implementations, the verification operation includes at least one of the following: a concurrent verification operation, a state verification operation, and an exception verification operation. The concurrent verification operation includes: during execution of the target test step, initiating a concurrent request to an endpoint involved in the target test step to obtain a second response result corresponding to the concurrent request; and the second response result corresponding to the concurrent request is used for comparison with the first response result of the target test step to verify the response content consistency of the target interface under concurrent access. The state verification operation includes: during execution of the target test step, continuously initiating a plurality of query requests to an endpoint involved in the target test step; recording a resource state corresponding to the target interface obtained by the query to obtain a second response result of the state verification operation; and the second response result of the state verification operation is used for comparison with a state transition process in the first execution result of the target test step to verify the behavior consistency of the target test step when triggering the state transition. The exception verification operation includes: during execution of the target test step, initiating a request injected with a preset exception condition to an endpoint involved in the target test step to obtain a second response result of the exception verification operation; and the second response result of the exception verification operation is used for comparison with the first response result of the target test step to determine whether the processing behavior of the target interface under the exception disturbance is consistent with the benchmark execution.

[0008] In this embodiment, by automatically and selectively implementing the concurrent verification, state verification, and exception verification operations during execution of the target test step, the “divergent thinking” and “experience intuition” of the test engineer for complex scenarios (such as high concurrency, state transition, and exception input) are solidified into executable and repeatable test actions. This makes the interface test no longer rely on manual construction of discrete special test scenarios, but can flexibly and accurately embed multi-dimensional deep verification in the basic function test process, thereby systematically improving the coverage rate and problem discovery capability of the automated test.

[0009] In some possible implementation manners, the test result is obtained based on a comparison of the consistency of the first response result and the second response result of the target test step, and the test result comprises: The test conclusion is determined according to at least one of the first verification result, the second verification result, the third verification result and the fourth verification result, the test conclusion comprises test success or test failure, and the test result comprises the test conclusion; In a case where the test conclusion is test failure, the failure root type is determined according to the abnormal verification result; The first verification result is obtained based on whether the consistency of the response fields in the first response result and the second response result is reasonable; the second verification result is obtained based on whether the first response duration in the first response result is within the first time threshold and whether the second response duration in the second response result is within the second time threshold; the third verification result is obtained based on whether the first data content returned by the first concurrent request in the first response result is consistent and whether the second data content returned by the second concurrent request in the second response result is consistent; and the fourth verification result is obtained based on whether the first abnormal status code matches the first error description information in the first response result and whether the second abnormal status code matches the second error description information in the second response result.

[0010] In this embodiment, intelligent verification is performed on at least one of the response field rationality, the response duration compliance, the concurrent data consistency and the abnormal code matching in the first response result and the second response result obtained through parallel verification, so that interface quality can be comprehensively evaluated from multiple aspects such as business logic correctness, performance, concurrent security and protocol compliance. When the test fails, the fault category can be accurately determined based on the specific verification dimension of the abnormality, so as to intelligently distinguish different types of problems such as product inherent defects, environmental occasional problems and test logic errors, thereby improving the accuracy of the test result and the problem positioning efficiency, and reducing the false positive rate.

[0011] In some possible implementation manners, the test result is obtained based on a comparison of the consistency of the first response result and the second response result of the target test step, and the test result comprises: A first self-consistency verification result of the first response result is obtained, wherein the first self-consistency verification result is used to indicate whether the response in the first response result meets expectations; The test result is obtained based on the first self-consistency verification result, the second response result and a comparison analysis of the consistency of the first response result and the second response result.

[0012] In this implementation, the first self-consistency verification result is introduced and included in the final result analysis. When the first self-consistency verification result indicates that it is normal, a credible benchmark premise is established for subsequent consistency comparison analysis, avoiding misleading conclusions based on an incorrect benchmark. At the same time, when the benchmark test itself fails, i.e., the first self-consistency verification result indicates anomalies, cross-analysis with the verification operation results can effectively distinguish between "inherent interface defects" and "sensitive defects that are only exposed under interference such as concurrency / abnormality," thus achieving intelligent preliminary classification of the root cause of the fault.

[0013] In some possible implementations, the method further includes the following when the test concludes as a test failure: For each test case, an association directed graph is constructed based on the baseboard management controller log, the test trace records in the first execution result, and the test trace records in the second execution result, according to the time window and content relevance. Any node in the association directed graph represents a log record in the baseboard management controller log, or represents an interaction record corresponding to an interaction process of initiating a request to the target interface and receiving a response in the test trace records. The edges in the association directed graph represent the association relationship between the nodes at both ends of the edge, and the edge weight represents the association strength between the nodes at both ends of the edge. Input information is fed into a large language model; the input information includes test requirements, first execution result, second execution result, and a directed graph of association. The large language model processes the input information by using a pre-set root cause classification prompt word template. The prompt word template is configured to guide the large language model to perform the following judgments: determine whether the root cause of the fault is an inherent defect of the product of the system to which the target interface belongs, a test agent misjudgment, or an occasional environmental problem; the test results include the fault root cause judgment results output by the large language model.

[0014] In this implementation, by linking the BMC logs and the interaction records corresponding to a single request to and response to the target interface into a relationship graph based on timeline and content relevance, and combining this with the analytical capabilities of a large language model, the root cause of the problem can be directly determined whether it is a defect in the product code, a misjudgment by the test agent, or a problem with the test environment. This enables rapid location of the root cause of the fault, thereby effectively improving the efficiency of troubleshooting.

[0015] In some possible implementations, the method further includes: Analyze the characteristics of each test step in the test case, including at least one of the following: the type of endpoint involved in the test step, the HTTP method, and the operation semantic category; Based on the characteristics of the test steps, determine whether there are any consistency risks that need to be verified in terms of concurrent access, state transition and / or exception handling when the test steps are executed. Based on the assessment of consistency risk, determine the type of verification operation corresponding to the test steps.

[0016] In this implementation, by analyzing the inherent characteristics of the test steps, the system intelligently predicts the potential risks of inconsistencies in the behavior of the target interface during the execution of the test steps, such as concurrent access, state transitions, and / or exception handling. Based on this, the system automatically selects the most suitable verification operation, realizing dynamic, context-based risk-driven testing. This enables the automated testing system to understand the test intent and potential problems, allowing it to autonomously make decisions and adopt the most suitable verification method for different operations based on their inherent risks.

[0017] Secondly, this disclosure provides an interface automated testing system, including an intelligent agent, a first test execution engine, and a second test execution engine; The intelligent agent is used to obtain at least one test case for the target interface; The first test execution engine is used to sequentially execute the test steps in each test case to obtain the first execution result, which includes the first response result of each test step. The second test execution engine is used to, independently and in parallel with the execution of any target test step in a test case, initiate at least one verification operation to the endpoints involved in the target test step to obtain a second execution result, which includes the second response results of each verification operation; the type of verification operation is dynamically determined based on the characteristics of the target test step. The intelligent agent is also used to obtain test results for each target test step based on the consistency comparison of the first response result and the second response result of the target test step; wherein, when the test result indicates that the test has failed, the test result at least identifies the type of failure root cause as an inherent defect in the product or an occasional environmental problem.

[0018] Thirdly, this disclosure provides an electronic device, including: A memory on which computer programs are stored; A processor is used to execute computer programs in memory to implement the interface automation testing method described in the first aspect.

[0019] Fourthly, this disclosure provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the interface automation testing method described in the first aspect.

[0020] Fifthly, this disclosure provides a computer program product, including a computer program that, when executed by a processor, implements the interface automation testing method described in the first aspect.

[0021] Other features and advantages of this disclosure will be described in detail in the following detailed description section. Attached Figure Description

[0022] The accompanying drawings are provided to further illustrate the present disclosure and form part of the specification. They are used together with the following detailed description to explain the present disclosure, but do not constitute a limitation thereof. In the drawings: Figure 1 This is a flowchart illustrating the interface automated testing method provided in the embodiments of this disclosure. Figure 1 ; Figure 2 This is a flowchart illustrating the interface automated testing method provided in the embodiments of this disclosure. Figure 2 ; Figure 3 This is a flowchart illustrating the interface automated testing method provided in the embodiments of this disclosure. Figure 3 ; Figure 4 This is a schematic diagram of the architecture of an interface automated testing system provided in an embodiment of this disclosure; Figure 5 This is a block diagram of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0023] The specific embodiments of this disclosure will be described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are for illustration and explanation only and are not intended to limit this disclosure.

[0024] In related technologies, during the testing of hardware management interfaces such as the Redfish interface, test cases are executed sequentially, and simple assertions are made on the responses of the test cases. The result of these assertions is then used as the final test result. For example, the HTTP status code in the response is checked; if the status code is 200, the test result is considered to have passed; otherwise, the test result is considered to have failed. That is, the test result obtained by this single execution path combined with simple assertions is either a pass or a failure. However, because this single-dimensional information input relies solely on the test result obtained from the sequential execution of test cases, this sequential single-execution path testing mode has limitations: when the test result indicates a failure, it is easy to misjudge test environment problems or test logic errors as inherent product defects, resulting in a high false positive rate and low reliability of the test results.

[0025] In view of this, embodiments of this disclosure provide an automated interface testing method. This method performs a refined comparative analysis of the results of a single standard execution of the same operation with parallel verification results obtained under dynamically applied different external conditions. It automatically determines whether the root cause of test failure stems from inherent product logic defects or from unstable external execution environments by utilizing the consistency or difference patterns between the two. For example, when only the parallel verification results show an anomaly, it can be identified that the anomaly is caused by an occasional environmental problem or a test logic error; when both the standard execution results and the parallel verification results show the same anomaly, it can be determined to be an inherent product defect. Therefore, it can effectively distinguish between inherent product defects and occasional environmental problems to a certain extent, thereby reducing the false alarm rate and improving the reliability of test results.

[0026] Figure 1 This is a flowchart illustrating the interface automated testing method provided in the embodiments of this disclosure. Figure 1 This method can be executed by an electronic device, which can be a terminal device or a server. That is, the method can be executed by the terminal device or the server alone, or by the terminal device and the server working together.

[0027] Taking the server as the execution entity and the Redfish interface as an example, such as Figure 1 As shown, the method may include the following steps.

[0028] S101. Obtain at least one test case for the target interface.

[0029] The target interface can refer to the hardware management interface to be tested. For example, the target interface could be the Redfish interface. Redfish is a standardized hardware management interface based on the HTTP protocol, primarily used for out-of-band management of data center hardware such as servers and storage devices. Through a RESTful API, it allows administrators to remotely monitor and control core hardware information such as power status, temperature, and firmware version using JSON-formatted data, achieving low-level hardware management independent of the host operating system. The Redfish interface supports standard HTTP methods.

[0030] In a possible implementation, after obtaining the test requirements for the target interface, these can be transformed into structured test intents. A comparative analysis of the current and previous versions of the Redfish Schema file is performed to identify changes such as newly added or modified interfaces, deleted interfaces, and modified fields, generating a Schema change list. Then, based on the test intents and the Schema change list, the generation of test cases is determined. Based on the defined test cases, complete test scripts are automatically generated. For example, the correct Redfish interface is called through endpoint mapping, parameter data is generated following Schema constraints, prerequisite dependencies are automatically handled, response and state changes are verified, and finally, an executable automated test script is output. The test requirements can refer to the specific verification requirements extracted from the Redfish official standard documentation or requirements document to guide test case generation.

[0031] It is understood that the code generated according to the testing requirements in this embodiment is essentially machine-executable code, i.e., the test scripts for the test cases, and the code executed subsequently is also the test scripts for the test cases. This embodiment does not specifically limit the method for generating the test scripts for the test cases.

[0032] S102. For each test case, execute the test steps in the test case sequentially to obtain the first execution result, which includes the first response result of each test step.

[0033] In a possible implementation, after at least one test case corresponding to the test requirement is generated, the at least one test case is executed sequentially. Each test case includes at least one test step; therefore, a test case can be understood as a sequence of test steps. The first execution result and execution trajectory of the entire sequential execution process can be recorded. The first execution result may include the execution time of each test step, the identifier of the object under test (such as BMC IP, firmware version number, etc.), the first response result, and indexes pointing to the detailed execution trajectory of each test step.

[0034] The first response result may include response fields, response duration, and status code. The execution trajectory can be understood as a data log that records the complete test interaction process and results of a test step in chronological order. For example, the execution trajectory corresponding to any test step may include a request sequence (such as the request headers, request body, and target endpoint of each HTTP request triggered by that test step), a response sequence (such as the response headers, response body, status code, and response duration of each HTTP request triggered by that test step), and precise timing (such as the millisecond-level timestamps for each HTTP request initiation and reception). The execution trajectory can be used for subsequent association graph construction, which will not be elaborated upon here.

[0035] The sequential execution of the at least one test case can be understood as the sequential execution of the test scripts used to implement the at least one test case.

[0036] S103. When executing any target test step in a test case, independently and in parallel with the execution of the target test step, initiate at least one verification operation to the endpoint involved in the target test step to obtain a second execution result, which includes the second response result of each verification operation; the type of verification operation is dynamically determined based on the characteristics of the target test step.

[0037] The characteristics of the target test step may include, but are not limited to, the type of endpoint involved in the target test step, the HTTP method, and the operation semantic category. The operation semantic category may include data read operations, data write operations, or state change operations, etc.

[0038] The verification operations can include at least one of the following: concurrent verification operations, state verification operations, and exception verification operations. For example, the result of a concurrent verification operation can be used to verify whether the response of the target interface in a concurrent scenario is consistent with the baseline execution result. The result of a state verification operation can be used to verify the consistency of the target interface's behavior when triggering state transitions while processing requests for the target test step; that is, to verify whether the state transition behavior of the target interface is consistent with the baseline execution when processing requests for the target test step. The result of an exception verification operation can be used to determine whether the target interface's processing behavior under abnormal disturbances is consistent with the baseline execution.

[0039] In a possible implementation, after obtaining each test case, the test steps within each test case can be analyzed. If it is determined that any test step has a consistency risk that needs to be verified during execution, then that test step is marked as a target test step, and at least one verification operation corresponding to that target test step is identified. When sequentially executing any target test step, on the one hand, the target test step is executed; on the other hand, independently and in parallel with the execution of the target test step, at least one corresponding exploratory verification operation is initiated to the endpoints involved in the target test step. During execution, the second execution result of the verification operation and the execution trajectory of the execution process are also recorded.

[0040] For example, if the target test step is to initiate a GET request, multiple verification operations such as concurrent verification (e.g., initiating 5 identical GET requests simultaneously to verify concurrency security) and status verification can be performed independently and in parallel with the execution of initiating a GET request, thus verifying the correctness of the interface behavior from different dimensions.

[0041] The present disclosure does not limit the implementation method of the verification operation corresponding to each target test step.

[0042] In some embodiments, a mapping relationship between request methods and verification operation types can be established in advance based on experience. For example, GET requests can be mapped to concurrent verification operations and exception verification operations, while PUT / DELETE requests can be mapped to status verification operations, concurrent verification operations, and exception verification operations. Based on this mapping relationship, one or more types of verification operations can be determined for each test case. This approach is simple to implement and requires no complex algorithms.

[0043] In some embodiments, test case characteristics (such as request method, resource type, historical failure rate, etc.) can be used as input to determine the combination of verification operations to be performed through preset rules or models. For example, for modification operations of critical resources, concurrent verification operations, status verification operations, and exception verification operations can be enabled simultaneously for comprehensive verification; for read-only operations, only concurrent verification operations and exception verification operations can be enabled.

[0044] In some embodiments, the characteristics of each test step in the test case can be analyzed, including at least one of the following: the type of endpoint involved in the test step, the HTTP method, and the operation semantic category; based on the characteristics of the test step, it is determined whether there are any consistency risks that need to be verified in terms of concurrent access, state transition, and / or exception handling when the test step is executed; and based on the determination result of the consistency risks, the type of verification operation corresponding to the test step is determined.

[0045] Among them, consistency risk can refer to the potential possibility that the behavior or response result of the target interface may deviate from the baseline (or expected) behavior of the target interface when processing the request of the test step due to the influence of external conditions (such as concurrency, state change and / or abnormal input).

[0046] Among them, the endpoint type can be used to indicate the sharing and competition attributes of resources, the HTTP method can be used to indicate the basic intent of the operation, and the operation semantic category can be used to define the actual impact of the operation at the business level.

[0047] For example, when a test step involves performing a write operation (such as PATCH) on a singleton or shared resource endpoint, multiple concurrent requests may compete to modify the same resource at the same time, causing the final system state or return response to depend on the uncontrollable execution sequence. In this case, it can be determined that there is a risk of concurrent access. Thus, the verification operation of this test step can be determined as a concurrent verification operation to verify whether the response of the target interface under concurrent pressure is consistent with that of a single execution.

[0048] When a test step is identified as a state-changing operation (e.g., sending a POST request to the / Actions / Reset endpoint to restart the server), this operation instructs the BMC to change the operating state of the hardware (e.g., the server). Correct state changes must follow a fixed sequence (e.g., first changing from "power on" to "power off," then back to "power on"). This indicates a potential risk of state transition errors. If the sequence is incorrect (e.g., getting stuck after directly changing from "power on" to "power off"), or if the intermediate states are incorrect, it indicates a serious product defect. Therefore, this test step can be classified as a state verification operation.

[0049] When test steps involve boundary value parameters, invalid input, or occur within critical business process chains, the interface response lacks a single, definitive "correct" benchmark. For example, an invalid password login might return either a 400 Bad Request or a 401 Unauthorized response, depending on the product design, and cannot be simply asserted as right or wrong as in normal functional testing. In such cases, an anomaly handling risk can be identified. Therefore, the verification operation in this test step can be designated as an anomaly verification operation to verify whether the interface's behavior under abnormal disturbances still maintains consistency with benchmark execution or design specifications.

[0050] In this embodiment of the disclosure, by analyzing the inherent characteristics of the test steps, the system intelligently predicts the potential risks of inconsistencies in the behavior of the target interface during the execution of the test steps, such as concurrent access, state transitions, and / or exception handling. Based on this, the system automatically selects the most suitable verification operation, thereby realizing dynamic, context-based risk-driven testing. This enables the automated testing system to understand the test intent and potential problems, and thus can autonomously make decisions and adopt the most suitable verification method for verification based on the inherent risks of different operations.

[0051] S104. For each target test step, the test result is obtained by comparing the consistency of the first response result and the second response result of the target test step; wherein, when the test result indicates that the test has failed, the test result shall at least identify the type of failure root cause as an inherent defect of the product or an occasional environmental problem.

[0052] In a possible implementation, for each target test step, if the first response result (e.g., status code 200, response body A) and the second response result (e.g., status code 200, response body A) are completely consistent in a predetermined dimension, the test is considered to have passed. Otherwise, it is considered to have failed.

[0053] For example, when inconsistency manifests as the first and second response results showing the same or logically related anomalies (e.g., the first response result is 500, and multiple concurrent verification requests also return 500 or 409 conflicting results), it can be identified as a product logic defect because the problem is consistently reproduced under both execution methods. When inconsistency manifests as the first response result being normal while the second response result is abnormal (e.g., the second response result is 200, but some requests in the concurrent verification operation return 503 service unavailable or timeout), it can be identified as an intermittent environmental issue.

[0054] In this embodiment, at least one test case is obtained for the target interface. Then, for each test case, the test steps in the test case are executed sequentially to obtain a first execution result, which includes a first response result for each test step. When executing any target test step in the test case, at least one verification operation is initiated independently and in parallel with the execution of the target test step to the endpoint involved in the target test step to obtain a second execution result, which includes a second response result for each verification operation; the type of verification operation is dynamically determined based on the characteristics of the target test step. Subsequently, for each target test step, a test result is obtained based on the consistency comparison between the first response result and the second response result of the target test step; wherein, when the test result indicates a test failure, the test result at least identifies the root cause of the failure as an inherent defect in the product or an occasional environmental problem. That is, this embodiment of the disclosure performs a refined comparative analysis of the single standard execution result of the same operation with the parallel verification results obtained under dynamically applied different external conditions, and uses the consistency or difference patterns between the two to automatically determine whether the root cause of the test failure is a logical defect in the product itself or caused by unstable factors in the external execution environment. For example, when only the second response result shows an anomaly, the anomaly can be identified as being caused by an occasional environmental problem or a test logic error; when both the first and second response results show the same anomaly, it can be determined to be an inherent defect in the product. Therefore, compared to existing single-channel assertion methods, the embodiments of this disclosure can effectively distinguish between inherent product defects and occasional environmental problems to a certain extent, thereby reducing the false alarm rate and improving the reliability of test results.

[0055] Understandably, the test results can also include third execution results of non-target test steps, which may include third response results of each non-target execution step. The test results can also include simple assertion results of the third response results. In this way, the type of failure root cause can be determined based on a consistency comparison between the first and second response results and the simple assertion results, further reducing the false positive rate.

[0056] In some possible implementations, the verification operation includes at least one of the following: concurrent verification operation, state verification operation, and exception verification operation.

[0057] In this embodiment of the disclosure, the concurrent verification operation may include: during the execution of the target test step, initiating a concurrent request to the endpoint involved in the target test step to obtain a second response result corresponding to the concurrent request; the second response result corresponding to the concurrent request may be used to compare with the first response result of the target test step to verify the consistency of the response content of the target interface under concurrent access.

[0058] For example, during the execution of the target test step, multiple identical requests are initiated concurrently to the same endpoint operated on by that step, independently and in parallel (e.g., using a thread pool to initiate N GET requests simultaneously). The responses to these concurrent requests are recorded, forming a second response result. Subsequently, the response of each concurrent request (or its main fields, status code) is compared one by one with the first response result of the target test step to determine whether the response content returned by the interface under concurrent access conditions is consistent with that of a single execution. For example, if all concurrent responses are consistent with the first response result, the interface is considered to behave consistently under concurrent read scenarios. If some concurrent responses are inconsistent with the first response result, it indicates a behavioral deviation under concurrent conditions. In this case, combined with the correctness of the first response result (indicating normal single-execution functionality), it can be inferred that the inconsistency is more likely caused by occasional environmental interference. If all concurrent responses are consistent with each other, but consistently show definite differences from the first response result, it indicates that the concurrent conditions may have triggered some systematic state observation or logical processing error within the interface, suggesting that this phenomenon points to an inherent logical defect in the product or a logical defect in the verification operation.

[0059] Optionally, the concurrent verification operation can also be used to verify the concurrent security of the target interface.

[0060] In one example, during the execution of the target test step, N threads are started independently and in parallel using ThreadPoolExecutor. Each thread continuously sends M read requests to the same resource endpoint of the target interface, and the response returned by each thread for each read request is recorded. This allows us to determine whether the responses returned by the N read requests are consistent at any given time, and whether the changes in the responses returned by the M consecutive requests from each thread are logical and expected. If at any time the responses returned by the N read requests are inconsistent, and / or the changes in the responses returned by the M consecutive requests from any thread are illogical and unexpected, then a concurrency security vulnerability exists in the target interface. The above execution process, judgment process, and judgment results can be recorded in the second execution result to improve the interpretability of subsequent test results.

[0061] In another example, during the execution of the target test step, execution can be performed independently and in parallel. This can be achieved by starting multiple read threads and one write thread simultaneously, allowing read and modify requests to occur concurrently, and recording the responses returned by these requests. If any read request's response contains missing fields, invalid values, or corrupted data structures, a static data vulnerability is identified in the target interface. If all read requests at the same time return inconsistent data, the target interface's synchronization or cache consistency mechanism is found to be flawed. If a write request fails due to concurrent reads and returns a non-standard conflict error or a system-level error, the target interface's concurrency conflict handling logic is found to be flawed.

[0062] In another example, during the execution of the target test step, multiple write threads can be started simultaneously, independently and in parallel with the execution of the target test step. If a write request returns a non-standard conflict error or a system-level error, it is determined that the concurrent write conflict handling logic of the target interface is defective.

[0063] In another example, during the execution of the target test step, the execution of multiple clients calling the same transactional operation interface independently and in parallel with the target test step can simulate the operation of multiple clients calling the same transactional operation interface. If multiple concurrent calls ultimately only create one task, or the returned task identifiers are duplicated or confused, it is determined that the transaction manager of the software system to which the target interface belongs has a concurrency vulnerability and cannot properly isolate requests under concurrent pressure.

[0064] In another example, during the execution of the target test step, a set of concurrent operations with circular dependencies can be executed independently and in parallel with that target test step. A circular dependency can be understood as at least two concurrent operations forming a circular blocking state where they mutually wait for and hold onto system resources, preventing all related operations from proceeding. If all or part of the requests from these at least two concurrent operations enter a prolonged unresponsive state until the test framework is forcibly interrupted due to a timeout, then the software system to which the target interface belongs is deemed to have a deadlock risk.

[0065] In this embodiment of the disclosure, the state verification operation may include: during the execution of the target test step, continuously initiating multiple query requests to the endpoints involved in the target test step; recording the resource status corresponding to the target interface obtained by the query, and obtaining a second response result of the state verification operation; the second response result of the state verification operation is used to compare with the state transition process in the first execution result of the target test step, so as to verify the consistency of the behavior of the target test step when triggering the state transition.

[0066] For example, during the execution of the target test step (such as a PATCH request modifying resource status), multiple query requests are continuously sent to the resource endpoint (e.g., frequent GET requests) during the triggering of its state transition. The resource status obtained from each query (such as status code, specific field values) is recorded, forming a second response result characterizing the state change process. Then, the recorded state change sequence (i.e., the second response result) is compared with the state transition process recorded in the first execution result of the target test step (such as the expected on->off->on change sequence) to verify the consistency of the Redfish interface's behavior when triggering state transitions.

[0067] Optionally, the state verification operation can also be used to verify the idempotency of the Redfish interface.

[0068] In one possible implementation, during the execution of the target test step, at least two identical insecure requests are continuously initiated to the target interface, independently and in parallel with the execution of the target test step. The at least two identical insecure requests are PUT requests or DELETE requests containing the same payload. The operation response of each insecure request is recorded, as well as the resource status of the target interface obtained by a GET request after each operation response. The static data portions in the operation responses of the at least two insecure requests are compared with the static data portions in the resource status of the target interface obtained by the GET request. If the static data portions in the operation responses are consistent and the static data portions in the resource status are consistent, then the operation corresponding to the insecure request of the target interface is determined to be idempotent.

[0069] In this context, "same payload" can be understood as "same request header and request body".

[0070] The static data portion of the operation response can include all data other than dynamic fields such as timestamps and request IDs. Similarly, the static data portion of the resource status can include all data other than dynamic fields such as timestamps, version identifiers, and automatically generated metadata fields.

[0071] Optionally, if the static data portion in the operation response is inconsistent and / or the static data portion in the resource status is inconsistent, it is determined that the operation corresponding to the insecure request of the target interface is not idempotent, and the corresponding difference information is recorded so that the cause can be analyzed based on the difference information later.

[0072] In another possible implementation, before initiating the insecure request, a snapshot of the original resource state corresponding to the target interface is obtained via a GET request. After executing the insecure request and completing idempotency verification, the original resource state snapshot is rewritten to the target interface via a PUT request. The consistency between the resource state after rollback via the PUT request and the original resource state snapshot in the static data portion is verified. If they are consistent, it can be determined that the target interface supports complete state rollback and has good test environment recoverability. If they are inconsistent, it can be determined that there is residual state in the target interface or the PUT request operation is incomplete. The corresponding difference information is recorded so that the cause can be analyzed based on the difference information later.

[0073] In this embodiment of the disclosure, the anomaly verification operation may include: during the execution of the target test step, initiating a request to the endpoint involved in the target test step to inject a preset anomaly condition, so as to obtain a second response result of the anomaly verification operation; the second response result of the anomaly verification operation is used to compare with the first response result of the target test step to determine whether the processing behavior of the target interface under anomaly disturbance is consistent with the baseline execution.

[0074] In one example, during the execution of a target test step (such as a resource creation request), a request injecting a pre-defined abnormal condition is sent to the same endpoint involved. This request embeds simulated abnormal perturbations (e.g., malformed JSON fields, excessively long string parameters, or instructions simulating network latency) on top of the normal business load. The interface's response to such abnormal requests is recorded, forming a second response result. Then, the response to the abnormal request (the second response result) is compared with the first response result of the target test step under normal conditions to determine whether the interface's handling behavior under abnormal perturbations is consistent with the baseline execution. For example, if the target interface returns an expected, standardized error response (such as a correct 4xx status code and clear error message) without causing a system-wide crash or data corruption, this indicates that the interface has robust exception handling logic. If the target interface returns non-standard, inconsistent, or ambiguous error messages (such as returning a 500 Internal Server Error for a business logic error), or if the abnormal request causes unexpected changes in the normal business data state, this indicates that the target interface has behavioral deviations. If the target interface does not return a valid response to an abnormal request (such as connection timeout or service unresponsiveness), or causes a service process exception, it indicates that the target interface lacks basic defensive programming or resource isolation mechanisms.

[0075] In another example, pre-defined exception conditions can be injected into the code during the execution of concurrent verification operations and / or status verification operations to verify the robustness of the system to which the Redfish interface belongs under abnormal scenarios.

[0076] The preset abnormal conditions may include, but are not limited to, network latency, forced concurrency conflicts, and resource lock contention. For example, network latency can be injected using the time.sleep function; forced concurrency conflicts can be multiple threads simultaneously performing write operations on the same field of the same resource; and resource lock contention can be initiating short-term high-concurrency requests to the same resource.

[0077] In this embodiment of the disclosure, by automatically and selectively implementing concurrent verification, state verification, and anomaly verification operations during the execution of the target test steps, the "divergent thinking" and "experience-based intuition" of test engineers for complex scenarios (such as high concurrency, state transitions, and abnormal inputs) are solidified into executable and repeatable test actions. This allows interface testing to no longer rely on manually constructing discrete, specialized test scenarios, but instead enables the flexible and precise embedding of multi-dimensional deep verification into the basic functional testing process, thereby systematically improving the coverage and problem detection capabilities of automated testing.

[0078] It is understood that more types of verification operations may be included, but this disclosure does not limit the scope of the embodiments.

[0079] Figure 2 This is a flowchart illustrating the interface automated testing method provided in the embodiments of this disclosure. Figure 2 .like Figure 3 As shown, step S104 above may include the following steps.

[0080] S201. Determine the test conclusion based on at least one of the first verification result, the second verification result, the third verification result, and the fourth verification result; the test conclusion includes whether the test was successful or failed, and the test result includes the test conclusion.

[0081] S202. If the test conclusion is a test failure, determine the type of failure root cause based on the verification results of the anomaly. The first verification result is obtained based on whether the consistency of the response fields in the first response result and the second response result is reasonable. The second verification result is obtained based on whether the first response duration in the first response result is within the first time consumption threshold and whether the second response duration in the second response result is within the second time consumption threshold. The third verification result is obtained based on whether the first data content returned by the first concurrent request in the first response result is consistent and whether the second data content returned by the second concurrent request in the second response result is consistent. The fourth verification result is obtained based on whether the first exception status code and the first error description information in the first response result match and whether the second exception status code and the second error description information in the second response result match.

[0082] In a possible implementation, for each target test step, a corresponding validator is selected to obtain a verification result based on the first response result and the corresponding second response result. The test conclusion is then determined based on the selected validator. The validator may include a first validator for obtaining the first verification result, a second validator for obtaining the second verification result, a third validator for obtaining the third verification result, and a fourth validator for obtaining the fourth verification result. The test result may include the test conclusion corresponding to each test step.

[0083] For example, for any target test step, consider selecting a first validator, a second validator, and a third validator. If the first verification result indicates that the consistency of the response fields in the first and second response results is reasonable, the second verification result indicates that the first response duration is within a first time consumption threshold and the second response duration is within a second time consumption threshold, and the third verification result indicates that the first data content returned by the first concurrent request (i.e., the target test step) is consistent and the second data content returned by the second concurrent request (i.e., the concurrent verification operation) is consistent, then the test conclusion is determined to be a successful test. If the first verification result indicates that the consistency of the response fields in the first and second execution results is unreasonable, the second verification result indicates that the first response duration is not within the first time consumption threshold and / or the second response duration is not within the second time consumption threshold, and / or the third verification result indicates that the first data content returned by the first concurrent request is inconsistent and / or the second data content returned by the second concurrent request is inconsistent, then the test conclusion is determined to be a failed test.

[0084] The consistency of response fields in the first and second execution results is reasonable. This can be understood as the difference in response fields between the first and second response results being consistent with expectations for the same target test step. For example, if the target test step is to send one GET request to the endpoint, the corresponding first response result is status code 200 and resource data. If the corresponding verification operation is to simultaneously send five identical GET requests to the same endpoint, and the corresponding second response result is status code 200 with consistent resource data, then this consistency is reasonable. Conversely, the inconsistency of response fields between the first and second execution results is unreasonable. This can be understood as an unreasonable difference between the first and second response results for the same target test step. For example, if the first response result returns status code 200 and resource data A for the GET request, while the second response result shows inconsistent resource data (500 status code or unequal resource data when the concurrent verification operation executes the same GET request), then this difference is unreasonable and may indicate a concurrency security issue with the interface.

[0085] In this context, a validator can be understood as executable code. This disclosure does not specifically limit the method of selecting a validator. For example, a validator can be selected through a test scenario, which may include, but is not limited to, performance testing scenarios, concurrency scenarios, and anomaly testing scenarios. For instance, in anomaly testing scenarios, a first validator and a fourth validator can be selected; in concurrency scenarios, a third validator is selected; and in performance testing scenarios, both a first validator and a second validator can be selected.

[0086] In a possible implementation, if the test conclusion is that the test failed, the fault category is determined based on the abnormal verification result. The abnormal verification result can be understood as the verification result that led to the test conclusion being that the test failed.

[0087] For example, consider selecting a first validator and a second validator. If the first verification result indicates that the consistency of the response fields in the first and second response results is unreasonable, and the second verification result indicates that the first response time is within a first time consumption threshold and the second response time is within a second time consumption threshold, and the response fields of the first response result meet expectations while the response fields of the second response result do not meet expectations, then the first verification result is an abnormal verification result, and the fault category could be an occasional environmental problem or a test logic error. If neither the response fields in the first nor the second response result meet expectations, then the fault category could be an inherent product defect.

[0088] For example, consider the selection of the fourth validator. If the fourth verification result indicates that the first exception status code is "404" and the first error description is "Page Not Found," meaning the first exception status code and the first error description match, and the second exception status code is "401" and the second error description is also "Page Not Found," but in reality, the description of "401" should be "Unauthorized," meaning the second exception status code and the second error description do not match. Therefore, the second error description is an abnormal verification result, and the fault category could be an intermittent environmental problem or a test logic error.

[0089] In this embodiment, intelligent verification of at least one dimension of the response field rationality, response duration compliance, concurrent data consistency, and exception code matching in the first response result and the second response result obtained through parallel verification allows for a comprehensive evaluation of interface quality from multiple levels, including business logic correctness, performance, concurrency security, and protocol compliance. When a test fails, the fault category can be accurately determined based on the specific exception verification dimension, thereby intelligently distinguishing different types of problems such as inherent product defects, occasional environmental issues, and test logic errors. This improves the accuracy of test results and the efficiency of problem localization, while reducing the false positive rate.

[0090] In some possible implementations, step S104 above may include: Obtain a first self-consistency verification result for the first response result, wherein the first self-consistency verification result is used to indicate whether the response in the first response result meets expectations; based on the first self-consistency verification result, the second response result, and a consistency comparison analysis between the first response result and the second response result, obtain the test result.

[0091] The results of the first self-consistency verification can serve as one of the reference bases for subsequent cross-validation analysis.

[0092] In one possible implementation, it can be first determined whether the response in the first response result meets expectations. If the response in the first response result meets expectations, it indicates that the benchmark of the first response result is reliable. At this point, the consistency comparison analysis results between the first response result and the second response result can be used as the main basis for generating the test results. The first self-consistency verification result can serve as a logical premise for confirming the validity of the consistency comparison analysis, thereby improving the reliability of the consistency comparison analysis conclusions.

[0093] For example, when the first response result is as expected but the second response result is abnormal, it can be preliminarily judged as an occasional environmental problem or a test logic problem.

[0094] In another possible implementation, even if the response in the first response is not as expected, a consistency comparison analysis is still performed, and the root cause type of failure is determined in conjunction with the first response result. When both the first and second response results show the same anomaly, it can be identified as an inherent product defect; when the first response result is abnormal while the second response result is normal, it can also be used as a basis for judging an occasional environmental problem. By comparing the parallel first and second response results, product defects and occasional environmental problems can be effectively distinguished, thereby improving the accuracy of the test results.

[0095] For example, if the first self-consistency verification result indicates that the first response result does not meet expectations (i.e., the baseline function fails), and the second response result also continues to fail under similar conditions as the first response result, with highly consistent failure characteristics (such as error codes and error messages), it indicates an inherent, deterministic core functional defect of the target interface, unrelated to external interference such as concurrent verification operations or state verification operations. The first self-consistency verification result directly identifies the main fault in this case. Conversely, if the second response result does not match the first response result, exhibiting intermittent success, failure, or varied failure modes, and this variability is clearly related to the specific applied interference (such as concurrent write operations or abnormal parameters), it indicates that the target interface function is unstable, its behavior depends on external concurrency, state, or abnormal conditions, exposing deep logical defects such as race conditions or resource synchronization. In this case, the first self-consistency verification result (indicating baseline anomaly) and the consistency comparison analysis results of the first and second response results (indicating behavior changes with interference) work together to achieve accurate identification and classification of this type of non-deterministic, interference-sensitive defect.

[0096] In this embodiment, a first self-consistency verification result is introduced and included in the final result analysis. When the first self-consistency verification result indicates normality, a credible benchmark premise is established for subsequent consistency comparison analysis, avoiding misleading conclusions based on an incorrect benchmark. At the same time, when the benchmark test itself fails, i.e., the first self-consistency verification result indicates anomalies, cross-analysis with the verification operation results can effectively distinguish between "inherent interface defects" and "sensitive defects exposed only under interference such as concurrency / abnormality," achieving intelligent preliminary classification of the root cause of the fault.

[0097] In some implementations, in order to further improve the accuracy of test results and diagnostic efficiency, the root cause of the fault can be further confirmed after performing the above step S202, so that the confirmed root cause of the fault is also included in the final test results.

[0098] For example, Figure 3 This is a flowchart illustrating the interface automated testing method provided in the embodiments of this disclosure. Figure 4 .like Figure 4 As shown, if the test result indicates that the test failed, the method may further include: S301. For each test case, construct an association directed graph based on the baseboard management controller log, the test trace records in the first execution result, and the test trace records in the second execution result, according to the time window and content relevance. Any node in the association directed graph represents a log record in the baseboard management controller log, or represents an interaction record corresponding to an interaction process in the test trace records that initiates a request to the target interface and receives a response. The edges in the association directed graph represent the association relationship between the nodes at both ends of the edge, and the edge weight represents the association strength between the nodes at both ends of the edge.

[0099] The Baseboard Management Controller (BMC) is a dedicated microcontroller chip on the server motherboard responsible for hardware monitoring and management. The BMC can be used in Redfish interface testing to verify whether test commands are actually executed at the hardware level. For example, when the Redfish interface returns "power off successfully," the BMC log can confirm whether the power module has actually been powered off. The BMC log may contain at least one of the following: power switching event records, temperature alarm records, hardware error records, and firmware operation records. The BMC log provides verification evidence of the true hardware execution in Redfish interface testing.

[0100] The test trace records in the first execution result can be understood as the key operation records in the detailed execution trajectory of each test step, pointed to by each index in the first execution result. For example, in the execution trajectory of a "create virtual machine" step, multiple key operation records such as "send authentication request to API gateway", "call resource orchestration service", and "wait for storage volume preparation" can be extracted.

[0101] The relevance may include, but is not limited to, semantic similarity and / or business logic similarity.

[0102] In a possible implementation, each interaction record in the test tracking log includes the corresponding time period information, and each log record in the BMC log also has a corresponding timestamp. Based on a preset time window, the temporal correlation between records is determined. For example, for a log record, it is determined whether its timestamp falls within the time period of a certain interaction record; for two log records, it is determined whether their timestamp difference is within the time window; for two interaction records, it is determined whether their time periods overlap. Subsequently, the relevance of time-related record pairs in terms of content is calculated, and record pairs with a relevance greater than a preset threshold are associated to construct a directed graph of association.

[0103] The size of the time window can be set according to the actual scenario, and this embodiment does not limit it.

[0104] S302. Input the input information into the large language model; the input information includes test requirements, first execution result, second execution result, and associated directed graph.

[0105] Optionally, the input information may also include a first exception response from a separate first execution result and a second exception response from a separate second execution result, so that the subsequent large language model can quickly identify the exception.

[0106] An abnormal response can be understood as a response that does not meet expectations. For example, for a certain request, the status code in a normal response should be 200, but the actual response returned has a status code of 400. This actual response is considered an abnormal response.

[0107] S303. Drive the large language model to process the input information through the preset root cause classification prompt word template. The prompt word template is configured to guide the large language model to perform the following judgment: determine whether the root cause of the fault is an inherent defect of the product of the system to which the target interface belongs, a test agent misjudgment, or an occasional environmental problem; the test result includes the fault root cause judgment result output by the large language model.

[0108] In one possible implementation, input information is filled into a pre-defined root cause classification prompt word template to obtain prompt words. These prompt words are then input into a large language model to obtain the fault root cause judgment result output by the large language model. The pre-defined root cause classification prompt word template may also include output requirements, which can include instructions for the large language model to make judgments in three categories: "product defect," "test agent misjudgment," and "test environment problem." This allows the large language model to output the fault root cause judgment result based on the input information.

[0109] In this embodiment, by associating the BMC logs and the interaction records corresponding to a single interaction process of initiating a request and receiving a response to the target interface into a correlation graph based on timeline and content relevance, and combining this with the analytical capabilities of a large language model, the root cause of the problem can be directly determined whether it is a defect in the product code, a misjudgment by the test agent, or a problem with the test environment. This enables rapid location of the root cause of the fault, thereby effectively improving the efficiency of fault diagnosis.

[0110] Optionally, the input information may also include system component context information. Correspondingly, the output requirements may also include requirements for instructing the large language model to locate the specific component causing the fault, thereby driving the large language model to output the fault root cause judgment result and the fault component location result based on the input information.

[0111] System component context information refers to a structured description of all hardware and software elements that may be related to a fault in a test scenario, and its purpose is to define the scope of diagnosis. For example, system component context information may include: relevant Redfish resource paths, involved hardware components (such as BMC, PMU, etc.), interacting software service modules, and their dependencies.

[0112] Optionally, the input information may also include quantitative analysis data and historical reference information. Quantitative analysis data can be fault evidence described numerically. For example, quantitative analysis data may include the correlation strength of a correlation graph (e.g., the correlation strength between "shutdown command" and "status unchanged" is 0.9), statistical results of confirmatory operation tests (e.g., the ratio of concurrent queries returning "On" to "Off" is 6:4), and performance deviation values ​​(e.g., the shutdown command takes 70 milliseconds longer than normal). Historical reference information can be past fault records and experience of the system. For example, historical reference information may include the number of similar historical cases (e.g., 3 similar cases in the knowledge base), the failure frequency of specific components (e.g., "power management unit" is a high-frequency failure component), and historical diagnostic accuracy (e.g., the large language model's accuracy rate for similar problems is 92%). Correspondingly, the output requirements may also include a quantitative confidence assessment of the fault component location results to instruct the large language model, enabling the large language model to output fault root cause judgment results, fault component location results, and the confidence level of the location results based on the input information.

[0113] In some possible implementations, for any test case, the minimum score among the scores of validators of the same type involved in the test case can be used as the score for that type of validator. Then, a weighted average is calculated based on the scores of each type of validator according to preset weights to obtain the consistency score of the test case. Only if the consistency score of the test case is less than a preset score threshold will the above steps S301-303 be executed for that test case, in order to save computing resources and shorten the test time.

[0114] The score corresponding to the validator can be a similarity score or a Boolean value. For example, consider selecting a first validator, a second validator, and a third validator. If the first verification result indicates that the consistency of the response fields in the first and second response results is reasonable, the score corresponding to the first validator can be a Boolean value of true, which can be quantified as "1". If the first verification result indicates that the consistency of the response fields in the first and second response results is unreasonable, a similarity calculation can be triggered. For example, the response fields in the first and second response results can be quantitatively analyzed from dimensions such as status code semantics, key headers, and response body structure to obtain a similarity score between 0 and 1 as the score corresponding to the first validator. If the second verification result indicates that the first response duration is within the first time threshold and the second response duration is within the second time threshold, the score corresponding to the second validator can be a Boolean value of true, which can be quantified as "1". If the second verification result indicates that the first response duration is not within the first time threshold and / or the second response duration is not within the second time threshold, the score corresponding to the second validator can be a Boolean value of false, which can be quantified as "0". If the third verification result indicates that the first data content returned by the first concurrent request is consistent with the second data content returned by the second concurrent request, then the score corresponding to the third verifier can be a Boolean value of true, which can be quantified as "1"; if the data content returned concurrently is inconsistent, then content similarity analysis is triggered, and a similarity score is output as the score corresponding to the third verifier.

[0115] In some possible implementations, the interface automated testing method provided in this disclosure may further include: Obtain confidence assessment index values, including but not limited to consistency comprehensive score, execution trajectory matching degree, large language model self-score, and historical accuracy; perform weighted calculation on the confidence assessment index values ​​to obtain the confidence of the test results.

[0116] The confidence level of the test results is used to characterize the reliability of the test results. The higher the confidence level, the higher the reliability of the test results.

[0117] Among them, the consistency comprehensive score can be used to characterize the degree of matching between the first execution result and the second execution result.

[0118] In a possible implementation, the overall consistency score can be determined based on the consistency score of each test case.

[0119] For example, the minimum consistency score among the individual test cases can be used as the overall consistency score. This low-tolerance decision-making approach can improve credibility.

[0120] The execution trajectory matching degree, also known as the trace matching degree, is used to characterize the degree to which the actual operation record executed during testing matches the expected standard pattern. For example, in the "system shutdown" test, if the standard pattern is "send shutdown command → wait → query status", but the test executes "send shutdown command → accidentally modify configuration → query status", the matching degree will be significantly reduced due to the inclusion of dangerous operations.

[0121] Among them, the large language model self-score can characterize the degree to which the large language model grasps the root causes of failures in its output.

[0122] Historical accuracy rate can be characterized by the frequency with which automated diagnostic conclusions for the same test case (or the same type of fault) are ultimately verified as correct in historical data. For example, if the system has performed 100 diagnoses for a fault such as "power management interface timeout", and 90 of the conclusions (such as "product defect - response timeout") are subsequently confirmed to be accurate, then its historical accuracy rate is 90%.

[0123] In a possible implementation, after obtaining the consistency comprehensive score, execution trajectory matching degree, large language model self-score, and historical accuracy, the consistency comprehensive score, execution trajectory matching degree, large language model self-score, and historical accuracy can be weighted and calculated to obtain the confidence level of the test results.

[0124] For example, the confidence level of the test results can satisfy: The confidence level of the test results = consistency score × 40% + execution trajectory matching degree × 30% + large language model self-score × 20% + historical accuracy × 10%.

[0125] In this embodiment of the disclosure, by acquiring and weighting multiple dimensions of confidence evaluation indicators, the reliability of test results is transformed from traditional subjective experience judgment to objective quantitative evaluation, thereby solving the pain points of traditional testing where the credibility of results cannot be measured and decision-making relies on human experience.

[0126] Optionally, to further improve the accuracy of the confidence level of the test results, the confidence level of the test results in the above steps can be fine-tuned based on the inherent attributes of the target interface, the consistency score of the test cases, the root causes of failures output by the large language model, and the historical pass rate, to obtain the final confidence level.

[0127] For example, if the inherent attributes of the target interface indicate that the target interface is a critical interface, the root cause of the failure output by the large language model is a misjudgment by the test agent, the consistency score is greater than or equal to a preset scoring threshold, and the historical accuracy is greater than a preset accuracy threshold, then the confidence evaluation result can satisfy: Confidence assessment result = confidence level of test result × 0.9 × 0.8 + 20 + 5.

[0128] Then, the confidence assessment result can be clamped so that the final confidence level is between 0 and 100.

[0129] In some possible implementations, after obtaining the final confidence level, the method may further include: performing automated hierarchical decision-making based on the final confidence level.

[0130] For example, if the final confidence level is greater than or equal to a preset confidence threshold, the system automatically outputs the test results and performs issue creation and knowledge base updates. If the final confidence level is less than the preset confidence threshold, the system automatically generates a task requiring manual review and highlights key information and decision suggestions. A test report is generated based on at least one of the above two decision types. The test report may include an execution summary, detailed logs, problem analysis, and improvement suggestions. Furthermore, a knowledge base and continuous learning mechanism can be established, storing each execution result in a structured manner according to four categories of information: API order, response mode, time consumption pattern, and error characteristics. Successful cases are stored as experience, and failed cases as lessons learned. When executing a new test, the system retrieves similar historical cases based on these four characteristics and extracts the optimal combination of practices as a recommended strategy, thereby achieving continuous accumulation of test knowledge and adaptive system optimization.

[0131] For example, Figure 5 This is a schematic diagram of the architecture of an automated interface testing system provided in an embodiment of this disclosure. Figure 5 As shown, the interface automated testing system 400 may include an intelligent agent 401, a first test execution engine 402, and a second test execution engine 403; The agent 401 is used to obtain at least one test case for the target interface; The first test execution engine 402 is used to sequentially execute the test steps in each test case to obtain a first execution result, which includes the first response result of each test step. The second test execution engine 403 is used to, independently and in parallel with the execution of any target test step in a test case, initiate at least one verification operation to the endpoint involved in the target test step to obtain a second execution result, the second execution result including the second response result of each verification operation; the type of verification operation is dynamically determined based on the characteristics of the target test step. The intelligent agent 401 is also used to obtain test results for each target test step based on the consistency comparison of the first response result and the second response result of the target test step; wherein, when the test result indicates that the test has failed, the test result at least identifies the type of failure root cause as an inherent defect in the product or an occasional environmental problem.

[0132] In some possible implementations, the verification operation includes at least one of the following: concurrent verification operation, state verification operation, and anomaly verification operation; The concurrent verification operation includes: during the execution of the target test step, initiating concurrent requests to the endpoints involved in the target test step to obtain the second response result corresponding to the concurrent request; the second response result corresponding to the concurrent request is used to compare with the first response result of the target test step to verify the consistency of the response content of the target interface under concurrent access. The state verification operation includes: during the execution of the target test step, continuously sending multiple query requests to the endpoints involved in the target test step; recording the resource status corresponding to the target interface obtained by the query, and obtaining the second response result of the state verification operation; the second response result of the state verification operation is used to compare with the state transition process in the first execution result of the target test step to verify the consistency of the target test step's behavior when triggering state transition. The exception verification operation includes: during the execution of the target test step, a request with preset exception conditions is sent to the endpoints involved in the target test step to obtain a second response result of the exception verification operation; the second response result of the exception verification operation is used to compare with the first response result of the target test step to determine whether the processing behavior of the target interface under exception disturbance is consistent with the baseline execution.

[0133] In some possible implementations, agent 401 is specifically used for: The test conclusion is determined based on at least one of the first verification result, the second verification result, the third verification result, and the fourth verification result; the test conclusion includes whether the test was successful or failed, and the test result includes the test conclusion. If the test conclusion is that the test failed, determine the type of the root cause of the failure based on the verification results of the exception; The first verification result is obtained based on whether the consistency of the response fields in the first response result and the second response result is reasonable; the second verification result is obtained based on whether the first response duration in the first response result is within the first time consumption threshold and whether the second response duration in the second response result is within the second time consumption threshold; the third verification result is obtained based on whether the first data content returned by the first concurrent request in the first response result is consistent and whether the second data content returned by the second concurrent request in the second response result is consistent; the fourth verification result is obtained based on whether the first abnormal status code and the first error description information in the first response result match and whether the second abnormal status code and the second error description information in the second response result match.

[0134] In some possible implementations, agent 401 is specifically used for: Obtain a first self-consistency verification result for the first response result, wherein the first self-consistency verification result is used to indicate whether the response in the first response result meets expectations; The test results are obtained based on the first self-consistency verification results, the second response results, and the consistency comparison analysis between the first and second response results.

[0135] In some possible implementations, when the test concludes as a test failure, agent 401 is specifically used to: For each test case, an association directed graph is constructed based on the baseboard management controller log, the test trace records in the first execution result, and the test trace records in the second execution result, according to the time window and content relevance. Any node in the association directed graph represents a log record in the baseboard management controller log, or represents an interaction record corresponding to an interaction process of initiating a request to the target interface and receiving a response in the test trace records. The edges in the association directed graph represent the association relationship between the nodes at both ends of the edge, and the edge weight represents the association strength between the nodes at both ends of the edge. Input information is fed into a large language model; the input information includes test requirements, first execution result, second execution result, and a directed graph of association. The large language model processes the input information by using a pre-set root cause classification prompt word template. The prompt word template is configured to guide the large language model to perform the following judgments: determine whether the root cause of the fault is an inherent defect of the product of the system to which the target interface belongs, a test agent misjudgment, or an occasional environmental problem; the test results include the fault root cause judgment results output by the large language model.

[0136] In some possible implementations, agent 401 is also used for: Analyze the characteristics of each test step in the test case, including at least one of the following: the type of endpoint involved in the test step, the HTTP method, and the operation semantic category; Based on the characteristics of the test steps, determine whether there are any consistency risks that need to be verified in terms of concurrent access, state transition and / or exception handling when the test steps are executed. Based on the assessment of consistency risk, determine the type of verification operation corresponding to the test steps.

[0137] Based on the same inventive concept, embodiments of this disclosure provide an electronic device, including: A memory on which computer programs are stored; The processor is used to execute computer programs in memory to implement the aforementioned interface automation testing methods.

[0138] ​ This is a block diagram illustrating an electronic device provided in an embodiment of this disclosure. ​As shown, the electronic device 500 may include a processor 501 and a memory 502. The electronic device 500 may also include one or more of a multimedia component 503, an input / output (I / O) interface 504, and a communication component 505. Specifically, the electronic device may be a server or terminal, or other device with data processing capabilities, and may be able to run automated interface testing methods.

[0139] The processor 501 controls the overall operation of the electronic device 500 to complete all or part of the steps in the aforementioned automated interface testing method. The memory 502 stores various types of data to support the operation of the electronic device 500. This data may include, for example, instructions for any application or method operating on the electronic device 500, and application-related data. The memory 502 can be implemented using any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read-Only Memory (EPROM), Programmable Read-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The multimedia component 503 may include a screen and an audio component. The screen may be, for example, a touchscreen, and the audio component is used for outputting and / or inputting audio signals. For example, the audio component may include a microphone for receiving external audio signals. The received audio signals may be further stored in memory 502 or transmitted via communication component 505. The audio component also includes at least one speaker for outputting audio signals. I / O interface 504 provides an interface between processor 501 and other interface modules, such as a keyboard, mouse, buttons, etc. These buttons may be virtual or physical buttons. Communication component 505 is used for wired or wireless communication between the electronic device 500 and other devices. Wireless communication, such as Wi-Fi, Bluetooth, Near Field Communication (NFC), 2G, 3G, 4G, NB-IoT, eMTC, or other 5G technologies, or combinations thereof, is not limited here. Therefore, the corresponding communication component 505 may include: a Wi-Fi module, a Bluetooth module, an NFC module, etc.

[0140] In an exemplary embodiment, the electronic device 500 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the aforementioned interface automation testing method.

[0141] In another exemplary embodiment, a computer-readable storage medium including program instructions is also provided, which, when executed by a processor, implement the steps of the interface automated testing method described above. For example, the computer-readable storage medium may be the memory 502 including program instructions described above, which may be executed by the processor 501 of the electronic device 500 to complete the interface automated testing method described above.

[0142] In another exemplary embodiment, a computer program product is also provided, the computer program product comprising a computer program executable by a programmable device, the computer program having a code portion for performing the above-described interface automation testing method when executed by the programmable device.

[0143] The preferred embodiments of this disclosure have been described in detail above with reference to the accompanying drawings. However, this disclosure is not limited to the specific details of the above embodiments. Within the scope of the technical concept of this disclosure, various simple modifications can be made to the technical solutions of this disclosure, and these simple modifications all fall within the protection scope of this disclosure.

[0144] It should also be noted that the various specific technical features described in the above embodiments can be combined in any suitable manner without contradiction. To avoid unnecessary repetition, this disclosure will not describe the various possible combinations separately.

[0145] Furthermore, various different embodiments of this disclosure can be combined in any way, as long as they do not violate the spirit of this disclosure, they should also be regarded as the content disclosed in this disclosure.

Claims

1. An automated interface testing method, characterized in that, The method includes: Obtain at least one test case for the target interface; For each of the test cases, the test steps in the test cases are executed sequentially to obtain a first execution result, which includes the first response result of each test step. When executing any target test step in the test case, at least one verification operation is initiated independently and in parallel with the execution of the target test step to the endpoint involved in the target test step, and a second execution result is obtained. The second execution result includes the second response result of each verification operation. The type of the verification operation is dynamically determined based on the characteristics of the target test step. For each of the target test steps, a test result is obtained based on the consistency comparison between the first response result and the second response result of the target test step; wherein, when the test result indicates that the test has failed, the test result at least identifies the root cause of the failure as an inherent defect in the product or an occasional environmental problem.

2. The method according to claim 1, characterized in that, The verification operation includes at least one of the following: concurrent verification operation, state verification operation, and anomaly verification operation; The concurrent verification operation includes: during the execution of the target test step, initiating a concurrent request to the endpoint involved in the target test step to obtain a second response result corresponding to the concurrent request; the second response result corresponding to the concurrent request is used to compare with the first response result of the target test step to verify the consistency of the response content of the target interface under concurrent access. The state verification operation includes: during the execution of the target test step, continuously initiating multiple query requests to the endpoints involved in the target test step; recording the resource status corresponding to the target interface obtained by the query, and obtaining a second response result of the state verification operation; the second response result of the state verification operation is used to compare with the state transition process in the first execution result of the target test step to verify the consistency of the target test step's behavior when triggering state transition; The anomaly verification operation includes: during the execution of the target test step, initiating a request to the endpoint involved in the target test step to inject a preset anomaly condition, so as to obtain a second response result of the anomaly verification operation; the second response result of the anomaly verification operation is used to compare with the first response result of the target test step to determine whether the processing behavior of the target interface under anomaly disturbance is consistent with the baseline execution.

3. The method according to claim 1, characterized in that, The test results are obtained by comparing the consistency of the first response result and the second response result based on the target test step, including: The test conclusion is determined based on at least one of the first verification result, the second verification result, the third verification result, and the fourth verification result; the test conclusion includes whether the test was successful or failed, and the test result includes the test conclusion. If the test conclusion is that the test failed, the root cause type of the failure shall be determined based on the verification results of the anomaly. The first verification result is obtained based on whether the consistency of the response fields in the first response result and the second response result is reasonable; the second verification result is obtained based on whether the first response duration in the first response result is within a first time consumption threshold and whether the second response duration in the second response result is within a second time consumption threshold; the third verification result is obtained based on whether the first data content returned by the first concurrent request in the first response result is consistent and whether the second data content returned by the second concurrent request in the second response result is consistent; and the fourth verification result is obtained based on whether the first abnormal status code and the first error description information in the first response result match and whether the second abnormal status code and the second error description information in the second response result match.

4. The method according to claim 1, characterized in that, The test results are obtained by comparing the consistency of the first response result and the second response result based on the target test step, including: Obtain a first self-consistency verification result for the first response result, wherein the first self-consistency verification result is used to indicate whether the response in the first response result meets expectations; The test results are obtained based on the first self-consistency verification results, the second response results, and the consistency comparison analysis between the first response results and the second response results.

5. The method according to claim 3, characterized in that, If the test result indicates that the test has failed, the method further includes: For each test case, a directed graph is constructed based on the baseboard management controller log, the test trace record of the first execution result, and the test trace record in the second execution result, according to the time window and content relevance. Any node in the directed graph represents a log record in the baseboard management controller log, or represents an interaction record corresponding to an interaction process of initiating a request to the target interface and receiving a response in the test trace record. The edges in the directed graph represent the association relationship between the nodes at both ends of the edge, and the edge weight represents the association strength between the nodes at both ends of the edge. The input information is fed into the large language model; the input information includes the test requirements, the first execution result, the second execution result, and the associated directed graph; The large language model is driven by a preset root cause classification prompt word template to process the input information. The prompt word template is configured to guide the large language model to perform the following judgment: determine whether the root cause of the failure is an inherent defect of the product, a test agent misjudgment, or an occasional environmental problem; the test result includes the root cause judgment result output by the large language model.

6. The method according to any one of claims 1-5, characterized in that, The method further includes: Analyze the characteristics of each test step in the test case, the characteristics including at least one of the following: the type of endpoint involved in the test step, the HTTP method, and the operation semantic category; Based on the characteristics of the test steps, determine whether there are any consistency risks that need to be verified in terms of concurrent access, state transition and / or exception handling when the test steps are executed. Based on the assessment result of the consistency risk, determine the type of verification operation corresponding to the test step.

7. An automated interface testing system, characterized in that, Includes an intelligent agent, a first test execution engine, and a second test execution engine; The intelligent agent is used to acquire at least one test case for the target interface; The first test execution engine is used to sequentially execute the test steps in each of the test cases to obtain a first execution result, wherein the first execution result includes a first response result for each test step. The second test execution engine is configured to, independently and in parallel with the execution of any target test step in the test case, initiate at least one verification operation to the endpoint involved in the target test step to obtain a second execution result, wherein the second execution result includes a second response result of each verification operation; the type of the verification operation is dynamically determined based on the characteristics of the target test step. The intelligent agent is also used to obtain a test result for each of the target test steps based on a consistency comparison of the first response result and the second response result of the target test step; wherein, when the test result indicates that the test has failed, the test result at least identifies the root cause of the failure as an inherent defect in the product or an occasional environmental problem.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program implements the steps of the method described in any one of claims 1-6.

9. An electronic device, characterized in that, include: A memory on which computer programs are stored; A processor for executing the computer program in the memory to implement the steps of the method according to any one of claims 1-6.

10. A computer program product, comprising a computer program, characterized in that, When executed by a processor, the computer program implements the steps of the method described in any one of claims 1-6.

Images