Automatic black box testing method and related device

Abstract

The invention provides an automatic black box testing method and a related device, and relates to the technical field of software, and the method comprises the following steps: obtaining a test task text of a to-be-tested application; analyzing the test task text through a large language model to obtain a test scheme of the to-be-tested application, the test scheme being composed of a plurality of test steps, and each test step being described by a natural language; and sequentially executing each test step in the test scheme on the to-be-tested application in a visual perception mode, and verifying an execution result of each test step in the test scheme. According to the method, a rigid script is replaced by dynamic planning and understanding ability of a large language model, highly-intelligent automatic testing is achieved, in addition, dependence on a code selector is replaced by visual perception, the problem of vulnerability of a traditional script is solved, testing continuity is guaranteed, and black box testing efficiency and accuracy are improved.

Description

Technical Field

[0001] This application relates to the field of software technology, and in particular to an automated black-box testing method and related apparatus. Background Technology

[0002] Black-box testing is an automated testing approach that allows testers to test software functionality at the application level without knowing the program's internal code structure and implementation details.

[0003] Traditional black-box testing heavily relies on fixed selectors for UI elements (such as IDs and XPaths). Even minor changes to the front-end code or UI layout can cause the scripts to malfunction, leading to frequent test interruptions and requiring significant manpower for script maintenance. Summary of the Invention

[0004] In view of the above problems, this application provides an automated black-box testing method and related apparatus to improve the efficiency and quality of black-box testing. The specific solution is as follows:

[0005] The first aspect of this application provides an automated black-box testing method, the automated black-box testing method comprising:

[0006] Obtain the test task text of the application under test;

[0007] The test plan for the application under test is obtained by parsing the test task text using a large language model. The test plan consists of multiple test steps, and each test step is described using natural language.

[0008] Each test step in the test plan is executed sequentially on the application under test using visual perception, and the execution result of each test step in the test plan is verified.

[0009] In one possible implementation, the step of parsing the test task text using a large language model to obtain the test plan for the application under test includes:

[0010] The operation sequence of the application under test is obtained by semantic understanding of the test task text through the large language model. The operation sequence contains multiple test operations, and each test operation has corresponding input parameters.

[0011] The test knowledge corresponding to each test operation in the operation sequence is retrieved sequentially from the black-box test knowledge base, which is a pre-set graph-based knowledge base;

[0012] The test plan is obtained by semantically understanding the input parameters and test knowledge corresponding to each test operation in the operation sequence using the large language model.

[0013] In one possible implementation, the step of sequentially executing each test step in the test plan on the application under test in a visually perceptual manner includes:

[0014] A test environment is created for the application under test using the sandbox management component;

[0015] For each test step in the test plan, a first interface image of the application under test after the previous test step is captured; the first interface image is analyzed based on computer vision to locate the UI element that matches the test step; and the UI element is tested in the test environment according to the test step.

[0016] In one possible implementation, the sandbox management component integrates a screen recording tool, and the automated black-box testing method further includes:

[0017] The execution process of all test steps in the test plan is recorded using the screen recording tool to generate a test video.

[0018] In one possible implementation, verifying the execution result of each test step in the test plan includes:

[0019] For each test step in the test plan, a second interface image of the application under test after the test step is obtained; the first interface image and the second interface image are compared based on computer vision to determine the execution result of the test step based on the comparison result.

[0020] In one possible implementation, the automated black-box testing method further includes:

[0021] Obtain the execution logs of all test steps in the test plan, and parse the execution logs using the large language model to obtain the test report of the application under test.

[0022] A second aspect of this application provides an automated black-box testing device, the automated black-box testing device comprising:

[0023] The task acquisition module is used to acquire the test task text of the application under test;

[0024] The task parsing module is used to parse the test task text using a large language model to obtain the test plan for the application under test. The test plan consists of multiple test steps, and each test step is described using natural language.

[0025] The test execution module is used to execute each test step in the test plan sequentially on the application under test in a visually perceptual manner, and to verify the execution result of each test step in the test plan.

[0026] A third aspect of this application provides a computer program product including computer-readable instructions that, when executed on an electronic device, cause the electronic device to implement the automated black-box testing method of the first aspect or any implementation thereof.

[0027] A fourth aspect of this application provides an electronic device, including at least one processor and a memory connected to the processor, wherein:

[0028] The memory is used to store computer programs;

[0029] The processor is used to execute the computer program so that the electronic device can implement the automated black-box testing method of the first aspect or any implementation thereof.

[0030] The fifth aspect of this application provides a computer storage medium carrying one or more computer programs, which, when executed by an electronic device, enable the electronic device to implement the automated black-box testing method described in the first aspect or any implementation thereof.

[0031] By employing the above technical solution, this application provides an automated black-box testing method and related apparatus, comprising: acquiring the test task text of the application under test; parsing the test task text using a large language model to obtain a test plan for the application under test, wherein the test plan consists of multiple test steps, and each test step is described using natural language; executing each test step in the test plan sequentially in a visually perceptual manner, and verifying the execution result of each test step in the test plan. This application uses the dynamic programming and understanding capabilities of a large language model to replace rigid scripts, achieving highly intelligent automated testing. Furthermore, by replacing the reliance on code selectors with visual perception, it solves the vulnerability problem of traditional scripts, ensures test continuity, and improves the efficiency and accuracy of black-box testing. Attached Figure Description

[0032] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the originals and elements are not necessarily drawn to scale.

[0033] Figure 1 A flowchart illustrating an automated black-box testing method provided in this application embodiment;

[0034] Figure 2 This is a partial flowchart illustrating an automated black-box testing method provided in an embodiment of this application;

[0035] Figure 3 This is another schematic flowchart of an automated black-box testing method provided in an embodiment of this application;

[0036] Figure 4 This is a schematic diagram of the structure of an automated black-box testing device provided in an embodiment of this application;

[0037] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0038] The embodiments of this application are described below with reference to the accompanying drawings. The terminology used in the implementation section of this application is for explaining specific embodiments only and is not intended to limit the scope of this application.

[0039] The embodiments of this application will now be described with reference to the accompanying drawings. Those skilled in the art will recognize that, with technological advancements and the emergence of new scenarios, the technical solutions provided in the embodiments of this application are equally applicable to similar technical problems.

[0040] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms are interchangeable where appropriate; this is merely a way of distinguishing objects with the same attributes in the embodiments of this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a series of elements is not necessarily limited to those elements, but may include other elements not explicitly listed or inherent to those processes, methods, products, or apparatuses.

[0041] Software testing is a crucial step in ensuring software quality, and its development has evolved from purely manual testing to automated testing. Automated testing aims to improve efficiency and coverage by replacing manual execution of repetitive testing tasks with program scripts. Black-box testing, a key component of automated testing, primarily focuses on script-based automation based on the UI (User Interface) object model, with representative technologies including Selenium for web applications and Appium for mobile applications. These tools simulate user behavior by locating UI elements in the page / application view and calling APIs (Application Programming Interfaces) to perform clicks, inputs, and other actions. However, with the rapid iteration of front-end technologies (such as React and Vue) and the increasing complexity of mobile application interfaces, the dynamism and volatility of UIs have become increasingly prominent, making the shortcomings of traditional automated testing methods increasingly apparent.

[0042] Traditional black-box testing relies on UI automation testing frameworks based on fixed selectors. The workflow of such frameworks is usually as follows: test engineers first manually analyze the UI structure of the application under test and write a unique "locator" or "selector" for each element that needs to be interacted with (such as input boxes, buttons, links) (for example, the HTML ID attribute id="login-button", or a complex XPath path / html / body / div[1] / div / button[1]). Then, in the test script, these elements are found and manipulated through the APIs provided by the framework (such as driver.findElement(By.id("login-button")).click()). The verification step of the test also usually relies on finding whether a specific element exists or whether its attribute is a certain expected value. The whole process is based on prior knowledge and hard coding of the internal code structure of the application (DOM tree or View tree). This has the following fundamental defects:

[0043] 1) The test scripts are tightly coupled with the UI implementation code. Once the front-end developers modify the ID, hierarchy, or class name of a UI element, even if the functionality and appearance remain unchanged, the corresponding selector will become invalid, causing the test script to fail. In agile development models, the UI changes frequently, and the testing team needs to spend a lot of time locating and fixing these invalid scripts, resulting in extremely high maintenance costs.

[0044] 2) Lack of generalization and adaptability. The script is "dead." It cannot understand the semantic equivalence between "Login" and "Sign In." If the button text changes, text-based positioning will fail. It also cannot handle unexpected UI events, such as suddenly appearing pop-up ads or permission request dialogs, unless additional processing logic is written for each possibility, making the script extremely bloated and complex.

[0045] 3) Fragmented cross-platform implementation. Selenium is dedicated to the web, while Appium is dedicated to mobile (iOS / Android). Although Appium attempts to unify the API, the underlying drivers and positioning mechanisms are completely different. A business process that covers both the web and app requires two separate sets of scripts and environments, making end-to-end automated testing impossible within a unified cognitive framework. This results in high technical barriers and management costs.

[0046] To address the shortcomings of traditional black-box testing, such as script fragility, lack of generalization and adaptability, and weak cross-platform capabilities, and to achieve more robust, intelligent, efficient, and unified automated testing, this application provides an automated black-box testing method. The automated black-box testing method of this application, as illustrated in the accompanying drawings, will be described in detail below.

[0047] See Figure 1 , Figure 1 This is a flowchart illustrating an automated black-box testing method provided in an embodiment of this application. Figure 1 As shown in the embodiment of this application, an automated black-box testing method can be encapsulated as an AI Agent, which may include steps S101 to S103. These steps are described in detail below.

[0048] S101, Obtain the test task text of the application under test.

[0049] In this embodiment, the application under test is an application to be subjected to black-box testing. The AI ​​Agent can obtain the test task text for the application under test. This test task text is described using natural language or a structured format and may include information such as the platform and version (e.g., Web Chrome, iOS 16, Android 13), a functional description of the function to be tested, test expectations, and test environment requirements. For example, the test task text might be: "Test the user login function. Use the account 'testuser' and password 'password123'. After verifying the login, the page should display 'Welcome'."

[0050] S102, the test task text is parsed using a large language model to obtain the test plan for the application under test. The test plan consists of multiple test steps, and each test step is described using natural language.

[0051] In this embodiment, a large language model is used as the core brain of the AI ​​Agent. With the help of the dynamic planning and understanding capabilities of the large language model, the test task text is input into the large language model so that the large language model can perform semantic understanding on the test task text to obtain the corresponding test plan. The test plan includes multiple test steps with temporal relationships, and each test step is described using natural language. That is to say, each test step in the test plan describes the operation intention rather than hard coding.

[0052] In one possible implementation, the large language model can combine the original task with enhanced knowledge to generate a structured list of semantic steps for the application under test. See also Figure 2 , Figure 2 This is a partial flowchart illustrating an automated black-box testing method provided in an embodiment of this application. Figure 2 As shown in the embodiment of this application, an automated black-box testing method is provided, wherein step S102, "parses the test task text through a large language model to obtain the test plan of the application under test", may include steps S201 to S203, which are described in detail below.

[0053] S201 uses a large language model to perform semantic understanding on the test task text to obtain the operation sequence of the application under test. The operation sequence contains multiple test operations, and each test operation has corresponding input parameters.

[0054] In this embodiment, the AI ​​Agent inputs the test task text into a large language model for preliminary semantic understanding and identifies it as the operation sequence required to achieve the functional description. The operation sequence includes multiple test operations with temporal relationships (such as login operation, jump operation, etc.), and each test operation has corresponding input parameters. For example, the input parameters of the login operation include account and password.

[0055] S202, sequentially retrieve the test knowledge corresponding to each test operation in the operation sequence from the black-box test knowledge base, which is a pre-built graph-based knowledge base.

[0056] In this embodiment, to generate high-quality test steps, the AI ​​Agent queries a pre-built, graph-based black-box testing knowledge base via MCP (Model Context Protocol). Combining the knowledge retrieved from this black-box knowledge base enables GraphRAG (Graph Retrieval-Augmented Generation) to enhance the black-box testing knowledge of large language models. The black-box testing knowledge base stores information such as general test process patterns, interaction logic of scene UI controls, test case generation criteria, historical test data, and best practices.

[0057] The black-box testing knowledge base is searched using each test operation in the application under test and operation sequence as the search criteria. In this way, knowledge related to each test operation is obtained from the black-box testing knowledge base, and this knowledge is used as the test knowledge corresponding to each test operation.

[0058] In practical applications, in addition to being a graph knowledge base, the black-box testing knowledge base can also be a vector database, enabling standard RAG to enhance knowledge of large language models.

[0059] S203 uses a large language model to semantically understand the input parameters and test knowledge corresponding to each test operation in the operation sequence to obtain a test plan.

[0060] In this embodiment, the AI ​​Agent combines the planning capabilities of a black-box testing knowledge base and a large language model to intelligently decompose high-level testing objectives (such as testing user login functionality) into a series of specific, executable semantic operation steps (e.g., "1. Open the application -> 2. Locate and click the 'My' button -> ..."), forming the final testing plan.

[0061] The input parameters and test knowledge corresponding to each test operation in the operation sequence are input into a large language model. This model performs semantic understanding to obtain multiple test steps described in natural language. One test operation corresponds to one test step. The following is an example of a test plan for generating a todo.list, including six test steps, step 1 to step 6:

[0062] Step 1. Open the app or navigate to the URL in your web browser; Step 2. If the function requires login, log in first; otherwise, log in (automatically or manually); Step 3. Navigate to the interface to be tested; Step 4. View the operable elements of the current function; Step 5. Generate multiple test cases based on the elements; Step 6. Test each test case one by one.

[0063] S103, execute each test step in the test plan sequentially in a visually perceptual manner for the application under test, and verify the execution result of each test step in the test plan.

[0064] In this embodiment, the AI ​​Agent can acquire screenshots of the application under test (such as web or platform screenshots) through an automated platform control component. It then uses its integrated computer vision model or multimodal large language model to identify the UI elements (such as input boxes and buttons) required for the current test step in the screenshot. Finally, it calls underlying automated tools (such as MCP Web for Web, MCP Mobile for iOS / Android) to perform precise simulated operations (such as clicking at specific coordinates, inputting text, or swiping) to complete the current test step. This approach relies on visual features and text content, rather than fragile ID or XPath selection, greatly enhancing the adaptability to UI changes. Furthermore, after each test step, the AI ​​Agent can acquire another screenshot for verification to determine if the expected effect has been achieved (e.g., after clicking the "Login" button, does the page display "Welcome, XXX" or redirect to the user center?). Of course, in practical applications, in addition to computer vision models or multimodal large language models for UI element recognition, it can also support ID or XPath-based UI element recognition, which can be implemented by calling relevant MCP interfaces.

[0065] In one possible implementation, the AI ​​Agent can automatically invoke and dynamically create a clean, isolated testing environment where each test step is performed using computer vision. See also Figure 3 , Figure 3 This is another schematic flowchart illustrating an automated black-box testing method provided in an embodiment of this application. Figure 3 As shown in the embodiment of this application, an automated black-box testing method is provided. In step S103, "execute each test step in the test plan sequentially in a visually perceptual manner" can include steps S301 to S302. These steps are described in detail below.

[0066] S301 creates a test environment for the application under test through the sandbox management component.

[0067] In this embodiment, the AI ​​Agent can query the built-in database to obtain information such as the access address, installation package version, and scripts required for environment initialization of the application under test. It automatically invokes the Sandbox management component to create a corresponding test environment for the platform under test. For example, during web testing, a containerized browser instance with built-in Playwright is launched and acts as the MCP Server, waiting for the Agent to call via the MCP protocol; during mobile testing, a specified version of an Android / iOS emulator is launched, the specified App is automatically installed, and MCP Mobile is run as the MCP Server. This ensures that each test is conducted in an isolated and reproducible environment, and the control handle of the test environment is returned to the AI ​​Agent.

[0068] In practical applications, the sandbox management component exposes an interface that conforms to MCP. The AI ​​Agent can control the web page in the test environment based on Playwright, or control the platform interface in the test environment based on the Mobile Agent.

[0069] S302, for each test step in the test plan, capture the first interface image of the application under test after the previous test step; analyze the first interface image based on computer vision to locate the UI element that matches the test step; and perform the test on the UI element in the test environment according to the test step.

[0070] In this embodiment, the AI ​​Agent acts as the overall commander, executing test steps one by one according to the test plan, and mimicking human testing behavior in each test step. For each test step in the test plan, the automated tool is instructed to capture the interface image of the application under test after the previous test step (i.e., the first interface image), locate the UI element matching the test step in the first interface image using a computer vision model or a multimodal large language model, and return its bounding box coordinates or the DOM element of the browser. Furthermore, according to the test operation in the test step, the automated tool is instructed to perform specific operations in the test environment, such as clicking once at the center point of the returned bounding box or typing text in the input box, which is completely independent of the application's internal code structure.

[0071] After the test is completed, the sandbox management component will shut down and clean up the browser instance or emulator used in this test, freeing up system resources.

[0072] In one possible implementation, a screen recording tool can be integrated into the sandbox management component to record the entire black-box testing process. To this end, this application provides an automated black-box testing method, in which the sandbox management component integrates a screen recording tool. This automated black-box testing method further includes the following steps:

[0073] Record the execution process of all test steps in the test plan using a screen recording tool to generate a test video.

[0074] In this embodiment of the application, the screen recording tool in the sandbox management component can be activated to record the execution process of all test steps in the test environment of the application under test, thereby generating a full video of the test process and outputting the test video.

[0075] In one possible implementation, the AI ​​Agent can perform computer vision-based verification of each test step. To this end, this application provides an automated black-box testing method, wherein step S103, "verifying the execution result of each test step in the test plan," can be performed using the following steps:

[0076] For each test step in the test plan, the second interface image of the application under test after the test step is obtained; the first interface image and the second interface image are compared based on computer vision, and the execution result of the test step is determined based on the comparison result.

[0077] In this embodiment, for each test step in the test plan, the AI ​​Agent command automation tool captures the interface image of the application under test after the test step (i.e., the second interface image). The first interface image and the second interface image are compared using a computer vision model or a multimodal large language model. Based on the comparison result, the execution result of the test step is determined to be either successful or failed. For example, if the test operation corresponding to the test step is a jump operation, the interface images before and after the jump can be compared to determine whether the jump was successful. Alternatively, key visual features in the interface image after the jump can be analyzed to make a judgment; this embodiment does not limit this approach. It should be noted that the first interface image and the second interface image can be the same or different; this embodiment does not limit this approach.

[0078] In addition, for verifications such as verify_presence, the AI ​​Agent can also combine technologies such as visual module OCR technology to retrieve relevant descriptive text or elements in the second interface image to complete the verification of this test step.

[0079] Furthermore, for test steps that fail, the AI ​​Agent can record information such as the second-interface image, expected result, and actual observations, and trigger the large language model to replan the test step. For example, the large language model can decide to change search keywords, handle unexpected pop-ups, or revert to the previous step for a retry. This closed-loop control mechanism based on visual feedback is key to achieving high robustness. For instance, if clicking "Login" results in a red "Incorrect Password" message instead of the expected homepage, after replanning by the large language model, it can execute either "Retry entering password" or "Click the Forgot Password link".

[0080] In one possible implementation, to facilitate users in summarizing the testing process, a comprehensive, human-readable report can also be generated. To this end, an automated black-box testing method provided in this application embodiment further includes the following steps:

[0081] Obtain the execution logs of all test steps in the test plan, and parse the execution logs using a large language model to obtain the test report of the application under test.

[0082] In this embodiment, the AI ​​Agent can obtain the execution logs of all test steps in the test plan and input the execution logs into the large language model. The large language model analyzes the execution logs to obtain a test report of the application under test. The test report includes the overall test result (pass / fail), execution details of each test step, screenshots of test steps that pass, and intelligent analysis of test steps that fail (e.g., "After clicking the 'login' button, the 'welcome' text could not be found, and the page remained on the login page. The possible reasons are incorrect password or service abnormality").

[0083] Based on the above description, the automated black-box testing method provided in this application replaces the reliance on code selectors with visual perception, solving the vulnerability problem of traditional scripts; it replaces the fragmented testing framework with a unified AI Agent, controlling multiple target platforms including Web, iOS, and Android through pluggable underlying drivers (such as MCP Web, MCP mobile), solving the complexity problem of cross-platform adaptation; and it replaces rigid script logic with the dynamic programming and understanding capabilities of a large language model, truly achieving highly intelligent automated testing. This application can replace manual black-box testing, improving testing efficiency and quality.

[0084] Specifically, this application employs computer vision to perceive and locate UI elements, rather than relying on fragile code selectors. Therefore, the testing process is no longer interrupted by front-end code refactoring or minor UI layout adjustments, significantly improving the robustness of test cases and substantially reducing long-term maintenance costs. Furthermore, leveraging the powerful planning and reasoning capabilities of large language models enables the testing system to understand complex test intentions and adapt to subtle UI changes. Moreover, by unifying the testing logic across different platforms such as Web and mobile devices under a single AI Agent, platform barriers are broken down, achieving truly integrated, intelligent, and end-to-end automated testing. This significantly improves testing efficiency and coverage, and the generated test reports, rich in context and visual evidence, facilitate faster problem localization.

[0085] The above describes an automated black-box testing method provided by the embodiments of this application. The following describes the apparatus for performing the above-described automated black-box testing method.

[0086] See Figure 4 , Figure 4 This is a schematic diagram of an automated black-box testing device provided in an embodiment of this application. Figure 4 As shown in the figure, an automated black-box testing device provided in this application includes:

[0087] The task acquisition module 401 is used to acquire the test task text of the application under test;

[0088] The task parsing module 402 is used to parse the test task text through a large language model to obtain the test plan of the application under test. The test plan consists of multiple test steps, and each test step is described using natural language.

[0089] The test execution module 403 is used to execute each test step in the test plan sequentially in a visually perceptual manner for the application under test, and to verify the execution results of each test step in the test plan.

[0090] In one possible implementation, the task parsing module 402 is specifically used for:

[0091] The operation sequence of the application under test is obtained by semantic understanding of the test task text through a large language model. The operation sequence contains multiple test operations, and each test operation has corresponding input parameters. The test knowledge corresponding to each test operation in the operation sequence is retrieved sequentially from the black-box test knowledge base, which is a pre-built graph-based knowledge base. The test plan is obtained by semantic understanding of the input parameters and test knowledge corresponding to each test operation in the operation sequence through the large language model.

[0092] In one possible implementation, the test execution module 403, which executes each test step in the test plan sequentially in a visually perceptual manner on the application under test, is specifically used for:

[0093] A test environment is created for the application under test using the sandbox management component; for each test step in the test plan, the first interface image of the application under test after the previous test step is captured; the first interface image is analyzed based on computer vision to locate the UI element that matches the test step; and the UI element is tested in the test environment according to the test step.

[0094] In one possible implementation, the sandbox management component integrates a screen recording tool, and the test execution module 403 is also used for:

[0095] Record the execution process of all test steps in the test plan using a screen recording tool to generate a test video.

[0096] In one possible implementation, the test execution module 403, used to verify the execution results of each test step in the test plan, is specifically used for:

[0097] For each test step in the test plan, the second interface image of the application under test after the test step is obtained; the first interface image and the second interface image are compared based on computer vision, and the execution result of the test step is determined based on the comparison result.

[0098] In one possible implementation, test execution module 403 is also used for:

[0099] Obtain the execution logs of all test steps in the test plan, and parse the execution logs using a large language model to obtain the test report of the application under test.

[0100] It should be noted that the detailed functions of each module in the embodiments of this application can be found in the corresponding disclosure of the above-mentioned automated black-box testing method embodiments, and will not be repeated here.

[0101] This application also provides an electronic device in its embodiments. See also... Figure 5 , Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device in this embodiment may include, but is not limited to, fixed terminals such as mobile phones, laptops, PDAs (personal digital assistants), PADs (tablet computers), desktop computers, etc. Figure 5 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0102] like Figure 5As shown, the electronic device may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 501, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 502 or a program loaded from a storage device 508 into a random access memory (RAM) 503. When the electronic device is powered on, the RAM 503 also stores various programs and data required for the operation of the electronic device. The processing unit 501, ROM 502, and RAM 503 are interconnected via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0103] Typically, the following devices can be connected to I / O interface 505: input devices 506 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 507 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 508 including, for example, memory cards, hard drives, etc.; and communication devices 509. Communication device 509 allows electronic devices to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 5 Electronic devices with various devices are shown, but it should be understood that it is not required to implement or have all of the devices shown. More or fewer devices may be implemented or have alternatively.

[0104] This application also provides a computer program product including computer-readable instructions, which, when executed on an electronic device, cause the electronic device to implement any of the automated black-box testing methods provided in this application.

[0105] This application also provides a computer-readable storage medium that carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement any of the automated black-box testing methods provided in this application.

[0106] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the device embodiment drawings provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0107] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk, or optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, training equipment, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0108] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.

[0109] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, training device, or data center to another website, computer, training device, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives (SSDs)).

Claims

1. An automated black-box testing method, characterized in that, The automated black-box testing method includes: Obtain the test task text of the application under test; The test plan for the application under test is obtained by parsing the test task text using a large language model. The test plan consists of multiple test steps, and each test step is described using natural language. Each test step in the test plan is executed sequentially on the application under test using visual perception, and the execution result of each test step in the test plan is verified. The step of parsing the test task text using a large language model to obtain the test plan for the application under test includes: The operation sequence of the application under test is obtained by semantic understanding of the test task text through the large language model. The operation sequence contains multiple test operations, and each test operation has corresponding input parameters. The test knowledge corresponding to each test operation in the operation sequence is retrieved sequentially from the black-box test knowledge base, which is a pre-set graph-based knowledge base; The test plan is obtained by semantically understanding the input parameters and test knowledge corresponding to each test operation in the operation sequence using the large language model.

2. The automated black-box testing method according to claim 1, characterized in that, The step of sequentially executing each test step in the test plan on the application under test in a visual perception manner includes: A test environment is created for the application under test using the sandbox management component; For each test step in the test plan, a first interface image of the application under test after the previous test step is captured; the first interface image is analyzed based on computer vision to locate the UI element that matches the test step; and the UI element is tested in the test environment according to the test step.

3. The automated black-box testing method according to claim 2, characterized in that, The sandbox management component integrates a screen recording tool, and the automated black-box testing method further includes: The execution process of all test steps in the test plan is recorded using the screen recording tool to generate a test video.

4. The automated black-box testing method according to claim 2, characterized in that, The verification of the execution result of each test step in the test plan includes: For each test step in the test plan, a second interface image of the application under test after the test step is obtained; the first interface image and the second interface image are compared based on computer vision to determine the execution result of the test step based on the comparison result.

5. The automated black-box testing method according to claim 1, characterized in that, The automated black-box testing method also includes: Obtain the execution logs of all test steps in the test plan, and parse the execution logs using the large language model to obtain the test report of the application under test.

6. An automated black-box testing device, characterized in that, The automated black-box testing device includes: The task acquisition module is used to acquire the test task text of the application under test; The task parsing module is used to parse the test task text using a large language model to obtain the test plan for the application under test. The test plan consists of multiple test steps, and each test step is described using natural language. The test execution module is used to execute each test step in the test plan sequentially on the application under test in a visually perceptual manner, and to verify the execution result of each test step in the test plan; The task parsing module is specifically used for: The operation sequence of the application under test is obtained by semantically understanding the test task text using the large language model. The operation sequence contains multiple test operations, and each test operation has corresponding input parameters. Test knowledge corresponding to each test operation in the operation sequence is sequentially retrieved from a black-box test knowledge base, which is a pre-set graph-based knowledge base. The test plan is obtained by semantically understanding the input parameters and test knowledge corresponding to each test operation in the operation sequence using the large language model.

7. A computer program product, characterized in that, It includes computer-readable instructions that, when executed on an electronic device, cause the electronic device to implement the automated black-box testing method as described in any one of claims 1 to 5.

8. An electronic device, characterized in that, It includes at least one processor and a memory connected to the processor, wherein: The memory is used to store computer programs; The processor is used to execute the computer program to enable the electronic device to implement the automated black-box testing method as described in any one of claims 1 to 5.

9. A computer storage medium, characterized in that, The storage medium carries one or more computer programs that, when executed by an electronic device, enable the electronic device to implement the automated black-box testing method as described in any one of claims 1 to 5.

Images