User interface test method and device, electronic equipment, medium and program product

Abstract

The invention provides a user interface testing method and device, electronic equipment, a medium and a program product, and can be applied to the technical field of big data and the technical field of artificial intelligence. The method comprises the following steps: acquiring a target test script, and performing semantic conversion on an operation instruction in the target test script by utilizing a first model to obtain a target test step set containing business semantic description; the target test step set is executed, in the process of executing the target test step set, the first model is used for determining the positioning mode of the target interface element according to the business semantic description, corresponding interaction operation is executed, and actual interface output is obtained; in response to execution failure of the target test step set, generating a diagnosis result by using a first model at least based on the actual interface output and the business semantic description; and inputting the diagnosis result into a second model, generating a modification code snippet by the second model, and applying the modification code snippet to the target test script.

Description

Technical Field

[0001] This application relates to the fields of big data technology and artificial intelligence technology, and more specifically to a method, apparatus, device, medium and program product for testing user interfaces. Background Technology

[0002] Existing automated testing systems for user interfaces (UIs) mainly include locator-driven interaction, image and optical character recognition (OCR) recognition, recording and playback, and hard-coded assertions, but all have significant limitations. Locator-driven methods are highly dependent on the front-end structure; minor UI adjustments can cause widespread script failure. Image and OCR recognition is affected by environmental factors such as resolution, font, and background, resulting in low recognition accuracy and an inability to understand interface semantics. While recording and playback can quickly generate scripts, it merely mechanically records operations, lacking intent understanding and adaptive capabilities. Hard-coded assertions struggle to handle dynamic data and semantic changes, lacking flexibility. Summary of the Invention

[0003] In view of the above problems, this application provides a method, apparatus, device, medium and program product for testing user interfaces.

[0004] According to a first aspect of this application, a method for testing a user interface is provided, the method comprising: obtaining a target test script; performing semantic transformation on operation instructions in the target test script using a first model to obtain a target test step set containing a business semantic description; executing the target test step set, wherein, during the execution of the target test step set, the first model is used to determine the positioning method of target interface elements according to the business semantic description, and corresponding interactive operations are performed to obtain actual interface output; in response to the failure of the execution of the target test step set, at least based on the actual interface output and the business semantic description, the first model is used to generate a diagnostic result; and the diagnostic result is input to a second model, the second model generates a modified code snippet, and the modified code snippet is applied to the target test script.

[0005] According to an embodiment of this application, the step of semantically transforming the operation instructions in the target test script using a first model to obtain a target test step set containing business semantic description includes: parsing the target test script to obtain the action type, initial locator, and context parameters corresponding to each operation instruction; inputting the action type, the initial locator, and the context parameters into the first model to generate a corresponding business semantic description and candidate feature set; and obtaining the target test step set based on the business semantic description and the candidate feature set.

[0006] According to an embodiment of this application, the step of determining the positioning method of the target interface element based on the business semantic description using the first model includes: for each step in the target test step set: generating multiple candidate positioning methods based on the business semantic description and candidate feature set using the first model; verifying and sorting the candidate positioning methods to obtain the target candidate positioning method; and determining the target candidate positioning method as the positioning method of the target interface element.

[0007] According to an embodiment of this application, determining the failure of the target test step set includes: inputting the actual interface output and the business semantic description into the first model to generate a semantic consistency score or judgment result; marking the corresponding target test step as an execution failure when the semantic consistency score is lower than a first preset threshold or the judgment result is inconsistent; and determining the execution failure of the target test step set in response to detecting at least one target test step marked as an execution failure.

[0008] According to an embodiment of this application, determining the failure of the target test step set includes: monitoring the running status of each target test step during the execution of the target test step set; in response to detecting a location failure, interaction anomaly, or output anomaly, collecting the corresponding running logs, screenshots of the interface at the time of the anomaly, and the business semantic description to construct a multimodal anomaly data packet; inputting the multimodal anomaly data packet into the first model and using the first model to generate a corresponding anomaly confidence score; and in response to the anomaly confidence score exceeding a second preset threshold, determining that the target test step set has failed to execute.

[0009] According to an embodiment of this application, generating a diagnostic result using the first model includes: inputting the multimodal abnormal data packet into the first model, and using the first model to perform fusion reasoning of text information and image information on the operation log, the screenshot of the interface at the abnormal moment, and the business semantic description to generate the diagnostic result.

[0010] According to an embodiment of this application, the step of inputting the diagnostic result into a second model and generating a modified code fragment by the second model includes: identifying the cause of execution failure by the second model based on the diagnostic result; generating at least two candidate modification schemes based on the cause of execution failure and generating corresponding candidate modified code fragments for the candidate modification schemes; performing static analysis or trial operation verification on the candidate modified code fragments and sorting them according to the execution success rate or confidence score, and obtaining the modified code fragments based on the sorting result.

[0011] According to an embodiment of this application, the method further includes: after applying the modified code snippet, re-executing the target test step set for repair verification; in response to the repair verification passing, storing the diagnostic result and the corresponding modified code snippet in a knowledge base; and in response to the repair verification failing, obtaining an update execution log, an update interface screenshot, and an update business semantic description, combining the update execution log, the update interface screenshot, and the update business semantic description into an update multimodal anomaly data packet, and performing re-diagnosis and re-repair based on the update multimodal anomaly data packet.

[0012] A second aspect of this application provides a user interface testing apparatus, the apparatus comprising: a data acquisition module, configured to: acquire a target test script, and semantically convert the operation instructions in the target test script using a first model to obtain a target test step set containing a business semantic description; a test step execution module, configured to: execute the target test step set, wherein, during the execution of the target test step set, the first model is used to determine the positioning method of target interface elements according to the business semantic description to execute corresponding interactive operations and obtain actual interface output; a diagnostic result generation module, configured to: in response to the failure of the execution of the target test step set, generate a diagnostic result using the first model based at least on the actual interface output and the business semantic description; and a code repair module, configured to: input the diagnostic result to a second model, generate a modified code fragment by the second model, and apply the modified code fragment to the target test script.

[0013] A third aspect of this application provides an electronic device comprising: one or more processors; and a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the method described above.

[0014] A fourth aspect of this application also provides a computer-readable storage medium having a computer program or instructions stored thereon, which, when executed by a processor, implement the steps of the above-described method.

[0015] The fifth aspect of this application also provides a computer program product, including a computer program or instructions that, when executed by a processor, implement the steps of the above-described method.

[0016] According to embodiments of this application, the test script is semantically transformed using a first model, converting the original test logic, which relied on fixed locators and instruction sequences, into a set of target test steps containing business semantic descriptions. This enables the system to understand the business intent of the test behavior, thus maintaining strong adaptability and portability even when the interface structure or element attributes change. During the test execution phase, the first model can dynamically determine the positioning method of the target interface elements based on the business semantic description and perform semantic-level judgment on the interaction results. When positioning fails or the output is inconsistent with the business semantics, the system can automatically identify the anomaly and generate diagnostic results, avoiding the reliance on fixed rules and manual assertions in traditional UI testing. By inputting the diagnostic results into a second model to generate modified code snippets and automatically applying them to the test script, intelligent repair and self-evolution of the test script are achieved, enabling the testing system to have self-diagnostic and self-repair capabilities, significantly reducing script maintenance costs, and improving the robustness and execution efficiency of the testing process. Attached Figure Description

[0017] The above-mentioned contents, other objects, features and advantages of this application will become clearer from the following description of embodiments with reference to the accompanying drawings, in which:

[0018] Figure 1 The illustration schematically depicts application scenarios of testing methods, apparatus, devices, media, and program products for user interfaces according to embodiments of this application;

[0019] Figure 2 A flowchart illustrating a user interface testing method according to an embodiment of this application is shown schematically;

[0020] Figure 3 The flowchart illustrates a method for semantically transforming operation instructions in a target test script using a first model, according to some exemplary embodiments of this application.

[0021] Figure 4 The illustration schematically shows a method for determining the failure of a set of target test steps according to some exemplary embodiments of this application;

[0022] Figure 5 This schematically illustrates a structural block diagram of a testing apparatus for a user interface according to an embodiment of this application; and

[0023] Figure 6 A block diagram schematically illustrates an electronic device suitable for implementing a user interface testing method according to an embodiment of this application. Detailed Implementation

[0024] The embodiments of this application will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of this application. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of this application for ease of explanation. However, it will be apparent that one or more embodiments may be implemented without these specific details. Furthermore, descriptions of well-known structures and technologies are omitted in the following description to avoid unnecessarily obscuring the concepts of this application.

[0025] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of this application. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0026] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0027] When using expressions such as "at least one of A, B and C", they should generally be interpreted in accordance with the meaning that is commonly understood by those skilled in the art (e.g., "a system having at least one of A, B and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B and C, etc.).

[0028] With increasingly complex software system architectures, richer user interfaces, and the widespread adoption of agile development and continuous integration, UI testing has become a crucial element in ensuring the correctness of application system functions, the consistency of user experience, and the quality of version iterations. UI automated testing typically simulates human operations to identify, interact with, and verify interface components, thereby quickly identifying interface interaction errors and business process anomalies during the development, regression, or canary release phases. Existing UI automated testing systems are usually multi-layered, complementary systems, mainly composed of four core technologies: locator-driven element interaction, image recognition and OCR assistance, recording and playback with script generation, and hard-coded assertion verification.

[0029] In traditional solutions, locator-driven element interaction is the most basic and common testing method. Test scripts use locators to find and identify UI components in the interface, establishing a unique address mapping for each target element. Based on this mapping, the system can automatically perform operations such as clicking, inputting, swiping, and reading text to verify the correctness of the UI interaction. However, this method is highly dependent on the stability of the front-end code structure and the Document Object Model (DOM) hierarchy. If the page structure is slightly adjusted, element attributes are renamed, or stylesheets are updated, the original locators may become invalid, leading to a large number of test scripts failing and batch errors, resulting in high maintenance costs and long debugging cycles.

[0030] To enhance positioning accuracy and assertion capabilities, some testing frameworks have incorporated image recognition and OCR technologies. These technologies assist in positioning by recognizing pre-captured UI component images or text content within specified screen areas. While this approach mitigates the vulnerabilities of pure locator methods to some extent, it remains essentially pixel-level recognition, significantly affected by factors such as the testing environment, resolution, font style, theme color scheme, and background complexity. Especially in scenarios with specific fonts, artistic fonts, mixed text and images, dynamic shadows, or complex backgrounds, the accuracy of traditional OCR algorithms drops markedly, easily leading to false positives or false negatives. Furthermore, image recognition methods can only perform static matching and cannot understand the logical hierarchy and semantic relationships between interface elements, lacking the ability to make business-level judgments.

[0031] Record-and-playback mechanisms are widely used to improve the efficiency of test script writing. Their basic principle is to generate corresponding automated scripts by recording user actions on the interface (such as clicks, input, scrolling, etc.), thereby achieving "no-code" or "low-code" testing. This method lowers the technical barrier for testers and facilitates the rapid construction of basic test cases. However, this mechanism is essentially just a mechanical recording of user "operational behavior" and cannot understand the user's "operational intent." Once the interface layout, interaction flow, or control hierarchy changes, the original recorded script becomes invalid and needs to be re-recorded. For rapidly iterating agile development environments, this method lacks reusability and adaptability.

[0032] On the other hand, hard-coded assertions are currently the most common verification method in UI testing. Testers directly write expected results (such as text values, colors, and states) in the script and compare the actual results with the expected values ​​during execution to determine whether the test passes. However, hard-coded assertions lack flexibility when dealing with dynamically generated data (such as timestamps, user nicknames, and personalized recommendations). To adapt to these changes, it is often necessary to write a large number of conditional statements or regular expression matching logic, increasing script complexity and maintenance costs. At the same time, this assertion method cannot perform "fuzzy judgments" or "semantic judgments," making it difficult to cope with the verification needs of dynamic interface changes or business semantic levels.

[0033] Based on this, embodiments of this application provide a user interface testing method, the method comprising: obtaining a target test script; using a first model to perform semantic transformation on the operation instructions in the target test script to obtain a target test step set containing business semantic descriptions; executing the target test step set, wherein, during the execution of the target test step set, the first model is used to determine the positioning method of target interface elements according to the business semantic descriptions to execute corresponding interactive operations and obtain actual interface output; in response to the failure of the execution of the target test step set, at least based on the actual interface outputs and the business semantic descriptions, using the first model to generate a diagnostic result; and inputting the diagnostic result into a second model, the second model generating modified code snippets, and applying the modified code snippets to the target test script. According to embodiments of this application, the test script is semantically transformed using a first model, converting the original test logic, which relied on fixed locators and instruction sequences, into a set of target test steps containing business semantic descriptions. This enables the system to understand the business intent of the test behavior, thus maintaining strong adaptability and portability even when the interface structure or element attributes change. During the test execution phase, the first model can dynamically determine the positioning method of the target interface elements based on the business semantic description and perform semantic-level judgment on the interaction results. When positioning fails or the output is inconsistent with the business semantics, the system can automatically identify the anomaly and generate diagnostic results, avoiding the reliance on fixed rules and manual assertions in traditional UI testing. By inputting the diagnostic results into a second model to generate modified code snippets and automatically applying them to the test script, intelligent repair and self-evolution of the test script are achieved, enabling the testing system to have self-diagnostic and self-repair capabilities, significantly reducing script maintenance costs, and improving the robustness and execution efficiency of the testing process.

[0034] It should be noted that the user interface testing methods, apparatus, devices, media, and program products defined in this application can be used in the fields of big data technology and artificial intelligence technology, and can also be used in a variety of fields other than big data technology and artificial intelligence technology. The application fields of the user interface testing methods, apparatus, devices, media, and program products provided in the embodiments of this application are not limited.

[0035] In the technical solution of this application, the user information (including but not limited to user personal information, user image information, user device information, such as location information) and data (including but not limited to data used for analysis, stored data, and displayed data) involved are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, storage, use, processing, transmission, provision, disclosure, and application of related data all comply with relevant laws, regulations, and standards, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entry points for users to choose to authorize or refuse.

[0036] In scenarios where personal information is used for automated decision-making, the methods, devices, and systems provided in this application all offer users corresponding operation entry points, allowing them to choose to agree to or reject the automated decision results; if the user chooses to reject, the process proceeds to expert decision-making. Here, "expert decision-making" refers to the decision-making activities of personnel who specialize in a particular field, possess specialized experience, knowledge, and skills, and have reached a certain level of professional expertise.

[0037] Figure 1 The illustration schematically depicts application scenarios of testing methods, apparatus, devices, media, and program products for user interfaces according to embodiments of this application.

[0038] like Figure 1 As shown, application scenario 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. The network 104 serves as a medium for providing a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0039] Users can use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 via the network 104 to receive or send messages, etc. Various communication client applications can be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social media platform software, etc. (for example only).

[0040] In the embodiments of this application, the first terminal device 101 can be an example of the first device, and the second terminal device 102 and / or the third terminal device 103 can be an example of at least one second device. The first device and the second device can communicate collaboratively through an internal client mechanism to implement the data distribution and rendering logic described in the user interface testing method.

[0041] In some embodiments, the first device and at least one second device may be different display modules, windows or screens on the same computing terminal (such as a host), or multiple physical devices that work together through a network, such as different client instances deployed on a desktop computer, tablet terminal or mobile device respectively.

[0042] The first terminal device 101, the second terminal device 102, and the third terminal device 103 can be various electronic devices with displays and support web browsing, including but not limited to smart mobile terminals, tablet computers, laptop computers, and desktop computers.

[0043] Server 105 can be a server that provides various services, such as a backend management server that supports websites browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103 (this is just an example). The backend management server can analyze and process data such as received user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal devices.

[0044] It should be noted that the user interface testing method provided in this application embodiment can generally be executed by server 105. Correspondingly, the user interface testing device provided in this application embodiment can generally be located in server 105. The user interface testing method provided in this application embodiment can also be executed by a server or server cluster that is different from server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or server 105. Correspondingly, the user interface testing device provided in this application embodiment can also be located in a server or server cluster that is different from server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or server 105.

[0045] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0046] The following will be based on Figure 1 The described scene, through Figures 2-4 The testing method for the user interface of the disclosed embodiments will be described in detail.

[0047] Figure 2 A flowchart illustrating a user interface testing method according to an embodiment of this application is shown schematically.

[0048] like Figure 2 As shown, the user interface testing method 200 of this embodiment includes operations S210 to S240.

[0049] In operation S210, the target test script is obtained, and the operation instructions in the target test script are semantically transformed using the first model to obtain a target test step set containing business semantic descriptions.

[0050] In the embodiments of this application, operation S210 can be executed by an artificial intelligence model (i.e., the first model), whose main function is to perform a semantic-layer structured reconstruction of traditional UI test scripts, thereby realizing the transformation of test scripts from "instruction-driven" to "semantic-driven". Specifically, the system can extract operation instructions from existing test script files and identify their corresponding business logic scenarios in combination with context information, thereby generating a set of target test steps that can express business intent.

[0051] In some embodiments, the first model can perform natural language parsing on the input script instructions and parameters based on a pre-trained language model or a domain-specific semantic model. The model first identifies the action type, object elements, and parameter relationships in each operation instruction, and then determines its corresponding business meaning through semantic embedding or contextual attention mechanisms. For example, for the script instruction `click(“btn_login”)`, the model can identify the business semantics of this operation as "execute login action"; for the instruction `input(“username”,“test_user”)`, the model can generate a business semantic description of "enter username". After conversion, the target test step set formed by the system not only records the specific operation actions but also includes the business objectives and contextual constraints of each step, enabling subsequent steps to be reused under different interface structures.

[0052] In some embodiments, the semantic transformation process can also be combined with the generation of candidate feature sets to enhance the model's understanding of the relationships between interface elements. When parsing the script, in addition to extracting explicit locator information, the system can also capture relevant visual labels, text content, component role attributes, hierarchical relationships, and contextual behavioral trajectories. The first model establishes semantic mapping relationships between elements based on these candidate features, enabling the generated target test steps to be decoupled from the interface structure. For example, when an interface update causes a change in the locator, the system can still identify the corresponding new element location based on the similarity matching between business semantics and candidate features, thereby improving the robustness of the script during interface evolution.

[0053] In some embodiments, the target test script can be obtained not only through file input, but also through real-time recorded interaction trajectories. When the user performs an operation, the system automatically records various interaction events (including clicks, inputs, swipes, view switching, etc.) and inputs this event data into the first model for semantic parsing in real time.

[0054] In some embodiments, semantic transformation can also be extended to multi-language and multi-platform scenarios. The first model can simultaneously process scripts generated by different front-end frameworks, generating a cross-platform consistent set of target test steps by uniformly modeling element attributes and operational semantics.

[0055] In operation S220, the target test step set is executed. During the execution of the target test step set, the first model is used to determine the positioning method of the target interface element according to the business semantic description, and the corresponding interactive operation is executed to obtain the actual interface output.

[0056] In the embodiments of this application, operation S220 is used to execute the target test step set after semantic transformation. The core of this operation is that the system no longer relies on traditional static locators to determine UI elements, but instead dynamically determines the positioning method of target interface elements based on business semantic descriptions through a first model. In other words, the model does not directly "find" a certain element, but rather "understands" the business purpose to be achieved in the current step, and then selects an appropriate positioning strategy based on contextual information. For example, when the business semantic is "click the login button", the model will not only search for buttons whose names or labels contain the word "login", but also combine the page structure, button hierarchy, display area, and visual features to make a comprehensive judgment to determine the interactive element on the current page that best fits the semantics of "login".

[0057] In some embodiments, when executing each target test step, the system may first receive the business semantic description and candidate feature set corresponding to that step from a first model. The candidate feature set may include the text content of elements, component type, positional relationship, color style, or visual region information, etc. The first model generates several candidate positioning expressions by performing weighted reasoning on these features, such as positioning methods based on text keywords, positioning methods based on component hierarchy paths, or positioning methods based on visual coordinates. The model then verifies and sorts these candidate expressions, selecting the one with the highest matching degree as the final positioning method, and executes the corresponding interactive operation.

[0058] In some embodiments, the first model can improve positioning accuracy through continuous context awareness and adaptive adjustment. During execution, the model can maintain an interface state representation to describe the page structure, interaction history, and element distribution. When an execution step requires clicking a specific button or input field, the model not only refers to the current semantic description but also utilizes contextual information from previous operations to determine the element's validity. For example, when the business semantic is "enter username," the model will automatically exclude "password input field" or "verification code input field," prioritizing input fields related to "username."

[0059] In some embodiments, the model can comprehensively utilize multiple perceptual signals for element localization. In addition to traditional DOM structure and text features, the system can also incorporate an image recognition module to detect the layout and position of UI components through screenshots or page renderings. The first model can jointly encode visual and semantic features to achieve "semantic-visual" bimodal matching. For example, when a button icon on a page is replaced with an image but the text label disappears, the model can identify the icon's functional attributes based on keywords such as "login" or "submit" in the semantic description, combined with visual features, thus still accurately performing the interactive operation.

[0060] In some embodiments, the execution process may further include real-time verification of the interaction results by the model. After completing a click, input, or toggle operation, the system can immediately capture interface response information, such as pop-up prompts, page redirects, or field changes. The first model can compare these response contents with the business semantic description to determine whether the current operation has achieved its expected results. For example, when the business semantic is "click to submit order," the model expects to detect a "payment successful" or "order generated" prompt on the page. If no matching semantic content is found, the step is deemed to have failed.

[0061] In some embodiments, the first model can also adaptively select the optimal interaction method according to different types of interface elements. For example, for button elements, the model performs click operations; for input elements, it performs text input or file upload operations; for dropdown menus or slider components, it performs interactions such as selection and dragging; for dynamically loaded content, the model can perform operations after determining that the element has been rendered.

[0062] In some embodiments, the system can perform cross-version execution for multi-language or multi-theme interfaces. When determining the positioning method, the first model not only considers the text content in the current language but can also use a multi-language embedding model to judge the semantic equivalence between different languages. For example, for the "Login" button in an English interface and the "登录" button in a Chinese interface, the model can recognize their semantic consistency and thus generate a unified positioning expression. Similarly, when the system theme or style changes, resulting in different button colors, icons, or layout positions, the model can still maintain stable execution through the matching of visual features and business semantics.

[0063] In some embodiments, to further improve the reliability during the execution phase, the system can save state information and generate an execution trace before and after each operation. The first model can analyze the time consumption of each step, page response latency, and possible abnormal signals based on these trace data, and thus dynamically adjust the execution strategy for subsequent steps. For example, when the system detects that a specific module is loading slowly, it can automatically extend the waiting time or switch to an alternative positioning method to ensure the continuity and stability of script execution.

[0064] In operation S230, in response to the failure of the target test step set, at least based on the actual interface output and the business semantic description, a diagnostic result is generated using the first model.

[0065] In the embodiments of the present application, after detecting the failure of the target test step set, the cause of the failure can be automatically analyzed and diagnosed. This operation can be performed by the first model, which comprehensively uses semantic information and actual execution output to generate a diagnostic result that can reflect the root cause of the failure. Specifically, when the system finds that a test step fails to achieve the expected effect during the execution phase, the first model does not simply analyze the error log or assertion result but conducts logical reasoning based on the difference between the business semantic description and the actual interface output to judge the cause of the failure at the semantic level. For example, when the test semantics is "the page should jump to the home page after clicking the login button", but the actual interface output remains on the login page, the model can infer that the cause of the failure may be that the button does not respond or the jump logic is abnormal, and generate corresponding prompts in the diagnostic result.

[0066] In some embodiments, the first model may employ a multimodal fusion analysis framework to comprehensively input anomaly information from different dimensions into the model to obtain more accurate diagnostic results. Model input may include: test execution logs (such as request / response status codes, operation exception stack traces), page screenshots (reflecting UI visual changes), and business semantic descriptions associated with the current step. The model extracts semantic and visual features through a text encoder and a visual encoder, respectively, and analyzes the correspondence between the two using a cross-modal attention mechanism. When the model detects a significant semantic deviation between the interface output and the business description, such as the expected button color change not occurring, page hierarchy not updating, or incorrect prompt information, the system can generate a diagnostic report containing the anomaly category (such as incorrect location, interaction not executed, or result not refreshed).

[0067] In some embodiments, the first model can combine historical test data for similar fault matching and diagnosis. The system maintains a large number of test failure samples and corresponding repair records in a knowledge base. When a new execution failure occurs, the model can retrieve the most similar historical cases based on the current semantic description and interface output features. By comparing feature similarity, the model can quickly infer possible root cause types, such as "element not loaded," "network request timeout," or "interface layout change," thereby achieving rapid diagnosis based on experience-based knowledge.

[0068] In some embodiments, the diagnostic process of the first model can also incorporate a causal reasoning mechanism. When the system detects multiple consecutive failures or abnormal outputs, the model can build a causal graph based on the execution sequence of the test step set to identify which step's anomaly is most likely the root cause. For example, in a test flow of "enter username—enter password—click login—load homepage," if clicking the login button does not trigger a redirect, the model can infer that the login logic anomaly is the primary failure point, rather than an input field error.

[0069] In some embodiments, diagnostic results can be generated not only as text reports but also in a structured format. The first model can organize fault information into a multi-dimensional result set, including: fault type, scope of impact, probability score, and recommended repair direction. For example, for a failed "click to submit order" operation, the model's diagnostic result might be expressed as: "Root cause type: location failure; Scope of impact: page checkout module; Confidence level: 0.92; Recommended repair direction: update the locator or delay the execution of the operation."

[0070] In some embodiments, the first model can also perform comprehensive diagnosis of chained anomalies. When multiple test steps are interdependent and the failure of a preceding step may lead to subsequent errors, the model can mark the fault propagation path when generating diagnostic results. For example, in a multi-page navigation process, if the failure to load the first page causes all subsequent element location to fail, the model can mark the "upstream failure dependency" relationship in the diagnostic output, suggesting that subsequent repairs should prioritize addressing the initial loading problem. This multi-level diagnostic result helps improve the efficiency and targeting of the overall system repair.

[0071] In operation S240, the diagnostic results are input into the second model, which generates a modified code snippet and applies the modified code snippet to the target test script.

[0072] In the embodiments of this application, the target test script can be automatically repaired after the diagnostic results are generated. Specifically, the system inputs the diagnostic results output by the first model into the second model. The second model generates modified code snippets for repair based on the diagnostic content and automatically applies these modified code snippets to the target test script, thereby forming a repaired version that can be executed again. This step realizes an automated closed loop from anomaly identification to repair implementation, enabling the testing system to complete the self-healing process of the test script without human intervention. The second model can be a specially fine-tuned large language model or code generation model. Its input includes the root cause description, anomaly type, and recommended repair direction from the diagnostic report, and its output is an executable code snippet that conforms to the testing framework specifications.

[0073] In some embodiments, the second model can generate modified code snippets based on a combination of rule constraints and deep generation. When performing this step, the system first categorizes the diagnostic results using template rules, such as locator failure, assertion logic errors, missing exception handling, and interaction delay mismatches. Then, it calls the corresponding repair templates or strategy libraries for different categories. Based on this, the second model further generates repair code that conforms to the context logic. For example, it automatically generates new locator expressions when a locator fails, reconstructs validation conditions when an assertion error occurs, and inserts wait statements or asynchronous processing logic when interaction delays are abnormal.

[0074] In some embodiments, the second model can be generated by transfer from historical repair cases. The system can maintain a knowledge base containing a large number of "diagnosis result - repair code" mapping samples. When a new diagnosis result is input, the model can retrieve the historical repair scheme most similar to the current anomaly type through similarity matching, and generate a new repair code fragment based on local modifications.

[0075] In some embodiments, when generating modified code snippets, the second model can also simultaneously output the reason for the fix and a description of the changes to enhance the interpretability of the fix. While generating code, the model can generate natural language explanations based on the diagnostic results, describing the problem addressed by the modification, the method of modification, and the expected effect. For example, it might output "This modification inserts a wait statement to resolve the location failure issue caused by page load delay," or "The assertion logic has been adjusted to match the new prompt text."

[0076] In some embodiments, the second model can generate and self-evaluate multiple remediation solutions. Upon receiving the diagnostic results, the model can generate several different modified code snippets at once and rank the candidate results using a built-in scoring mechanism (such as code consistency scoring, executability prediction, or confidence assessment). The system can automatically apply the solution with the highest score while retaining other candidate solutions for manual reference or subsequent verification. Through this diversified generation and automatic screening mechanism, the system can significantly improve the stability and success rate of the remediation results.

[0077] In some embodiments, the second model can also perform semantic verification and static analysis after the repair code is generated to prevent the repair from introducing new errors. The system can perform syntax checks, dependency verification, and context consistency analysis on the modified fragments before code application to ensure that the generated modifications are compatible with the original script and do not affect unrelated logic modules. For example, when the modification involves location expressions, the model can verify whether the modified element is uniquely identifiable in the current interface structure; when the modification involves business logic, the model can detect whether the variable scope and dependencies are correct. If the verification fails, the system can automatically fall back to the suboptimal candidate repair solution.

[0078] In some embodiments, the second model can also self-optimize based on the repair execution results. After the repair code snippet is applied, the system re-executes the test script and generates feedback signals based on the execution logs and verification results. If the test passes, the model can store the repair solution in the knowledge base, forming a new "diagnosis-repair" sample; if the test still fails, the system can re-input the new anomaly information into the first model for re-diagnosis, and iteratively generate a new repair solution through the second model, thereby achieving closed-loop self-learning and continuous optimization. Through this feedback learning mechanism, the second model can accumulate experience in long-term operation, continuously improving the repair quality and intelligence level.

[0079] In some embodiments, the second model can also generate cross-platform fix code by combining adaptation strategies for different languages ​​and frameworks. When the test system involves multiple development environments, the model can automatically generate fix code with corresponding syntax based on the target scripting language.

[0080] According to embodiments of this application, the original test script is semantically transformed using a first model, converting traditional scripts that rely on locators and fixed commands into a set of target test steps with business semantic descriptions. This achieves a shift from syntax-driven to semantic-driven scripting. The first model dynamically determines the location of target interface elements and performs comprehensive matching based on page structure, context, and visual features, enabling intelligent recognition and interaction of UI elements. This overcomes the limitations of fragile locators and easily invalidated rules in traditional UI automation, giving the testing process semantic understanding and context awareness. Even if the interface layout or identifiers are adjusted, the system can still automatically complete interactive operations and obtain accurate actual interface output, significantly improving test robustness and execution success rate. When a test step fails, the first model performs multimodal fusion analysis based on the actual interface output and business semantic description, identifying the root cause of the anomaly from a semantic perspective, improving the accuracy and interpretability of anomaly identification. The diagnostic results are input into a second model, which automatically generates corresponding modified code snippets and applies them to the test script, achieving automatic script repair and re-execution. This reduces manual maintenance costs and improves the adaptability and intelligence of automated testing.

[0081] The following describes in detail the testing method of the user interface of the present application embodiment by way of preferred embodiments.

[0082] In the embodiments of this application, in order to transform traditional test scripts that rely on fixed instructions and locators into a set of target test steps with business semantic descriptions, and to realize the semantic understanding and context adaptation of automated test scripts, the following method is also provided.

[0083] Figure 3 The flowchart illustrates a method for semantically transforming operation instructions in a target test script using a first model, according to some exemplary embodiments of this application.

[0084] like Figure 3 As shown, the method for semantically transforming operation instructions in the target test script using the first model includes operations S310 to S330.

[0085] In operation S310, the target test script is parsed to obtain the action type, initial locator, and context parameters corresponding to each operation instruction.

[0086] In the embodiments of this application, the system can extract the original operation sequence from existing UI automation test scripts, such as click, input, scroll, switch, and assert instructions. The parsing process can analyze the script statements line by line to identify the action type, target object, and parameter dependencies of each operation. For example, in a login process script, the action type of the instruction "input('username','test_user')" is "input", the initial locator is "username", and the context parameter is "test_user".

[0087] In some embodiments, the parsing process is not limited to syntax-level analysis; it can also incorporate runtime context information such as the dependency order of instructions, variable references, and page states. The system can identify the relationships between elements at different steps by parsing log files or script execution snapshots, thus supplementing the dynamic context that traditional parsing cannot capture. For example, when a script contains the logic of "clicking button A and then clicking button B," the model can identify that button B depends on the activation state of button A, and then incorporate this dependency into subsequent semantic generation.

[0088] In some embodiments, the parsing process can also semantically expand unstructured script comments or naming rules by combining static code analysis with semantic segmentation technology. If the variable name or comment contains business keywords (such as "submit order", "pay", "search for products"), the system can extract them as potential business clues, providing richer feature inputs for subsequent semantic modeling.

[0089] In operation S320, the action type, the initial locator, and the context parameters are input into the first model to generate a corresponding business semantic description and candidate feature set.

[0090] In the embodiments of this application, the first model can be a fine-tuned large language model or a multimodal semantic model, used to extract business semantics from operational features. The first model first receives the action type, initial locator, and context parameters of the operation instruction, generates a unified representation through a semantic encoding network, and then uses a context attention mechanism to infer the business purpose behind the operation. For example, for the instruction "click('btn_submit')", the model can identify that the operation belongs to the business semantics of "submitting a form"; for the instruction "input('password','123456')", the model can identify its business semantics as "entering a password".

[0091] In another embodiment, the first model can perform transfer learning by combining semantic samples from an existing test knowledge base. The system can utilize the "instruction-business semantic" correspondence already labeled in historical script data to pre-train the model to adapt to the expression patterns of a specific domain. For example, "click('confirm_order')" corresponds to "confirm order" in an e-commerce system, while it corresponds to "submit application" in a government affairs system. Through domain-adaptive training, the model can generate more accurate semantic descriptions in different business scenarios.

[0092] In some embodiments, in addition to outputting a business semantic description, the first model also generates a candidate feature set for subsequent location of interface elements. The candidate feature set may include the element's text label, type (button, input box, dropdown list), parent hierarchy, visual features, component role attributes, relative position, and contextual features of elements at the same level. For example, the first model may map the "login button" to the candidate features {"text": "login", "type": "Button", "role": "SubmitAction"}. The introduction of the candidate feature set allows the subsequent location process to no longer rely on a single identifier, but rather on multi-feature fusion matching, thereby improving the robustness of the location process.

[0093] In operation S330, the target test step set is obtained based on the business semantic description and the candidate feature set.

[0094] In the embodiments of this application, the system performs semantic recombination and structured mapping on the operation sequence in the original script based on the business semantic description and candidate feature set generated by the first model, forming a target test step set. Each target test step includes a business objective, action type, candidate feature set, and contextual constraints. For example, the three instructions "enter username—enter password—click login" in the original script can be integrated into a high-level business step to perform the login operation, and its internal actions and interface elements are bound through the candidate feature set. This process improves the abstraction level of the test script, and the system can dynamically select elements and paths based on the semantic description in subsequent execution stages.

[0095] In some embodiments, the system can persist the generated target test step set as a semantic template for reuse in subsequent test tasks. When similar business processes (such as registration, payment, and search) reappear, the system can directly call the existing semantic template to automatically generate test scripts that conform to the new interface structure, significantly reducing manual writing and maintenance costs. Furthermore, the target test step set can also be used for visual display, enabling test engineers to understand the test logic in natural language, facilitating review and version management.

[0096] In some embodiments, the system can perform consistency verification on the target test step set to ensure that the model-generated results match the original script intent. This verification can be achieved by comparing the semantic description generated by the model with the operation results in the execution log. When the difference exceeds a threshold, the system can trigger manual review or retraining. Through the above mechanism, the target test step set not only improves interpretability but also ensures semantic consistency and business correctness.

[0097] According to embodiments of this application, a semantic reconstruction of traditional test scripts is achieved. Compared to existing technologies that rely solely on locators or hard-coded logic, embodiments of this application enable the system to understand the business meaning of test behaviors, thereby improving script readability and maintainability. By introducing business semantic descriptions, test scripts can maintain consistency across different versions and language interfaces; by generating candidate feature sets, the system can accurately match target elements even when the interface structure changes, avoiding large-scale script failures. Furthermore, through semantic templating and context verification mechanisms, high-level abstraction and automatic reuse of test logic can be achieved, enabling automated testing to move from instruction execution to semantic understanding, significantly reducing maintenance costs and the need for manual intervention while improving testing efficiency.

[0098] In the embodiments of this application, the first model is used to determine the location method of the target interface element based on the business semantic description. This solves the problems of strong locator dependence and easy failure in traditional UI automated testing, enabling the system to accurately identify and operate the target element even when the interface layout or element attributes change. This process is completed by the first model based on semantic understanding, candidate feature matching, and dynamic verification mechanisms, realizing the transformation from static location to semantic-driven location.

[0099] In the embodiments of this application, the system can take the corresponding business semantic description and candidate feature set as input for each step in the target test step set. The first model first performs semantic parsing on the business semantic description, converting it into a high-level expression for the element recognition task. For example, when the business semantic description is "click the login button", the first model parses the description as needing to find "an interactive button with login semantics" on the current page. Subsequently, the first model generates multiple candidate positioning methods based on the candidate feature set (including element text, component type, visual labels, hierarchical relationships, parent node attributes, adjacent element features, etc.), such as positioning coordinates based on visual regions, semantic positioning based on text matching, or dynamic identifier positioning based on role attributes.

[0100] In some embodiments, multiple candidate positioning methods can be verified and their credibility evaluated one by one. Verification can be performed through methods such as simulated clicks, component detection, or page snapshot analysis. The first model can compare the degree of matching between each candidate positioning method and the expected business semantics, and calculate its credibility score. For example, if multiple buttons contain the text "Login," but only one button is located in the main operation area and has a click event binding, then this positioning method has the highest credibility. The system sorts the candidate positioning methods according to their credibility and selects the highest-ranked one as the target candidate positioning method.

[0101] In some embodiments, to address interface version updates or cross-device compatibility issues, the system can also assist in localization through a multimodal matching mechanism. When generating candidate localization methods, the first model can simultaneously utilize visual information from interface screenshots and semantic information from the DOM structure for fusion analysis. When text or attributes change, the model can identify target elements through visual features (such as color, shape, and layout position), thereby ensuring that the localization process does not rely on a single text identifier and improving the system's robustness across different interface themes or language versions.

[0102] In the embodiments of this application, during the test execution phase, the first model can automatically determine whether the execution results of the test steps or set of steps meet expectations based on semantic consistency analysis, thereby realizing intelligent monitoring and anomaly identification of the UI automated testing process.

[0103] Figure 4 The illustration shows a schematic diagram of a method for determining the failure of a set of target test steps according to some exemplary embodiments of this application.

[0104] like Figure 4 As shown, the method for determining the failure of the target test step set may include operations S410 to S430.

[0105] In operation S410, the actual interface output and the business semantic description are input into the first model to generate a semantic consistency score or judgment result.

[0106] In the embodiments of this application, during the execution of the target test step set, the system can capture the actual interface output in real time, such as the DOM structure of the operation result page, screenshot content, visual text, or status flags. Subsequently, this actual interface output, along with the corresponding business semantic description of the step, is input into the first model to generate a semantic consistency score or logical judgment result. The first model acts as a semantic reviewer in this process, using semantic alignment and intent matching mechanisms to determine whether the actual execution result meets the logical expectations of the business semantics.

[0107] In another embodiment, the first model can directly output a Boolean decision result instead of continuous scoring. The first model uses logical reasoning or rule enhancement mechanisms to determine whether there is a semantic conflict between the current interface state and the expected business result. For example, when the expected behavior is "form submission successful," but the interface output includes an "input error" message, the first model directly outputs an "inconsistent" decision result. This approach simplifies subsequent logical processing and is suitable for business scenarios with clear results and a high degree of standardization, such as order creation, data submission, or form validation.

[0108] In operation S420, when the semantic consistency score is lower than the first preset threshold or the determination result is inconsistent, the corresponding target test step is marked as execution failure.

[0109] In the embodiments of this application, the system determines the execution status of each target test step based on the output score or judgment result. When the semantic consistency score is lower than a first preset threshold, or the model output judgment result is "inconsistent", the system automatically marks the target test step as an execution failure.

[0110] In some embodiments, the determination of the first preset threshold can be automatically adjusted based on the statistical distribution of historical test data. The system can collect the semantic score distribution of different types of test steps during long-term operation and optimize the threshold setting according to the false positive rate. For example, in form-type tests, the score threshold can be set to 0.8, while in visual inspection tests, it can be relaxed to 0.7.

[0111] In operation S430, in response to detecting at least one target test step marked as execution failure, it is determined that the target test step set has failed.

[0112] In the embodiments of this application, when the system detects that at least one target test step has been marked as failed, the entire set of target test steps can be determined to have failed. This design follows the principle of the weakest link in automated testing, meaning that any failure of a critical step may result in the test objective not being achieved. Based on this, the system can trigger subsequent anomaly diagnosis, submitting the semantic description of the failed step, execution logs, and interface output to the model for root cause analysis.

[0113] Compared to existing detection methods that rely on hard-coded assertions or pixel-level comparisons, the embodiments of this application can understand the correctness of the results at the semantic level, exhibiting stronger generalization and context adaptability. The model performs semantic consistency analysis by fusing textual, visual, and structural features, effectively identifying logical errors and latent anomalies, reducing missed detections and false positives.

[0114] In the embodiments of this application, the determination of the failure of the target test step set can not only be based on semantic consistency analysis, but also combined with a runtime multimodal anomaly monitoring mechanism to achieve higher precision anomaly detection. Specifically, during the execution of the target test step set, the system can monitor the running status of each test step in real time. When a location failure, interaction anomaly, or output anomaly is detected, the system automatically triggers an anomaly data collection and analysis process to achieve intelligent execution failure determination based on multimodal data.

[0115] In some embodiments, the system continuously captures execution process information for each target test step through a monitoring module, including operation response time, element visibility, interface loading status, and interaction event feedback. When the model detects that a certain indicator exceeds a preset threshold or an abnormal event occurs, such as element positioning timeout, unresponsive click operation, or missing output content, the system immediately generates an abnormal event flag and enters the abnormal collection phase. This monitoring mechanism can cover multiple potential abnormal types in UI automation execution, achieving comprehensive capture of both explicit and implicit errors.

[0116] In some embodiments, in response to the detection of an abnormal event, the system can automatically collect the corresponding runtime log, a screenshot of the interface at the time of the abnormality, and a business semantic description associated with the step, thereby constructing a multimodal abnormal data packet containing text and image information. The runtime log records system calls, exception stack traces, and operation feedback information to reflect the underlying execution behavior; the screenshot provides the visual state when the abnormality occurs, used to identify UI-level errors; and the business semantic description explains the expected business intent of the step. By combining these three elements, the system can achieve semantic expression of abnormalities at the data level, providing subsequent model analysis with global contextual information.

[0117] In some embodiments, multimodal anomaly data packets are input into a first model for anomaly analysis. The model evaluates the anomaly type and severity by fusing textual features from runtime logs with visual features from screenshots. The first model may employ a multimodal Transformer architecture or a cross-modal attention mechanism to extract event sequence features from logs, extract visual anomaly region information from screenshots, and semantically correlate them with business semantic descriptions. The model ultimately outputs an anomaly confidence score to quantify the probability that the current anomaly will cause the test to fail.

[0118] In some embodiments, an anomaly confidence score can be compared with a second preset threshold to determine whether the target test step set has failed. When the score exceeds the second preset threshold, the system automatically determines that the current test step set has failed and triggers an anomaly diagnosis process. The threshold can be dynamically adjusted according to different types of business scenarios. For example, stricter judgment criteria can be set for critical path steps (such as login, payment, and submission), while relatively lenient thresholds can be used for non-critical steps (such as pop-up windows or page scrolling). This hierarchical strategy ensures that the system has higher detection accuracy in sensitive scenarios while avoiding test interruptions caused by false positives.

[0119] In some embodiments, the system can also store multimodal anomaly data packets and their corresponding anomaly confidence scores in a knowledge base for subsequent model optimization and experience transfer. When the system detects anomaly events with similar characteristics again in subsequent tests, it can quickly locate historical cases through similarity matching and improve judgment efficiency by referring to previous scoring results. As the knowledge base continues to expand, the model can gradually learn the distribution patterns of anomaly behaviors under different business types, achieving more accurate and interpretable anomaly judgment capabilities.

[0120] Through the embodiments of this application, the multimodal anomaly detection mechanism introduces semantic understanding and cross-modal analysis capabilities into traditional test monitoring. Compared with traditional methods that rely solely on log keywords or static assertion rules, the embodiments of this application can comprehensively consider the correlation between execution behavior, visual state, and business intent, thereby identifying complex anomalies at the semantic level. This mechanism effectively reduces the false negative rate and the false positive rate, improves the adaptability of the test system to dynamic interfaces, asynchronous operations, and multi-device scenarios, and provides a high-confidence decision-making basis for subsequent anomaly diagnosis and automatic repair.

[0121] In some embodiments, the process of generating diagnostic results using the first model aims to perform intelligent root cause analysis on anomalies occurring during test execution. Specifically, after determining that the target test step set has failed, the system can input the constructed multimodal anomaly data package into the first model to perform fusion inference analysis. As mentioned above, the multimodal anomaly data package includes runtime logs, screenshots of the interface at the time of the anomaly, and business semantic descriptions related to that step. These three elements together constitute a panoramic data input of the anomaly scenario, enabling the model to comprehensively understand the causes of the anomaly from different dimensions.

[0122] In this process, the first model can employ a multimodal neural network architecture, including a text encoder and a visual encoder. The model first extracts textual features from the runtime logs to identify system error messages, event sequences, and call chain anomalies. Simultaneously, it performs visual feature extraction on anomaly screenshots to capture visual issues such as missing UI elements, layout misalignment, or rendering errors. Subsequently, the first model uses a cross-modal attention mechanism to semantically align textual and image features, and combines this with business semantic descriptions for contextual reasoning, thereby generating diagnostic results that reflect the anomaly type, triggering conditions, and potential root causes.

[0123] In some embodiments, the diagnostic results may include anomaly categories (such as location errors, interaction failures, data loading timeouts, etc.), fault trigger points, and suggested repair directions. Unlike traditional analysis methods that rely solely on log keyword matching, embodiments of this application achieve unified diagnosis of logical errors, visual errors, and cross-module anomalies through semantic reasoning. Through this fusion reasoning mechanism, the system can identify the root causes of problems with higher accuracy and interpretability, providing a reliable basis for generating targeted modification code snippets for subsequent repair models, thereby significantly improving the self-diagnostic capabilities and intelligence level of the testing system.

[0124] In the embodiments of this application, the process of generating modified code snippets using the second model is used to achieve automatic repair of test scripts. Specifically, after obtaining the diagnostic results generated by the first model, the system inputs the diagnostic results into the second model, which performs semantic analysis and repair planning on the anomaly type and its root cause, thereby generating modified code snippets that can be directly applied to the target test script. This process constructs an intelligent closed loop from "fault understanding" to "automatic repair," enabling the testing system to have self-repair and continuous evolution capabilities.

[0125] In some embodiments, the second model can first identify the cause of execution failure based on the diagnostic results, determining whether the failure belongs to a type such as locator failure, logical assertion error, element loading anomaly, or environmental dependency problem. The model can extract key failure features and generate a remediation intent representation through semantic analysis of the root cause description and log summary in the diagnostic report. For example, when the diagnostic result indicates "invalid location expression," the model can identify the remediation intent as "update element location method."

[0126] In some embodiments, the second model generates at least two candidate modification schemes based on the identified reasons for execution failure. Each candidate scheme corresponds to a possible remediation strategy, such as refactoring the location path, adding a waiting mechanism, updating the assertion logic, or replacing the compatibility function. The model can generate corresponding candidate modification code snippets for each scheme based on its natural language to code generation capabilities.

[0127] In some embodiments, the system performs static analysis or trial run verification on the generated candidate modified code snippets to evaluate their effectiveness and stability. Static analysis can check code syntax, variable scope, and dependency integrity; trial run verification can execute the modified script in a sandbox environment to obtain execution success rate or error feedback information. Based on the analysis and verification results, the system assigns a reliability score to each candidate modified code snippet, sorts them by score, and selects the scheme with the highest score as the final modified code snippet.

[0128] Through the above embodiments, the automatic repair mechanism based on the second model can generate multiple candidate repair solutions for different anomaly types and perform self-verification, thereby reducing manual intervention while ensuring the correctness of the repair. This method significantly improves the stability and maintenance efficiency of test scripts, enabling the automated testing system to possess intelligent features of self-learning and self-repair.

[0129] In the embodiments of this application, after applying the modified code snippet, the repair result can be automatically verified to ensure that the generated repair solution can effectively resolve the execution anomalies in the target test script. Specifically, after applying the modified code snippet generated by the second model to the target test script, the system will re-execute the target test step set and monitor the execution status and interface response of the script after repair. By re-execution, the system can verify whether the modification has eliminated the faults in the previous test, thereby realizing an automated closed-loop repair verification process.

[0130] In some embodiments, during the repair verification phase, the system performs semantic consistency analysis on the test execution results and compares them with the original business semantic description. When the actual interface output after re-execution remains logically consistent with the business semantic description and no further exception flags are triggered, the system determines that the repair verification has passed. At this point, the system stores the corresponding diagnostic results and the final modified code snippets in the knowledge base, forming a structured record of "exception type - diagnostic result - repair solution". Through this mechanism, the system can gradually accumulate repair experience for different types of exceptions, enabling the direct reuse of historical solutions for subsequent similar problems, significantly shortening future repair time and improving the model's self-learning capability.

[0131] In another embodiment, when the repair verification fails, the system can automatically collect new execution data and initiate a re-diagnosis process. Specifically, when execution fails, the system will re-acquire updated runtime logs, screenshots of the interface at the time of the anomaly, and the corresponding business semantic description of that step, combining these three into an updated multimodal anomaly data packet. This updated data packet reflects new anomaly information or residual issues that appear after the application repair. The system then re-inputs this updated multimodal anomaly data packet into the first model, performs semantic understanding and cross-modal reasoning analysis, and generates new diagnostic results.

[0132] In some embodiments, the system can invoke the second model again based on the new diagnostic results, generate updated modified code snippets, and reapply them to the target test script to complete the re-repair operation. If the repair still fails, the system can repeatedly execute the above process until the repair verification is successful or the preset number of iterations is reached. Through this cyclical diagnosis and re-repair mechanism, the system forms a self-learning "execution-verification-diagnosis-repair" closed-loop process, enabling it to continuously optimize the model's judgment accuracy and repair effect in multiple iterations.

[0133] Through the above embodiments, the repair verification and re-diagnosis mechanism achieves full automation and continuous optimization of the test script repair process. Compared with traditional manual verification, the embodiments of this application can automatically determine the effectiveness of the repair without human intervention, and accumulate experience into the knowledge base to form a long-term learning effect, thereby significantly improving the system's self-evolution capability and the reliability of automated testing.

[0134] Corresponding to the above-described user interface testing method, embodiments of this application also provide a user interface testing apparatus.

[0135] Figure 5 A schematic block diagram of a test apparatus for a user interface according to an embodiment of this application is shown.

[0136] like Figure 5 As shown, the user interface testing device 500 of this embodiment includes a data acquisition module 510, a test step execution module 520, a diagnostic result generation module 530, and a code repair module 540.

[0137] The data acquisition module 510 can be used to acquire a target test script, and use a first model to perform semantic transformation on the operation instructions in the target test script to obtain a target test step set containing business semantic descriptions. In one embodiment, the data acquisition module 510 can be used to execute the operation S210 described above, which will not be repeated here.

[0138] The test step execution module 520 can be used to execute the target test step set. During the execution of the target test step set, the first model is used to determine the positioning method of the target interface elements based on the business semantic description, so as to execute the corresponding interactive operation and obtain the actual interface output. In one embodiment, the test step execution module 520 can be used to execute the operation S220 described above, which will not be repeated here.

[0139] The diagnostic result generation module 530 can be used to generate a diagnostic result using the first model in response to the failure of the target test step set execution, based at least on the actual interface output and the business semantic description. In one embodiment, the diagnostic result generation module 530 can be used to perform the operation S230 described above, which will not be repeated here.

[0140] The code repair module 540 can be used to input the diagnostic results into the second model, which generates modified code snippets and applies the modified code snippets to the target test script. In one embodiment, the code repair module 540 can be used to perform the operation S240 described above, which will not be repeated here.

[0141] According to an embodiment of this application, the data acquisition module 510 can also be used to parse the target test script, obtain the action type, initial locator and context parameters corresponding to each operation instruction; input the action type, the initial locator and the context parameters into the first model to generate the corresponding business semantic description and candidate feature set; and obtain the target test step set based on the business semantic description and the candidate feature set.

[0142] According to an embodiment of this application, the test step execution module 520 can also be used for each step in the target test step set: generating multiple candidate positioning methods based on the business semantic description and candidate feature set using the first model; verifying and sorting the candidate positioning methods to obtain the target candidate positioning method, and determining the target candidate positioning method as the positioning method of the target interface element.

[0143] According to an embodiment of this application, the user interface testing device 500 may include a failure determination module. The failure determination module can be used to input the actual interface output and the business semantic description into the first model to generate a semantic consistency score or judgment result; when the semantic consistency score is lower than a first preset threshold or the judgment result is inconsistent, the corresponding target test step is marked as an execution failure; and in response to detecting at least one target test step marked as an execution failure, the target test step set is determined to have failed.

[0144] According to an embodiment of this application, the failure determination module can also be used to monitor the running status of each target test step during the execution of the target test step set; in response to detecting a location failure, interaction anomaly, or output anomaly, collect the corresponding running logs, screenshots of the interface at the time of the anomaly, and the business semantic description to construct a multimodal anomaly data packet; input the multimodal anomaly data packet into the first model and use the first model to generate a corresponding anomaly confidence score; and in response to the anomaly confidence score exceeding a second preset threshold, determine that the execution of the target test step set has failed.

[0145] According to an embodiment of this application, the diagnostic result generation module 530 can also be used to input the multimodal abnormal data packet into the first model, and use the first model to perform fusion reasoning of text information and image information on the operation log, the screenshot of the interface at the abnormal moment and the business semantic description to generate the diagnostic result.

[0146] According to an embodiment of this application, the code repair module 540 can also be used to identify the cause of execution failure by the second model based on the diagnostic results; generate at least two candidate modification schemes based on the cause of execution failure, and generate corresponding candidate modified code fragments for the candidate modification schemes; perform static analysis or trial operation verification on the candidate modified code fragments, and sort them according to the execution success rate or confidence score, and obtain the modified code fragments based on the sorting results.

[0147] According to an embodiment of this application, the code repair module 540 can also be used to re-execute the target test step set for repair verification after applying the modified code snippet; in response to the repair verification passing, store the diagnostic result and the corresponding modified code snippet in the knowledge base; and in response to the repair verification failing, obtain the update execution log, the update interface screenshot and the update business semantic description, combine the update execution log, the update interface screenshot and the update business semantic description into an update multimodal anomaly data packet, and perform re-diagnosis and re-repair based on the update multimodal anomaly data packet.

[0148] According to embodiments of this application, any multiple modules among the data acquisition module 510, test step execution module 520, diagnostic result generation module 530, and code repair module 540 can be combined into one module, or any one of these modules can be split into multiple modules. Alternatively, at least some of the functions of one or more of these modules can be combined with at least some of the functions of other modules and implemented in one module. According to embodiments of this application, at least one of the data acquisition module 510, test step execution module 520, diagnostic result generation module 530, and code repair module 540 can be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or implemented in hardware or firmware by any other reasonable means of integrating or packaging the circuitry, or implemented in software, hardware, or firmware, or in any suitable combination of any of these three implementation methods. Alternatively, at least one of the data acquisition module 510, the test step execution module 520, the diagnostic result generation module 530, and the code repair module 540 may be implemented at least partially as a computer program module, which can perform corresponding functions when the computer program module is run.

[0149] Figure 6 A block diagram schematically illustrates an electronic device suitable for implementing a user interface testing method according to an embodiment of this application.

[0150] like Figure 6 As shown, an electronic device 600 according to an embodiment of this application includes a processor 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage portion 608 into a random access memory (RAM) 603. The processor 601 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 601 may also include onboard memory for caching purposes. The processor 601 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of this application.

[0151] RAM 603 stores various programs and data required for the operation of electronic device 600. Processor 601, ROM 602, and RAM 603 are interconnected via bus 604. Processor 601 executes various operations of the method flow according to embodiments of this application by executing programs in ROM 602 and / or RAM 603. It should be noted that the programs may also be stored in one or more memories other than ROM 602 and RAM 603. Processor 601 may also execute various operations of the method flow according to embodiments of this application by executing programs stored in said one or more memories.

[0152] According to embodiments of this application, the electronic device 600 may further include an input / output (I / O) interface 605, which is also connected to a bus 604. The electronic device 600 may also include one or more of the following components connected to the input / output (I / O) interface 605: an input section 606 including a keyboard, mouse, etc.; an output section 607 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 608 including a hard disk, etc.; and a communication section 609 including a network interface card such as a LAN card, modem, etc. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the input / output (I / O) interface 605 as needed. A removable medium 611, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 610 as needed so that computer programs read from it can be installed into the storage section 608 as needed.

[0153] This application also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs, which, when executed, implement the method according to the embodiments of this application.

[0154] According to embodiments of this application, the computer-readable storage medium can be a non-volatile computer-readable storage medium, such as including but not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this application, the computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to embodiments of this application, the computer-readable storage medium may include ROM 602 and / or RAM 603 and / or one or more memories other than ROM 602 and RAM 603 described above.

[0155] Embodiments of this application also include a computer program product comprising a computer program containing program code for performing the methods shown in the flowchart. When the computer program product is run on a computer system, the program code is used to enable the computer system to implement the user interface testing method provided in the embodiments of this application.

[0156] When the computer program is executed by the processor 601, it performs the functions defined in the system / apparatus of this application embodiment. According to the embodiments of this application, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0157] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and downloaded and installed via the communication section 609, and / or installed from the removable medium 611. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0158] In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 609, and / or installed from the removable medium 611. When the computer program is executed by the processor 601, it performs the functions defined in the system of this application embodiment. According to the embodiments of this application, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0159] According to embodiments of this application, program code for executing the computer programs provided in the embodiments of this application can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can be executed entirely on the user's computing device, partially on the user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0160] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0161] Those skilled in the art will understand that the features described in the various embodiments of this application can be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in this application. In particular, the features described in the various embodiments of this application can be combined and / or combined in various ways without departing from the spirit and teachings of this application. All such combinations and / or combinations fall within the scope of this application.

[0162] The embodiments of this application have been described above. However, these embodiments are merely illustrative and not intended to limit the scope of this application. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. Without departing from the scope of this application, those skilled in the art can make various substitutions and modifications, all of which should fall within the scope of this application.

Claims

1. A method for testing a user interface, characterized in that, The method includes: Obtain the target test script, and use the first model to perform semantic transformation on the operation instructions in the target test script to obtain a target test step set containing business semantic descriptions; The target test step set is executed, wherein, during the execution of the target test step set, the first model is used to determine the positioning method of the target interface element according to the business semantic description, and the corresponding interactive operation is executed to obtain the actual interface output; In response to the failure of the target test step set, a diagnostic result is generated using the first model, based at least on the actual interface output and the business semantic description; and The diagnostic results are input into the second model, which generates a modified code snippet and applies the modified code snippet to the target test script.

2. The method according to claim 1, characterized in that, The step of semantically transforming the operation instructions in the target test script using the first model to obtain a target test step set containing business semantic descriptions includes: Parse the target test script to obtain the action type, initial locator, and context parameters corresponding to each operation instruction; The action type, the initial locator, and the context parameters are input into the first model to generate a corresponding business semantic description and candidate feature set; and Based on the business semantic description and the candidate feature set, the target test step set is obtained.

3. The method according to claim 2, characterized in that, The method of determining the location of target interface elements based on the business semantic description using the first model includes: For each step in the target test step set: based on the business semantic description and candidate feature set, generate multiple candidate positioning methods using the first model; verify and sort the candidate positioning methods to obtain the target candidate positioning method, and determine the target candidate positioning method as the positioning method of the target interface element.

4. The method according to claim 1, characterized in that, The determination of the failure of the target test step set includes: The actual interface output and the business semantic description are input into the first model to generate a semantic consistency score or judgment result. When the semantic consistency score is lower than a first preset threshold or the determination result is inconsistent, the corresponding target test step is marked as execution failure; and In response to detecting at least one target test step marked as having failed to execute, it is determined that the target test step set has failed to execute.

5. The method according to claim 1, characterized in that, The determination of the failure of the target test step set includes: During the execution of the target test step set, the running status of each target test step is monitored; In response to the detection of positioning failure, interaction abnormality, or output abnormality, the corresponding operation log, screenshot of the interface at the time of the abnormality, and the business semantic description are collected to construct a multimodal abnormal data packet. The multimodal anomaly data packets are input into the first model, and the first model is used to generate corresponding anomaly confidence scores; and In response to the abnormal confidence score exceeding a second preset threshold, it is determined that the execution of the target test step set has failed.

6. The method according to claim 5, characterized in that, The step of generating diagnostic results using the first model includes: The multimodal anomaly data packet is input into the first model, and the first model is used to perform text and image information fusion reasoning on the operation log, the screenshot of the interface at the abnormal moment, and the business semantic description to generate the diagnostic result.

7. The method according to claim 1, characterized in that, The step of inputting the diagnostic results into the second model, and having the second model generate modified code snippets, includes: Based on the diagnostic results, the second model identifies the reasons for the execution failure; Based on the reasons for the execution failure, at least two candidate modification schemes are generated, and corresponding candidate modification code fragments are generated for the candidate modification schemes. The candidate modified code snippets are subjected to static analysis or trial operation verification, and are sorted according to execution success rate or confidence score. The modified code snippets are obtained based on the sorting results.

8. The method according to claim 1, characterized in that, The method further includes: After applying the modified code snippet, the target test set is re-executed to verify the fix; Upon successful repair verification, the diagnostic results and the corresponding modified code snippets are stored in the knowledge base; as well as In response to the failure of the repair verification, the update execution log, the update interface screenshot, and the update business semantic description are obtained. The update execution log, the update interface screenshot, and the update business semantic description are combined into an update multimodal anomaly data packet, and re-diagnosis and re-repair are performed based on the update multimodal anomaly data packet.

9. A testing device for a user interface, characterized in that, The device includes: The data acquisition module is used to: acquire the target test script, and use the first model to perform semantic transformation on the operation instructions in the target test script to obtain a target test step set containing business semantic descriptions; The test step execution module is used to: execute the target test step set, wherein, during the execution of the target test step set, the first model is used to determine the positioning method of the target interface element according to the business semantic description, the corresponding interactive operation is executed, and the actual interface output is obtained; The diagnostic result generation module is configured to: in response to the failure of the target test step set execution, generate a diagnostic result using the first model, based at least on the actual interface output and the business semantic description; and The code repair module is used to: input the diagnostic results into the second model, generate modified code snippets from the second model, and apply the modified code snippets to the target test script.

10. An electronic device, comprising: One or more processors; Memory, used to store one or more computer programs. The characteristic feature is that the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1 to 8.

11. A computer-readable storage medium having a computer program or instructions stored thereon, characterized in that, When the computer program or instructions are executed by a processor, they implement the steps of the method according to any one of claims 1 to 8.

12. A computer program product, comprising a computer program or instructions, characterized in that, When the computer program or instructions are executed by a processor, they implement the steps of the method according to any one of claims 1 to 8.

Images