AI-agent based ui automated testing script self-repairing and evolving system and method

Abstract

This invention discloses an AI-Agent-based self-repair and evolution system and method for UI automated test scripts, relating to the field of artificial intelligence. It addresses the difficulties in script maintenance in existing technologies. A multimodal perception module connects to the user interface flow and test execution flow of the application under test. An intelligent analysis and decision-making module receives signals from the multimodal perception module and generates repair decisions. A secure execution verification module executes the instructions of the intelligent analysis and decision-making module and verifies the results. A knowledge learning and evolution module stores and processes cases and optimizes system intelligence. The system integrates with a CI / CD server through an application programming interface (API) and is automatically triggered when tests fail. Advantages include: convenient management, reduced maintenance burden; intelligent and accurate repair, ensuring the robustness of test assets; and continuous evolution capabilities, enabling knowledge accumulation and value-added. It not only handles current problems but also identifies high-frequency failure modes and proactively proposes optimization suggestions for the script architecture.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence, and in particular to a self-healing and evolution system and method for UI automation test scripts based on AI-Agent. Background Technology

[0002] With the widespread adoption of agile software development and continuous delivery (CI / CD) models, UI (user interface) automated testing has become a crucial step in ensuring the quality of front-end applications such as web and mobile devices. However, current mainstream automated testing frameworks (such as Selenium, Appium, and Cypress) and related script maintenance methods have revealed a series of fundamental flaws when facing the rapidly iterating modern UI. This results in fragile test scripts, high maintenance costs, and severely restricts the return on investment of automated testing. The main defects and shortcomings are as follows:

[0003] 1. Vulnerability and High Maintenance Costs of Test Scripts. Current technologies heavily rely on the identification of static locators for UI elements (such as XPath and CSS selectors). Once the application's front-end UI changes (e.g., element ID changes, layout structure adjustments, control type replacements), these locators become invalid, causing test scripts to fail. Maintainers must manually locate the changes, find new locators, and update the scripts—a tedious, time-consuming, and error-prone process. Statistics show that in rapidly iterating products, test script maintenance can account for over 60% of the total cost of automated testing.

[0004] 2. Lack of self-healing capabilities and low level of automation. Some commercial tools and open-source solutions on the market attempt to provide script-based repair functionality, but these are essentially simple replacements based on predefined rules or fixed patterns (e.g., attempting to replace an ID with a name attribute when it becomes invalid). These methods have serious limitations: Rigid and unintelligent: It cannot understand the semantic structure and business logic of the UI, and is completely powerless to deal with complex structural changes (such as the reconstruction of the entire module or changes in the interaction process).

[0005] Lack of reasoning ability: It is unable to determine whether a newly found element is the correct replacement for the original element in the business context, which may result in the script performing incorrect operations after being repaired.

[0006] Passive response: It is triggered only after the script fails to execute. It is a post-event remedy and cannot proactively predict or prevent script failures caused by UI changes during the development phase or before deployment.

[0007] 3. Lack of continuous evolution and optimization capabilities. Existing test scripts are static, one-off products. They cannot learn from historical successes / failures, nor can they synchronously optimize their test logic and positioning strategies as the product UI evolves. These shortcomings include: Unable to accumulate knowledge: Each repair is an independent event, and experience in solving similar problems cannot be accumulated and reused.

[0008] Unable to adapt to style evolution: When the UI design language or interaction mode of a product undergoes a holistic evolution, the old scripts need to be almost completely rewritten, and a smooth transition cannot be achieved through local adjustments.

[0009] Single positioning strategy: It usually adopts a single or fixed positioning strategy, and cannot dynamically evaluate and adopt the most stable and robust hybrid positioning scheme (such as combining semantic, visual, layout and other multimodal information) in the current page environment.

[0010] 4. Limitations of Existing Intelligent Solutions. In recent years, although some research has attempted to introduce machine learning or computer vision into UI testing, it often focuses on single-point problems (such as image recognition replacing localization) or is merely a supplement to traditional locators. These solutions fail to construct an intelligent entity (Agent) with a closed loop of perception, decision-making, execution, and learning, thus failing to achieve full-process autonomy from "identifying element failure" to "understanding the intent of change," then to "formulating and executing repair strategies," and finally to "optimizing future strategies." Summary of the Invention

[0011] The purpose of this invention is to provide a self-repairing and evolution system and method for UI automation test scripts based on AI-Agent, so as to solve the problems existing in the prior art.

[0012] The AI-Agent-based UI automation test script self-repair and evolution system described in this invention includes a multimodal perception module, an intelligent analysis and decision-making module, a secure execution verification module, and a knowledge learning and evolution module. The multimodal perception module accesses the user interface flow and test execution flow of the application under test; the intelligent analysis and decision module receives signals from the multimodal perception module and generates repair decisions; the security execution verification module executes the instructions of the intelligent analysis and decision module and verifies the results; and the knowledge learning and evolution module stores and processes cases and optimizes system intelligence. The system is integrated with the CI / CD server via an application programming interface and is automatically triggered when a test fails.

[0013] The self-repair and evolution method of UI automation test script based on AI-Agent described in this invention is accomplished using the system described above.

[0014] The AI-Agent-based UI automation test script self-repair and evolution system and method described in this invention have the following technical advantages: 1. Convenient management reduces maintenance burden, enabling autonomous maintenance of UI automated test scripts. When changes to the application interface cause scripts to malfunction, the system can automatically detect, intelligently analyze, securely repair, and verify the results, without requiring test engineers to manually locate, analyze, and modify the code. This fundamentally liberates testers from repetitive and tedious script maintenance work, allowing them to focus on higher-level test design and business verification.

[0015] 2. Intelligent and precise repair ensures the robustness of test assets. Through multimodal perception and intelligent reasoning based on large language models, it can understand the context and semantic intent of UI changes, surpassing matching based on simple rules or fixed patterns. The system-generated repair solutions (such as optimal locator selection) not only enable scripts to resume operation, but also ensure that the repaired scripts remain correct at the business logic level through stability assessment algorithms and regression verification. This effectively avoids secondary errors or logical deviations that may be introduced by traditional manual repair, significantly improving the long-term robustness and reliability of test assets.

[0016] 3. Possessing continuous evolution capabilities, enabling knowledge accumulation and value enhancement, its core difference from one-off repair tools lies in its built-in learning and evolution mechanism. The system continuously accumulates experience through a knowledge graph of repair cases and uses this data to continuously optimize the core reasoning model and strategy evaluation algorithm. This allows the system to become smarter with use, not only handling current problems but also identifying high-frequency failure modes and proactively proposing optimization suggestions for the script architecture (such as promoting the adoption of more stable positioning strategies). This drives the entire test suite to continuously evolve towards greater stability and maintainability, transforming test scripts from high-maintenance-cost liabilities into self-improving intelligent assets. Attached Figure Description

[0017] Figure 1 This is a schematic diagram of the system described in this invention.

[0018] Figure 2 This is a flowchart illustrating the method described in this invention.

[0019] Figure 3 This is a schematic diagram of the workflow of the intelligent analysis and decision-making module described in this invention. Detailed Implementation

[0020] like Figure 1As shown, the AI-Agent-based UI automated test script self-repair and evolution system of this invention includes a multimodal perception module, an intelligent analysis and decision-making module, a secure execution verification module, and a knowledge learning and evolution module. The multimodal perception module accesses the user interface flow and test execution flow of the application under test. The intelligent analysis and decision-making module receives signals from the multimodal perception module and generates repair decisions. The secure execution verification module executes the instructions of the intelligent analysis and decision-making module and verifies the results. The knowledge learning and evolution module stores and processes cases and optimizes the system's intelligence.

[0021] The multimodal perception module is used to capture real-time state changes of the user interface, and its structure includes: Structured listening units: By embedding hooks in browsers or testing frameworks, they monitor changes in the properties, structure, and events of Document Object Model (DOM) elements.

[0022] Visual perception unit: Based on computer vision models, it analyzes screenshots of the interface during the testing process to identify changes in the style and layout of the visual rendering layer.

[0023] Context correlator: Aligns and merges the raw events captured by the structured listening unit and the visual perception unit according to timestamps and spatial locations to generate a composite change event description with business context (such as associated test steps).

[0024] The intelligent analysis and decision-making module, as the core AI-Agent, is used to understand changes and formulate remediation strategies. Its structure includes: The core inference engine consists of a large language model (LLM) fine-tuned through software engineering, test scripts, and UI design corpora, used to understand the semantics and underlying intent of complex change events (such as feature upgrades, defect fixes, and style refactoring).

[0025] Dynamic positioning strategy evaluator: When it is necessary to find a locator for a new element, it simultaneously generates candidate positioning strategies based on XPath, CSS selectors, text content, visual features and custom attributes, and scores and sorts them according to preset stability, uniqueness and performance weights.

[0026] Repair and Evolution Strategy Generator: Based on the intent judgment of the inference engine and the scoring results of the location strategy dynamic evaluator, it generates atomic code modification instructions; and when high-frequency failure modes are identified, it proposes higher-order script architecture evolution suggestions (such as introducing data-driven testing and recommending the addition of stable test attributes).

[0027] The secure execution verification module is used to apply the repair and verify its effectiveness in an isolated environment, and its structure includes: Sandboxed execution environment: Creates an independent copy of the test environment for each fix attempt to ensure that the main test assets are not contaminated.

[0028] Automated verification unit: Automatically runs the fixed test cases and corresponding upstream and downstream related test cases in the sandbox, and collects the execution logs of the entire chain.

[0029] Regression test unit: Based on the data and logical dependencies between test cases, intelligently select and execute upstream and downstream test cases associated with the fix test case to perform regression testing to verify that the fix has not introduced any new defects or side effects.

[0030] Results Comparison and Decision Maker: The repaired test results are compared with historical success benchmarks to ensure that business verification points are equivalent, thereby determining whether the repair was successful.

[0031] The knowledge learning and evolution module is used to achieve continuous system optimization, and its structure includes: Repair Case Knowledge Graph: Successful repair cases are stored in the form of a graph database. Nodes include UI elements, change patterns, location strategies, and code snippets. Edges are used to describe the relationships between nodes. For example, "Element A" uses "Strategy B", and "Strategy B" is applicable to "Pattern C".

[0032] Model and strategy continuous optimization pipeline: High-quality repair cases are used as new training data to incrementally fine-tune the inference engine; at the same time, the scoring weights of the dynamic evaluator of the localization strategy are dynamically adjusted based on the long-term failure statistics of each localization strategy.

[0033] The system is integrated with the CI / CD server via an application programming interface (API) and is automatically triggered when a test fails.

[0034] like Figure 2 and Figure 3 As shown, the self-repair and evolution method of UI automation test script based on AI-Agent described in this invention is based on the system and includes the following steps: S1. Change Awareness Steps: The multimodal awareness module continuously monitors the UI of the application under test. When the test script execution is interrupted due to a malfunctioning UI element, or when the CI / CD server pushes new front-end code for deployment, the system is triggered to start.

[0035] S2. Information Acquisition and Analysis Steps: The multimodal perception module captures the current interface state and synchronously acquires it with the interface baseline snapshot from the last successful execution. The interface baseline snapshot comes from historical records. The context association unit fuses the acquired document object model changes with visual differences to generate a structured change event description, which is then sent to the intelligent analysis and decision-making module.

[0036] S3. Intelligent Decision-Making Steps: The inference engine analyzes change events to determine whether the current event is an element-level update, component replacement, or process refactoring. If it is an element-level change, the location strategy dynamic evaluator generates and evaluates several candidate locators for the corresponding element. If it is a component replacement or process refactoring, the core inference engine analyzes its business impact scope and logic, and generates corresponding component adjustment or process modification plans. The repair and evolution strategy generator outputs specific code repair instructions (such as replacing locators or adjusting waiting logic) or architecture optimization suggestions based on the analysis results.

[0037] The algorithm of the dynamic evaluator for the positioning strategy is as follows: For the target UI element, feature sets in three dimensions—XPath, text content, and visual template—are obtained through document object model parsing, optical character recognition, and feature matching. For each candidate locator, a uniqueness score, a stability score, and a performance score are calculated. The uniqueness score represents the confidence that the corresponding element can be uniquely located on the current page. The stability score represents the robustness against common attribute changes of the corresponding element. The performance score is an estimate of the time complexity of the search operation. Based on feedback from the knowledge learning evolution module and the actual efficiency of each strategy in history, a comprehensive score of the three scores is dynamically weighted and calculated. The positioning strategy with the highest comprehensive score is selected as the preferred recommendation.

[0038] The decision logic of the inference engine is as follows: the input includes prompts containing the changed element, change type, page URL, and associated test business intent. The output is a classification of the changes, such as "batch changes to class names caused by component library version upgrade", and / or a repair priority, such as "high - need to be repaired immediately and the team notified".

[0039] S4. Security Execution and Verification Steps: The security execution verification module modifies the test scripts using generated remediation instructions within the sandbox environment. The remediated scripts and related regression test cases are automatically run in the sandbox. The result comparison and decision-making system compares the execution results with historical benchmarks to verify whether business functionality has been restored.

[0040] The specific steps for result comparison are as follows: Check the pass and fail status of the test cases; compare the similarity of screenshots of key business checkpoints; and verify the consistency of key data fields in network interface requests and responses. The screenshot similarity comparison uses the Structural Similarity Index (SSIM) algorithm. A successful repair is only determined when all comparison items are within a preset tolerance threshold.

[0041] S5. Result Judgment and Learning Steps: If verification passes, the knowledge learning and evolution module will structure the current repair case and store it in the repair case knowledge graph, and use it to optimize the internal model. The repair case includes the change context, the adopted strategy, and the final code. The passed repair scripts will be merged back into the main test code library to complete this automatic repair. If verification fails, the system will record the reason for the failure and re-analyze it or upgrade it to an event requiring manual handling.

[0042] S6. Steps of Cycle and Evolution: The system runs continuously. The knowledge learning and evolution module periodically analyzes and repairs patterns in the case knowledge graph. When the failure rate of any type of positioning strategy is found to be higher than the threshold, or the change frequency of any UI component is higher than the threshold, it proactively initiates script architecture evolution and optimization suggestions, such as prompting the development team to add a unified data-testid test attribute to relevant components.

[0043] For those skilled in the art, various other corresponding changes and modifications can be made based on the technical solutions and concepts described above, and all such changes and modifications should fall within the protection scope of the claims of this invention.

Claims

1. A UI automation test script self-repair and evolution system based on AI-Agent, characterized in that, It includes a multimodal perception module, an intelligent analysis and decision-making module, a secure execution verification module, and a knowledge learning and evolution module; The multimodal perception module accesses the user interface flow and test execution flow of the application under test; the intelligent analysis and decision module receives signals from the multimodal perception module and generates repair decisions; the security execution verification module executes the instructions of the intelligent analysis and decision module and verifies the results; and the knowledge learning and evolution module stores and processes cases and optimizes system intelligence. The system is integrated with the CI / CD server via an application programming interface and is automatically triggered when a test fails.

2. The self-repair and evolution system for UI automation test scripts based on AI-Agent as described in claim 1, characterized in that, The multimodal sensing module is used to capture real-time state changes of the user interface, and its structure includes: Structured listening unit: By embedding hooks in the browser or test framework, it listens for changes in the properties, structure, and events of document object model elements; Visual perception unit: Based on a computer vision model, it analyzes screenshots of the interface during the testing process and identifies style and layout changes in the visual rendering layer. Context correlator: Aligns and merges the raw events captured by the structured listening unit and the visual perception unit according to timestamps and spatial locations to generate a composite change event description with business context.

3. The self-repair and evolution system for UI automation test scripts based on AI-Agent as described in claim 2, characterized in that, The intelligent analysis and decision-making module, as the core AI-Agent, is used to understand changes and formulate remediation strategies. Its structure includes: Core inference engine: It consists of a large language model that has been fine-tuned through software engineering, test scripts and UI design corpus, and is used to understand the semantics and underlying intent of complex change events; Dynamic evaluator for positioning strategies: When it is necessary to find locators for new elements, it simultaneously generates candidate positioning strategies based on XPath, CSS selectors, text content, visual features and custom attributes, and scores and sorts them according to preset stability, uniqueness and performance weights. Repair and Evolution Strategy Generator: Based on the intent judgment of the inference engine and the scoring results of the location strategy dynamic evaluator, it generates atomic code modification instructions; and when high-frequency failure modes are identified, it proposes higher-order script architecture evolution suggestions.

4. The self-repairing and evolution system for UI automation test scripts based on AI-Agent as described in claim 3, characterized in that, The secure execution verification module is used to apply the repair and verify its effectiveness in an isolated environment, and its structure includes: Sandboxed execution environment: Creates a separate copy of the test environment for each fix attempt; Automated verification unit: Automatically runs the patched test cases and corresponding upstream and downstream related test cases in the sandbox, and collects the full-link execution log; Regression test unit: Based on the data and logical dependencies between test cases, intelligently select and execute upstream and downstream test cases associated with the fix test case to perform regression testing to verify that the fix has not introduced any new defects or side effects; Results Comparison and Decision Maker: The repaired test results are compared with historical success benchmarks to determine whether the repair was successful.

5. The self-repair and evolution system for UI automation test scripts based on AI-Agent as described in claim 4, characterized in that, The knowledge learning and evolution module is used to achieve continuous system optimization, and its structure includes: Repair Case Knowledge Graph: Successful repair cases are stored in the form of a graph database. Nodes include UI elements, change patterns, location strategies, and code snippets, while edges are used to describe the relationships between nodes. Model and policy continuous optimization pipeline: Repair cases are used as new training data to incrementally fine-tune the inference engine; at the same time, the scoring weights of the localization policy dynamic evaluator are dynamically adjusted based on the long-term failure statistics of each localization policy.

6. A self-healing and evolution method for UI automation test scripts based on AI-Agent, characterized in that, This is accomplished using the system described in any one of claims 1-5.

7. The self-repair and evolution method for UI automation test scripts based on AI-Agent as described in claim 6, characterized in that, Includes the following steps: S1. Change perception steps: The multimodal perception module continuously monitors the UI of the application under test; The system is triggered to start when the test script execution is interrupted due to the failure of UI elements, or when the CI / CD server pushes new front-end code for deployment. S2. Information Acquisition and Analysis Steps: The multimodal perception module captures the current interface state and synchronously acquires it with the interface baseline snapshot from the last successful execution; the interface baseline snapshot comes from historical records; The context association tool integrates the collected changes in the document object model with visual differences to generate a structured description of the change event, which is then sent to the intelligent analysis and decision-making module. S3. Intelligent decision-making steps: The inference engine analyzes change events and determines whether the current event is an element-level update, component replacement, or process refactoring; If it is an element-level change, the location strategy dynamic evaluator generates and evaluates several candidate locators for the corresponding element; if it is a component replacement or process refactoring, the inference engine will analyze the scope and logic of the business impact and generate corresponding component adjustment or process modification plans; the repair and evolution strategy generator outputs specific code repair instructions or architecture optimization suggestions based on the analysis results. S4. Steps for Secure Execution and Verification: In the sandbox environment, the secure execution verification module modifies the test scripts using the generated repair instructions; it then automatically runs the repaired scripts and related regression test cases within the sandbox. The results comparison and decision-making system compares the results with historical benchmarks to verify whether business functions have been restored. S5. Result Judgment and Learning Steps: If the verification passes, the knowledge learning and evolution module will structure the current repair case and store it in the repair case knowledge graph, and use it to optimize the internal model; the repair case includes the change context, the adopted strategy, and the final code; the passed repair scripts will be merged back into the main test code library to complete this automatic repair; if the verification fails, the system will record the reason for the failure, re-analyze it, or upgrade it to an event that requires manual handling; S6. Steps of Cycle and Evolution: The system runs continuously; the knowledge learning and evolution module periodically analyzes and repairs the patterns in the case knowledge graph. When the failure rate of any type of positioning strategy is higher than the threshold, or the change frequency of any UI component is higher than the threshold, it actively initiates script architecture evolution and optimization suggestions.

8. The self-repair and evolution method for UI automation test scripts based on AI-Agent according to claim 7, characterized in that, In step S3, the algorithm of the positioning strategy dynamic evaluator is as follows: for the target UI element, feature sets in three dimensions, namely XPath, text content, and visual template, are obtained through document object model parsing, optical character recognition, and feature matching; and uniqueness score, stability score, and performance score are calculated for each candidate locator. The uniqueness score is the confidence level that the corresponding element can be uniquely located on the current page; The stability score is the robustness against changes in common properties of the corresponding element; The performance score is an estimate of the time complexity of performing the search operation; Based on feedback from the knowledge learning evolution module and the actual effectiveness of each strategy in history, a comprehensive score of the three scores is dynamically weighted and calculated. The positioning strategy with the highest overall score is selected as the preferred recommendation.

9. The self-repair and evolution method for UI automation test scripts based on AI-Agent according to claim 8, characterized in that, In step S3, the decision logic of the inference engine is as follows: the input includes prompts containing the changed element, the change type, the page URL, and the associated test business intent; the output is the classification of the change and / or the priority of its repair.

10. The self-repair and evolution method for UI automation test scripts based on AI-Agent according to claim 9, characterized in that, In step S4, the specific steps for result comparison are as follows: check the pass and failure status of the test cases, compare the similarity of screenshots of key business checkpoints, and whether the key data fields of network interface requests and responses are consistent; wherein, the comparison of screenshot similarity is performed using the Structural Similarity Index (SSIM) algorithm; only when all comparison items are within the preset tolerance threshold is the repair considered successful.

Images