An automated self-healing UI testing method and system based on multi-agent AI

Abstract

The application discloses a kind of automated self-healing UI test method and system based on multi-agent AI, belong to automated UI test technical field, including: through test inference interpretation class, original test script is semantized, and output expected intent;When test fails, multiple-source data is synchronously collected and multi-modal feature is extracted, fault propagation graph is constructed according to expected intent and multi-modal feature, fault propagation analysis is carried out using graph attention network, and fault diagnosis result is obtained;According to fault diagnosis result, repair strategy is negotiated and decided using deep Q network, greedy strategy and byzantine fault tolerance algorithm by distributed agent collaborative mechanism, and feedback is obtained;Based on feedback, repair strategy and test inference interpretation class are optimized by meta-learning algorithm, and dynamically adjusted;Through the construction test inference interpretation class, test intent is deeply understood, and distributed agent collaboration is used to complete fault diagnosis and automatic repair, improve semantic understanding ability, diagnostic accuracy and repair success rate.

Description

Technical Field

[0001] This application belongs to the field of automated UI testing technology, and specifically relates to an automated self-healing UI testing method and system based on multi-agent AI. Background Technology

[0002] The current UI automation testing framework has the following specific technical shortcomings:

[0003] The problem of missing semantic understanding: Traditional test scripts only record the sequence of operations (such as "click the button with ID 'submit'"), and cannot understand the business semantics of the operations (such as "submit order"), which leads to the inability to intelligently adapt to UI changes.

[0004] Insufficient diagnostic depth: The existing framework can only locate the cause of surface failure (such as element not found), and cannot trace the deep system failure (such as API timeout caused by database deadlock).

[0005] Fixed repair strategy: Traditional self-healing solutions use preset rules and cannot dynamically adjust strategies based on historical repair results. Summary of the Invention

[0006] To address the issues of missing semantic understanding, insufficient diagnostic depth, and rigid repair strategies, this application provides an automated self-healing UI testing method based on multi-agent AI. Its improvements include:

[0007] The original test script is semantically interpreted by the test reasoning interpretation class, and the expected intent is output.

[0008] When a test fails, multi-source data is collected synchronously and multimodal features are extracted. A fault propagation graph is constructed based on the expected intent and multimodal features. A graph attention network is used to perform fault propagation analysis to obtain fault diagnosis results.

[0009] Based on the fault diagnosis results, a repair strategy is negotiated and decided through a distributed intelligent agent cooperative mechanism using a deep Q-network, a greedy strategy, and a Byzantine fault-tolerant algorithm, and feedback is obtained.

[0010] Based on feedback, the repair strategy and test reasoning explanation class are optimized and dynamically adjusted using a meta-learning algorithm.

[0011] Optionally, the test reasoning interpretation class includes the following data structures:

[0012] A semantic relationship graph stores the mapping relationship between UI elements and business concepts;

[0013] Operate the ontology library and define standardized business operation categories;

[0014] Contextual memory saves the sequence of historical operations;

[0015] The intent pattern library stores common operation intent templates.

[0016] Optionally, the step of semanticizing the original test script through the test reasoning interpretation class and outputting the expected intent includes:

[0017] Input the original test script, and extract the operation type, target element, and context through test reasoning interpretation class;

[0018] Calculate semantic similarity and determine semantic operations by querying semantic relationship graphs or by reasoning based on NLP models;

[0019] Based on the Bayesian inference algorithm, the probability of business intent is calculated according to semantic operations and contextual history.

[0020] The output is a semantic test instruction containing business intent, operation, context, expected result and alternative strategy as the expected intent.

[0021] Optionally, when the test fails, multi-source data is collected synchronously and multi-modal features are extracted. A fault propagation graph is constructed based on the expected intent and multi-modal features. A graph attention network is used to perform fault propagation analysis to obtain fault diagnosis results, including:

[0022] When a test fails, UI data, APM data, log data, and code change data are collected synchronously within the time window of the test failure event. Based on semantic test instructions, features are extracted and DOM structure feature vectors, visual feature vectors, and layout feature vectors are fused to obtain multimodal features.

[0023] Based on the expected intent and multimodal features, system components are treated as nodes, multimodal features are used as node feature vectors, and dependencies between system components are used as edges to construct a fault propagation graph. Based on the attention weights of each node and its neighboring nodes, a graph attention network is used to perform fault propagation analysis, determine the root cause of the fault, and obtain the fault diagnosis result.

[0024] The system components include specific units at the software and infrastructure levels, specifically including microservice / backend service nodes, UI element / frontend component nodes, and infrastructure resource nodes.

[0025] Optionally, the step of negotiating and deciding on a repair strategy based on the fault diagnosis results using a distributed intelligent agent cooperative mechanism employing a deep Q-network, a greedy strategy, and a Byzantine fault-tolerant algorithm, and obtaining feedback, includes:

[0026] Construct a communication protocol between multiple intelligent agents, and realize communication and cooperation between distributed intelligent agents through message queues, shared memory, and message signature and verification mechanisms;

[0027] Each agent proposes a repair strategy based on the fault diagnosis results using a deep Q-network and an ε-greedy strategy, and a consensus on the repair strategy is reached through a Byzantine fault-tolerant algorithm.

[0028] The agreed-upon remediation strategy is executed in an isolated environment, and the remediation results are comprehensively scored through functional verification, performance evaluation, and stability testing, and feedback is generated.

[0029] Optionally, the step of dynamically adjusting the repair strategy and test inference explanation class based on feedback through a meta-learning algorithm includes:

[0030] The feedback is stored in the experience buffer, a preset number of historical feedback data are sampled from the experience buffer, successful repair patterns are discovered, key factors are extracted and similar patterns are clustered.

[0031] By employing a model-agnostic meta-learning algorithm, an inner loop gradient descent is performed on the support set, and an outer loop evaluation is performed on the query set to update the meta-model parameters, enabling rapid adaptation and optimization of the repair strategy and test inference explanation class.

[0032] Based on the same inventive concept, this application also provides an automated self-healing UI testing system based on multi-agent AI, the improvement of which includes:

[0033] The test reasoning interpretation layer is used to semantically interpret the original test script through the test reasoning interpretation class and output the expected intent;

[0034] The multimodal data acquisition layer is used to simultaneously collect multi-source data and extract multimodal features when the test fails. Based on the expected intent and multimodal features, a fault propagation graph is constructed, and a graph attention network is used to perform fault propagation analysis to obtain fault diagnosis results.

[0035] The agent cooperative execution layer is used to negotiate and decide on repair strategies based on fault diagnosis results through a distributed agent cooperative mechanism using deep Q-networks, greedy strategies, and Byzantine fault-tolerant algorithms, and to obtain feedback.

[0036] The meta-learning optimization layer is used to dynamically adjust the repair strategy and test reasoning explanation class based on feedback through a meta-learning algorithm.

[0037] Optionally, the test reasoning interpretation layer includes a test reasoning interpretation class, which contains the following core components:

[0038] A semantic relationship graph is used to store the mapping relationship between UI elements and business concepts;

[0039] Operation ontology library, used to define standardized business operation categories;

[0040] Contextual memory is used to save historical operation sequences;

[0041] The intent pattern library stores common operation intent templates.

[0042] Optionally, the step of semanticizing the original test script through the test reasoning interpretation class and outputting the expected intent includes:

[0043] Input the original test script, and extract the operation type, target element, and context through test reasoning interpretation class;

[0044] Calculate semantic similarity and determine semantic operations by querying semantic relationship graphs or by reasoning based on NLP models;

[0045] Based on the Bayesian inference algorithm, the probability of business intent is calculated according to semantic operations and contextual history.

[0046] The output is a semantic test instruction containing business intent, operation, context, expected result and alternative strategy as the expected intent.

[0047] Optionally, when the test fails, multi-source data is collected synchronously and multi-modal features are extracted. A fault propagation graph is constructed based on the expected intent and multi-modal features. A graph attention network is used to perform fault propagation analysis to obtain fault diagnosis results, including:

[0048] When a test fails, UI data, APM data, log data, and code change data are collected synchronously within the time window of the test failure event. Based on semantic test instructions, features are extracted and DOM structure feature vectors, visual feature vectors, and layout feature vectors are fused to obtain multimodal features.

[0049] Based on the expected intent and multimodal features, system components are treated as nodes, multimodal features are used as node feature vectors, and dependencies between system components are used as edges to construct a fault propagation graph. Based on the attention weights of each node and its neighboring nodes, a graph attention network is used to perform fault propagation analysis, determine the root cause of the fault, and obtain the fault diagnosis result.

[0050] The system components include specific units at the software and infrastructure levels, specifically including microservice / backend service nodes, UI element / frontend component nodes, and infrastructure resource nodes.

[0051] Optionally, the step of negotiating and deciding on a repair strategy based on the fault diagnosis results using a distributed intelligent agent cooperative mechanism employing a deep Q-network, a greedy strategy, and a Byzantine fault-tolerant algorithm, and obtaining feedback, includes:

[0052] Construct a communication protocol between multiple intelligent agents, and realize communication and cooperation between distributed intelligent agents through message queues, shared memory, and message signature and verification mechanisms;

[0053] Each agent proposes a repair strategy based on the fault diagnosis results using a deep Q-network and an ε-greedy strategy, and a consensus on the repair strategy is reached through a Byzantine fault-tolerant algorithm.

[0054] The agreed-upon remediation strategy is executed in an isolated environment, and the remediation results are comprehensively scored through functional verification, performance evaluation, and stability testing, and feedback is generated.

[0055] Optionally, the step of dynamically adjusting the repair strategy and test inference explanation class based on feedback through a meta-learning algorithm includes:

[0056] The feedback is stored in the experience buffer, a preset number of historical feedback data are sampled from the experience buffer, successful repair patterns are discovered, key factors are extracted and similar patterns are clustered.

[0057] Through a model-independent meta-learning algorithm, an inner loop gradient descent is performed on the support set, and an outer loop evaluation is performed on the query set to update the meta-model parameters. This allows for rapid adaptation and optimization of the repair strategy and test inference explanation class, with dynamic adjustments.

[0058] Furthermore, this application also provides a computing device, comprising: at least one processor and a memory;

[0059] The memory is used to store one or more programs;

[0060] When the one or more programs are executed by the one or more processors, an automated self-healing UI testing method based on multi-agent AI, as described above, is implemented.

[0061] In another aspect, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed, implements the above-described automated self-healing UI testing method based on multi-agent AI.

[0062] Compared with the prior art, this application has the following advantages:

[0063] This application provides an automated self-healing UI testing method and system based on multi-agent AI, comprising: semanticizing the original test script through a test inference explanation class to output the expected intent; when the test fails, synchronously collecting multi-source data and extracting multimodal features, constructing a fault propagation graph based on the expected intent and multimodal features, using a graph attention network to perform fault propagation analysis, and obtaining fault diagnosis results; based on the fault diagnosis results, negotiating and deciding on repair strategies through a distributed intelligent agent collaborative mechanism using a deep Q-network, a greedy strategy, and a Byzantine fault-tolerant algorithm, and obtaining feedback; and optimizing the repair strategy and the test inference explanation class based on the feedback using a meta-learning algorithm.

[0064] By constructing a test reasoning interpretation class, a deep understanding of test instructions is achieved, enabling intelligent mapping between test intent and actual execution. A distributed intelligent agent collaborative mechanism is adopted to complete fault diagnosis and automatic repair, thereby improving semantic understanding capabilities, increasing diagnostic accuracy, improving repair success rate, and reducing maintenance costs.

[0065] Other features and advantages of this application will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the application. The objectives and other advantages of this application may be realized and obtained by means of the structures pointed out in the description, claims and drawings. Attached Figure Description

[0066] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0067] Figure 1 The implementation flowchart of an automated self-healing UI testing method based on multi-agent AI provided in this application is shown;

[0068] Figure 2 The system architecture diagram of an automated self-healing UI testing system based on multi-agent AI provided in this application is shown. Detailed Implementation

[0069] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0070] This application aims to solve the fundamental technical problem of "vulnerability caused by test scripts' inability to understand business semantics" in UI automated testing. By constructing a test reasoning and interpretation class with semantic understanding capabilities, it achieves intelligent mapping between test intent and actual execution, and builds a complete self-healing system on this basis.

[0071] Example 1

[0072] This application provides an automated self-healing UI testing method based on multi-agent AI, such as... Figure 1 ,include:

[0073] The original test script is semantically interpreted by the test reasoning interpretation class, and the expected intent is output.

[0074] When a test fails, multi-source data is collected synchronously and multimodal features are extracted. A fault propagation graph is constructed based on the expected intent and multimodal features. A graph attention network is used to perform fault propagation analysis to obtain fault diagnosis results.

[0075] Based on the fault diagnosis results, a repair strategy is negotiated and decided through a distributed intelligent agent cooperative mechanism using a deep Q-network, a greedy strategy, and a Byzantine fault-tolerant algorithm, and feedback is obtained.

[0076] Based on feedback, the repair strategy and test reasoning explanation class are optimized and dynamically adjusted using a meta-learning algorithm.

[0077] Optionally, the test reasoning interpretation class includes the following data structures:

[0078] A semantic relationship graph stores the mapping relationship between UI elements and business concepts;

[0079] Operate the ontology library and define standardized business operation categories;

[0080] Contextual memory saves the sequence of historical operations;

[0081] The intent pattern library stores common operation intent templates.

[0082] Optionally, the step of semanticizing the original test script through the test reasoning interpretation class and outputting the expected intent includes:

[0083] Input the original test script, and extract the operation type, target element, and context through test reasoning interpretation class;

[0084] Calculate semantic similarity and determine semantic operations by querying semantic relationship graphs or by reasoning based on NLP models;

[0085] Based on the Bayesian inference algorithm, the probability of business intent is calculated according to semantic operations and contextual history.

[0086] The output is a semantic test instruction containing business intent, operation, context, expected result and alternative strategy as the expected intent.

[0087] Step 1: Parsing the original instruction

[0088] Input: Original test script instructions

[0089] Extract the following information using the test reasoning interpretation class:

[0090] Extraction operation type: click;

[0091] Extract the target element: ID="submit-btn";

[0092] Extracting the context: The preceding operation was filling out a form.

[0093] Step 2: Semantic Mapping Query

[0094] return max(scores, key=lambda x: x[1])[0]

[0095] Semantic similarity is calculated and semantic operations are determined by querying the semantic relationship graph or by reasoning based on NLP models; the semantic similarity calculation formula is as follows:

[0096]

[0097] in: This represents the feature vector obtained by encoding the context information of the current test operation. This represents the feature vector of candidate business operations retrieved from the operation ontology. This represents the semantic similarity score, used to measure the degree of matching between the current operation and the business intent, ranging from [0,1]. Represents the dot product operation of vectors; The modulus (or Euclidean norm) of the vector is used to calculate the cosine similarity between the context feature vector and the candidate operation feature vector.

[0098] Step 3: Business Intent Reasoning

[0099] Intent recognition algorithm based on Bayesian inference:

[0100] Calculate prior probabilities ;

[0101] Calculate the likelihood probability ;

[0102] The Bayesian formula is used to calculate the intent probability (posterior probability) for a given operation and context.

[0103] Normalization.

[0104] Bayesian inference formula:

[0105]

[0106] Where: I represents business intent (e.g., "submit order"), O represents observed operation (e.g., "click submit button"), C represents historical context, and P(I|O,C) represents the probability of intent given operation and context.

[0107] Step 4: Semantic Description Generation

[0108] Output: The semantic test instruction description includes:

[0109] "Intent": "Complete order submission",

[0110] "Operation": "Click the submit button",

[0111] "context": "User has filled in order information",

[0112] "Expected Outcome": "Order created successfully",

[0113] "fallback strategies": ["Use text positioning", "Use relative positioning"].

[0114] The generated semantic test instructions will be input as expected intents to the agent cooperative execution layer (ACEL) for comparative analysis with the observed behavior fed back by the multimodal data acquisition layer (MDCL) to determine the root cause of the failure.

[0115] Optionally, when the test fails, multi-source data is collected synchronously and multi-modal features are extracted. A fault propagation graph is constructed based on the expected intent and multi-modal features. A graph attention network is used to perform fault propagation analysis to obtain fault diagnosis results, including:

[0116] When a test fails, UI data, APM data, log data, and code change data are collected synchronously within the time window of the test failure event. Based on semantic test instructions, features are filtered and extracted. DOM structure features, visual features, and layout features of the UI page are extracted to form a UI feature vector; response time fluctuations and throughput changes are extracted from the APM data to form an APM feature vector; and the frequency of abnormal keywords and error stack traces are extracted from the log data to form a log feature vector. The above vectors are then fused to obtain multimodal features.

[0117] Based on the expected intent and multimodal features, system components are treated as nodes, multimodal features are used as node feature vectors, and dependencies between system components are used as edges to construct a fault propagation graph. Based on the attention weights of each node and its neighboring nodes, a graph attention network is used to perform fault propagation analysis, determine the root cause of the fault, and obtain the fault diagnosis result.

[0118] The system components include specific units at the software and infrastructure levels, specifically including microservice / backend service nodes, UI element / frontend component nodes, and infrastructure resource nodes.

[0119] Data Acquisition and Preprocessing:

[0120] Step 1: Synchronous Acquisition of Multi-Source Data

[0121] In this embodiment, the time window for the test failure event is defined as 30 seconds before and 10 seconds after the time of the test failure.

[0122] It collects various types of data in parallel, including UI data, APM data, log data, and code change data.

[0123] Step 2: Feature Extraction and Vectorization

[0124] For each data type, key features are extracted and converted into a unified vector representation:

[0125] Extracted using UI feature extraction algorithms:

[0126] DOM structure features (extracting DOM tree depth, total number of elements, and dynamic elements);

[0127] Visual features (extracted using CNN);

[0128] Layout features (extracting layout complexity);

[0129] The output includes a unified feature vector comprising DOM structural features, visual features, and layout features, i.e., multimodal features. Layout complexity calculation formula:

[0130]

[0131] in, Indicates the first The probability of a layout pattern occurring, where n represents the total number of different layout patterns. This represents the layout entropy value, reflecting the complexity of the page layout.

[0132] Association analysis based on graph neural networks:

[0133] Step 1: Construct a fault propagation graph

[0134] Add nodes (various system components). The node content includes node ID, node type, and node feature vector.

[0135] Add edges (dependencies between components), traverse all call chains, and traverse each call segment in the chain to obtain the edge content, which includes the caller, the callee, the call duration, and the correlation between the caller and the callee.

[0136] The "nodes (system components)" in the fault propagation diagram include specific units at the software and infrastructure levels, specifically including the following three categories:

[0137] Microservice / backend service nodes: services extracted from APM data in the code, such as "order service" and "payment service".

[0138] UI Elements / Front-end Component Nodes: Although the diagram mainly describes APM, UI elements (such as buttons and forms) are also part of the diagram in multimodal fusion, used to associate front-end operations with back-end responses.

[0139] Infrastructure resource nodes: Components involved in the log layer, such as database instances (connection pool exhaustion), message queues, etc.

[0140] Specific examples:

[0141] When the "Submit Order" test fails, the failure propagation graph may include:

[0142] Frontend: Order Submission Button → Order Page Component → API Call Layer;

[0143] Backend: API Gateway → Order Service → Payment Service → Order Database;

[0144] Infrastructure: Redis cache → message queue.

[0145] Step 2: Graph Neural Network Inference

[0146] Using Graph Attention Networks (GAT) for Fault Propagation Analysis:

[0147] The three parameters for initializing a neural network are: input dimension, hidden layer dimension, and output dimension.

[0148] Create a multi-head attention layer: Input dimension parameters and use 4 attention heads;

[0149] Creating a propagation layer: This involves stacking two layers.

[0150] Create a classification layer: output the probability that each node is the root cause;

[0151] The system takes the input node feature vector and edge content as inputs, calculates the importance weights between nodes through a multi-head attention layer, simulates fault propagation through a propagation layer, and outputs the probability distribution of each input node as the root cause through a classification layer.

[0152] Attention weight , representing the importance of the feature information of neighbor node j to the central node i. In fault analysis scenarios, it represents the probability or intensity of a fault propagating from node j to the central node i. Calculation formula:

[0153]

[0154] in, , representing the original attention coefficient, , representing the original association strength (energy value) between the center node i and its neighbor node j, which has not yet been probability normalized, and LeakyReLu represents the non-linear activation function. The feature vector representing the center node i. This represents the feature vector of neighbor node j. This represents a learnable weight matrix used to perform linear transformations on node features. The parameter vector representing the attention mechanism. express transpose, Let represent the set of neighbors of the central node i, referring to all system components that have a direct dependency relationship (connected by edges) with the central node i, and k represents the set of neighbors of the central node i. This represents the original attention score between the center node i and its neighbor k. This indicates a vector concatenation operation.

[0155] Optionally, the step of negotiating and deciding on a repair strategy based on the fault diagnosis results using a distributed intelligent agent cooperative mechanism employing a deep Q-network, a greedy strategy, and a Byzantine fault-tolerant algorithm, and obtaining feedback, includes:

[0156] Construct a communication protocol between multiple intelligent agents, and realize communication and cooperation between distributed intelligent agents through message queues, shared memory, and message signature and verification mechanisms;

[0157] Each agent proposes a repair strategy based on the fault diagnosis results using a deep Q-network and an ε-greedy strategy, and a consensus on the repair strategy is reached through a Byzantine fault-tolerant algorithm.

[0158] The agreed-upon remediation strategy is executed in an isolated environment, and the remediation results are comprehensively scored through functional verification, performance evaluation, and stability testing, and feedback is generated.

[0159] Intelligent agent communication protocol:

[0160] First, initialize the definitions: priority message queue, shared memory space, and Byzantine fault-tolerant consensus mechanism;

[0161] The broadcast parameters required to send a message are: sender object, message type, and message content;

[0162] The message content to be sent includes: message ID, sender, message type, message content, timestamp, and priority;

[0163] Use message signing to prevent tampering;

[0164] Enqueue messages according to priority and return message IDs.

[0165] Collaborative decision-making algorithm for repair strategies:

[0166] Step 1: Strategy Proposal Phase

[0167] Each agent proposes a repair strategy based on its own area of ​​expertise:

[0168] Determine the fault type (if it is a UI change fault), convert the fault information into a neural network vector, output the reward value of each action through a deep Q network, explore randomly with probability ε using an ε-greedy strategy, and select the optimal action as the repair strategy with probability 1-ε.

[0169] The repair strategy includes: strategy type, repair steps, confidence level, and repair time.

[0170] Q-value update formula (deep Q-network):

[0171]

[0172] Where s represents the current state (fault characteristics), and a represents the repair action taken. Let represent the action value function, which indicates the effectiveness score of adopting a certain repair strategy 'a' under the current fault scenario 's'. This indicates the state at the next moment, referring to the system environment state after the repair operation is performed. Indicates 'in the next state' The candidate action variable is r, which represents the immediate reward (whether the repair was successful). Indicates the discount factor. This represents the learning rate.

[0173] Step 2: Consensus Reaching Phase

[0174] A consensus on repair strategies was reached using the Byzantine fault-tolerant algorithm.

[0175] Collect the list of repair strategies submitted by each agent, vote within a finite maximum number of rounds, verify the legality of each repair strategy, accumulate the same repair strategies according to their weights, check if any repair strategy has obtained 2 / 3 support, if so, a consensus is reached, and the repair strategy that has reached the consensus is returned, otherwise continue to the next round.

[0176] Fix the closed-loop mechanism for execution and verification:

[0177] Step 1: Repair Execution

[0178] Receive the consensus repair strategy and test context, create an isolated sandbox environment, execute the repair according to the agreed-upon repair strategy, analyze failed elements to generate repair patches, apply the patches in the sandbox, execute verification tests, and return the patch and a success flag if the test passes; otherwise, return an empty set and a failure flag, and finally clean up the sandbox resources. Step 2: Verification and Feedback

[0179] The results of executing the consensus-reached repair strategy in the isolated environment will be input into a three-layer verification system: functional correctness, performance impact, and stability. A comprehensive score will be calculated (where functional correctness accounts for 50%, performance impact accounts for 30%, and stability accounts for 20%). Feedback data (including the repair strategy used, failure scenarios, comprehensive score, repair time, and resource consumption) will be generated and stored in the meta-learning buffer.

[0180] Optionally, the step of dynamically adjusting the repair strategy and test inference explanation class based on feedback through a meta-learning algorithm includes:

[0181] The feedback is stored in the experience buffer, a preset number of historical feedback data are sampled from the experience buffer, successful repair patterns are discovered, key factors are extracted and similar patterns are clustered.

[0182] Through a model-independent meta-learning algorithm, an inner loop gradient descent is performed on the support set, and an outer loop evaluation is performed on the query set to update the meta-model parameters. This allows for rapid adaptation and optimization of the repair strategy and test inference explanation class, with dynamic adjustments.

[0183] Experience playback and pattern recognition:

[0184] The framework uses a meta-learning optimizer to accumulate and transform repair experience.

[0185] First, a historical experience playback mechanism is constructed: a preset size (e.g., 1000 historical experience samples) is randomly sampled from the experience buffer, and high-confidence successful cases with repair success rate scores higher than a preset threshold (e.g., 0.8) are selected through a confidence filtering mechanism.

[0186] Secondly, perform multidimensional feature extraction and representation: for the selected successful cases, extract and construct a triplet feature knowledge representation consisting of fault context features, self-healing repair strategies and key influencing factors.

[0187] Finally, policy clustering and knowledge abstraction are implemented: clustering algorithms are used to aggregate triples with similar feature distributions, abstracting discrete self-healing experiences into a general repair pattern library, providing structured knowledge support for rapid repair of new fault scenarios. The MAML algorithm enables rapid policy adaptation.

[0188] To address multi-dimensional UI changes and complex fault scenarios, the Model Independent Meta-Learning (MAML) algorithm is used to optimize the repair strategy, enabling the framework to quickly adapt to new environments with very few attempts.

[0189] First, an inner-loop adaptive process based on the task support set is executed: for newly emerging fault repair tasks, a task support set is constructed from historical experience. During the inner loop, the initial parameter vector of the current meta-model is utilized. By performing a small number of iterative updates on the support set using the gradient descent algorithm, adaptive parameters for specific fault scenarios are calculated. .

[0190] Secondly, perform outer loop optimization based on the task query set: adjust the above adaptive parameters. Applied to a task query set, it evaluates the performance of the repair strategy and calculates the meta-loss. Through an outer loop mechanism, it adjusts the global meta-model parameters based on the cumulative meta-loss from multiple tasks. Perform backpropagation updates.

[0191] Finally, it enables rapid policy transfer across scenarios: through the above-mentioned two-layer gradient optimization mechanism, the framework learns a feature initialization with high generalization ability. When faced with unseen system faults, it can achieve accurate adaptation of the repair strategy with only a very small number of gradient steps.

[0192] The objective function of MAML (outer loop optimization objective):

[0193]

[0194] Adaptive parameters The update formula (inner loop adaptive logic) is as follows:

[0195]

[0196] in, This represents the optimal global initial parameter vector obtained through meta-learning training, which is the "best starting point" for the model when facing a new self-healing task. Represents the initial parameters, the global initial parameter vector of the meta-model in the current training round (pre-adaptation state); This represents the adaptive parameter, i.e., the parameter specific to a particular task. The parameter vector (adapted state) obtained after inner loop gradient update; Indicates the distribution of self-healing tasks The first sample obtained from the middle Each specific self-healing task covers multiple dimensions of fault types and repair scenarios; Indicates the task The defined loss function is used to quantify the deviation between the output of the self-healing strategy and the actual success criteria. Indicates based on initial parameters The constructed global policy generation model; Indicates based on adaptive parameters Build for task Specialized model instances; This indicates the learning step size of the inner loop, used to control the rate at which the model converges quickly to a single task. This indicates that the parameter vector The first-order gradient operator is calculated. The following specific example illustrates the workflow of the entire system:

[0197] Scenario: Order submission functionality test on an e-commerce website fails.

[0198] Step 1: Test Failure Detection and Semantic Understanding

[0199] Original failure: When the system was executing the order submission action, the positioning was interrupted due to the invalidation of the target control ID attribute, which triggered the underlying driver's element not found error.

[0200] Semantic understanding: Intent pattern matching is performed using the intent pattern library in the test reasoning interpretation class, mapping the underlying code error to business logic semantics: The user is interrupted when attempting to execute the core "submit order" operation after completing product selection and address entry.

[0201] Step 2: Multimodal Data Acquisition

[0202] UI layer: The DOM tree structure change is detected, and the button ID changes from "submit-order" to "place-order";

[0203] APM layer: Detected a 200% increase in order-service response time;

[0204] Log layer: Detected a database connection pool exhaustion warning.

[0205] Step 3: Root Cause Analysis

[0206] Graph neural network analysis results:

[0207] Root cause 1 (confidence 0.7): UI element ID change;

[0208] Root cause 2 (confidence level 0.3): Backend service performance degradation.

[0209] Step 4: Multi-agent negotiation

[0210] RepairAgent proposes: Change the locator to a new ID;

[0211] MonitorAgent proposes: Add performance monitoring assertions;

[0212] Consensus decision: Perform locator fixes and add performance warnings.

[0213] Step 5: Repair Execution and Verification

[0214] The system generates and executes a highly robust self-healing script:

[0215] Multi-dimensional feature relocation: The find_element_smart mechanism sets the new ID as the primary strategy and the business semantic text as the alternative, ensuring that a single attribute change does not interrupt the path.

[0216] Dynamic performance verification: The performance_monitor monitors the backend response while performing the repair, avoiding failure repairs that result in "functionality working but business slowness".

[0217] Step 6: Meta-learning update

[0218] Successful repair cases are recorded and stored in the experience replay pool. The query set loss for this task is calculated based on a meta-learning algorithm, and the outer loop gradient is updated according to the aforementioned formula (Note: referring to the MAML formula) to optimize the global initial parameters. The system detected a pattern of frequent UI changes in this module and adjusted the adaptive parameters. In the future, when faced with similar scenarios, it can converge to the semantic localization strategy more quickly.

[0219] This application has the following significant advantages over the prior art:

[0220] In typical business system UI automation testing scenarios, compared with traditional rule-based self-healing testing solutions, the method of this application shows higher semantic understanding accuracy and repair success rate in multiple test projects. It can significantly improve the adaptability of test scripts to UI changes and effectively shorten the fault diagnosis and repair time, thereby reducing manual maintenance costs.

[0221] Significantly improved semantic understanding capabilities: Through the understanding mechanism of test reasoning interpretation classes, the system can accurately understand more than 95% of business interaction semantics, making the test scripts more than 10 times more adaptable to business changes.

[0222] Accurate and rapid root cause diagnosis: The multimodal fusion algorithm based on graph neural networks can locate more than 85% of system-level complex faults, and the average diagnosis time is reduced from 2 hours to less than 3 minutes.

[0223] Significantly enhanced repair success rate: The introduction of a MAML-based meta-learning strategy for optimization has increased the automatic repair success rate from 30% in traditional solutions to over 85%, achieving true self-evolution and self-healing.

[0224] Reduced manual maintenance costs: The automated repair loop replaces approximately 80% of routine manual maintenance tasks, significantly reducing the repetitive workload of engineers.

[0225] Example 2

[0226] Based on the same inventive concept, this application also provides an automated self-healing UI testing system based on multi-agent AI, comprising:

[0227] The test reasoning interpretation layer is used to semantically interpret the original test script through the test reasoning interpretation class and output the expected intent;

[0228] The multimodal data acquisition layer is used to simultaneously collect multi-source data and extract multimodal features when the test fails. Based on the expected intent and multimodal features, a fault propagation graph is constructed, and a graph attention network is used to perform fault propagation analysis to obtain fault diagnosis results.

[0229] The agent cooperative execution layer is used to negotiate and decide on repair strategies based on fault diagnosis results through a distributed agent cooperative mechanism using deep Q-networks, greedy strategies, and Byzantine fault-tolerant algorithms, and to obtain feedback.

[0230] The meta-learning optimization layer is used to dynamically adjust the repair strategy and test reasoning explanation class based on feedback through a meta-learning algorithm.

[0231] The system architecture of this embodiment is as follows: Figure 2 After the test begins, the original test script is semantically parsed to obtain semantic test instructions as the expected intent. Upon failure of the detection, a fault event is triggered, capturing multi-source data within a time window. Feature extraction and filtering yield multimodal feature data. Root cause analysis is performed on the expected intent and multimodal feature data to generate a diagnostic report. A consensus on a repair strategy is reached through agent negotiation, executed, and verified. Feedback is input into the meta-learning model for updates, optimizing the parameters of the repair strategy and test inference explanation classes. This process includes the following modules:

[0232] Test Reasoning Interpretation Layer (TRIL): This is the entry point, responsible for converting test scripts into semantic business operation descriptions;

[0233] Multimodal Data Acquisition Layer (MDCL): This layer is triggered and collects multi-source data such as UI, APM, and logs in real time based on the TRIL context when a test fails.

[0234] The Agent Cooperative Execution Layer (ACEL) is the core processing unit. It receives the intent from TRIL and the data from MDCL, performs diagnosis and repair, and involves the collaborative work of multiple specialized AI agents.

[0235] Meta-learning optimization layer (MLOL): This is the background support layer. It receives the repair results from ACEL, updates the policy model, and in turn influences the subsequent decisions of TRIL and ACEL, and is responsible for continuously optimizing the diagnosis and repair strategy.

[0236] Optionally, the test reasoning interpretation layer includes a test reasoning interpretation class, which contains the following core components:

[0237] A semantic relationship graph is used to store the mapping relationship between UI elements and business concepts;

[0238] Operation ontology library, used to define standardized business operation categories;

[0239] Contextual memory is used to save historical operation sequences;

[0240] The intent pattern library stores common operation intent templates.

[0241] Optionally, the step of semanticizing the original test script through the test reasoning interpretation class and outputting the expected intent includes:

[0242] Input the original test script, and extract the operation type, target element, and context through test reasoning interpretation class;

[0243] Calculate semantic similarity and determine semantic operations by querying semantic relationship graphs or by reasoning based on NLP models;

[0244] Based on the Bayesian inference algorithm, the probability of business intent is calculated according to semantic operations and contextual history.

[0245] The output is a semantic test instruction containing business intent, operation, context, expected result and alternative strategy as the expected intent.

[0246] Optionally, when the test fails, multi-source data is collected synchronously and multi-modal features are extracted. A fault propagation graph is constructed based on the expected intent and multi-modal features. A graph attention network is used to perform fault propagation analysis to obtain fault diagnosis results, including:

[0247] When a test fails, UI data, APM data, log data, and code change data are collected synchronously within the time window of the test failure event. Based on semantic test instructions, features are extracted and DOM structure feature vectors, visual feature vectors, and layout feature vectors are fused to obtain multimodal features.

[0248] Based on the expected intent and multimodal features, system components are treated as nodes, multimodal features are used as node feature vectors, and dependencies between system components are used as edges to construct a fault propagation graph. Based on the attention weights of each node and its neighboring nodes, a graph attention network is used to perform fault propagation analysis, determine the root cause of the fault, and obtain the fault diagnosis result.

[0249] The system components include specific units at the software and infrastructure levels, specifically including microservice / backend service nodes, UI element / frontend component nodes, and infrastructure resource nodes.

[0250] Optionally, the step of negotiating and deciding on a repair strategy based on the fault diagnosis results using a distributed intelligent agent cooperative mechanism employing a deep Q-network, a greedy strategy, and a Byzantine fault-tolerant algorithm, and obtaining feedback, includes:

[0251] Construct a communication protocol between multiple intelligent agents, and realize communication and cooperation between distributed intelligent agents through message queues, shared memory, and message signature and verification mechanisms;

[0252] Each agent proposes a repair strategy based on the fault diagnosis results using a deep Q-network and an ε-greedy strategy, and a consensus on the repair strategy is reached through a Byzantine fault-tolerant algorithm.

[0253] The agreed-upon remediation strategy is executed in an isolated environment, and the remediation results are comprehensively scored through functional verification, performance evaluation, and stability testing, and feedback is generated.

[0254] Optionally, the step of dynamically adjusting the repair strategy and test inference explanation class based on feedback through a meta-learning algorithm includes:

[0255] The feedback is stored in the experience buffer, a preset number of historical feedback data are sampled from the experience buffer, successful repair patterns are discovered, key factors are extracted and similar patterns are clustered.

[0256] Through a model-independent meta-learning algorithm, an inner loop gradient descent is performed on the support set, and an outer loop evaluation is performed on the query set to update the meta-model parameters. This allows for rapid adaptation and optimization of the repair strategy and test inference explanation class, with dynamic adjustments.

[0257] Example 3

[0258] Based on the same inventive concept, this application also provides an electronic device. The electronic device of this application includes at least one processor and at least one storage medium electrically connected to the processor. The storage medium is electrically connected to the processor, wherein the storage medium stores instructions executable by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the method described above.

[0259] Example 4

[0260] Based on the same inventive concept, this application also provides a storage medium storing instructions executable by at least one processor, the instructions being executed by at least one processor to enable at least one processor to perform the method described above.

[0261] Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. An automated self-healing UI testing method based on multi-agent AI, characterized in that, include: The original test script is semantically interpreted by the test reasoning interpretation class, and the expected intent is output. When a test fails, multi-source data is collected synchronously and multimodal features are extracted. A fault propagation graph is constructed based on the expected intent and multimodal features. A graph attention network is used to perform fault propagation analysis to obtain fault diagnosis results. Based on the fault diagnosis results, a repair strategy is proposed through a distributed intelligent agent collaborative mechanism using a deep Q-network and a greedy strategy. A consensus on the repair strategy is reached through a Byzantine fault-tolerant algorithm, and feedback is obtained. Based on feedback, the repair strategy and test reasoning explanation class are optimized and dynamically adjusted using the MAML algorithm.

2. The automated self-healing UI testing method based on multi-agent AI according to claim 1, characterized in that, The test reasoning interpretation class contains the following data structures: A semantic relationship graph stores the mapping relationship between UI elements and business concepts; Operate the ontology library and define standardized business operation categories; Contextual memory saves the sequence of historical operations; The intent pattern library stores common operation intent templates.

3. The automated self-healing UI testing method based on multi-agent AI according to claim 2, characterized in that, The step of semanticizing the original test script through test reasoning interpretation and outputting the expected intent includes: Input the original test script, and extract the operation type, target element, and context through test reasoning interpretation class; Calculate semantic similarity and determine semantic operations by querying semantic relationship graphs or by reasoning based on NLP models; Based on the Bayesian inference algorithm, the probability of business intent is calculated according to semantic operations and contextual history. The output is a semantic test instruction containing business intent, operation, context, expected result and alternative strategy as the expected intent.

4. The automated self-healing UI testing method based on multi-agent AI according to claim 1, characterized in that, When a test fails, multi-source data is collected synchronously and multimodal features are extracted. A fault propagation graph is constructed based on the expected intent and multimodal features. A graph attention network is used to perform fault propagation analysis to obtain fault diagnosis results, including: When a test fails, UI data, APM data, log data, and code change data are collected synchronously within the time window of the test failure event. Based on semantic test instructions, features are extracted and DOM structure feature vectors, visual feature vectors, and layout feature vectors are fused to obtain multimodal features. Based on the expected intent and multimodal features, system components are treated as nodes, multimodal features are used as node feature vectors, and dependencies between system components are used as edges to construct a fault propagation graph. Based on the attention weights of each node and its neighboring nodes, a graph attention network is used to perform fault propagation analysis, determine the root cause of the fault, and obtain the fault diagnosis result. The system components include specific units at the software and infrastructure levels, specifically including microservice / backend service nodes, UI element / frontend component nodes, and infrastructure resource nodes.

5. The automated self-healing UI testing method based on multi-agent AI according to claim 1, characterized in that, Based on the fault diagnosis results, a repair strategy is proposed using a distributed intelligent agent collaborative mechanism with a deep Q-network and a greedy strategy. A consensus on the repair strategy is reached through a Byzantine fault-tolerant algorithm, and feedback is obtained, including: Construct a communication protocol between multiple intelligent agents, and realize communication and cooperation between distributed intelligent agents through message queues, shared memory, and message signature and verification mechanisms; Each agent proposes a repair strategy based on the fault diagnosis results using a deep Q-network and an ε-greedy strategy, and a consensus on the repair strategy is reached through a Byzantine fault-tolerant algorithm. The agreed-upon remediation strategy is executed in an isolated environment, and the remediation results are comprehensively scored through functional verification, performance evaluation, and stability testing, and feedback is generated.

6. The automated self-healing UI testing method based on multi-agent AI according to claim 1, characterized in that, The optimization and dynamic adjustment of the repair strategy and test reasoning explanation class based on feedback using the MAML algorithm includes: The feedback is stored in the experience buffer, a preset number of historical feedback data are sampled from the experience buffer, successful repair patterns are discovered, key factors are extracted and similar patterns are clustered. The model-independent MAML algorithm performs inner loop gradient descent on the support set and outer loop evaluation on the query set to update meta-model parameters, enabling rapid adaptation and optimization of the repair strategy and test inference explanation class, and dynamic adjustment.

7. An automated self-healing UI testing system based on multi-agent AI, characterized in that, include: The test reasoning interpretation layer is used to semantically interpret the original test script through the test reasoning interpretation class and output the expected intent; The multimodal data acquisition layer is used to simultaneously collect multi-source data and extract multimodal features when the test fails. Based on the expected intent and multimodal features, a fault propagation graph is constructed, and a graph attention network is used to perform fault propagation analysis to obtain fault diagnosis results. The agent cooperative execution layer is used to propose repair strategies based on the fault diagnosis results through a distributed agent cooperative mechanism using a deep Q-network and a greedy strategy. A consensus on the repair strategies is reached through a Byzantine fault-tolerant algorithm, and feedback is obtained. The meta-learning optimization layer is used to dynamically adjust the repair strategy and test inference explanation class based on feedback using the MAML algorithm.

8. The automated self-healing UI testing system based on multi-agent AI according to claim 7, characterized in that, The test reasoning interpretation layer includes a test reasoning interpretation class, which contains the following core components: A semantic relationship graph is used to store the mapping relationship between UI elements and business concepts; Operation ontology library, used to define standardized business operation categories; Contextual memory is used to save historical operation sequences; The intent pattern library stores common operation intent templates.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the automated self-healing UI testing method based on multi-agent AI as described in any one of claims 1-6.

10. An electronic device, characterized in that, It includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store computer programs; When a processor executes a program stored in memory, it implements the steps of the automated self-healing UI testing method based on multi-agent AI as described in any one of claims 1-6.

Images