Test script self-healing method, system, electronic device, and storage medium

Abstract

This application belongs to the field of script testing technology, and relates to a test script self-healing method, system, electronic device, and storage medium. By acquiring error information and forming error context information when a test script fails to execute, it provides a structured basis for subsequent processing. In the visual self-healing process, the error context information is analyzed, and candidate elements are screened and verified to automatically identify and handle script failure points. This closed-loop process can automatically update and re-execute failed scripts without manual intervention, thereby reducing the workload of manual location and modification, and reducing the maintenance cost of front-end test scripts. Simultaneously, by using error context information and verification results to guide script adjustments, the root cause of problems can be quickly and accurately identified, significantly improving the efficiency of fault analysis and handling compared to traditional manual troubleshooting.

Description

Technical Field

[0001] This application belongs to the field of script testing technology, specifically relating to a test script self-healing method, system, electronic device, and storage medium. Background Technology

[0002] In front-end automated testing and software testing practices, the maintenance and execution efficiency of test scripts have always been technical challenges. During execution, test scripts often fail due to differences in the software being tested or the system environment. Current technologies typically rely on manual error analysis, modification of test scripts, or re-recording of the test process to ensure the successful execution of test cases.

[0003] However, this approach has significant limitations: 1) High maintenance costs: Visual UI automation testing frameworks such as Airtest (Airtest is a cross-platform UI automation testing framework released by NetEase. This framework includes the Airtest framework based on image recognition and the Poco framework based on UI control search. With the dedicated Airtest IDE editor, it can help developers generate test scripts and other code, supporting automated testing on multiple platforms such as Android and Windows) rely heavily on the stability of image recognition for their execution logic. The effectiveness of visual anchors directly determines the success or failure of test cases. When the mini-program version is iterated or the container interface is redesigned, the image matching logic of the original script will directly fail. It is necessary to re-record the process and update the screenshot resources and coordinate parameters in the Airtest IDE editor. Otherwise, the test cases will not be able to execute at all. Even if the script passes the initial debugging stage, subsequent environmental heterogeneity (such as differences in mobile phone models, WeChat version iterations, and minor adjustments to mini-program UI details) will still cause test cases to fail randomly. The phenomenon of "occasional failure" becomes the norm, which greatly increases the continuous maintenance cost of the script. 2) Low efficiency in troubleshooting and script repair: Repairing faulty scripts heavily relies on manual intervention by specialists. Furthermore, the error reporting mechanism of Airtest has significant flaws, only outputting superficial prompts such as "No matching image found," failing to accurately pinpoint underlying issues in the mini-program (such as page loading timeouts, abnormal element rendering, or front-end logic blocking). This vague error information not only increases the difficulty of locating the root cause of the fault but also makes it difficult for novice debuggers to get to the heart of the problem, further lengthening the script repair cycle and hindering overall testing efficiency.

[0004] In the process of developing this application, the inventors discovered that the prior art has at least the above-mentioned problems, which seriously restrict the efficient execution and sustainable maintenance of front-end automated testing. New methods are needed to improve the efficiency of test script maintenance and troubleshooting capabilities. Summary of the Invention

[0005] To address the aforementioned issues, this application provides a test script self-healing method, system, electronic device, and storage medium to achieve intelligent updating and re-execution of test scripts, thereby reducing script maintenance costs and improving troubleshooting efficiency.

[0006] To address the aforementioned technical problems, one technical solution adopted in this application is: providing a test script self-healing method, the method comprising: when a test script execution failure is detected, obtaining error information; when the error information meets preset visual self-healing conditions, executing a visual self-healing process until a preset termination condition is met; wherein, the visual self-healing process comprises: obtaining error context information based on the error information; detecting abnormal interface screenshots in the error context information based on a preset image recognition detection model to obtain target element data; performing candidate element filtering operations on the target element data to obtain candidate repair element information; verifying the candidate repair element information to obtain a verification result; when the verification result meets preset repair conditions, updating the test script based on the candidate repair element information and re-executing the test script to obtain an execution result.

[0007] To solve the above-mentioned technical problems, another technical solution adopted in this application is: providing a test script self-healing system to execute the above-mentioned test script self-healing method. The system includes: a unified scheduling and execution management module, an anomaly capture module, a visual self-healing module, an image recognition and detection module, and an anomaly analysis and candidate element selection module. The anomaly capture module is connected to the unified scheduling and execution management module and the visual self-healing module respectively. The visual self-healing module is also connected to the unified scheduling and execution management module, the image recognition and detection module, and the anomaly analysis and candidate element selection module. The image recognition and detection module is also connected to the anomaly analysis and candidate element selection module.

[0008] To solve the above-mentioned technical problems, another technical solution adopted in the embodiments of this application is: to provide an electronic device, including: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the above-mentioned method.

[0009] To solve the above-mentioned technical problems, another technical solution adopted in the embodiments of this application is: to provide a non-volatile computer-readable storage medium that stores computer-executable instructions, which, when executed by an electronic device, cause the electronic device to perform the above-mentioned method.

[0010] Unlike related technologies, this application provides a test script self-healing method, system, electronic device, and storage medium. When a test script fails to execute, error information is obtained and error context information is formed, providing a structured basis for subsequent processing. Then, in the visual self-healing process, the error context information is analyzed, candidate elements are screened, and verified to automatically identify and handle potential failure points in the test script. Through this closed-loop process, failed scripts can be automatically updated and re-executed without manual intervention, reducing the workload of manually locating errors and modifying scripts, effectively lowering the maintenance cost of front-end test scripts. Simultaneously, since the verification results of the error context information and candidate repair element information can be directly used to guide script adjustments, the root cause of problems can be identified more quickly and accurately, significantly improving the efficiency of fault analysis and processing compared to traditional methods relying on manual troubleshooting. Attached Figure Description

[0011] One or more embodiments are illustrated by way of example with reference to the accompanying drawings. These illustrations do not constitute a limitation on the embodiments. Elements having the same reference numerals in the drawings are denoted as similar elements. Unless otherwise stated, the figures in the drawings are not to be limited by scale.

[0012] Figure 1 This is a schematic diagram of the structure of a test script self-healing system provided in an embodiment of this application; Figure 2 This is a flowchart of a test script self-healing method provided in an embodiment of this application; Figure 3 This is a flowchart of a visual self-healing process provided in an embodiment of this application; Figure 4 This is a schematic diagram illustrating an error message acquisition method provided in an embodiment of this application; Figure 5 This is a schematic diagram illustrating the acquisition of an element sub-screenshot provided in an embodiment of this application; Figure 6 This is a schematic diagram illustrating a test script update provided in an embodiment of this application; Figure 7 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0013] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and thoroughly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application. It should be noted that, unless otherwise specified, the various features in the embodiments of this application can be combined with each other, and all are within the protection scope of this application.

[0014] When an element is described as "connected" to another element, it can be directly connected to the other element, or there may be one or more intervening elements between them.

[0015] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and are not used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and are not limited in number; for example, a first object can be one or more.

[0016] Unless otherwise defined, all technical and scientific terms used in this specification have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the scope of this application.

[0017] Please see Figure 1 , Figure 1 This is a schematic diagram of the structure of a test script self-healing system provided in an embodiment of this application. Figure 1 As shown, the test script self-healing system 100 includes: a unified scheduling and execution management module 10, an anomaly capture module 20, a visual self-healing module 30, an image recognition and detection module 40, and an anomaly analysis and candidate element selection module 50. The anomaly capture module 20 is connected to the unified scheduling and execution management module 10 and the visual self-healing module 30. The visual self-healing module 30 is also connected to the unified scheduling and execution management module 10, the image recognition and detection module 40, and the anomaly analysis and candidate element selection module 50. The image recognition and detection module 40 is also connected to the anomaly analysis and candidate element selection module 50. These modules work together to complete the test script self-healing function.

[0018] The unified scheduling and execution management module 10 is used for centralized scheduling, loading, execution, and management of test scripts. Its main functions include: 1) Multi-mode script scheduling: Supports both scheduled and immediate execution modes for test scripts. It supports configuration of retry and self-healing retries for single scripts to ensure execution stability. 2) Dynamic script loading: Automatically scans test scripts (such as .py files) in specified directories and multi-level subdirectories, filtering invalid or private files (starting with an underscore). It sorts and dynamically loads scripts by directory level, providing a unified entry point for subsequent scheduling and execution. 3) Layered retry mechanism: Designs multi-level retry logic that does not interfere with each other, distinguishing between script-level retries and self-healing retry scenarios. Script-level retries: Handle non-page change issues such as network fluctuations and service anomalies when visual self-healing conditions are not triggered. Self-healing retry: Activated when visual self-healing conditions are triggered (e.g., when the touch operation in the Airtest test script fails, and the target image cannot be matched due to changes in the tested software or mini-program page). This controls the scope of self-healing attempts and avoids blocking the overall script execution. 4) Global scheduling strategy: Manages the execution lifecycle of all scripts, including execution, exception handling, self-healing triggering, and execution status control. Supports two types of scheduling rules: "execute immediately upon service startup" and "execute on a schedule." Dynamic loading of script modules is achieved based on importlib (importlib is a dynamic import module of the Python standard library that can dynamically load and execute corresponding Python scripts or modules based on the string name of the module during program execution). Global control is provided for batch execution, recording the number of self-healing attempts and execution results for each script. Resetting counters and running status is supported to ensure the independence and isolation of multiple script executions. Simultaneously, execution reports of all Airtest test scripts are aggregated, providing a global visualization view of the execution results. 5) Intelligent Error Analysis and Self-Healing Trigger: Extract error code lines (e.g., the Airtest test script operation is touch) and failure operation context from the exception stack, and provide them to the visual self-healing module to trigger self-healing. 6) Test Script Initialization Processing: To facilitate test script maintenance and provide comparable standard script information for subsequent visual self-healing, the specified test script directory is initialized. For image names in the Airtest test script that are meaningless, after optical character recognition, the script is refreshed, image files are renamed, and then standardized and persistently saved.

[0019] The exception capture module 20 is responsible for monitoring the execution process of the test script, capturing various exception information and error interface screenshots, and generating structured error context information, including the script location where the exception occurred, the type of failed operation, associated template file information, and the original click coordinates. This module ensures image clarity and storage efficiency by optimizing screenshot parameters, providing accurate data for subsequent image analysis and self-healing, and provides standardized error information to the visual self-healing module 30.

[0020] The visual self-healing module 30 performs automated repair and verification based on error information and candidate repair element information, identifying and attempting to repair operations that may cause script failure. The module includes a repair verification mechanism that ensures accurate positioning and effective interaction of candidate elements through screen recognition and operation verification. If verification is successful, a repair result is generated and the original script is updated; if verification fails, the module tries the next candidate solution according to a preset self-healing retry strategy or records the failure information, providing a basis for manual intervention, thereby achieving a closed loop for the test script from anomaly detection to automatic repair.

[0021] The image recognition and detection module 40 performs structured analysis on abnormal interface screenshots, including target region detection, text recognition, and semantic classification, removing irrelevant icons or background information. The module outputs the coordinates and text information of interface elements, providing basic data for the selection of candidate repair elements. It also supports screenshot segmentation processing, multi-threaded calls, and coordinate normalization transformation to ensure accurate docking with the coordinate system of the test script and to ensure the integrity and usability of candidate element data.

[0022] The anomaly analysis and candidate element selection module 50 is used for: when the candidate element set meets the preset score calculation conditions, performing text matching on the candidate element set based on the preset text length matching degree to obtain text matching information; performing spatial position matching on the candidate element set based on the preset spatial coordinate matching degree to obtain spatial position matching information; calculating the first score result of the candidate element set based on the text matching information and spatial position matching information; when the candidate element set does not meet the score calculation conditions, calculating the second score result of the candidate element set based on the preset text similarity calculation rules; and performing sorting and optimization operations on the candidate element set based on the preset matching priority rules, the first score result, and the second score result to generate candidate repair element information.

[0023] Understandably, the anomaly analysis and candidate element selection module 50 enters the candidate element screening stage based on the target element data output by the image recognition and detection module 40. First, it performs preliminary screening of the target element data (e.g., removing icon-type element sub-screenshots and element sub-screenshots that do not contain valid Chinese text content after secondary OCR recognition), obtaining the screened element sub-screenshots and their corresponding second region coordinate information and second text information. Subsequently, it performs standardization processing (e.g., text standardization, coordinate normalization, image standardization) to generate a candidate element set.

[0024] After standardization, a candidate element set is generated. Each candidate element contains a standardized sub-screenshot, second region coordinate information, and second text information, serving as the basis for subsequent matching. Based on whether the candidate text has a complete match with the target text, the candidate elements are divided into two processing paths. The candidate text is the content corresponding to the second text information, and the target text is the text information obtained from the target template image in the error context through OCR recognition. The first processing method: When the candidate text contains all characters of the target text and there is a substring matching relationship, the preset score calculation conditions are met. The text length matching degree and spatial coordinate matching degree are calculated separately, and the two are weighted and fused to generate the first score result. The second processing method: When the score calculation conditions are not met, text similarity calculation rules are used. These rules include Levenshtein edit distance and LCS similarity, which are fused to obtain the second score result.

[0025] Subsequently, the first and second scores are normalized and then sorted hierarchically according to a matching priority rule: first, sorted by the first score, then by the second score, and further merged to generate a comprehensive score. Finally, the top N candidate elements are extracted based on the comprehensive score (e.g., selecting candidates with scores above 85), forming candidate repair element information, which includes sub-screenshots, coordinate information, text information, and the comprehensive score. Finally, the candidate repair element information is output to the visual self-healing module 30 for subsequent verification and script repair.

[0026] It is understood that the modules in the test script self-healing system 100 provided in this embodiment form a closed-loop process. The unified scheduling and execution management module 10 monitors the execution of the test script and captures abnormal information. The abnormal capture module 20 generates structured error context information and provides it to the visual self-healing module 30. The visual self-healing module 30 calls the image recognition and detection module 40 and the abnormal analysis and candidate element selection module 50 to complete the generation and verification of candidate repair elements. Finally, the results are fed back to the unified scheduling and execution management module 10 to realize the automated repair and re-execution of the test script.

[0027] Please see Figure 2 , Figure 2This is a flowchart illustrating a test script self-healing method provided in an embodiment of this application. This test script self-healing method is applied to the aforementioned test script self-healing system 100. For example... Figure 2 As shown, the method includes steps S1-S2: S1: When a test script fails to execute, retrieve the error message.

[0028] The unified scheduling and execution management module first scans the specified test script directory, automatically loads all Python script files that meet the criteria, and supports multi-level directory structure recognition while filtering private script files (files starting with an underscore). For screenshot Template objects in the Airtest test script (Template is a Python class in Airtest used to represent an image template), the module uses regular expressions to match whether the image name contains a recognizable identifier. If the filename is meaningless, it calls OCR to recognize the image content and renames the image file according to the recognition result. At the same time, it updates the corresponding Template reference in the script to ensure that the image reference is consistent with the actual file, providing reliable reference data for subsequent script execution and visual self-healing.

[0029] During script execution, each operation is monitored in real time. When a script operation fails (e.g., the Airtest touch operation cannot match the target image), the exception capture module activates the runtime exception capture mechanism to parse the error log. The error log is analyzed using dedicated regular expression rules to extract error information. This error information includes, but is not limited to: the path of the script file where the exception occurred, the specific script line number where the exception occurred, the name of the target template image, the original click coordinates, and the image matching threshold parameters. This information is used to identify the operation object and location where the exception occurred, supporting subsequent visual self-healing judgment. It is understood that the dedicated regular expression rules used in this embodiment are a set of pattern matching rules customized for Airtest test script error logs (a specially designed set of regular expressions (a text pattern matching tool used to extract content that conforms to specific rules from a string)). These rules are used to extract the aforementioned error information from the error log, thereby converting unstructured log information into structured error data, providing a data foundation for subsequent error context construction and visual self-healing processing. It should be noted that this embodiment does not limit the dedicated regular expression rules and can be adapted according to the actual application scenario.

[0030] S2: When the error message meets the preset visual self-healing conditions, execute the visual self-healing process until the preset termination conditions are met.

[0031] Please see Figure 3 , Figure 3 This is a flowchart of a visual self-healing process provided in an embodiment of this application. For example... Figure 3 As shown, the visual self-healing process includes steps S21-S25: S21: Obtain the error context information based on the error message.

[0032] Specifically, based on the error message, error context information is obtained, including: parsing the erroneous lines of code in the test script based on the error message; extracting the target template image information and the original click coordinate information based on the erroneous lines of code; obtaining a screenshot of the abnormal page when the test script fails to execute; and using the target template image information, the original click coordinate information, and the abnormal page screenshot as error context information.

[0033] The system determines whether the preset visual self-healing conditions are met based on the error message. The visual self-healing conditions refer to the conditions that trigger the visual self-healing process when the page changes. For example, when the error type is TemplateNotFoundError, it means that the page of the tested software or mini-program has changed, thereby triggering the visual self-healing process.

[0034] Only when the error message meets the visual self-healing conditions will the system further obtain error context information to provide complete structured data for the visual self-healing module. The error context information includes the following: (1) Parse the error code line in the test script to accurately locate the location of the operation failure; (2) Extract the target template image information and the original click coordinate information according to the error code line to provide benchmark data for subsequent matching; among them, the target template image information refers to the original template image data used for image matching operation during the execution of the test script. The template image comes from the screenshot resources referenced by the image operations such as touch or exists in the Airtest script and is used as the benchmark visual features for UI element recognition; the original click coordinate information refers to the interaction position information of the target UI element recorded during the execution of the test script. The coordinate information usually comes from the click position parameters generated by the image matching result or record recording mechanism during the execution of Airtest and is used to describe the spatial position of the target UI element in the screen coordinate system; (3) At the moment when the test script fails to execute, obtain a complete screenshot of the abnormal page through the snapshot function of Airtest and use optimized screenshot parameter configuration (such as quality 90% (quality 90% is the encoding quality parameter when the screenshot image is saved or compressed, usually used for JPEG) Compression quality settings (maximum screenshot size 1200 pixels) achieve the optimal balance between image clarity and storage efficiency. Screenshot files are categorized and stored in a dedicated error directory, with unique filenames generated using timestamps to ensure the integrity of abnormal screenshot data. Finally, the system unifies the target template image information, original click coordinate information, and abnormal page screenshots as error context information, and manages and stores them in a standardized manner, providing a complete data foundation for target element detection and candidate element screening based on image recognition in subsequent step S22. While acquiring error context information, the system initializes the original test script and screenshots, including OCR recognition and semantic filename encoding, to ensure accurate positioning of templates and elements in the subsequent self-healing process, providing reliable input data for closed-loop self-healing.

[0035] By obtaining error context information from error messages, the system can accurately pinpoint the specific operation point where the test script failed to execute, and acquire structured data such as target template image information, original click coordinates, and screenshots of the abnormal page. This provides complete and accurate input for the subsequent visual self-healing process. This processing method transforms abnormal information during script execution into data that can be directly used for automated repair, avoiding the tediousness and uncertainty of manually analyzing error logs, thereby improving the efficiency and accuracy of error analysis. Furthermore, standardized error context information and screenshot processing enable the visual self-healing module to quickly match target elements under different scripts and UI change scenarios, ensuring the reliability and repeatability of automated self-healing operations.

[0036] S22: Based on the preset image recognition and detection model, detect the abnormal interface screenshot in the error context information to obtain the target element data.

[0037] Specifically, based on a preset image recognition detection model, abnormal interface screenshots in error context information are detected to obtain target element data, including: based on the image recognition detection model, interface element detection is performed on the abnormal interface screenshots to obtain multiple interface element regions and corresponding first region coordinate information; text recognition processing is performed on the interface element regions to obtain first text information corresponding to the interface element regions; the first region coordinate information is associated with the first text information to generate interface element detection results; and target element data is generated based on the interface element detection results.

[0038] The process involves generating target element data based on the interface element detection results. This includes: dividing the abnormal interface screenshot into multiple element sub-screenshots based on the interface element detection results, and generating second region coordinate information and second text information corresponding to each element sub-screenshot; wherein the second region coordinate information is obtained based on the first region coordinate information; and the second text information is obtained by performing optical character recognition on the sub-screenshots; and associating and storing the element sub-screenshots with the corresponding second region coordinate information and second text information to generate target element data.

[0039] In the visual self-healing process, the system first receives an abnormal interface screenshot from the error context information. This screenshot is a screen image captured by the front-end UI automation testing framework Airtest when the test script fails to execute, used to completely record the current state of the page. The system calls the preset image recognition and detection model Omniparser-v2 framework for processing. This framework integrates a multi-model fusion strategy to ensure the accuracy of element detection and semantic understanding capabilities. Specifically, the system inputs the abnormal interface screenshot into the preset image recognition and detection model for processing. Interface elements refer to interactive or identifiable objects that constitute the UI page, including visual components such as buttons, input boxes, menu controls, text display areas, and icons. The specific implementation process is as follows: (1) Interface element detection and fusion processing: The system uses the YOLO v8n model in the module to perform target detection on abnormal interface screenshots, and obtains multiple interface element regions and their corresponding first region coordinate information. Among them, the interface element region refers to the rectangular visual region detected by the YOLO v8n model in the image, which is used to represent the spatial range of an independent functional unit in the UI interface; the first region coordinate information refers to the spatial position coordinates of the interface element region in the original screenshot, which is usually represented by the bounding box of the upper left and lower right corners, and is used to describe the geometric position of the UI element in the image.

[0040] Simultaneously, the system uses the Paddle OCR model within the module to perform text recognition processing on the screenshot of the abnormal interface, obtaining the first text information. The first text information refers to the set of text content recognized by the Paddle OCR model from the screenshot of the abnormal interface; the first text information contains only semantic text content and does not contain any coordinate information; the first text information is used to describe the semantic expression of readable text in the interface, such as button text, prompts, or input box placeholders.

[0041] Furthermore, the system uses the Florence-2 model in the module to perform semantic classification of interface elements, which is used to identify different types of UI elements (such as buttons, input boxes, text display elements, and icons) and filter out non-critical UI elements.

[0042] The final interface element detection result is generated, which consists of the following three types of information: interface element area (UI visual area), first area coordinate information (YOLO detection coordinates), and first text information (OCR text content).

[0043] (2) Abnormal Interface Screenshot Region Segmentation: Based on the generated interface element detection results, the system performs element-by-element region segmentation processing on the abnormal interface screenshot based on the interface element region and its corresponding first region coordinate information, so as to achieve a fine mapping from the global image to the local element image. Specifically, the system sequentially traverses each interface element region in the interface element detection results and obtains its corresponding first region coordinate information. The first region coordinate information is used to represent the spatial position range of the interface element region in the abnormal interface screenshot, usually represented in the form of a rectangular bounding box, including the coordinates of the upper left corner and the lower right corner.

[0044] After obtaining the coordinate information, the system calls the Python Imaging Library (PIL, the most mainstream and mature image processing library in the Python ecosystem, whose core functions include image reading, saving, resizing, cropping, rotating, filtering, pixel manipulation, and format conversion, and is widely used in automated image processing, image analysis, batch image processing, and other scenarios) to perform image cropping on the abnormal interface screenshot. Through a region-based cropping method based on coordinate ranges, each interface element region is divided into independent image blocks from the original screenshot, thereby generating element sub-screenshots that correspond one-to-one with each interface element region. These element sub-screenshots represent the local visual information of a single UI element, containing only the pixel-level representation of that element in the original screenshot, and excluding other irrelevant UI region information to reduce interference from subsequent recognition and matching.

[0045] After image cropping, the system further establishes a mapping relationship between the element sub-screenshots and the coordinate information of the first region to ensure that each sub-screenshot can be traced back to its spatial position in the original interface. Based on this, the system performs coordinate system transformation on the first region coordinate information to generate second region coordinate information. The second region coordinate information refers to the result of converting the first region coordinate information from a pixel coordinate system with the top-left corner of the image as the origin to a standardized coordinate system with the screen center as the origin, consistent with the Airtest test script, to adapt to subsequent automated click execution logic. Specifically, the coordinate transformation process includes calculating the center point of the original bounding box coordinates and normalizing them in conjunction with screen resolution parameters to obtain a standard coordinate representation with the screen center as the reference origin.

[0046] Finally, the element sub-screenshot is bound to the corresponding second region coordinate information, which is used as the basic input data for the subsequent visual self-healing module to locate the target and repair the script.

[0047] (3) Secondary OCR Optimization and Text Information Generation: To improve text recognition accuracy and solve the problem of insufficient classifier accuracy, the system performs secondary OCR recognition on the split element sub-screens. If the first OCR fails to recognize effective text content or the recognition confidence is lower than the preset threshold, the system re-performs optical character recognition processing on the element sub-screen to obtain second text information. The second text information refers to the semantic result of the text obtained based on the local image content recognition of the element sub-screen. It only represents the text content itself and does not contain any coordinate information or spatial location information. It is used to supplement or enhance the semantic expression ability of the UI element. In terms of data association, the second text information is bound and stored with the corresponding element sub-screen and its second region coordinate information to form a three-element association structure of "visual region - spatial location - semantic text" to ensure that each UI element has complete semantic description ability. In addition, the system filters element sub-screens that do not recognize any effective text information to reduce the interference of invalid data on the subsequent candidate element screening process, thereby improving the accuracy and stability of the overall visual self-healing process.

[0048] (4) Target Element Data Generation and Storage: The split element sub-screenshots, second region coordinate information, and second text information are uniformly structured and encapsulated to generate target element data. Target element data refers to the structured data set used as input to the visual self-healing module. Its components include: element sub-screenshots: representing local visual image information of a single UI interface; second region coordinate information: representing the spatial position information of the UI element in the standardized coordinate system; and second text information: representing the OCR semantic text information corresponding to the UI element. These three types of data constitute the core components of the target element data, used to describe the visual characteristics, spatial distribution, and semantic attributes of each UI element in the abnormal interface. During storage, the system standardizes the second region coordinate information, mapping it uniformly to a coordinate system with the screen center as the origin, ensuring consistency with the coordinate system executed by the Airtest test script, thereby supporting the accurate execution of subsequent automated click operations. The final generated target element data serves as the standard input to the visual self-healing module, used for subsequent candidate element filtering, target matching verification, and script automatic repair processes, providing a complete data foundation for achieving closed-loop self-healing of the UI automated test script.

[0049] It's worth noting that the system also incorporates a thread-safe global model manager and a globally shared model instance manager. Multi-threaded safety management of GPU access is achieved through `threading.Lock` (a thread-lock mechanism provided by Python to control concurrent access to shared resources by multiple threads, preventing resource contention and ensuring consistency in GPU inference execution). Inference computation is performed under the protection of the model lock, effectively avoiding CUDA context conflicts and ensuring the stability of model calls in multi-threaded scenarios. Furthermore, a multi-threaded safe call mechanism for Omniparser-v2 is designed, reusing the model lock to ensure thread-safe access to GPU models; limiting the maximum number of concurrent tasks (e.g., 5); and maintaining real-time statistics on task execution (e.g., total number of tasks, number of successes, number of failures, execution time), balancing call efficiency and system stability.

[0050] By systematically processing abnormal interface screenshots within error context information using a pre-defined image recognition detection model, the system transforms raw UI screenshots into structured, operable target element data. In this process, the system accurately identifies various element regions within the interface and their corresponding coordinate information, and combines text recognition technology to obtain the text content of each interface element, thereby generating interface element detection results. Furthermore, based on the interface element detection results, the abnormal interface screenshot is region-segmented to obtain multiple element sub-screenshots along with their corresponding normalized coordinates and text information, which are then associated and stored to form complete target element data. Through this method, the visual self-healing module can quickly identify UI changes based on reliable structured data without manual intervention, and generate accurate information that can be used for subsequent candidate element selection and script repair, thereby significantly reducing the maintenance cost of front-end test scripts and improving the efficiency and accuracy of automated testing. In addition, this method improves the accuracy and robustness of element recognition through multi-model fusion, secondary OCR optimization, and coordinate normalization processing, enabling the system to adapt to diverse and dynamically changing UI scenarios, providing a solid data foundation for the self-healing capabilities of automated scripts.

[0051] S23: Perform a candidate element filtering operation on the target element data to obtain candidate repair element information.

[0052] The process of filtering candidate elements from the target element data to obtain candidate repair element information includes: filtering the target element data to obtain the filtered element sub-screenshots and their corresponding second region coordinate information and second text information; standardizing the filtered element sub-screenshots and their corresponding second region coordinate information and second text information to generate a candidate element set; and obtaining the candidate repair element information based on the candidate element set.

[0053] The process of obtaining candidate repair element information based on the candidate element set includes: when the candidate element set meets the preset score calculation conditions, performing text matching on the candidate element set based on the preset text length matching degree to obtain text matching information; performing spatial location matching on the candidate element set based on the preset spatial coordinate matching degree to obtain spatial location matching information; calculating the first score result of the candidate element set based on the text matching information and the spatial location matching information; when the candidate element set does not meet the score calculation conditions, calculating the second score result of the candidate element set based on the preset text similarity calculation rules; and extracting candidate element information from the candidate element set based on the preset matching priority rules, the first score result, and the second score result to generate candidate repair element information.

[0054] The system performs a candidate element filtering operation on the target element data generated in step S22 to obtain candidate repair element information for subsequent test script self-healing. The target element data includes a sub-screenshot of each interface element, normalized second region coordinate information, and text information corresponding to the element sub-screenshot (second text information). The specific implementation process includes: (1) Preliminary screening and standardization: The system first performs preliminary screening of the target element data. The screening rules include: removing sub-screenshots of icon elements, because these elements usually do not involve text interaction in operation; removing sub-screenshots of elements that do not contain valid Chinese text content after secondary OCR recognition, in order to ensure the validity of the text matched later.

[0055] The remaining valid element sub-screenshots and their corresponding second region coordinate information and second text information are standardized, including: text standardization: removing special characters, spaces, or invalid symbols from the text to make the text usable for precise matching; coordinate normalization: adjusting the original second region coordinate information to a unified coordinate system (e.g., coordinates with the center of the screen as the origin) to facilitate comparison with the expected coordinates of the target operation; image standardization: unifying the element sub-screenshot size, resolution, and format to ensure consistency in subsequent processing and display.

[0056] After standardization, a candidate element set is generated, where each candidate element contains a standardized sub-screenshot, second region coordinate information, and second text information. The candidate element set is the foundation dataset for subsequent text and spatial matching operations.

[0057] Within the candidate element set, the system categorizes candidate elements into two processing paths based on whether the target text can be completely matched within the candidate text. Candidate text refers to the text content corresponding to the second text information in the candidate element set, while target text refers to the text content obtained after optical character recognition (OCR) processing of the target template image information in the error context information, used to represent the UI semantic information corresponding to the original template.

[0058] (2) First processing method: When a candidate element meets the preset score calculation conditions, it enters this path. The score calculation conditions refer to the criteria that the candidate text contains all the character information of the target text, and that there is a complete substring matching relationship between the target text and the candidate text. For candidate elements that meet these conditions, the system calculates the first score result based on the following two indicators: Text length matching calculation: The system performs text length matching analysis on the candidate element set. Text length matching refers to the closeness between the length of a candidate element's text and the length of the target operation text, used to quantify the structural consistency between the candidate and target texts. In essence, for any candidate element, its text length matching degree is defined as the inverse function of the difference between the candidate and target text lengths; or, in a normalized manner, as a length similarity score, ranging from 0 to 1; where the closer the value is to 1, the closer the text structure is to the target text. By calculating the length closeness between each candidate element's text and the target text, text matching information is generated. This text matching information refers to the matching score between the candidate element's text and the target text in the length dimension, used to represent the degree of matching at the text structure level. Text matching information plays a core role in prioritizing candidate elements, helping the system quickly filter out elements whose text best matches the target operation.

[0059] Spatial Position Matching Calculation: The system compares the normalized coordinates of candidate elements with the expected coordinates of the target operation to calculate the spatial coordinate matching degree. The spatial coordinate matching degree refers to the degree of distance matching between the candidate element's position in the interface space and the expected position of the target operation. In essence, the system calculates the spatial deviation between the candidate element's coordinates and the original click coordinates in the error context information based on Euclidean distance, and normalizes this deviation to obtain a spatial matching score: the smaller the distance, the higher the spatial coordinate matching degree; the spatial coordinate matching degree ranges from 0 to 1. The candidate element coordinates are the coordinates corresponding to the coordinates of the second region in the candidate element set. Spatial position matching information is generated through the spatial position matching degree. This spatial position matching information is a score used to characterize the consistency between the candidate element's position in the interface layout and the target operation position. Used in conjunction with text matching information, it provides a multi-dimensional evaluation standard for candidate element ranking.

[0060] Comprehensive Ranking: The system weights and fuses text matching information and spatial location matching information to generate a matching score for each candidate element, and then sums these scores to obtain the first score result. In terms of weight design, the weight of spatial location matching degree can be increased according to the actual strategy to ensure the spatial accuracy of the operation target.

[0061] (3) Second processing method: When a candidate element does not meet the above scoring conditions, it enters this path. The system uses text similarity calculation rules to perform matching analysis on this type of candidate element. Among them, the text similarity calculation rules refer to the rule system for quantitative calculation based on the semantic similarity between candidate text and target text, including Levenshtein edit distance calculation and longest common subsequence (LCS) similarity calculation, and generates a unified similarity evaluation model through weighted fusion. Based on this rule, the system calculates the second score result of the candidate element set. Among them, the second score result refers to the matching score result obtained based on text semantic similarity calculation, which is used to characterize the semantic closeness between candidate element and target text.

[0062] (4) Generation of candidate repair element information: The system performs unified fusion and sorting of the first score result and the second score result to generate candidate repair element information. Specifically, the system first performs unified scoring space mapping processing on the two types of candidate elements, so that the first score result and the second score result are normalized to the same scoring range (0~1).

[0063] During the sorting process, the system uses a matching priority rule for hierarchical processing. That is, candidate elements that meet the score calculation conditions are sorted in descending order according to the first score result; and candidate elements that do not meet the score calculation conditions are sorted in descending order according to the second score result.

[0064] Based on the hierarchical ranking, the system further performs a weighted fusion of the first and second score results to obtain a comprehensive similarity score, which is then sorted in descending order of score.

[0065] Finally, the system extracts the top N candidate elements based on the sorted similarity scores to form the final candidate repair element information. The candidate repair element information refers to the final set of candidate operation objects used by the visual self-healing module, and each item contains at least: a standardized element sub-screenshot, standardized second region coordinate information, standardized second text information, and a similarity score.

[0066] (5) Output and data closure: The system outputs the candidate repair element information to the visual self-healing module, providing accurate operation objects and matching data for the subsequent verification step S24.

[0067] By performing two-layer filtering and standardization on the target element data, irrelevant icon elements and invalid text information are effectively removed, so that the candidate element set has high semantic purity and structural consistency at the data level. Secondly, in the candidate element scoring stage, the system performs triage based on whether the candidate elements meet the preset scoring conditions: when there are candidates in the candidate element set that completely or highly match the target text, the system prioritizes a comprehensive score based on text length matching degree and spatial coordinate matching degree. The text length matching degree is used to characterize the consistency between the candidate text and the target text in terms of character length structure, and the spatial coordinate matching degree is used to characterize the degree of spatial deviation between the position of the candidate element in the interface space and the original click coordinates. The two are weighted and fused to generate the first score result, thereby achieving a preferred ranking that takes into account both structural matching and spatial consistency. When there are no candidates in the candidate element set that meet the complete target text matching condition, the system further switches to the text similarity calculation rule. By calculating the Levenshtein edit distance and the longest common subsequence (LCS) similarity between the candidate element OCR text and the target text, the two similarity indicators are fused to generate the second score result, so as to achieve effective identification and compensation ranking of weakly matched or semantically similar elements. Finally, the system performs a unified fusion and sorting of the first and second score results based on preset matching priority rules, prioritizing the retention of complete matching candidate elements. It then combines the comprehensive score results to extract the top N candidate elements, thereby generating candidate repair element information. This approach elevates the candidate element selection process from a single-rule matching to a collaborative decision-making mechanism of "structural matching + spatial matching + semantic similarity." This not only improves robustness in scenarios involving partial UI changes but also significantly reduces the misjudgment rate caused by template failures or text changes, thereby enhancing the success rate and stability of the visual self-healing module's automatic repair in complex front-end interface scenarios.

[0068] S24: Verify the information of the candidate repair elements and obtain the verification results.

[0069] S25: When the verification result meets the preset repair conditions, update the test script according to the candidate repair element information, and re-execute the test script to obtain the execution result.

[0070] In this embodiment, the verification of candidate repair element information to obtain verification results includes: generating a template file corresponding to the candidate repair element information based on the candidate repair element information, and performing format verification on the template file to obtain format verification results; when the format verification results meet the preset verification conditions, verifying the candidate repair element information based on the template file to obtain verification results.

[0071] During the visual self-healing process, the system verifies each candidate solution based on the candidate repair element information output in step S23. The specific process is as follows: 1) Template File Generation and Format Validation: A new template file is generated based on the candidate element sub-screenshots and their corresponding coordinate information for subsequent screen recognition. The template file is an image resource file generated based on the candidate element sub-screenshots, used as a visual template for UI element matching in Airtest. Each template file corresponds one-to-one with a candidate repair element, replacing the invalid template image in the original test script. Simultaneously, the coordinate information of the second region is bound to the template file for subsequent script click positioning.

[0072] The system performs format validation on the generated template file and the coordinate information of the second region to determine whether they meet preset validation conditions. These preset validation conditions include the existence of the template file, the validity of the template file format, the accessibility of the template file path, and whether the coordinate information of the second region is within the current screen coordinate range and conforms to the coordinate representation rules with the screen center as the origin. Format validation refers to the process of checking whether the template file and coordinate data meet preset structural specifications. These specifications include: the existence of the template file in the specified path, whether the template file is a valid image format (such as PNG / JPG), whether the template file path is accessible to the system, and whether the coordinate information of the second region conforms to the Airtest script coordinate format (i.e., a two-dimensional coordinate structure with the screen center as the origin). When both the template file and the coordinate information of the second region meet the preset validation conditions, the generated format validation result is "pass"; otherwise, the generated format validation result is "fail". The format validation result is the output judgment after checking the above structural specifications, used to characterize whether the template file and coordinate information meet the basic conditions for entering the validation stage, and its value includes "pass" or "fail".

[0073] 2) Verification Test: When the format verification result is passed, the system calls the Airtest's exists function based on the template file to perform screen matching detection on the UI elements to determine whether the UI elements corresponding to the template file can be recognized on the current screen. When the template file can be accurately recognized, a touch simulation click operation is further performed based on the coordinate information of the second region to verify the interactivity and responsiveness of the target UI element in spatial position.

[0074] 3) Verification Result Determination and Candidate Element Processing: When the template file can be successfully recognized and the click operation can trigger the expected UI interaction (preset repair conditions), the candidate repair element is determined to meet the preset repair conditions, i.e., the verification result is successful, and the system identifies the candidate repair element as a usable repair solution. When the candidate repair element does not meet the preset repair conditions, i.e., the verification result is unsuccessful, the system selects the next candidate repair element in the order of the candidate repair element set for repeated verification until the preset termination conditions are met. The preset termination conditions include: all candidate repair elements have been traversed, there are no usable candidate repair elements, or the number of self-healing retries reaches a preset threshold.

[0075] Based on the verification results confirmed in step S24, the system updates and re-executes the test script according to the candidate element verification results: 1) Script update: For successfully verified candidate elements, the generated template file path and second region coordinate information replace the corresponding template file and coordinate information in the original script. This ensures that the update process does not affect other unchanged script operations, and a modification log is recorded for traceability. 2) Test script re-execution: The updated test script is re-executed under the control of the unified scheduling and execution management module. During execution, it continues to monitor for new exceptions. If a new exception occurs, the visual self-healing process can be triggered again until the preset termination conditions are met. 3) Execution result output: The execution result of the self-healed script is output, including whether the operation was successfully completed, the verification status of each candidate element, and the final script execution status. If all candidate element verifications fail or the retry limit is reached, failure information is recorded and manual intervention is required.

[0076] By performing template recognition and click response verification on candidate repair element information, the identifiability and interaction validity of candidate elements can be confirmed in a real-world operating environment, thereby selecting truly usable repair solutions. Upon successful verification, the test script is automatically updated and re-executed, enabling the script to adapt to changes in interface elements and resume normal operation. This closed-loop processing mechanism of verification and update not only improves the reliability and accuracy of repair solutions but also reduces the workload of manual troubleshooting and script modification, thus enhancing the maintenance efficiency and execution stability of automated test scripts.

[0077] The following section uses a specific example, taking the automated testing of the front-end UI of a financial product as an application scenario, to explain in detail the implementation process of the test script self-healing method provided in this application embodiment. In this scenario, automated test scripts are used to perform daily scheduled regression testing on the main processes of the mini-program. When UI elements change due to front-end page updates, a visual self-healing mechanism automatically repairs the test scripts. The specific implementation process is as follows: After system startup, the initialization phase begins. The unified scheduling and execution management module reads the pre-configured test script directory via command interaction, such as the insurance product application process automation scripts stored in the cases directory. The system parses all Airtest test scripts in this directory and loads them into the scheduling task list.

[0078] During initialization, the system performs unified parsing of Template screenshot resources in the script. It scans all image names corresponding to Airtest Template objects in the script using regular expressions and determines whether they are meaningless names generated by the system (e.g., screenshot names starting with "tp"). For such screenshots, the system uses OCR technology to recognize the text content and renames the images using the recognized text, while automatically updating the Template reference paths in the script. This process ensures that the template image names in the test script reflect the actual UI element content, providing foundational data for text matching and candidate element filtering during subsequent visual self-healing processes. After initialization, the unified scheduling and execution management module starts scheduled tasks, such as executing all automated test scripts daily and generating a test report after execution.

[0079] During a test execution, when the script reached line N of a certain test script, due to a front-end page update, the text for the button used to select risk warnings on the original health declaration page changed from "Partially Unqualified" to "Partially Unqualified for the Health Declaration-Free Plan" (e.g., ...). Figure 4 (As shown). Because the template in the script is still based on the old page screenshot for matching, Airtest cannot recognize the button when performing the click operation, thus triggering a TemplateNotFoundError exception.

[0080] The unified scheduling and execution management module continuously monitors for anomalies during script execution. When the aforementioned error is detected, the anomaly capture module immediately records the error log and captures a screenshot of the abnormal interface on the current page. Simultaneously, it parses the error log content to extract information such as the script code line where the failed operation occurred, the target template image, and the original click coordinates. Subsequently, the abnormal page screenshot, target template image, and original click coordinates are used as error context information and passed to the visual self-healing process.

[0081] After receiving the error context information, the visual self-healing module transmits the abnormal interface screenshot to the Omniparser-v2 image recognition and detection module for interface parsing. In this module, the system uses the YOLO v8n model to perform object detection on the abnormal interface screenshot, obtaining multiple interface element regions and their corresponding first region coordinates. Simultaneously, the system uses the Paddle OCR model to perform text recognition processing on the abnormal interface screenshot, obtaining the first text information. Furthermore, the system uses the Florence-2 model to perform semantic classification processing on interface elements (interface elements refer to interactive or identifiable objects that constitute a UI page, including buttons, input boxes, menu controls, text display areas, and icons, etc.) to identify different types of UI elements (such as buttons, input boxes, text display areas, and icons) and filter out non-critical UI elements. Finally, the system generates interface element detection results, which consist of three types of information: interface element regions (UI visual regions), first region coordinate information (YOLO detection coordinates), and first text information (OCR text content).

[0082] Subsequently, based on the interface element detection results, the system performs image cropping on the abnormal interface screenshots using the PIL image processing library. By employing a coordinate-range-based region cropping method, each interface element region is segmented from the original screenshot into independent image blocks, thereby generating element sub-screenshots corresponding one-to-one to each interface element region (e.g., ...). Figure 5 As shown in the diagram, a mapping relationship is established between the element sub-screenshots and the coordinate information of the first region to ensure that each element sub-screenshot can be traced back to its spatial position in the original interface. Based on this, the system performs coordinate system transformation on the first region coordinate information to generate the second region coordinate information. Furthermore, for each element sub-screenshot, OCR recognition is performed again to obtain the corresponding text content, i.e., the second text information. Finally, the split element sub-screenshots, the second region coordinate information, and the second text information are uniformly structured and encapsulated to generate the target element data.

[0083] After obtaining the target element data, the system enters the candidate element filtering stage. The system first performs preliminary filtering of the target element data (e.g., removing sub-screenshots of icon elements and sub-screenshots of elements that do not contain valid Chinese text content after secondary OCR recognition), obtaining the filtered element sub-screenshots and their corresponding second region coordinate information and second text information. Subsequently, it performs standardization processing (e.g., text standardization, coordinate normalization, image standardization) to generate a candidate element set.

[0084] After standardization, a candidate element set is generated. Each candidate element contains a standardized sub-screenshot, second region coordinate information, and second text information, serving as the basis for subsequent matching. The system categorizes candidate elements into two processing paths based on whether the candidate text has a complete match with the target text. The candidate text corresponds to the second text information, while the target text is the text information obtained from the target template image in the error context through OCR recognition. The first processing method: When the candidate text contains all characters of the target text and a substring match exists, the preset score calculation conditions are met. The system calculates the text length matching degree and spatial coordinate matching degree separately, and then weights and fuses them to generate the first score result. The second processing method: When the score calculation conditions are not met, text similarity calculation rules are used. These rules include Levenshtein edit distance and LCS similarity, which are then fused to obtain the second score result.

[0085] During the candidate repair element generation stage, the system normalizes the first and second score results and sorts them hierarchically according to a matching priority rule: first, sorting by the first score result, then by the second score result, and further merging them to generate a comprehensive score. Finally, the top N candidate elements are extracted based on the comprehensive score (e.g., selecting candidates with scores above 85), forming candidate repair element information, which includes sub-screenshots, coordinate information, text information, and the comprehensive score. Subsequently, the system outputs the candidate repair element information to the visual self-healing module for subsequent verification and script repair.

[0086] The visual self-healing module verifies candidate repair elements sequentially based on their information. The system first generates a new template file from the candidate element's sub-screenshot and verifies the validity of its file path and coordinate information. Then, it verifies whether the template can be correctly recognized on the current interface using the `Airtest.exists` function and performs a simulated click operation to determine if the expected UI interaction behavior is triggered. For example... Figure 6 As shown, if a candidate element is successfully verified, the template file and coordinate information corresponding to that element replace the old template information in the original script, and the unified scheduling and execution management module re-executes the test script. If the script executes successfully, the self-healing repair content is recorded and the test report is updated. If verification fails, the system selects the next candidate element from the candidate list to continue verification and script update operations. When all candidate elements fail to complete the process or the maximum number of self-healing attempts set by the system is reached, the system stops the automatic repair process and records a failure log to prompt manual intervention for maintenance.

[0087] The test script self-healing method provided in this application automatically captures abnormal information during the execution of the test script and constructs error context information based on the abnormal information, including target template image information, original click coordinate information, and screenshots of abnormal pages, providing an accurate data foundation for subsequent interface element identification and anomaly analysis. Subsequently, a preset image recognition detection model is used to perform interface element detection and text recognition on the screenshots of abnormal pages, obtaining the coordinate and text information corresponding to each element area in the interface. Structured target element data is then formed through screenshot splitting and coordinate normalization, enabling the system to comprehensively acquire the distribution of interface elements on the current page. Based on this, candidate element screening and standardization are performed on the target element data, and candidate elements are comprehensively scored and ranked using multi-dimensional matching indicators such as text length matching degree, text similarity, and spatial coordinate matching degree, thereby selecting candidate repair element information. Furthermore, template recognition verification and click response verification are performed on the candidate repair elements to confirm their identifiability and interactive effectiveness in the actual interface environment. After successful verification, the corresponding template file and coordinate information in the test script are automatically updated, and the script is re-executed, thereby achieving automatic repair and continued operation of the test script. Through the closed-loop processing mechanism of execution-anomaly detection-candidate element analysis-verification-script update-re-execution described above, this application can automatically locate new operable interface elements and repair test scripts in the event of minor page changes such as changes in front-end interface text and layout adjustments. This reduces script execution failures caused by interface changes, thereby significantly reducing the manual maintenance cost of front-end UI automated test scripts and improving the continuous availability of test scripts and the overall operating efficiency of the automated testing system.

[0088] This application also provides an electronic device 200, please refer to... Figure 7 This diagram illustrates the hardware structure of an electronic device 200 capable of performing the methods described in the embodiments above. The electronic device 200 includes: at least one processor 210; and a memory 220 communicatively connected to the at least one processor 210. Figure 7 Taking a processor 210 as an example, the memory 220 stores instructions executable by at least one processor 210. These instructions, when executed by at least one processor 210, enable the at least one processor 210 to execute the test script self-healing method of the above embodiment. The processor 210 and the memory 220 can be connected via a bus or other means. Figure 7 Taking the example of a connection between China and Israel via a bus.

[0089] The memory 220, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules, such as the program instructions / modules corresponding to the test script self-healing method in the embodiments of this application. The processor 210 executes various functional applications and data processing of the server by running the non-volatile software programs, instructions, and modules stored in the memory 220, thereby implementing the test script self-healing method of the above embodiments.

[0090] Memory 220 may include a stored program area and a stored data area, wherein the stored program area may store the operating system and applications required for at least one function; the stored data area may store data created based on the use of the computing device, etc. Furthermore, memory 220 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, memory 220 may optionally include memory remotely located relative to processor 210, and these remote memories may be connected to the computing device via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0091] One or more modules are stored in memory 220 and, when executed by one or more processors 210, execute the test script self-healing method of the above embodiments.

[0092] It should be noted that the electronic device in the embodiments of this application may be a server, desktop computer, laptop computer or other computing device capable of running automated testing programs and image recognition programs. It executes program instructions in the memory through the processor to realize the above-mentioned test script self-healing method.

[0093] The above-described product can execute the methods provided in the embodiments of this application, and has the corresponding functional modules and beneficial effects for executing the methods. Technical details not described in detail in this embodiment can be found in the test script self-healing method of the embodiments of this application.

[0094] This application provides a non-volatile computer-readable storage medium storing computer-executable instructions. These instructions are executed by one or more processors to enable at least one processor to execute the test script self-healing method described in the above embodiments. For example, the non-volatile computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a compact disc read-only memory (CDROM), magnetic tape, floppy disk, or optical data storage device, etc.

[0095] This application provides a computer program product, which includes a computer program stored on a non-volatile computer-readable storage medium. The computer program includes program instructions, which, when executed by an electronic device, enable the electronic device to execute the test script self-healing method in any of the above method embodiments.

[0096] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0097] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and not to limit them; under the concept of this application, the technical features of the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other variations of different aspects of this application as described above, which are not provided in detail for the sake of brevity; although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. A self-healing method for test scripts, characterized in that, The method includes: When a test script fails to execute, retrieve the error message. When the error message meets the preset visual self-healing conditions, the visual self-healing process is executed until the preset termination conditions are met; wherein, the visual self-healing process includes: Based on the error message, obtain the error context information; Based on a preset image recognition and detection model, abnormal interface screenshots in the error context information are detected to obtain target element data; Perform a candidate element filtering operation on the target element data to obtain candidate repair element information; The candidate repair element information is verified to obtain the verification results; When the verification result meets the preset repair conditions, the test script is updated according to the candidate repair element information, and the test script is re-executed to obtain the execution result.

2. The test script self-healing method according to claim 1, characterized in that, The step of obtaining error context information based on the error message includes: Based on the error message, parse the erroneous lines of code in the test script; Based on the error code line, extract the target template image information and the original click coordinate information; Obtain a screenshot of the exception page when the test script fails to execute; The target template image information, the original click coordinate information, and the screenshot of the abnormal page are used as the error context information.

3. The test script self-healing method according to claim 1, characterized in that, The preset image recognition detection model detects abnormal interface screenshots in the error context information to obtain target element data, including: Based on the image recognition and detection model, the abnormal interface screenshot is subjected to interface element detection to obtain multiple interface element regions and the corresponding first region coordinate information. Perform text recognition processing on the interface element area to obtain the first text information corresponding to the interface element area; The coordinate information of the first region is associated with the first text information to generate the interface element detection result; Based on the interface element detection results, the target element data is generated.

4. The test script self-healing method according to claim 3, characterized in that, The step of generating the target element data based on the interface element detection results includes: Based on the interface element detection results, the abnormal interface screenshot is divided into multiple element sub-screenshots, as well as second region coordinate information and second text information corresponding to the element sub-screenshots; wherein, the second region coordinate information is obtained based on the first region coordinate information; the second text information is obtained by performing optical character recognition on the sub-screenshots; The element sub-screenshot is associated and stored with the corresponding second region coordinate information and second text information to generate the target element data.

5. The test script self-healing method according to claim 4, characterized in that, The step of performing candidate element filtering on the target element data to obtain candidate repair element information includes: The target element data is filtered to obtain the filtered element sub-screenshots and the corresponding second region coordinate information and second text information; The filtered element sub-screenshots and their corresponding second region coordinate information and second text information are standardized to generate a candidate element set; Based on the set of candidate elements, the information of the candidate repair elements is obtained.

6. The test script self-healing method according to claim 5, characterized in that, The step of obtaining the candidate repair element information based on the candidate element set includes: When the candidate element set meets the preset score calculation conditions, text matching is performed on the candidate element set based on the preset text length matching degree to obtain text matching information; spatial position matching is performed on the candidate element set based on the preset spatial coordinate matching degree to obtain spatial position matching information; and a first score result of the candidate element set is calculated based on the text matching information and the spatial position matching information. When the candidate element set does not meet the score calculation conditions, a second score result of the candidate element set is calculated based on the preset text similarity calculation rules; Based on the preset matching priority rules, the first score result, and the second score result, the candidate element set is extracted to generate the candidate repair element information.

7. A test script self-healing system, executing the test script self-healing method according to any one of claims 1 to 6, characterized in that, The system includes a unified scheduling and execution management module, an anomaly capture module, a visual self-healing module, an image recognition and detection module, and an anomaly analysis and candidate element selection module. The anomaly capture module is connected to the unified scheduling and execution management module and the visual self-healing module. The visual self-healing module is also connected to the unified scheduling and execution management module, the image recognition and detection module, and the anomaly analysis and candidate element selection module. The image recognition and detection module is also connected to the anomaly analysis and candidate element selection module.

8. The test script self-healing system according to claim 7, characterized in that, The anomaly analysis and candidate element selection module is used for: When the candidate element set meets the preset score calculation conditions, text matching is performed on the candidate element set based on the preset text length matching degree to obtain text matching information; Based on a preset spatial coordinate matching degree, spatial position matching is performed on the candidate element set to obtain spatial position matching information; Based on the text matching information and the spatial location matching information, calculate the first score result of the candidate element set; When the candidate element set does not meet the score calculation conditions, a second score result of the candidate element set is calculated based on the preset text similarity calculation rules; Based on the preset matching priority rules, the first score result, and the second score result, the candidate element set is sorted and optimized to extract the candidate repair element information.

9. An electronic device, characterized in that, include: At least one processor; as well as, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the method according to any one of claims 1-6.

10. A non-volatile computer-readable storage medium, characterized in that, The non-volatile computer-readable storage medium stores computer-executable instructions that, when executed by an electronic device, cause the electronic device to perform the method described in any one of claims 1-6.

Images