Method and apparatus for positioning interface elements

Abstract

This application discloses a method and apparatus for locating interface elements. The method includes: acquiring an image of the interface upon initial loading and using it as a reference image; extracting and recording features from the reference image using a trained element localization model; and using the extracted interface elements as reference interface elements; acquiring an image during application runtime or interface update and using it as the current image; extracting and recording features from the current image using a trained element localization model; and using the extracted interface elements as the current interface elements; if it is determined that the feature information of the current interface element has changed relative to the feature information of the reference interface element, then outputting a recognition result; and updating the feature information of the reference interface element to the feature information of the current interface element based on the recognition result, and updating the corresponding test script to locate the interface elements. Using this application, dynamic identification and adjustment of element localization can be achieved during test runtime, reducing the need for frequent maintenance.

Description

Technical Field

[0001] This application relates to the field of computer software technology, and in particular to a method and apparatus for locating interface elements. Background Technology

[0002] Traditional UI (User Interface) automated testing methods mainly rely on static element locators, such as IDs (Identifiers) and XPath (XML Path Language), to locate UI elements. However, when the UI changes, these locators often become ineffective, causing test scripts to fail to execute properly and requiring frequent maintenance.

[0003] Therefore, how to provide a new method for locating interface elements is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0004] This application addresses the shortcomings of existing methods by proposing a method and apparatus for locating interface elements. This solves the problem that when the UI changes, UI elements may also be adjusted, and traditional static element locators may fail, causing test scripts to fail to execute properly and requiring frequent maintenance.

[0005] Firstly, embodiments of this application provide a method for locating interface elements, including:

[0006] The image of the interface when it is first loaded is obtained and used as the reference image. The trained element localization model is used to extract and record the features of the interface elements in the reference image, and the extracted interface elements are used as the reference interface elements.

[0007] The system acquires an image during application operation or interface update and uses it as the current image. It then uses the trained element localization model to extract and record features of the interface elements in the current image and uses the extracted interface elements as the current interface elements.

[0008] If it is determined that the feature information of the current interface element has changed relative to the feature information of the reference interface element, the recognition result is output.

[0009] Based on the recognition results, the feature information of the baseline interface element is updated to the feature information of the current interface element, and the corresponding test script is updated to locate the interface element.

[0010] In one possible implementation, if it is determined that the feature information of the current interface element has changed relative to the feature information of the reference interface element, the recognition result is output, including:

[0011] Using the element feature library of the trained element localization model, the feature information of the current interface elements is compared and analyzed with the feature information of the baseline interface to obtain the analysis results; the element feature library includes the feature information of the elements and the change patterns of the elements.

[0012] The correctness of the analysis results is determined by the confidence score output by the trained element localization model.

[0013] If the confidence score is greater than the preset threshold, the analysis result is determined to be correct, and the recognition result is output.

[0014] In one possible implementation, the element feature library of the trained element localization model is used to compare and analyze the feature information of the current interface elements with the feature information of the baseline interface to obtain the analysis results, including:

[0015] Using the element feature library of the trained element localization model, the feature information of the current interface element and the feature information of the baseline interface element are compared to obtain the analysis results; the feature information includes at least one of the following: attribute information, layout information, visual features and semantic information;

[0016] The attribute information includes at least one of the following: label, element ID, and class name; the layout information includes at least one of the following: relative position, hierarchy, and path; the visual features include at least one of the following: font, color, shape, size, and style; and the semantic information includes the text content.

[0017] One possible implementation also includes:

[0018] The trained element localization model is embedded into the automated testing tool for dynamic identification and adjustment of element localization during test runtime.

[0019] In one possible implementation, the corresponding test script is updated, including:

[0020] Generate a test report; the test report includes: the state of the interface elements before and after the change, the time of the change, and the frequency of the change;

[0021] Use a script update tool to read and parse the change information in the test report, and retrieve the corresponding test script for updating; or, automatically replace the feature information of the current interface element with the corresponding position in the test script.

[0022] In one possible implementation, the method also includes:

[0023] Generate a feedback dataset, which includes labeled error types and feature data of input interface elements;

[0024] Based on the feedback dataset, the element localization model was retrained.

[0025] In one possible implementation, the training process of the element localization model includes:

[0026] Acquire multi-interface image data from both desktop and mobile devices;

[0027] Feature extraction is performed on multi-interface image data to obtain feature information; the feature information includes attribute information, layout information, visual features and semantic information.

[0028] Automatically label feature information and generate the formatted dataset required for training;

[0029] The formatted dataset is input into the convolutional neural network model to learn and record the feature information and change patterns of the elements, and to build an element feature library.

[0030] Based on the element feature library, the convolutional neural network model is adjusted, and the trained convolutional neural network model is used as the element localization model.

[0031] Secondly, embodiments of this application provide a positioning device for interface elements, comprising:

[0032] The first acquisition module is configured to acquire the image of the interface when it is first loaded and use it as a reference image, use the trained element localization model to extract and record the features of the interface elements in the reference image, and use the extracted interface elements as the reference interface elements.

[0033] The second acquisition module is configured to acquire the image when the application is running or the interface is updated and use it as the current image, use the trained element localization model to extract and record the features of the interface elements in the current image, and use the extracted interface elements as the current interface elements.

[0034] The output module is configured to output the recognition result if it is determined that the feature information of the current interface element has changed relative to the feature information of the reference interface element.

[0035] The update module is configured to update the feature information of the baseline interface element to the feature information of the current interface element based on the recognition result, and update the corresponding test script to locate the interface element.

[0036] Thirdly, embodiments of this application provide a computer-readable storage medium storing computer program instructions, which, when executed by a processor, implement the steps of any of the methods described above.

[0037] Fourthly, embodiments of this application provide an electronic device, including a memory and a processor, wherein the memory stores an executable program, and the processor executes the executable program to perform the steps of any of the methods described above.

[0038] The interface element localization method provided in this application can dynamically identify and adjust element localization. It acquires an image from the initial interface loading and uses it as a reference image. A trained element localization model is used to extract and record features from the reference image, and the extracted interface elements are used as reference interface elements. Then, an image is acquired during application runtime or interface updates and used as the current image. The trained element localization model is used to extract and record features from the current image, and the extracted interface elements are used as the current interface elements. If it is determined that the feature information of the current interface element has changed relative to the feature information of the reference interface element, a recognition result is output. Based on the recognition result, the feature information of the reference interface element is updated to the feature information of the current interface element, and the corresponding test script is updated to locate the interface elements. This method enables dynamic identification and adjustment of element localization during test runtime, ensuring continuous and stable test scripts, reducing frequent maintenance needs due to interface adjustments, minimizing manual modifications, significantly reducing maintenance costs, and improving test efficiency and system stability. Attached Figure Description

[0039] Figure 1 A flowchart illustrating a method for locating interface elements provided in an embodiment of this application;

[0040] Figure 2 Provided for the embodiments of this application Figure 1 A flowchart of the S300 process;

[0041] Figure 3 Provided for the embodiments of this application Figure 1 A flowchart illustrating the test script for updating the S400 in China;

[0042] Figure 4 A schematic diagram illustrating the process of retraining the element localization model through a feedback mechanism, as provided in an embodiment of this application;

[0043] Figure 5 This is a schematic diagram of the structure of a positioning device for interface elements provided in an embodiment of this application. Detailed Implementation

[0044] Various embodiments and features of this application are described herein with reference to the accompanying drawings.

[0045] It should be understood that various modifications can be made to the embodiments described herein. Therefore, the above description should not be considered as limiting, but merely as an example of embodiments. Other modifications within the scope and spirit of this application will be apparent to those skilled in the art.

[0046] The accompanying drawings, which are included in and form part of this specification, illustrate embodiments of the present application and, together with the general description of the present application given above and the detailed description of the embodiments given below, serve to explain the principles of the present application.

[0047] These and other features of this application will become apparent from the following description of preferred forms of embodiments given as non-limiting examples, with reference to the accompanying drawings.

[0048] It should also be understood that although this application has been described with reference to some specific examples, those skilled in the art can certainly implement many other equivalent forms of this application.

[0049] The above and other aspects, features and advantages of this application will become more apparent when taken in conjunction with the accompanying drawings and in view of the following detailed description.

[0050] Specific embodiments of this application are described thereafter with reference to the accompanying drawings; however, it should be understood that the claimed embodiments are merely examples of this application, which can be implemented in various ways. Well-known and / or repeated functions and structures are not described in detail to avoid unnecessary or redundant details that could obscure the application. Therefore, the specific structural and functional details claimed herein are not intended to be limiting, but merely serve as the basis and representative basis for the claims to teach those skilled in the art to use this application in a variety of substantially any suitable detailed structures.

[0051] This specification may use the phrases “in one embodiment,” “in another embodiment,” “in yet another embodiment,” or “in other embodiments,” all of which may refer to one or more of the same or different embodiments according to this application.

[0052] The inventors of this application discovered through research that when the UI interface changes, there may be adjustments to the page layout, changes to labels or attributes, etc. Since some traditional element locators such as ID and XPath are based on fixed page structures or element attributes to locate UI elements, once these characteristics change, the test script cannot find the target element. When the element locator fails, testers need to modify the element locator, rewrite some test code, and perform other maintenance operations on the script. These operations are time-consuming and labor-intensive, and require testers to have a certain understanding of UI structure and coding skills.

[0053] Therefore, the interface element positioning method provided in this application aims to solve the problem in the prior art that when the UI interface changes, the UI elements may also be adjusted, and the traditional static element locator may fail, causing the test script to fail to execute normally, thus requiring frequent maintenance.

[0054] The following describes in detail, with reference to the accompanying drawings, a method for locating interface elements provided in an embodiment of this application. Figure 1 This is a flowchart illustrating a method for locating interface elements provided in an embodiment of this application. Figure 1 This diagram illustrates a flowchart of a method for locating interface elements according to an embodiment of this application. This method is applied to electronic devices, which may exemplarily include personal computers, mobile phones, or laptops. See also... Figure 1 A method for locating interface elements according to an embodiment of this application includes the following steps:

[0055] S100: Obtain the image of the interface when it is first loaded and use it as the reference image. Use the trained element localization model to extract and record the features of the interface elements in the reference image, and use the extracted interface elements as the reference interface elements.

[0056] For example, the reference image is an image obtained when the electronic device first loads its interface. This image serves as a reference image, used subsequently to compare element features with images acquired in real time if the interface of the electronic device changes, thereby achieving dynamic recognition and adjustment of element positioning. The trained element positioning model extracts and records features for each interface element in the reference image. The feature information of each interface element may include attributes, layout information, visual features, and semantic information, and each extracted interface element is used as a reference interface element.

[0057] S200: Acquire the image during application operation or interface update and use it as the current image. Use the trained element localization model to extract and record the features of the interface elements in the current image, and use the extracted interface elements as the current interface elements.

[0058] The interface is monitored in real time, capturing images as the application runs or the interface updates. For example, when an electronic device's application runs or the interface updates, interface images are captured periodically or after specific trigger events, and these captured images are used as the current image. A trained element localization model is used to extract and record features for each interface element in the current image. The feature information of each interface element can include attributes, layout information, visual features, and semantic information. Each extracted interface element is used as the current interface element for subsequent dynamic identification and adjustment of element localization, thus achieving the localization of interface elements.

[0059] S300: If it is determined that the feature information of the current interface element has changed relative to the feature information of the reference interface element, the recognition result is output.

[0060] For example, the trained element localization model performs real-time monitoring and change detection on the interface. When it detects that the feature information of the current interface element has changed relative to the feature information of the reference interface element, it identifies the changed feature information. For example, it can identify the element's attributes, layout information, visual features, and semantic information respectively, and output the identification results. The identification results can include feature change information, such as changes in the element's attributes and changes in the element's visual features.

[0061] S400: Based on the recognition results, update the feature information of the baseline interface element to the feature information of the current interface element, and update the corresponding test script to locate the interface element.

[0062] For example, based on the output feature change information, the feature information of the baseline interface element is updated to the feature information of the current interface element. For instance, if the visual features of the baseline interface element change, the visual features of the baseline interface element are updated to the visual features of the current interface element for subsequent testing, and the corresponding test script is updated synchronously to achieve real-time positioning of the interface element.

[0063] The interface element localization method provided in this application utilizes a trained element localization model to identify changes in the feature information of the current interface element relative to the feature information of a reference interface element, and outputs the identification result. Based on the identification result, the feature information of the reference interface element is updated to the feature information of the current interface element, and the corresponding test script is updated to locate the interface element. This application embodiment can dynamically identify and adjust element localization during test execution, ensuring the continuous stability of the test script. This allows the test script to adapt to most changes in the interface, reducing frequent maintenance needs caused by interface adjustments, minimizing manual modifications, significantly reducing maintenance costs, and improving testing efficiency and system stability.

[0064] Moreover, the interface element positioning method provided in this application embodiment can ensure that the test script can continue to run when the UI changes by using real-time recognition and positioning technology, thereby improving the efficiency and stability of automated test execution.

[0065] In some embodiments, see Figure 2 As shown, if it is determined that the feature information of the current interface element has changed relative to the feature information of the reference interface element, the recognition result is output, including:

[0066] S301: Using the element feature library of the trained element localization model, the feature information of the current interface element is compared and analyzed with the feature information of the baseline interface to obtain the analysis results; wherein, the element feature library includes the feature information of the element and the change pattern of the element;

[0067] For example, the element feature library of the trained element localization model records the feature information of the elements and their change patterns. By comparing and analyzing the feature information of the current interface elements with the feature information of the benchmark interface, the analysis results can be obtained, which include feature change information.

[0068] S302: The correctness of the analysis results is determined by the confidence score output by the trained element localization model;

[0069] S303: If the confidence score is greater than the preset threshold, the analysis result is determined to be correct, and the identification result is output.

[0070] For example, this element localization model is built upon a real-time recognition algorithm. This algorithm utilizes GPU acceleration to process visual data and is applied in real-time during testing to dynamically identify UI elements. Furthermore, this real-time recognition algorithm can perform real-time element recognition for object detection algorithms based on convolutional neural networks (CNNs), such as YOLO or SSD. Lightweight networks (such as MobileNet) are introduced to reduce computational overhead while maintaining high detection accuracy. The implementation can be as follows: inputting the current image acquired in real-time, initializing weights using a pre-trained model, performing transfer learning on UI features, processing page visual data in parallel via GPU, and detecting UI elements in real-time. Post-detection processing steps (such as non-maximum suppression) are optimized to reduce latency.

[0071] This real-time recognition algorithm strikes a balance between real-time performance and accuracy. It reduces the number of parameters by using a lightweight model and achieves rapid inference through GPU acceleration. Optimizing the input to the neural convolutional network model, such as adjusting the page screenshot to a fixed resolution, reduces computational load and enables real-time element recognition. Multi-dimensional feature fusion, such as visual and positional features, improves recognition accuracy. Dynamically adjusting the preset confidence threshold ensures reliable detection under different UI environments, further enhancing the accuracy of interface element recognition.

[0072] In some embodiments, the feature library of the trained element localization model is used to compare and analyze the feature information of the current interface elements with the feature information of the baseline interface to obtain the analysis results, including:

[0073] Using the element feature library of the trained element localization model, the feature information of the current interface element and the feature information of the baseline interface element are compared to obtain the analysis results; the feature information includes at least one of the following: attribute information, layout information, visual features and semantic information;

[0074] The attribute information includes at least one of the following: label, element ID, and class name; the layout information includes at least one of the following: relative position, hierarchy, and path; the visual features include at least one of the following: font, color, shape, size, and style; and the semantic information includes the text content.

[0075] For example, difference analysis can include visual difference analysis, structural difference analysis, and attribute difference analysis. Visual difference analysis compares the visual features of interface elements, such as color, shape, and size, to detect changes in visual appearance. Structural difference analysis can examine changes to nodes in the DOM structure, including additions, deletions, and modifications. Attribute difference analysis can identify changes through variations in tags, IDs, class names, and other attributes.

[0076] The following explanation uses visual difference analysis as an example.

[0077] By capturing screenshots and visually comparing them, we can verify changes in the state and position of interface elements, ensuring that changes in interface elements are identified and adjusted in a timely manner.

[0078] Acquiring the current image: Browser automation tools, such as Selenium, can be used to capture a screenshot of the page in real time and use it as the current image. The acquired current image can then be saved in a standard image format.

[0079] Example code:

[0080] driver.save_screenshot("current_page.png")

[0081] Regarding visual comparison: OpenCV is used to process screenshots and compare the visual features of the current image with a reference image. This includes: element detection and difference analysis; element detection: extracting features such as shape, color, and position for each UI element. Difference analysis: detecting whether elements have moved or changed through image registration and template matching.

[0082] Example code:

[0083] import cv2

[0084] # Read the current screenshot and reference screenshot

[0085] current = cv2.imread("current_page.png")

[0086] reference = cv2.imread("reference_page.png")

[0087] # Convert to grayscale image

[0088] current_gray = cv2.cvtColor(current, cv2.COLOR_BGR2GRAY)

[0089] reference_gray = cv2.cvtColor(reference, cv2.COLOR_BGR2GRAY)

[0090] # Calculate the difference

[0091] diff = cv2.absdiff(current_gray, reference_gray)

[0092] This application utilizes computer vision technology (such as OpenCV) to compare and analyze the current image, checking for changes in the state and position of UI elements. This visual comparison method ensures that element positioning and test scripts are adjusted promptly when the UI changes, increasing robustness to dynamic UI changes.

[0093] The embodiments of this application use multi-dimensional features and visual contrast technology, which can effectively improve the accuracy of UI element recognition and reduce test failures caused by misidentification.

[0094] The correctness of the analysis results is determined by the confidence score output by the element localization model. If the confidence score is greater than a preset threshold, the analysis result is considered correct, and the recognition result is output. Based on the recognition result, the feature information of the baseline interface element is updated to the feature information of the current interface element, and the corresponding test script is updated to locate the interface element.

[0095] Optionally, the preset threshold for confidence can be dynamically adjusted to reduce the possibility of false recognition. Ensuring reliable detection under different UI environments can improve the accuracy of interface element recognition.

[0096] Optionally, if the confidence score is greater than a preset threshold, the analysis result is determined to be correct, and the identification result is output, including:

[0097] If the confidence score is greater than the preset threshold, the analysis result is considered correct.

[0098] The correctness of the analysis results is reconfirmed based on pre-set logic or business rules.

[0099] If the analysis results are confirmed to be correct, the identification results will be output.

[0100] This embodiment reduces false alarms by reconfirming the analysis results.

[0101] This application utilizes AI technology, particularly deep learning and computer vision, to develop a real-time recognition algorithm capable of dynamically identifying and locating UI elements in real time, adapting to frequently changing page layouts. It learns and recognizes the features of interface elements by training a convolutional neural network model, and continuously optimizes and adjusts it to adapt to UI changes. Specifically, the trained element location model not only relies on traditional DOM attributes but also incorporates visual features (such as color, shape, and layout), improving its adaptability to dynamic interface elements.

[0102] In some embodiments, before acquiring the image of the interface at the time of initial loading and using it as a reference image, the method further includes:

[0103] The trained element localization model is embedded into the automated testing tool for dynamic identification and adjustment of element localization during test runtime.

[0104] For example, the trained element localization model can be embedded into the Selenium testing framework to dynamically identify and adjust element localization during test execution, ensuring the continuity and stability of the test script and reducing manual modifications.

[0105] Selenium is a commonly used automated testing framework that primarily interacts with the browser through locators (such as IDs, XPath, and CSS selectors) to complete UI testing. Its operation involves predefined locators in the script, which then executes tests according to a sequence of operations. If a locator fails, the script needs to be manually adjusted. However, it cannot adapt to dynamic changes in UI elements, relies on static rules, and lacks intelligent adjustment capabilities.

[0106] Embedding the trained element localization model into a testing framework, such as Selenium, allows for dynamic identification and automatic adjustment of element localization during test runtime, ensuring script continuity and stability.

[0107] For example, the element location model can be embedded into the automated testing tool Selenium by extending the interface, as follows:

[0108] In the Selenium test script, add an interface to call the element localization model. During test runtime, capture the current image and pass it as input to the element localization model. The model outputs the recognition results, such as the bounding box of a button. Convert the recognition results into a Selenium-supported locator, such as the relative path of the interface element and its center point coordinates. Then, automatically update the locator in the test script to avoid manual modification.

[0109] By embedding the trained element localization model into the automated testing tool, the system dynamically adjusts the localization strategy based on feedback from the model during test execution. Even if page elements change, the test script maintains continuity and stability. This solves the problem of traditional testing frameworks being unable to adapt to dynamic page changes.

[0110] In some embodiments, see Figure 3 Update the corresponding test scripts, including:

[0111] S401: Generate a test report; the test report includes: the state of the interface elements before and after the change, the time of the change, and the frequency of the change;

[0112] S402: Use the script update tool to read and parse the change information in the test report, and retrieve the corresponding test script for update; or, automatically replace the feature information of the current interface element with the corresponding position in the test script.

[0113] For example, data visualization tools (such as Tableau and D3.js) can be used to generate intuitive and informative test reports, which can then be sent to the management platform for team reference. The test reports detail the detected element changes, allowing for the gradual refinement and improvement of the element location model to achieve higher success rates.

[0114] In addition, during test execution, the test status can be monitored in real time. Specifically, monitoring tools such as Prometheus can be used to capture and alert on abnormal events, and automated logging and analysis tools such as ELK Stack can be set up for test results and system performance.

[0115] The automated test script will be updated with information about the elements that have been changed. There are two ways to update the automated test script.

[0116] The first method is to use a script update tool: use an automated update tool to read and automatically parse each change information in the test report, and call the corresponding test script to perform the update.

[0117] The second approach utilizes locators for automatic replacement: For example, regarding XPath updates, tools can be used to reconstruct the XPath with new locations, attributes, etc. For instance, stability can be improved by generating a more robust XPath (such as / / button[contains(text(), 'Submit')] instead of an absolute path). Regarding ID and other attribute replacement: the new ID or other identifying attributes can be automatically replaced with the corresponding locations in the test script.

[0118] This application embodiment uses an element positioning model to identify changes in interface elements and generate a test report. Automated tools are then used to dynamically adjust elements in the test script based on the changed element information in the test report. This allows for dynamic identification and adjustment of element positioning during test execution, ensuring the continuity and stability of the test script and reducing manual modifications.

[0119] In some embodiments, see Figure 4 The method also includes:

[0120] S501: Generate a feedback dataset, which includes labeled error types and feature data of input interface elements;

[0121] S502: Retrain the element localization model based on the feedback dataset.

[0122] For example, if the recognition result output by the element localization model is inconsistent with the expectation, an error case is recorded. This error case includes: error type and input data. Error types include localization deviation, classification error, etc., and input data includes page screenshots and element features, etc. A feedback dataset is generated, and the problem is labeled so that the element localization model can be adjusted and optimized.

[0123] The feedback dataset can be used to compare error cases with the original training data to identify features or abnormal patterns not covered by the model. A new round of data collection and annotation can then be designed based on the error distribution.

[0124] In this embodiment, a feedback dataset is provided for adjusting the element localization model. After the test is run, the actual test results are compared with the expected results to analyze the correctness and stability of the element localization model. This can help optimize the element localization model or application. Based on the feedback dataset, the element localization module is retrained to achieve continuous learning and improvement of the element localization model.

[0125] This application's embodiments utilize a dynamic feedback mechanism to not only adjust the element localization strategy in real time but also automatically optimize the model's recognition accuracy based on test results. By generating feedback datasets, data augmentation, and incremental learning techniques, the system continuously optimizes the element localization model, ensuring its efficiency and stability in practical applications.

[0126] Furthermore, through continuous training and optimization of the element positioning model, the system can better adapt to different UI change scenarios, improving the versatility and adaptability of the testing framework.

[0127] In some embodiments, the training process of the element localization model includes:

[0128] Acquire multi-interface image data from both desktop and mobile devices;

[0129] Feature extraction is performed on multi-interface image data to obtain feature information; the feature information includes attribute information, layout information, visual features and semantic information.

[0130] Automatically label feature information and generate the formatted dataset required for training;

[0131] The formatted dataset is input into the convolutional neural network model to learn and record the feature information and change patterns of the elements, and to build an element feature library.

[0132] Based on the element feature library, the convolutional neural network model is adjusted, and the trained convolutional neural network model is used as the element localization model.

[0133] For example, a detailed explanation of data acquisition and model training is provided below:

[0134] Data Acquisition: Collect a large amount of UI element data, including labels, positions, styles, paths, etc., and build a data acquisition tool to automatically capture page snapshots to obtain the current image in real time.

[0135] Feature extraction: An extended feature extraction method is adopted, which extracts not only DOM attributes, but also styles (such as color and font) and layout information (such as relative position and hierarchy).

[0136] It supports snapshot capture on multiple devices and at multiple resolutions, adapts to PC and mobile pages, and achieves cross-platform adaptation for data collection. Through multi-page batch snapshot and data collection functions, it can improve data preparation efficiency and automatically annotate UI elements to generate the formatted dataset required for training.

[0137] Use tools such as Selenium and Puppeteer to load the target page and capture a snapshot to obtain the current image;

[0138] Example (Selenium):

[0139] from selenium import webdriver

[0140] driver = webdriver.Chrome()

[0141] driver.get("URL")

[0142] driver.save_screenshot("screenshot.png")

[0143] driver.quit()

[0144] This application's embodiments extend the feature extraction capabilities (such as tags, paths, and styles) of traditional frameworks through a dedicated, cross-platform UI element data acquisition tool. It supports the acquisition of page snapshots from multiple devices and resolutions, and can automatically annotate UI elements to generate the formatted dataset required for training. The development of this tool automates large-scale data acquisition and processing, significantly improving the efficiency of training data generation.

[0145] By parsing the DOM to extract features such as tags, positions, and styles, and then using OpenCV to process the current image, visual features such as shape and color are extracted.

[0146] Example (DOM extraction):

[0147] element = driver.find_element_by_id("example")

[0148] element_id = element.get_attribute("id")

[0149] element_location = element.location

[0150] Example (visual feature extraction):

[0151] import cv2

[0152] img = cv2.imread('screenshot.png')

[0153] gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)

[0154] edges = cv2.Canny(gray, 30, 100)

[0155] Automatically label feature information and generate the formatted dataset required for training;

[0156] Model training: Select a suitable model type, such as using a convolutional neural network to learn and record the feature information and change patterns of elements, and establish an element feature library; based on the element feature library, adjust the convolutional neural network model, and use the trained convolutional neural network model as the element localization model.

[0157] To strike a balance between accuracy and real-time performance, lightweight models (such as MobileNet) are preferred. If sufficient data is available, deep models are chosen; otherwise, simplified structures are used to avoid overfitting.

[0158] By combining attribute information, layout information, visual features, and semantic information, multi-dimensional feature fusion is achieved. A convolutional neural network model is trained using a labeled, formatted dataset to capture UI change patterns and learn element change patterns. The trained convolutional neural network model serves as an element localization model, enabling dynamic identification and adjustment of element positioning during testing, reducing manual modifications.

[0159] The element localization model based on real-time recognition algorithm provided in this application embodiment can achieve fast reasoning by using GPU acceleration while ensuring high recognition accuracy. It can dynamically recognize UI elements while ensuring real-time performance, thus optimizing the efficiency of the testing framework.

[0160] Based on the same inventive concept, embodiments of this application provide a positioning device for interface elements, see [link to relevant documentation]. Figure 5 As shown, Figure 5 This is a schematic diagram of a positioning device for an interface element provided in an embodiment of this application. The positioning device includes:

[0161] The first acquisition module 100 is configured to acquire the image of the interface when it is first loaded and use it as a reference image, use the trained element localization model to extract and record the features of the interface elements of the reference image, and use the extracted interface elements as the reference interface elements.

[0162] The second acquisition module 200 is configured to acquire an image during application operation or interface update and use it as the current image, use the trained element localization model to extract and record the features of the interface elements in the current image, and use the extracted interface elements as the current interface elements.

[0163] The output module 300 is configured to output the recognition result if it is determined that the feature information of the current interface element has changed relative to the feature information of the reference interface element.

[0164] The update module 400 is configured to update the feature information of the baseline interface element to the feature information of the current interface element based on the recognition result, and update the corresponding test script to locate the interface element.

[0165] The interface element localization device provided in this application embodiment utilizes a trained element localization model to identify changes in the feature information of the current interface element relative to the feature information of a reference interface element, and outputs the identification result. Based on the identification result, the feature information of the reference interface element is updated to the feature information of the current interface element, and the corresponding test script is updated to locate the interface element. This application embodiment can dynamically identify and adjust element localization during test execution, ensuring the continuous stability of the test script. This allows the test script to adapt to most changes in the interface, reducing frequent maintenance needs caused by interface adjustments, minimizing manual modifications, significantly reducing maintenance costs, and improving testing efficiency and system stability.

[0166] In some embodiments, the output module 300 is further configured to use the element feature library of the trained element localization model to compare and analyze the feature information of the current interface element with the feature information of the benchmark interface to obtain the analysis result; wherein, the element feature library includes the feature information of the element and the change pattern of the element.

[0167] The correctness of the analysis results is determined by the confidence score output by the trained element localization model.

[0168] If the confidence score is greater than the preset threshold, the analysis result is determined to be correct, and the recognition result is output.

[0169] In some embodiments, the output module 300 is specifically configured to use the element feature library of the trained element localization model to perform a difference analysis on the feature information of the current interface element and the feature information of the reference interface element to obtain the analysis result; the feature information includes at least one of the following: attribute information, layout information, visual features and semantic information;

[0170] The attribute information includes at least one of the following: label, element ID, and class name; the layout information includes at least one of the following: relative position, hierarchy, and path; the visual features include at least one of the following: font, color, shape, size, and style; and the semantic information includes the text content.

[0171] In some embodiments, the device further includes:

[0172] The embedding module is configured to embed the trained element localization model into the automated testing tool for dynamic identification and adjustment of element localization during test runtime.

[0173] In some embodiments, the update module 400 is further configured to generate a test report; wherein the test report includes: the state of the interface element's feature information before and after the change, the change time, and the change frequency;

[0174] Use a script update tool to read and parse the change information in the test report, and retrieve the corresponding test script for updating; or, automatically replace the feature information of the current interface element with the corresponding position in the test script.

[0175] In some embodiments, the device further includes:

[0176] The feedback module is configured to generate a feedback dataset, which includes labeled error types and feature data of input interface elements.

[0177] The first training module is configured to retrain the element localization model based on the feedback dataset.

[0178] In some embodiments, the device further includes a second training module, specifically configured as follows:

[0179] Acquire multi-interface image data from both desktop and mobile devices;

[0180] Feature extraction is performed on multi-interface image data to obtain feature information; the feature information includes attribute information, layout information, visual features and semantic information.

[0181] Automatically label feature information and generate the formatted dataset required for training;

[0182] The formatted dataset is input into the convolutional neural network model to learn and record the feature information and change patterns of the elements, and to build an element feature library.

[0183] Based on the element feature library, the convolutional neural network model is adjusted, and the trained convolutional neural network model is used as the element localization model.

[0184] This application also provides a computer-readable storage medium storing computer program instructions, which, when executed by a processor, implement the steps of the method described above.

[0185] This application also provides an electronic device, including a memory and a processor, wherein the memory stores an executable program, and the processor executes the executable program to perform the steps of the method described above.

[0186] The above embodiments are merely exemplary embodiments of this application and are not intended to limit this application. The scope of protection of this application is defined by the claims. Those skilled in the art can make various modifications or equivalent substitutions to this application within its substance and scope of protection, and such modifications or equivalent substitutions should also be considered to fall within the scope of protection of this application.

Claims

1. A method for locating interface elements, characterized in that, include: The image of the interface when it is first loaded is obtained and used as the reference image. The interface elements of the reference image are extracted and recorded using the trained element localization model. The extracted interface elements are used as the reference interface elements. The image obtained during application operation or interface update is used as the current image. The trained element localization model is used to extract and record the features of the interface elements in the current image, and the extracted interface elements are used as the current interface elements. If it is determined that the feature information of the current interface element has changed relative to the feature information of the reference interface element, the recognition result is output, including: using the element feature library of the trained element localization model, comparing and analyzing the feature information of the current interface element and the feature information of the reference interface to obtain the analysis result; wherein, the element feature library includes the feature information of the element and the change pattern of the element; Based on the recognition results, the feature information of the baseline interface element is updated to the feature information of the current interface element, and the corresponding test script is updated to locate the interface element.

2. The method according to claim 1, characterized in that, If it is determined that the feature information of the current interface element has changed relative to the feature information of the reference interface element, then the recognition result is output, including: The correctness of the analysis results is determined by the confidence score output by the trained element localization model. If the confidence score is greater than a preset threshold, the analysis result is determined to be correct, and the identification result is output.

3. The method according to claim 2, characterized in that, Using the element feature library of the trained element localization model, the feature information of the current interface element is compared and analyzed with the feature information of the baseline interface to obtain the analysis results, including: Using the element feature library of the trained element localization model, the feature information of the current interface element and the feature information of the reference interface element are analyzed to obtain the analysis results; the feature information includes at least one of the following: attribute information, layout information, visual features and semantic information; The attribute information includes at least one of the following: label, element ID, and class name; the layout information includes at least one of the following: relative position, hierarchy, and path; the visual features include at least one of the following: font, color, shape, size, and style; and the semantic information includes text content.

4. The method according to claim 1, characterized in that, The method further includes: The trained element localization model is embedded into the automated testing tool for dynamic identification and adjustment of element localization during test runtime.

5. The method according to claim 1, characterized in that, The test script corresponding to the update includes: Generate a test report; wherein the test report includes: the state of the interface elements before and after the change, the time of change, and the frequency of change; Using a script update tool, the change information in the test report is read and parsed, and the corresponding test script is retrieved for updating; or, the feature information of the current interface element is automatically replaced with the corresponding position in the test script.

6. The method according to claim 1, characterized in that, The method further includes: Generate a feedback dataset, which includes labeled error types and feature data of input interface elements; Based on the feedback dataset, the element localization model is retrained.

7. The method according to any one of claims 1 to 6, characterized in that, The training process of the element localization model includes: Acquire multi-interface image data from both desktop and mobile devices; Feature extraction is performed on the multi-interface image data to obtain feature information; wherein, the feature information includes attribute information, layout information, visual features and semantic information; The feature information is automatically labeled, and a formatted dataset required for training is generated; The formatted dataset is input into a convolutional neural network model to learn and record the feature information and change patterns of the elements, and to build an element feature library. Based on the element feature library, the convolutional neural network model is adjusted, and the trained convolutional neural network model is used as the element localization model.

8. A positioning device for interface elements, characterized in that, include: The first acquisition module is configured to acquire the image of the interface when it is first loaded and use it as a reference image, use the trained element localization model to extract and record the features of the interface elements of the reference image, and use the extracted interface elements as reference interface elements. The second acquisition module is configured to acquire an image during application operation or interface update and use it as the current image, use a trained element localization model to extract and record features of the interface elements in the current image, and use the extracted interface elements as the current interface elements. The output module is configured to output a recognition result if it is determined that the feature information of the current interface element has changed relative to the feature information of the reference interface element. This includes: using the element feature library of the trained element localization model, comparing and analyzing the feature information of the current interface element and the feature information of the reference interface to obtain an analysis result; wherein, the element feature library includes the feature information of the element and the change pattern of the element. The update module is configured to update the feature information of the baseline interface element to the feature information of the current interface element based on the recognition result, and update the corresponding test script to locate the interface element.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer program instructions that, when executed by a processor, implement the steps of the method as described in any one of claims 1 to 7.

10. An electronic device comprising a memory and a processor, the memory storing an executable program, the processor executing the executable program to perform the steps of the method as claimed in any one of claims 1 to 7.

Images