Interface element positioning method and device

Abstract

The invention discloses an interface element positioning method and device, and the method comprises the steps: obtaining an image when an interface is loaded for the first time, taking the image as a reference image, carrying out the feature extraction and recording of the reference image through a trained element positioning model, and taking the extracted interface element as a reference interface element; acquiring an image when the application runs or the interface is updated and taking the image as a current image, performing feature extraction and recording on the current image by utilizing the trained element positioning model, and taking the extracted interface element as a current interface element; if it is determined that the feature information of the current interface element changes relative to the feature information of the reference interface element, an identification result is output; and according to the identification result, updating the feature information of the reference interface element as the feature information of the current interface element, and updating the corresponding test script to position the interface element. By adopting the method and the device, element positioning can be dynamically identified and adjusted during test operation, and frequent maintenance requirements are reduced.

Description

Technical Field

[0001] The present application relates to the field of computer software technology, and in particular to a method and device for positioning interface elements. Background Art

[0002] Traditional UI (User Interface) automated testing methods mainly rely on static element locators, such as ID (Identifier), XPath (XML Path Language), etc. to locate UI elements. However, when the UI interface changes, these locators often become invalid, causing the test script to fail to execute normally, thus requiring frequent maintenance.

[0003] In view of this, how to provide a new method for positioning interface elements is an urgent problem to be solved by those skilled in the art. Summary of the invention

[0004] In view of the shortcomings of the existing methods, the present application proposes a method and device for positioning interface elements to solve the problem in the prior art that when the UI interface changes, the UI elements may also be adjusted, and the traditional static element locator may fail, resulting in the inability to execute the test script normally, thereby requiring frequent maintenance.

[0005] In a first aspect, an embodiment of the present application provides a method for positioning an interface element, comprising:

[0006] The image when the interface is first loaded is obtained and used as the reference image. The trained element positioning model is used to extract and record the features of the interface elements of the reference image, and the extracted interface elements are used as the reference interface elements.

[0007] Obtain an image when the application is running or the interface is updated and use it as the current image, use the trained element positioning model to extract and record the features of the interface elements of the current image, and use the extracted interface elements as the current interface elements;

[0008] If it is determined that the feature information of the current interface element has changed relative to the feature information of the reference interface element, outputting a recognition result;

[0009] According to the recognition result, the feature information of the reference interface element is updated to the feature information of the current interface element, and the corresponding test script is updated to locate the interface element.

[0010] In a possible implementation, if it is determined that the feature information of the current interface element has changed relative to the feature information of the reference interface element, then outputting a recognition result includes:

[0011] Using the element feature library of the trained element positioning model, the feature information of the current interface element and the feature information of the reference interface are compared and analyzed to obtain the analysis result; wherein the element feature library includes the feature information of the element and the change mode of the element;

[0012] The correctness of the analysis results is determined by outputting confidence scores through the trained element positioning model;

[0013] If the confidence score is greater than a preset threshold, the analysis result is determined to be correct and the recognition result is output.

[0014] In a possible implementation, the feature information of the current interface element and the feature information of the reference interface are compared and analyzed using the element feature library of the trained element positioning model to obtain analysis results, including:

[0015] Using the element feature library of the trained element positioning model, the feature information of the current interface element and the feature information of the reference interface element are differentially analyzed to obtain an analysis result; the feature information includes at least one of: attribute information, layout information, visual features and semantic information;

[0016] Among them, the attribute information includes at least one item: tag, element ID and class name; the layout information includes at least one item: relative position, hierarchical relationship and path; the visual feature includes at least one item: font, color, shape, size and style; the semantic information includes text content.

[0017] In a possible implementation, it also includes:

[0018] The trained element positioning model is embedded into the automated testing tool to dynamically identify and adjust element positioning during test runtime.

[0019] In a possible implementation, updating the corresponding test script includes:

[0020] Generate a test report; wherein the test report includes: the status of the characteristic information of the interface element before and after the change, the change time and the change frequency;

[0021] Use the script update tool to read and parse the change information in the test report, and call the corresponding test script for update; or automatically replace the feature information of the current interface element to the corresponding position in the test script.

[0022] In a possible implementation, the method further includes:

[0023] Generate a feedback data set, which includes annotated error types and input interface element feature data;

[0024] Retrain the element positioning model based on the feedback dataset.

[0025] In a possible implementation, the training process of the element positioning model includes:

[0026] Obtain multi-interface image data from PC and mobile terminals;

[0027] Extracting features from the multi-interface image data to obtain feature information; wherein the feature information includes attribute information, layout information, visual features and semantic information;

[0028] Automatically annotate feature information and generate formatted data sets required for training;

[0029] Input the formatted data set into the convolutional neural network model, learn and record the characteristic information of the elements and the change patterns of the elements, and establish an element feature library;

[0030] Based on the element feature library, the convolutional neural network model is adjusted, and the trained convolutional neural network model is used as the element positioning model.

[0031] In a second aspect, an embodiment of the present application provides a positioning device for an interface element, including:

[0032] A first acquisition module is configured to acquire an image when the interface is first loaded and use it as a reference image, extract and record the interface elements of the reference image using the trained element positioning model, and use the extracted interface elements as reference interface elements;

[0033] A second acquisition module is configured to acquire an image when the application is running or the interface is updated and use it as the current image, use the trained element positioning model to extract and record the features of the interface elements of the current image, and use the extracted interface elements as the current interface elements;

[0034] an output module configured to output a recognition result if it is determined that the feature information of the current interface element has changed relative to the feature information of the reference interface element;

[0035] The updating module is configured to update the characteristic information of the reference interface element to the characteristic information of the current interface element according to the recognition result, and to update the corresponding test script to locate the interface element.

[0036] In a third aspect, an embodiment of the present application provides a computer-readable storage medium, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the steps of any of the above methods are implemented.

[0037] In a fourth aspect, an embodiment of the present application provides an electronic device, including a memory and a processor, wherein an executable program is stored in the memory, and the processor executes the executable program to perform the steps of any of the above methods.

[0038] The method for positioning interface elements provided in the embodiment of the present application can dynamically identify and adjust the positioning of elements, by obtaining an image when the interface is first loaded and using it as a reference image, using a trained element positioning model to extract and record features of the reference image, and using the extracted interface elements as reference interface elements; obtaining an image when the application is running or the interface is updated and using it as the current image, using a trained element positioning model to extract and record features of the current image, and using the extracted interface elements as the current interface elements; if it is determined that the feature information of the current interface element has changed relative to the feature information of the reference interface element, then output the recognition result; based on the recognition result, update the feature information of the reference interface element to the feature information of the current interface element, and update the corresponding test script to locate the interface element. It can realize dynamic identification and adjustment of element positioning during test operation, ensure the continuity and stability of the test script, reduce the frequent maintenance requirements caused by interface adjustment, reduce manual modifications, significantly reduce maintenance costs, and improve test efficiency and system stability. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 A schematic diagram of a flow chart of a method for positioning an interface element provided in an embodiment of the present application;

[0040] Figure 2 Provided in the embodiments of this application Figure 1 A schematic diagram of the process flow of S300 in FIG.

[0041] Figure 3 Provided in the embodiments of this application Figure 1 A flowchart of the test script corresponding to the update of S400;

[0042] Figure 4 A schematic diagram of a process for retraining an element positioning model through a feedback mechanism provided in an embodiment of the present application;

[0043] Figure 5 A schematic diagram of the structure of a positioning device for an interface element provided in an embodiment of the present application. DETAILED DESCRIPTION

[0044] Various aspects and features of the present application are described herein with reference to the accompanying drawings.

[0045] It should be understood that various modifications may be made to the embodiments of the present application. Therefore, the above description should not be considered as limiting, but only as an example of an embodiment. Other modifications within the scope and spirit of the present application will occur to those skilled in the art.

[0046] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate embodiments of the present application and, together with the general description of the present application given above and the detailed description of the embodiments given below, serve to explain the principles of the present application.

[0047] These and other characteristics of the present application will become apparent from the following description of a preferred form of embodiment given as a non-limiting example with reference to the accompanying drawings.

[0048] It should also be understood that although the present application has been described with reference to some specific examples, those skilled in the art will be able to readily implement many other equivalent forms of the present application.

[0049] The above and other aspects, features and advantages of the present application will become more apparent in view of the following detailed description when taken in conjunction with the accompanying drawings.

[0050] Specific embodiments of the present application are described hereinafter with reference to the accompanying drawings; however, it should be understood that the embodiments applied for are merely examples of the present application, which may be implemented in a variety of ways. Well-known and / or repeated functions and structures are not described in detail to avoid unnecessary or redundant details that obscure the present application. Therefore, the specific structural and functional details applied for herein are not intended to be limiting, but merely serve as a basis and representative basis for the claims to teach those skilled in the art to use the present application in a variety of ways with substantially any suitable detailed structure.

[0051] This specification may use the phrases "in one embodiment," "in another embodiment," "in yet another embodiment," or "in other embodiments," all of which may refer to one or more of the same or different embodiments according to the present application.

[0052] The inventor of the present application has discovered through research that when the UI interface changes, page layout adjustments, label or attribute changes, etc. may occur. Since some traditional element locators such as ID and XPath implement UI element positioning based on fixed page structures or element attributes, once these features change, the test script will not be able to find the target element. When the element locator fails, the tester needs to modify the element locator, rewrite part of the test code, etc. to perform maintenance operations on the script. These operations are time-consuming and labor-intensive, and require the tester to have a certain understanding of UI structure and coding skills.

[0053] Therefore, the method for positioning interface elements provided in the present application is intended to solve the problem in the prior art that when the UI interface changes, the UI elements may also be adjusted, and the traditional static element locator may fail, resulting in the inability to execute the test script normally, thus requiring frequent maintenance.

[0054] A method for positioning an interface element provided in an embodiment of the present application is described in detail below with reference to the accompanying drawings. Figure 1 A schematic diagram of a flow chart of a method for positioning an interface element provided in an embodiment of the present application, Figure 1 The flowchart of a method for locating an interface element provided by an embodiment of the present application is shown. The method for locating an interface element is applied to an electronic device, which may include a personal computer, a mobile phone, or a notebook. Figure 1 A method for positioning an interface element in an embodiment of the present application includes the following steps:

[0055] S100: obtaining an image when the interface is first loaded and using it as a reference image, extracting and recording features of interface elements of the reference image using a trained element positioning model, and using the extracted interface elements as reference interface elements;

[0056] Exemplarily, the reference image is an image obtained when the interface of the electronic device is first loaded. This image is used as a reference image. If the interface of the electronic device changes later, the reference image is used as a reference image to compare element features with the image obtained in real time later, so as to realize dynamic recognition and adjustment of element positioning. The trained element positioning model is used to extract and record the features of each interface element of the reference image. The feature information of each interface element may include attributes, layout information, visual features, semantic information, etc. Each extracted interface element is used as a reference interface element.

[0057] S200: acquiring an image when the application is running or the interface is updated and using it as the current image, extracting and recording features of interface elements of the current image using the trained element positioning model, and using the extracted interface elements as the current interface elements;

[0058] Monitor the interface in real time and obtain images in real time when the application is running or the interface is updated. For example, when the application of the electronic device is running or the interface is updated, capture the interface image regularly or after a specific trigger event, and use the captured image as the current image. Use the trained element positioning model to extract and record the features of each interface element in the current image. The feature information of each interface element may include attributes, layout information, visual features, and semantic information. Use each extracted interface element as the current interface element for subsequent dynamic identification and adjustment of element positioning to achieve positioning of interface elements.

[0059] S300: If it is determined that the feature information of the current interface element has changed relative to the feature information of the reference interface element, outputting a recognition result;

[0060] Exemplarily, the trained element positioning model performs real-time monitoring and change detection on the interface. When it is detected that the feature information of the current interface element has changed relative to the feature information of the baseline interface element, the changed feature information is identified. For example, the element's attributes, layout information, visual features, and semantic information can be identified separately, and the identification result can be output. The identification result may include feature change information, such as change information of the element's attributes, change information of the element's visual features, etc.

[0061] S400: According to the recognition result, the feature information of the reference interface element is updated to the feature information of the current interface element, and the corresponding test script is updated to locate the interface element.

[0062] Exemplarily, based on the output feature change information, the feature information of the baseline interface element is updated to the feature information of the current interface element. For example, if the visual features of the baseline interface element change, the visual features of the baseline interface element are updated to the visual features of the current interface element for subsequent testing, and the corresponding test script is synchronously updated to achieve real-time positioning of the interface elements.

[0063] The method for positioning interface elements provided in the embodiment of the present application utilizes a trained element positioning model to identify changes in the feature information of the current interface element relative to the feature information of the reference interface element, and then outputs a recognition result; based on the recognition result, the feature information of the reference interface element is updated to the feature information of the current interface element, and the corresponding test script is updated to position the interface element. The embodiment of the present application can realize dynamic identification and adjustment of element positioning during test execution, ensuring the continuity and stability of the test script, so that the test script can adapt to most changes in the interface, reducing the frequent maintenance requirements caused by interface adjustments, reducing manual modifications, significantly reducing maintenance costs, and improving test efficiency and system stability.

[0064] Moreover, the method for locating interface elements provided in the embodiment of the present application utilizes real-time recognition and positioning technology to ensure that the test script can continue to run when the UI changes, thereby improving the efficiency and stability of automated test operation.

[0065] In some embodiments, see Figure 2 As shown, if it is determined that the feature information of the current interface element has changed relative to the feature information of the reference interface element, the recognition result is output, including:

[0066] S301: using the element feature library of the trained element positioning model, compare and analyze the feature information of the current interface element with the feature information of the reference interface to obtain an analysis result; wherein the element feature library includes the feature information of the element and the change mode of the element;

[0067] Exemplarily, the element feature library of the trained element positioning model records the feature information of the element and its change pattern. By comparing and analyzing the feature information of the current interface element with the feature information of the reference interface, an analysis result can be obtained, which includes feature change information.

[0068] S302: Determine the correctness of the analysis result by outputting a confidence score through the trained element positioning model;

[0069] S303: If the confidence score is greater than a preset threshold, the analysis result is determined to be correct, and the recognition result is output.

[0070] Exemplarily, the element positioning model is built based on a real-time recognition algorithm, which uses GPU to accelerate the processing of visual data and is applied in real time in the test to dynamically identify UI elements. Further, the real-time recognition algorithm can be a target detection algorithm based on a convolutional neural network (CNN) (such as YOLO or SSD) for real-time element recognition. A lightweight network (such as MobileNet) is introduced to reduce computational overhead while maintaining high detection accuracy. The implementation method can be: input the current image acquired in real time, initialize the weights using a pre-trained model, perform transfer learning for UI features, and process page visual data in parallel through the GPU to detect UI elements in real time. Optimize post-detection processing steps (such as non-maximum suppression) to reduce delays.

[0071] The real-time recognition algorithm can strike a balance between real-time and accuracy by using lightweight models to reduce the number of parameters and combining GPU acceleration to achieve fast reasoning. Optimizing the input of the neural convolutional network model, such as adjusting the page screenshot to a fixed resolution, can reduce the amount of calculation and achieve real-time element recognition. Through multi-dimensional feature fusion, such as vision and position, the recognition accuracy can be improved. Dynamically adjusting the preset threshold of confidence to ensure reliable detection in different UI environments can improve the accuracy of interface element recognition.

[0072] In some embodiments, the feature information of the current interface element and the feature information of the reference interface are compared and analyzed using the element feature library of the trained element positioning model to obtain analysis results, including:

[0073] Using the element feature library of the trained element positioning model, the feature information of the current interface element and the feature information of the reference interface element are differentially analyzed to obtain an analysis result; the feature information includes at least one of: attribute information, layout information, visual features and semantic information;

[0074] Among them, the attribute information includes at least one item: tag, element ID and class name; the layout information includes at least one item: relative position, hierarchical relationship and path; the visual feature includes at least one item: font, color, shape, size and style; the semantic information includes text content.

[0075] For example, difference analysis can include visual difference analysis, structural difference analysis, attribute difference analysis, etc. Visual difference analysis can compare the visual features of interface elements, such as color, shape, size, etc., to detect changes in visual appearance. Structural difference analysis can check changes in nodes in the DOM structure, including additions, deletions, and modifications. Attribute difference analysis can identify changes through changes in tags, IDs, class names, and other attributes.

[0076] The following is an explanation using visual difference analysis as an example.

[0077] Verify the status and position changes of interface elements through screenshot capture and visual comparison to ensure that changes in interface elements are identified and adjusted in a timely manner.

[0078] Acquisition of the current image: You can use browser automation tools such as Selenium to capture page screenshots in real time and use them as the current image, and save the acquired current image in a standard image format.

[0079] Sample code:

[0080] driver.save_screenshot("current_page.png")

[0081] About visual comparison: Use OpenCV to process screenshots and compare the visual features of the current image with the reference image. This includes: element detection and difference analysis; Element detection: Extract shape, color, position and other features of each UI element. Difference analysis: Detect whether the element has moved or changed through image registration and template matching.

[0082] Sample code:

[0083] import cv2

[0084] #Read the current screenshot and reference screenshot

[0085] current=cv2.imread("current_page.png")

[0086] reference=cv2.imread("reference_page.png")

[0087] #Convert to grayscale image

[0088] current_gray=cv2.cvtColor(current,cv2.COLOR_BGR2GRAY)

[0089] reference_gray=cv2.cvtColor(reference,cv2.COLOR_BGR2GRAY)

[0090] # Calculate the difference

[0091] diff=cv2.absdiff(current_gray,reference_gray)

[0092] The embodiment of the present application uses computer vision technology (such as OpenCV) to compare and analyze the current image to check the state and position changes of UI elements. Through this visual comparison method, it can ensure that when the UI changes, the element positioning and test scripts are adjusted in time, increasing the robustness to dynamic changes of the UI.

[0093] The embodiments of the present application use multi-dimensional features and visual comparison technology, which can effectively improve the accuracy of UI element recognition and reduce test failures caused by misrecognition.

[0094] The correctness of the analysis result is determined by the confidence score output by the element positioning model. If the confidence score is greater than the preset threshold, the analysis result is determined to be correct and the recognition result is output. Based on the recognition result, the feature information of the reference interface element is updated to the feature information of the current interface element, and the corresponding test script is updated to locate the interface element.

[0095] Optionally, the preset threshold of the confidence level can be adjusted dynamically to reduce the possibility of misidentification. Ensuring reliable detection in different UI environments can improve the accuracy of interface element recognition.

[0096] Optionally, if the confidence score is greater than a preset threshold, the analysis result is determined to be correct, and a recognition result is output, including:

[0097] If the confidence score is greater than a preset threshold, the analysis result is determined to be correct;

[0098] Combine the preset logic or business rules to reconfirm the correctness of the analysis results;

[0099] If the analysis result is confirmed to be correct again, the recognition result is output.

[0100] This embodiment can reduce false positives by reconfirming the analysis results.

[0101] The embodiments of the present application utilize AI technology, especially deep learning and computer vision, to develop a real-time recognition algorithm that can dynamically identify and locate UI elements in real time to adapt to frequently changing page layouts. The convolutional neural network model is trained to learn and identify the characteristics of interface elements, and to adapt to changes in the UI through continuous optimization and adjustment. In particular, the trained element positioning model not only relies on traditional DOM attributes, but also combines visual features (such as color, shape, layout, etc.) to improve the adaptability to dynamic interface elements.

[0102] In some embodiments, before acquiring the image of the initial loading interface and using it as the reference image, the method further includes:

[0103] The trained element positioning model is embedded into the automated testing tool to dynamically identify and adjust element positioning during test runtime.

[0104] For example, the trained element positioning model can be embedded into the test framework Selenium to dynamically identify and adjust element positioning when the test is running, ensuring the continuity and stability of the test script and reducing manual modifications.

[0105] Selenium is a commonly used automated testing framework that mainly interacts with the browser through locators (such as ID, XPath, and CSS selectors) to complete UI testing. Its working method includes: pre-defining locators in the script and executing tests according to the operation sequence. If the locator fails, the script needs to be adjusted manually. It cannot adapt to dynamic changes in UI elements, relies on static rules, and lacks intelligent adjustment capabilities.

[0106] Embedding the trained element positioning model into a test framework, such as Selenium, can dynamically identify and automatically adjust element positioning during test execution, thus ensuring the continuity and stability of the script.

[0107] Exemplarily, the element positioning model is embedded into the automated testing tool Selenium by extending the interface, as follows:

[0108] In the Selenium test script, add an interface to call the element positioning model. Capture the current image while the test is running and pass it as input to the element positioning model. The element positioning model outputs recognition results, such as the bounding box of a button. Convert the recognition results to locators supported by Selenium, such as the relative path of the interface element and the center point coordinates. Then, automatically update the locators in the test script to avoid manual modification.

[0109] After embedding the trained element positioning model into the automated testing tool, during the test run, the system dynamically adjusts the positioning strategy based on the feedback from the element positioning model. Even if the page elements change, the test script can still maintain continuity and stability. This solves the problem that the traditional testing framework cannot adapt to dynamic page changes.

[0110] In some embodiments, see Figure 3 , update the corresponding test scripts, including:

[0111] S401: Generate a test report; wherein the test report includes: the status of the characteristic information of the interface element before and after the change, the change time and the change frequency;

[0112] S402: Using a script update tool, read and parse the change information in the test report, and call the corresponding test script for update; or, automatically replace the feature information of the current interface element to the corresponding position in the test script.

[0113] For example, data visualization tools (such as Tableau, D3.js) can be used to generate intuitive and informative test reports, which can be sent to the management platform for the team's reference. The test report lists the detected element change information in detail, and through these element change information, the success rate of element positioning using the element positioning model can be gradually improved.

[0114] In addition, while the test is running, you can monitor the test status in real time. Specifically, you can use monitoring tools such as Prometheus to capture and alert abnormal events, and set up automated logs and analysis tools for test results and system performance, such as ELK Stack.

[0115] The automated test script is updated with the detected changed element information. The automated test script can be updated in two ways.

[0116] The first method is to use the script update tool: Use the automated update tool to read and automatically parse each change information in the test report, and call the corresponding test script for update.

[0117] The second way is to use locator automatic replacement: For example, regarding XPath update, use tools to reconstruct XPath with new locations, attributes, etc. For example, you can improve stability by generating relatively more robust XPaths (such as / / button[contains(text(),'Submit')] instead of absolute paths). Regarding ID and other attribute replacement: Automatically replace the new ID or other identification attributes to the corresponding location in the test script.

[0118] The embodiment of the present application uses an element positioning model to identify information about changes in interface elements and generate a test report, and uses an automated tool to obtain the changed element information in the test report and dynamically adjust the elements in the test script. The element positioning can be dynamically identified and adjusted during the test run to ensure the continuity and stability of the test script and reduce manual modifications.

[0119] In some embodiments, see Figure 4 , the method further comprises:

[0120] S501: Generate a feedback data set, where the feedback data set includes annotated error types and input interface element feature data;

[0121] S502: Retrain the element positioning model according to the feedback data set.

[0122] For example, if the recognition result output by the element positioning model is inconsistent with expectations, an error case is recorded, which includes: error type and input data, error type includes positioning deviation, classification error, etc., input data includes page screenshots and element features, etc. A feedback data set is generated, and the problem is marked so that the element positioning model can be adjusted and optimized.

[0123] The feedback dataset can be used to compare the error cases with the original training data to find out the feature information or abnormal patterns that are not covered by the element location model. A new round of data collection and annotation plan can be designed based on the error distribution.

[0124] In this embodiment, a feedback data set is provided for adjusting the element positioning model. After the test is run, the actual test results are compared with the expected results to analyze the correctness and stability of the element positioning model, which can help optimize the element positioning model or application. The element positioning module is retrained based on the feedback data set to achieve continuous learning and improvement of the element positioning model.

[0125] The embodiment of the present application uses a dynamic feedback mechanism to not only adjust the element location strategy in real time, but also automatically optimize the recognition accuracy of the model based on the test results. By generating feedback data sets, data enhancement and incremental learning technology, the system can continuously optimize the element location model to ensure the efficiency and stability of the model in practical applications.

[0126] In addition, through continuous training and optimization of the element positioning model, the system can better adapt to different UI change scenarios, improving the versatility and adaptability of the test framework.

[0127] In some embodiments, the training process of the element positioning model includes:

[0128] Obtain multi-interface image data from PC and mobile terminals;

[0129] Extracting features from the multi-interface image data to obtain feature information; wherein the feature information includes attribute information, layout information, visual features and semantic information;

[0130] Automatically annotate feature information and generate formatted data sets required for training;

[0131] Input the formatted data set into the convolutional neural network model, learn and record the characteristic information of the elements and the change patterns of the elements, and establish an element feature library;

[0132] Based on the element feature library, the convolutional neural network model is adjusted, and the trained convolutional neural network model is used as the element positioning model.

[0133] For example, the detailed description of data collection and model training is as follows:

[0134] Data collection: Collect a large amount of UI interface element data, including labels, positions, styles, paths and other information, and establish data collection tools to automatically capture page snapshots to obtain the current image in real time.

[0135] Feature extraction: Using extended feature extraction, in addition to extracting DOM attributes, it also extracts styles (such as color, font), layout information (such as relative position, hierarchical relationship), etc.

[0136] It supports snapshot collection of multiple devices and multiple resolutions, adapts to PC and mobile pages, and realizes cross-platform adaptation of data collection. Through multi-page batch snapshot and data collection functions, it can improve data preparation efficiency, automatically annotate UI elements, and generate formatted data sets required for training.

[0137] Use tools such as Selenium and Puppeteer to load the target page and take a snapshot to obtain the current image;

[0138] Example (Selenium):

[0139] from selenium import webdriver

[0140] driver = webdriver.Chrome()

[0141] driver.get("http: / / example.com")

[0142] driver.save_screenshot("screenshot.png")

[0143] driver.quit()

[0144] The embodiment of the present application uses a dedicated, cross-platform UI element data collection tool to expand the feature extraction capabilities of the traditional framework (such as labels, paths, styles), support multi-device, multi-resolution page snapshot collection, and automatically annotate UI elements to generate formatted data sets required for training. The development of this tool realizes the automation of large-scale data collection and processing, greatly improving the efficiency of generating training data.

[0145] Through DOM parsing, it extracts features such as tags, positions, styles, etc., and combines OpenCV to process the current image and extract visual features such as shape, color, etc.

[0146] Example (DOM extraction):

[0147] element=driver.find_element_by_id("example")

[0148] element_id=element.get_attribute("id")

[0149] element_location=element.location

[0150] Example (visual feature extraction):

[0151] import cv2

[0152] img=cv2.imread('screenshot.png')

[0153] gray=cv2.cvtColor(img,cv2.COLOR_BGR2GRAY)

[0154] edges=cv2.Canny(gray,30,100)

[0155] Automatically annotate feature information and generate formatted data sets required for training;

[0156] Model training: Select a suitable model type, such as using a convolutional neural network to learn and record the characteristic information of elements and the change patterns of elements, and establish an element feature library; based on the element feature library, adjust the convolutional neural network model, and use the trained convolutional neural network model as the element positioning model.

[0157] In order to balance the accuracy and real-time performance of recognition, lightweight models (such as MobileNet) are preferred. If the amount of data is sufficient, a deep model is selected, otherwise a simplified structure is used to avoid overfitting.

[0158] Combine attribute information, layout information, visual features and semantic information to achieve multi-dimensional feature fusion. Train the convolutional neural network model through the annotated formatted data set to capture the UI change rules and learn the element change pattern. The trained convolutional neural network model is used as an element positioning model to realize dynamic recognition and adjustment of element positioning during testing, reducing manual modification.

[0159] The element positioning model structure constructed based on the real-time recognition algorithm provided in the embodiment of the present application can use GPU acceleration to achieve fast reasoning while ensuring high recognition accuracy. It can dynamically identify UI elements while ensuring real-time performance, thereby optimizing the efficiency of the test framework.

[0160] Based on the same inventive concept, the present application embodiment provides a positioning device for interface elements, see Figure 5 As shown, Figure 5 A schematic diagram of a device for positioning an interface element provided in an embodiment of the present application, the device comprising:

[0161] A first acquisition module 100 is configured to acquire an image when the interface is first loaded and use it as a reference image, extract and record the interface elements of the reference image using the trained element positioning model, and use the extracted interface elements as reference interface elements;

[0162] The second acquisition module 200 is configured to acquire an image when the application is running or the interface is updated and use it as the current image, use the trained element positioning model to extract and record the features of the interface elements of the current image, and use the extracted interface elements as the current interface elements;

[0163] an output module 300 configured to output a recognition result if it is determined that the feature information of the current interface element has changed relative to the feature information of the reference interface element;

[0164] The updating module 400 is configured to update the characteristic information of the reference interface element to the characteristic information of the current interface element according to the recognition result, and update the corresponding test script to locate the interface element.

[0165] The positioning device of the interface element provided in the embodiment of the present application utilizes the trained element positioning model to identify the changes in the characteristic information of the current interface element relative to the characteristic information of the reference interface element, and then outputs the recognition result; based on the recognition result, the characteristic information of the reference interface element is updated to the characteristic information of the current interface element, and the corresponding test script is updated to locate the interface element. The embodiment of the present application can realize dynamic identification and adjustment of element positioning during test operation, ensuring the continuity and stability of the test script, so that the test script can adapt to most changes in the interface, reducing the frequent maintenance requirements caused by interface adjustments, reducing manual modifications, significantly reducing maintenance costs, and improving test efficiency and system stability.

[0166] In some embodiments, the output module 300 is further configured to use the element feature library of the trained element positioning model to compare and analyze the feature information of the current interface element with the feature information of the reference interface to obtain an analysis result; wherein the element feature library includes the feature information of the element and the change pattern of the element;

[0167] The correctness of the analysis results is determined by outputting confidence scores through the trained element positioning model;

[0168] If the confidence score is greater than a preset threshold, the analysis result is determined to be correct and the recognition result is output.

[0169] In some embodiments, the output module 300 is specifically configured to use the element feature library of the trained element positioning model to perform a difference analysis on the feature information of the current interface element and the feature information of the reference interface element to obtain an analysis result; the feature information includes at least one of: attribute information, layout information, visual features and semantic information;

[0170] Among them, the attribute information includes at least one item: tag, element ID and class name; the layout information includes at least one item: relative position, hierarchical relationship and path; the visual feature includes at least one item: font, color, shape, size and style; the semantic information includes text content.

[0171] In some embodiments, the apparatus further comprises:

[0172] The embedding module is configured to embed the trained element positioning model into the automated testing tool so as to dynamically identify and adjust the element positioning during test runtime.

[0173] In some embodiments, the update module 400 is further configured to generate a test report; wherein the test report includes: the state of the characteristic information of the interface element before and after the change, the change time and the change frequency;

[0174] Use the script update tool to read and parse the change information in the test report, and call the corresponding test script for update; or automatically replace the feature information of the current interface element to the corresponding position in the test script.

[0175] In some embodiments, the apparatus further comprises:

[0176] A feedback module configured to generate a feedback data set, the feedback data set including annotated error types and input interface element feature data;

[0177] The first training module is configured to retrain the element positioning model according to the feedback data set.

[0178] In some embodiments, the device further includes a second training module, which is specifically configured as follows:

[0179] Obtain multi-interface image data from PC and mobile terminals;

[0180] Extracting features from the multi-interface image data to obtain feature information; wherein the feature information includes attribute information, layout information, visual features and semantic information;

[0181] Automatically annotate feature information and generate formatted data sets required for training;

[0182] Input the formatted data set into the convolutional neural network model, learn and record the characteristic information of the elements and the change patterns of the elements, and establish an element feature library;

[0183] Based on the element feature library, the convolutional neural network model is adjusted, and the trained convolutional neural network model is used as the element positioning model.

[0184] The embodiment of the present application further provides a computer-readable storage medium, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the steps of the method described above are implemented.

[0185] An embodiment of the present application further provides an electronic device, including a memory and a processor, wherein the memory stores an executable program, and the processor executes the executable program to perform the steps of the method described above.

[0186] The above embodiments are only exemplary embodiments of the present application and are not intended to limit the present application. The protection scope of the present application is defined by the claims. Those skilled in the art may make various modifications or equivalent substitutions to the present application within the essence and protection scope of the present application, and such modifications or equivalent substitutions shall also be deemed to fall within the protection scope of the present application.

Claims

1. A method for positioning an interface element, characterized in that: include: Acquire an image when the interface is first loaded and use it as a reference image, use the trained element positioning model to extract and record the features of the interface elements of the reference image, and use the extracted interface elements as reference interface elements; Acquire an image when the application is running or the interface is updated and use it as the current image, use the trained element positioning model to extract and record the features of the interface elements of the current image, and use the extracted interface elements as the current interface elements; If it is determined that the characteristic information of the current interface element has changed relative to the characteristic information of the reference interface element, outputting a recognition result; According to the recognition result, the feature information of the reference interface element is updated to the feature information of the current interface element, and the corresponding test script is updated to locate the interface element.

2. The method according to claim 1, characterized in that If it is determined that the feature information of the current interface element has changed relative to the feature information of the reference interface element, outputting a recognition result includes: Using the element feature library of the trained element positioning model, the feature information of the current interface element and the feature information of the reference interface are compared and analyzed to obtain an analysis result; wherein the element feature library includes the feature information of the element and the change pattern of the element; Determining the correctness of the analysis result by outputting a confidence score of the trained element positioning model; If the confidence score is greater than a preset threshold, the analysis result is determined to be correct and the recognition result is output.

3. The method according to claim 2, characterized in that Using the element feature library of the trained element positioning model, the feature information of the current interface element and the feature information of the reference interface are compared and analyzed to obtain analysis results, including: Using the element feature library of the trained element positioning model, the feature information of the current interface element and the feature information of the reference interface element are differentially analyzed to obtain an analysis result; the feature information includes at least one of: attribute information, layout information, visual features and semantic information; Among them, the attribute information includes at least one item: tag, element ID and class name; the layout information includes at least one item: relative position, hierarchical relationship and path; the visual feature includes at least one item: font, color, shape, size and style; the semantic information includes text content.

4. The method according to claim 1, characterized in that The method further comprises: The trained element positioning model is embedded into the automated testing tool to dynamically identify and adjust element positioning during test runtime.

5. The method according to claim 1, characterized in that The test script corresponding to the update includes: Generate a test report; wherein the test report includes: the state of the characteristic information of the interface element before and after the change, the change time and the change frequency; Utilize the script update tool to read and parse the change information in the test report, and call the corresponding test script for updating; or automatically replace the feature information of the current interface element to the corresponding position in the test script.

6. The method according to claim 1, characterized in that The method further comprises: Generate a feedback data set, wherein the feedback data set includes annotated error types and input interface element feature data; The element positioning model is retrained according to the feedback data set.

7. The method according to any one of claims 1 to 6, characterized in that: The training process of the element positioning model includes: Obtain multi-interface image data from PC and mobile terminals; Extracting features from the multi-interface image data to obtain feature information; wherein the feature information includes attribute information, layout information, visual features and semantic information; Automatically annotating the feature information and generating a formatted data set required for training; Inputting the formatted data set into a convolutional neural network model, learning and recording characteristic information of elements and change patterns of elements, and establishing an element characteristic library; Based on the element feature library, the convolutional neural network model is adjusted, and the trained convolutional neural network model is used as the element positioning model.

8. A positioning device for interface elements, characterized in that: include: A first acquisition module is configured to acquire an image when the interface is first loaded and use it as a reference image, extract and record the interface elements of the reference image using the trained element positioning model, and use the extracted interface elements as reference interface elements; A second acquisition module is configured to acquire an image when the application is running or the interface is updated and use it as the current image, use the trained element positioning model to extract and record the features of the interface elements of the current image, and use the extracted interface elements as the current interface elements; an output module configured to output a recognition result if it is determined that the feature information of the current interface element has changed relative to the feature information of the reference interface element; An updating module is configured to update the characteristic information of the reference interface element to the characteristic information of the current interface element according to the recognition result, and to update the corresponding test script to locate the interface element.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer program instructions, and when the computer program instructions are executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

10. An electronic device, comprising a memory and a processor, wherein an executable program is stored in the memory, and the processor executes the executable program to perform the steps of any one of the methods according to claims 1 to 7.

Images