A control positioning method, device and equipment in automation testing

Abstract

The application provides a control positioning method, device and equipment in automation testing, the method comprises the following steps: first layer positioning of a target control based on the resource identifier of the target control; when the first layer positioning is successful, determining the first position of the target control in the user interface; when the first layer positioning fails, second layer positioning of the target control based on the text content of the target control; when the second layer positioning is successful, determining the second position of the target control in the user interface; when the second layer positioning fails, third layer positioning of the target control based on a multi-scale image template; when the third layer positioning is successful, determining the third position of the target control in the user interface; when the third layer positioning fails, fourth layer positioning of the target control based on the screen scale coordinates; when the fourth layer positioning is successful, determining the fourth position of the target control in the user interface. Through the technical scheme of the application, accurate positioning of the control is realized.

Description

Technical Field

[0001] This application relates to the field of automated testing technology, and in particular to a control positioning method, apparatus and device in automated testing. Background Technology

[0002] Automated testing is a software testing method that uses code to replace manual labor and automatically execute software testing tasks. Automated testing can offload repetitive, tedious, large-scale, and error-prone manual testing work (such as repeatedly clicking, entering data, and checking results) to machines, thereby improving testing efficiency and ensuring product quality.

[0003] Automated testing of the UI (User Interface) refers to simulating real user actions on the user interface, such as clicking, swiping, and typing, by writing scripts or using automation tools to automatically verify whether the software functions correctly and whether the user interface displays properly. Automated UI testing involves the following processes: Locating: Finding the target control (such as a "Login" button) in the user interface. Acting: Performing an action on the target control (such as clicking or typing text). Asserting: Verifying whether the result of the action meets expectations (such as whether it successfully redirects to the homepage or whether an error message pops up).

[0004] Clearly, locating the target control within the user interface—that is, positioning the target control—is a crucial operation for performing automated UI testing. Failure to accurately locate the target control will prevent automated UI testing from being achieved. However, current technologies do not provide accurate positioning results for the target control. Summary of the Invention

[0005] This application provides a method for locating controls in automated testing, the method comprising: The target control is located at the first level based on its resource identifier; If the first-level positioning is successful, the first position of the target control in the user interface is determined, where the first position represents the position of the control that matches the resource identifier; if the first-level positioning fails, the target control is positioned in a second level based on its text content. If the second-level positioning is successful, the second position of the target control in the user interface is determined, where the second position represents the position of the control that matches the text content; if the second-level positioning fails, the target control is positioned in a third level based on a multi-scale image template. If the third-level positioning is successful, the target control is determined to have a third position in the user interface, where the third position represents the position of the control that matches the multi-scale image template; if the third-level positioning fails, the target control is positioned to have a fourth level based on screen ratio coordinates. If the fourth-level positioning is successful, the fourth position of the target control in the user interface is determined, where the fourth position represents the control position that matches the screen ratio coordinates.

[0006] This application provides a control positioning device for automated testing, the device comprising: The first positioning module is used to perform a first-level positioning of the target control based on the resource identifier of the target control; if the first-level positioning is successful, the first position of the target control in the user interface is determined, where the first position represents the position of the control that matches the resource identifier. The second positioning module is used to perform a second-level positioning of the target control based on the text content of the target control if the first-level positioning fails; and to determine the second position of the target control in the user interface if the second-level positioning is successful, wherein the second position represents the position of the control that matches the text content. The third positioning module is used to perform third-level positioning of the target control based on a multi-scale image template when the second-level positioning fails; and to determine the third position of the target control in the user interface when the third-level positioning is successful, wherein the third position represents the control position that matches the multi-scale image template. The fourth positioning module is used to perform fourth-level positioning of the target control based on screen ratio coordinates when the third-level positioning fails; and to determine the fourth position of the target control in the user interface when the fourth-level positioning is successful, wherein the fourth position represents the control position that matches the screen ratio coordinates.

[0007] This application provides an electronic device, including: a processor and a machine-readable storage medium, wherein the machine-readable storage medium stores machine-executable instructions that can be executed by the processor; the processor is used to execute the machine-executable instructions to implement the control positioning method in the above-mentioned automated testing.

[0008] This application provides a computer program product, which includes a computer program that, when executed by a processor, implements the control positioning method in the above-mentioned automated testing.

[0009] This application provides a machine-readable storage medium storing machine-executable instructions that can be executed by a processor; wherein the processor is used to execute the machine-executable instructions to implement the control positioning method in the automated testing example of this application.

[0010] As can be seen from the above technical solutions, in the embodiments of this application, during the automated testing of the UI, the target control is positioned at a first level based on its resource identifier; if the first level positioning is successful, the target control is positioned at a first position in the user interface, where the first position represents the position of the control that matches the resource identifier; if the first level positioning fails, the target control is positioned at a second level based on its text content; if the second level positioning is successful, the target control is positioned at a second position in the user interface, where the second position represents the position of the control that matches the text content; if the second level positioning fails, the target control is positioned at a third level based on a multi-scale image template; if the third level positioning is successful, the target control is positioned at a third position in the user interface, where the third position represents the position of the control that matches the multi-scale image template; if the third level positioning fails, the target control is positioned at a fourth level based on screen ratio coordinates; if the fourth level positioning is successful, the target control is positioned at a fourth position in the user interface, where the fourth position represents the position of the control that matches the screen ratio coordinates. In the automated testing of UI, the above-mentioned method proposes an adaptive control positioning method based on priority degradation. This method achieves accurate control positioning through hierarchical degradation, making it suitable for scenarios with frequent application iterations, diverse device resolutions, and high requirements for test script stability, especially for regression testing and functional verification of long-term operational apps such as online education, e-commerce, and instant messaging. In this method, multiple positioning strategies are ordered according to stability and accuracy, forming an ordered positioning sequence chain: resource identifier positioning - text content positioning - multi-scale image template positioning - screen ratio coordinate positioning. Controls must be positioned strictly according to this sequence chain to achieve accurate positioning. When a high-priority positioning method fails, it automatically tries the next lower priority method until the control is positioned, ensuring both fast and accurate positioning. Furthermore, for multi-scale image template positioning, adaptive scale scaling and confidence gradient adjustment are introduced, enabling the entire positioning process to maintain a high success rate under various app versions and device resolutions, adapting to devices with different resolutions. Attached Figure Description

[0011] Figure 1 This is a flowchart illustrating the control location method in the automated testing of this application; Figure 2 This is a schematic diagram showing the arrangement of the four positioning methods in this application; Figure 3 This is a flowchart illustrating the resource identification and positioning method in this application; Figure 4 This is a flowchart illustrating the text content location method in this application; Figure 5 This is a flowchart illustrating the multi-scale image template matching method in this application; Figure 6 This is a flowchart illustrating the screen ratio coordinate positioning method in this application; Figure 7 This is a schematic diagram of the control positioning device in the automated testing of this application; Figure 8 This is a hardware structure diagram of an electronic device according to one embodiment of this application. Detailed Implementation

[0012] This application proposes a control location method for automated testing. This method can be applied to electronic devices such as personal computers, laptops, smartphones, servers, IoT devices, cloud devices, edge devices, and network devices; the type of electronic device is not limited. See also... Figure 1 The diagram shown illustrates a flowchart of a control location method in automated testing. This method may include: Step 101: Perform the first-level positioning of the target control based on the resource identifier of the target control.

[0013] Step 102: If the first-level positioning is successful, determine the first position of the target control in the user interface. The first position can represent the position of the control that matches the resource identifier.

[0014] Step 103: If the first-level positioning fails (i.e., the position of the control cannot be determined based on the resource identifier), perform a second-level positioning of the target control based on the text content of the target control.

[0015] Step 104: If the second-level positioning is successful, determine the second position of the target control in the user interface. The second position can represent the position of the control that matches the text content.

[0016] Step 105: If the second-level positioning fails (i.e., the position of the control cannot be determined based on the text content), the target control is positioned in the third level based on the multi-scale image template.

[0017] Step 106: If the third-level positioning is successful, determine the third position of the target control in the user interface. The third position can represent the position of the control that matches the multi-scale image template.

[0018] Step 107: If the third-level positioning fails (i.e., the position of the control cannot be determined based on the multi-scale image template), perform fourth-level positioning of the target control based on the screen ratio coordinates.

[0019] Step 108: If the fourth-level positioning is successful, determine the fourth position of the target control in the user interface. The fourth position can represent the control position that matches the screen ratio coordinates.

[0020] In one example, the first-level location of a target control based on its resource identifier may include, but is not limited to: obtaining the resource identifier of the target control from an automated test script used for automated testing of the user interface; the resource identifier being a unique identifier for the target control; querying the interface control tree using the target control's resource identifier to perform the first-level location of the target control; wherein the interface control tree includes multiple nodes, each node corresponding to a control, and storing the control's resource identifier and position. If the target node in the interface control tree includes the target control's resource identifier, the position stored in the target node is the target control's first position in the user interface; if the node in the interface control tree does not include the target control's resource identifier, the first-level location fails.

[0021] In one example, second-level positioning of a target control based on its text content can include, but is not limited to: obtaining the text content of the target control from an automated test script; the text content includes text attributes or content descriptions; the text attributes are the text displayed to the user on the user interface, and the content descriptions are the descriptive text used for accessibility features of the target control; querying the interface control tree through the text content of the target control to perform second-level positioning of the target control; wherein the interface control tree includes multiple nodes, each node corresponding to a control, and the node stores the text content and position of the control. If the text content of the target node in the interface control tree matches the text content of the target control, the position stored in the target node is the second position; if the text content of any node in the interface control tree does not match the text content of the target control, the second-level positioning fails.

[0022] In one example, third-layer localization of a target control based on multi-scale image templates may include, but is not limited to: determining K confidence scores based on K candidate template images of the target control at K scales and the image to be recognized, where K can be a positive integer greater than 1, and the image to be recognized can be a screenshot of the user interface. The first candidate threshold is selected from the configured candidate threshold set as the current candidate threshold; the candidate thresholds in the set are sorted from largest to smallest. Then, each confidence score is traversed sequentially; if a confidence score greater than the current candidate threshold is encountered, the third position is determined based on the matching result corresponding to that confidence score; if none of the K confidence scores are greater than the current candidate threshold, it is determined whether the current candidate threshold is the last candidate threshold in the candidate threshold set; if yes, the third-layer localization fails; if no, the next candidate threshold after the current candidate threshold is used as the current candidate threshold, and the operation of traversing each confidence score sequentially is returned.

[0023] In one example, the process of acquiring K candidate template images of the target control at different scales may include, but is not limited to: acquiring a reference scaling factor, which is determined based on the pixel height of the current device and the pixel height of the template device; wherein, the current device is the device used to display the user interface, and the template device is the device used to acquire the initial template image of the target control; acquiring K candidate scaling factors, which include a first type of scaling factor, a second type of scaling factor, and a third type of scaling factor; the first type of scaling factor is less than the reference scaling factor, the second type of scaling factor is equal to the reference scaling factor, and the third type of scaling factor is greater than the reference scaling factor; for each candidate scaling factor, scaling the initial template image of the target control based on the candidate scaling factor to obtain the candidate template image corresponding to the candidate scaling factor.

[0024] In one example, the process of determining the confidence score based on the candidate template image and the image to be recognized for each candidate template image specifically includes: traversing a sliding window on the image to be recognized with a preset step size; determining the similarity corresponding to each sliding window position, where the similarity is the similarity between the candidate template image and the sub-image at that sliding window position in the image to be recognized; normalizing the matching metric matrix of the image to be recognized, where the matching metric matrix includes the similarity corresponding to each sliding window position in the image to be recognized; and determining the maximum similarity in the normalized matching metric matrix as the confidence score.

[0025] In one example, fourth-level positioning of a target control based on screen ratio coordinates may include, but is not limited to: obtaining the screen ratio coordinates of the target control on a base interface, where the screen ratio coordinates include horizontal and vertical ratio coordinates; wherein the horizontal ratio coordinate is the ratio of the horizontal coordinate of the target control on the base interface to the width of the base interface, and the vertical ratio coordinate is the ratio of the vertical coordinate of the target control on the base interface to the height of the base interface; obtaining the width and height of the user interface; determining the reference horizontal coordinate of the target control in the user interface based on the product of the horizontal ratio coordinate and the width of the user interface; determining the reference vertical coordinate of the target control in the user interface based on the product of the vertical ratio coordinate and the height of the user interface; and determining the fourth position of the target control in the user interface based on the reference horizontal and reference vertical coordinates, where the fourth position includes the reference horizontal and reference vertical coordinates.

[0026] As can be seen from the above technical solutions, in the embodiments of this application, during the automated testing of the UI, the target control is positioned at a first level based on its resource identifier; if the first level positioning is successful, the target control is positioned at a first position in the user interface, where the first position represents the position of the control that matches the resource identifier; if the first level positioning fails, the target control is positioned at a second level based on its text content; if the second level positioning is successful, the target control is positioned at a second position in the user interface, where the second position represents the position of the control that matches the text content; if the second level positioning fails, the target control is positioned at a third level based on a multi-scale image template; if the third level positioning is successful, the target control is positioned at a third position in the user interface, where the third position represents the position of the control that matches the multi-scale image template; if the third level positioning fails, the target control is positioned at a fourth level based on screen ratio coordinates; if the fourth level positioning is successful, the target control is positioned at a fourth position in the user interface, where the fourth position represents the position of the control that matches the screen ratio coordinates. In the automated testing of UI, the above-mentioned method proposes an adaptive control positioning method based on priority degradation. This method achieves accurate control positioning through hierarchical degradation, making it suitable for scenarios with frequent application iterations, diverse device resolutions, and high requirements for test script stability, especially for regression testing and functional verification of long-term operational apps such as online education, e-commerce, and instant messaging. In this method, multiple positioning strategies are ordered according to stability and accuracy, forming an ordered positioning sequence chain: resource identifier positioning - text content positioning - multi-scale image template positioning - screen ratio coordinate positioning. Controls must be positioned strictly according to this sequence chain to achieve accurate positioning. When a high-priority positioning method fails, it automatically tries the next lower priority method until the control is positioned, ensuring both fast and accurate positioning. Furthermore, for multi-scale image template positioning, adaptive scale scaling and confidence gradient adjustment are introduced, enabling the entire positioning process to maintain a high success rate under various app versions and device resolutions, adapting to different resolution devices.

[0027] The technical solutions described above in the embodiments of this application will be explained below in conjunction with specific application scenarios.

[0028] Automated testing of the user interface (UI) refers to simulating real user actions on the user interface, such as clicking, swiping, and entering text, by writing scripts or using automation tools to automatically verify whether the software functions correctly and whether the user interface displays properly. The automated testing of the UI involves the following processes: Location: Locating the target control (such as a "Login" button) in the user interface. Action: Performing an action on the target control (such as clicking or entering text). Assertion: Verifying whether the result after the action meets expectations (such as whether it successfully redirects to the homepage or whether an error message pops up). Clearly, locating the target control in the user interface is the key operation for performing automated UI testing.

[0029] In response to the above findings, this application proposes a control positioning method in automated testing (such as automated testing of user interfaces, such as automated testing of user interfaces for mobile devices). This method is an adaptive control positioning method based on priority degradation, which achieves accurate control positioning through hierarchical degradation.

[0030] In this embodiment, the controls, also referred to as elements, can be basic interactive controls used to receive direct user actions such as clicks, selections, and inputs, such as Button, CheckBox, RadioButton, Switch, Slider, and TextField / EditText. They can also be content display controls used to show users text, images, progress, and other information, such as TextView / Label, ImageView, ProgressBar, and Toast / Snackbar. Furthermore, they can be navigation and organization controls used for switching and navigating between different pages, views, or functional modules, such as TabBar / Tabs, Navigation Bar, Menu, List / ListView, and GridView. It can also be a container and layout control, acting as a container to hold and arrange other controls, determining the overall structure of the user interface, such as LinearLayout, FrameLayout, RelativeLayout, ScrollView, and CardView. It can also be a feedback and prompt control, used to display operation results, warnings, notifications, or confirmation information to the user, such as Alert / Dialog and Action Sheet.

[0031] This embodiment relates to the field of mobile UI automation testing technology, specifically to an automated testing scenario on Android devices, which implements robust control localization through a multi-strategy hierarchical degradation approach. It is suitable for scenarios with frequent application iterations, diverse device resolutions, and high requirements for test script stability, especially for regression testing and functional verification of long-term operational apps such as online education, e-commerce, and instant messaging.

[0032] This embodiment proposes an adaptive control localization method based on priority degradation. Multiple control localization strategies are sorted according to stability and accuracy, forming an ordered localization sequence chain: resource identifier localization - text content localization - multi-scale image template localization - screen ratio coordinate localization. When a high-priority localization strategy fails, a lower-priority strategy is automatically tried. Furthermore, for multi-scale image template localization, adaptive scale scaling and confidence gradient adjustment are introduced, ensuring a high success rate across various app versions and device resolutions.

[0033] In this embodiment, the problem of test link interruption caused by the failure of a single positioning strategy can be solved, the problem of image template matching being unable to adapt to devices with different resolutions can be solved, the problem of inconsistency between coordinates and pixels under high DPI displays can be solved, and the problem of high maintenance cost of test scripts across versions can be solved.

[0034] This application proposes a control location method in automated testing, which includes a four-layer priority degradation location strategy chain (location order chain). This chain comprises four location methods (four-layer location methods, also known as four location strategies), arranged from highest to lowest location stability (location accuracy). For example, these four location methods are resource identifier location method, text content location method, multi-scale image template location method, and screen ratio coordinate location method. Based on the analysis of these four location methods, this embodiment provides their order.

[0035] See Figure 2 The diagram shows the arrangement of the four positioning methods. The first-level positioning method is the resource-id positioning method. This method is characterized by its high stability and fast positioning speed; therefore, it is chosen as the first-level positioning method.

[0036] When the resource identifier positioning method fails to locate the target control, it is downgraded to the second-level positioning method, which is the text content positioning method (exact matching of text or content-desc). The text content positioning method can achieve accurate positioning of the control and is faster (better than the third-level positioning method). Therefore, the text content positioning method is used as the second-level positioning method.

[0037] When the text content positioning method fails to locate the target control, it is downgraded to the third-level positioning method, which is the multi-scale image template positioning method (i.e., the adaptive multi-scale image template matching method). The multi-scale image template positioning method can achieve accurate positioning of the control, and its accuracy is better than that of the fourth-level positioning method. Therefore, the multi-scale image template positioning method is used as the third-level positioning method.

[0038] When the multi-scale image template localization method fails to locate the target control, it degrades to a fourth-level localization method, which is the screen-scale coordinate localization method. While the screen-scale coordinate localization method can locate the control, its accuracy is relatively low, and it is used as a last resort.

[0039] By configuring resource identifier positioning method, text content positioning method, multi-scale image template positioning method and screen ratio coordinate positioning method, each positioning method has an independent timeout control when it is executed. The degradation process in the positioning strategy chain is completely transparent to the caller. The caller only needs to describe which control to locate (hereinafter referred to as the target control) and does not need to care about which specific method to use to locate the target control.

[0040] First, there's the resource-id-based positioning method. For example, a first-level positioning of the target control can be performed based on its resource-id. If the first-level positioning is successful, the target control's first position in the user interface is determined; this first position can represent the position of the control matching that resource-id. If the first-level positioning fails, it can be downgraded to a text content positioning method.

[0041] In one example, the resource identifier of the target control can be used as the first-level location method, as it is the fastest (approximately 8-15ms per query), with strong determinism and high accuracy. However, with app iterations, developers may rename the resource identifiers of the target controls for refactoring purposes. According to actual project statistics, approximately 15%-25% of resource identifiers change with each app version update, causing test scripts that rely on these resource identifiers to fail in batches. This requires manual troubleshooting and repair, resulting in extremely high maintenance costs. Therefore, using the resource identifier of the target control for the first-level location may fail.

[0042] See Figure 3 The diagram shown illustrates a resource identification and location method, which may include: Step 301: Obtain the resource identifier of the target control from the automated test script. The automated test script is used to perform automated testing of the user interface, and the resource identifier is the unique identifier of the target control.

[0043] In one example, during automated testing, an automated test script can be obtained, and the automated testing process can be implemented through the automated test script. There are no restrictions on the content of this automated test script.

[0044] For example, automated test scripts can include resource identifiers for target controls, and these resource identifiers are unique identifiers for the target controls. For instance, a resource identifier is a property assigned by the system to a target control in the user interface to uniquely identify that target control; that is, resource identifiers are unique.

[0045] Step 302: Query the interface control tree through the resource identifier of the target control to perform the first-level positioning of the target control. The first-level positioning is used to locate the first position of the target control in the user interface.

[0046] In one example, during the automated testing of the user interface, a UI control tree (or control tree for short) can be maintained in advance. The UI control tree can include multiple nodes. For each node in the UI control tree, the node corresponds to a control (UI control), and the node stores the control's resource identifier (such as resource-id) and key attributes such as the control's position (such as coordinates, bounding rectangle).

[0047] Taking the View object as an example, each node in the View Tree stores various types of information in memory. This information may include, but is not limited to: resource identifier, used to uniquely identify the control's ID; control type (class / type); content-desc, content description, often used for accessibility; position information, such as the control's bounding rectangle on the screen, the coordinates of the control's center point, and the control's actual width and height; state information, such as whether it is interactive, clickable, selected, or checked; content information, such as the text content displayed by the control and input prompt text; and hierarchical relationships, such as parent node references and child node lists.

[0048] In summary, for each node in the UI control tree, the node stores the resource identifier of the control and the position of the control. Based on this, the UI control tree can be queried using the resource identifier of the target control to determine whether a target node corresponding to the resource identifier of the target control exists in the UI control tree.

[0049] Step 303: Determine whether there is a target node in the interface control tree that corresponds to the resource identifier of the target control.

[0050] If so, that is, if the target node exists in the interface control tree, then step 304 can be executed; If not, i.e. the target node does not exist in the interface control tree, then step 305 can be executed.

[0051] For example, traverse each node of the interface control tree in sequence. If a node includes the resource identifier of the target control, then the node is taken as the target node, that is, the target node of the interface control tree includes the resource identifier of the target control, the traversal process ends, and step 304 is executed; if all nodes do not include the resource identifier of the target control, that is, all nodes of the interface control tree do not include the resource identifier of the target control, step 305 is executed.

[0052] Step 304: Determine the location where the target node is stored as the first position of the target control in the user interface.

[0053] For example, if the target node in the interface control tree includes the resource identifier of the target control, the first-level positioning is considered successful. If the first-level positioning is successful, the location where the target node is stored is determined as the first position of the target control in the user interface; the first position represents the position of the control that matches the resource identifier.

[0054] Step 305: Determine if the first-level positioning failed. For example, if none of the nodes in the UI control tree contain the resource identifier of the target control, then the first-level positioning failed. Alternatively, in the event of a first-level positioning failure, immediately downgrade to the second-level positioning. Or, if the query result for the resource identifier is empty, record a failure log and immediately proceed to the second-level positioning without waiting, thus avoiding the default timeout.

[0055] Second, regarding text content localization methods. For example, a second layer of localization can be performed on the target control based on its text content. If the second layer of localization is successful, the second position of the target control in the user interface can be determined, which can represent the position of the control that matches the text content. If the second layer of localization fails, it can be downgraded to multi-scale image template matching localization.

[0056] In one example, when resource identifier location fails (due to changes in resource identifiers caused by app version iterations), the system attempts to locate the target control through its text content. Compared to resource identifiers, text content changes less frequently during product iterations. For instance, a button's resource identifier might change in each version, but the button's text content (such as confirmation) typically remains unchanged.

[0057] See Figure 4 The diagram shown illustrates a method for locating text content. This method may include: Step 401: Obtain the text content of the target control from the automated test script. The automated test script is used to perform automated testing of the user interface. The text content includes text attributes or content descriptions.

[0058] In one example, during automated testing, an automated test script can be obtained, and the automated testing process can be implemented using the automated test script. The automated test script may include the text content of the target control, and based on this, the text content of the target control can be extracted from the automated test script.

[0059] In one example, the text content of a target control can be its text property, which is the text displayed to the user on the user interface. For instance, if a target control (such as a button) displays "Login," then the text property of that target control is "Login."

[0060] In one example, the text content of the target control can be a content description of the target control, which is a descriptive text for the target control's accessibility features, such as the descriptive text set for an image control.

[0061] Step 402: Query the interface control tree through the text content of the target control to perform a second-level positioning of the target control. The second-level positioning is used to locate the second position of the target control in the user interface.

[0062] In one example, during the automated testing of the user interface, a UI control tree (or control tree for short) can be maintained in advance. The UI control tree can include multiple nodes. For each node in the UI control tree, the node corresponds to a control (UI control), and the node stores the control's text content (such as text attributes or content description) and key attributes such as the control's position (such as coordinates or bounding rectangle).

[0063] Taking the View object as an example, each node in the View Tree stores various types of information in memory. This information may include, but is not limited to: resource identifier, used to uniquely identify the control's ID; control type (class / type); content-desc, content description, often used for accessibility; position information, such as the control's bounding rectangle on the screen, the coordinates of the control's center point, and the control's actual width and height; state information, such as whether it is interactive, clickable, selected, or checked; content information, such as the text content displayed by the control and input prompt text; and hierarchical relationships, such as parent node references and child node lists.

[0064] In summary, for each node in the UI control tree, the node stores the text content of the control and the position of the control. Based on this, the UI control tree can be queried by the text content of the target control to determine whether a target node corresponding to the text content of the target control exists in the UI control tree.

[0065] Step 403: Determine whether there is a target node in the interface control tree that corresponds to the text content of the target control.

[0066] If so, i.e., the target node exists in the interface control tree, then step 404 can be executed; If not, i.e. the target node does not exist in the interface control tree, then step 405 can be executed.

[0067] For example, traverse each node of the interface control tree in turn. If the text content of a certain node matches the text content of the target control, then the node is taken as the target node. That is, the text content of the target node of the interface control tree matches the text content of the target control, and the traversal process ends, and step 404 is executed; if the text content of all nodes does not match the text content of the target control, then step 405 is executed.

[0068] For example, a match between the text content of a node and the text content of a target control means that the text content of the node is the same as the text content of the target control, or that the similarity between the text content of the node and the text content of the target control is greater than a threshold. A mismatch between the text content of a node and the text content of a target control means that the text content of the node is different from the text content of the target control, or that the similarity between the text content of the node and the text content of the target control is not greater than a threshold.

[0069] Step 404: Determine the location where the target node is stored as the second position of the target control in the user interface.

[0070] For example, when the text content of a target node in the interface control tree matches the text content of a target control, the second-level positioning is considered successful. If the second-level positioning is successful, the target node is stored at its second position in the user interface; that is, the target node is stored at the second position. Here, the second position represents the position of the control that matches the text content of the target control.

[0071] Step 405: Determine if the second-level positioning failed. For example, if the text content of all nodes in the interface control tree does not match the text content of the target control, then the second-level positioning failed. For example, in the case of a second-level positioning failure, directly degrade to the third-level positioning. For example, if the query result for the text content is empty, record the failure log and immediately proceed to the third-level positioning to avoid the default timeout wait.

[0072] For example, in multilingual scenarios, a text mapping table can be maintained to record the mapping relationship between the current language and the text content, thereby reducing the maintenance cost of multilingual scripts.

[0073] Third, regarding multi-scale image template matching methods, for example, a third-level localization of the target control can be performed based on a multi-scale image template. If the third-level localization is successful, the third position of the target control in the user interface can be determined, representing the position of the control that matches the multi-scale image template. If the third-level localization fails, it can be downgraded to screen-scale coordinate localization.

[0074] In one example, image template matching locates controls by comparing the similarity between the current screenshot and a pre-stored screenshot of the control (i.e., the template image). Image template matching is intuitive and does not rely on any control properties. However, the template image is captured at a specific resolution. When switching to a device with a different resolution, the pixel size changes, and the matching similarity drops sharply. For example, if the resolution difference between the current device (i.e., the test device) and the template device (the device from which the template was captured) exceeds 20%, the matching success rate will drop below 60%, thus affecting the reliability of control location.

[0075] In response to the above findings, this embodiment employs an adaptive multi-scale image template matching method, see [link to relevant documentation]. Figure 5 The diagram shown illustrates a multi-scale image template matching method, which may include: Step 501: Obtain the initial template image of the target control.

[0076] In one example, a user interface can be displayed on a template device (template capture device), and a screenshot of the user interface can be taken to obtain a reference image on the template device. Then, a sub-image of the target control is captured from the reference image, and this sub-image serves as the initial template image for the target control. For instance, the position of the target control on the reference image can be obtained; the method of obtaining this position is not limited. Based on the position of the target control on the reference image, the initial template image of the target control is captured from the reference image.

[0077] Clearly, the initial template image of the target control was captured from a template device (such as a 1080p screen), but the current device used for the test run might be a 720p or 1440p screen, or even a macOS Retina screen (where logical resolution and physical resolution differ by a factor of two). When directly using the initial template image of the target control for matching, the inconsistent pixel dimensions result in low matching similarity.

[0078] Step 502: Obtain the reference scaling factor. The reference scaling factor can be determined based on the pixel height of the current device and the pixel height of the template device. The current device can be a device used to display the user interface, and the template device can be a device used to obtain the initial template image of the target control.

[0079] For example, the current device could be the device currently being tested, meaning the user interface is displayed on the current device. During automated testing, it is necessary to locate the target controls on the user interface.

[0080] The template device can be a template capture device, that is, to take a screenshot of the user interface displayed on the template device to obtain a reference image, and then capture the initial template image of the target control from the reference image.

[0081] For example, during the operation of the current device, a screenshot of the current device's user interface can be taken, and a standard reference control (such as a status bar control) with a known logical size can be located from the screenshot of the current device. That is, the actual pixel height of the standard reference control can be determined from the screenshot of the current device.

[0082] Furthermore, when displaying the user interface on a template device, a screenshot of the user interface can be taken to obtain a reference image on the template device. A standard reference control (such as a status bar control) with a known logical size is then located from the reference image; that is, the expected pixel height of the standard reference control is determined from the reference image.

[0083] Based on this, the ratio between the actual pixel height of the standard reference control and its expected pixel height is determined as the reference scaling factor for the current device. Clearly, this ratio is also the ratio between the pixel height of the current device and the pixel height of the template device. Thus, the reference scaling factor can be determined based on the pixel heights of the current device and the template device. The above is merely an example of determining the reference scaling factor; it only needs to be related to the pixel heights of the current device and the template device.

[0084] For example, the reference scaling factor can be denoted as `scale_factor`. If the template device is a standard Android device, and the current device is also a standard Android device, then `scale_factor` ≈ 1.0; if the current device is a macOS Retina display device, then `scale_factor` ≈ 2.0; if the current device is a Windows device with 150% DPI, then `scale_factor` ≈ 1.5. DPI (Dots Per Inch) is a metric describing screen pixel density. High-DPI screens have more pixels in the same physical area, and the operating system scales them to maintain consistent physical dimensions of UI elements. Retina displays are high-pixel-density screens used on devices, where the physical pixel resolution is twice the logical coordinate resolution (i.e., `scale_factor=2`).

[0085] Step 503: Obtain K candidate scaling factors, which include a first type of scaling factor, a second type of scaling factor, and a third type of scaling factor; the first type of scaling factor is less than the reference scaling factor, the second type of scaling factor is equal to the reference scaling factor, and the third type of scaling factor is greater than the reference scaling factor.

[0086] For example, the number of first-class scaling factors can be at least one, such as 1, 2, 3, etc. Taking two first-class scaling factors as an example, there is no limit to the number of first-class scaling factors, as long as the first-class scaling factor is less than the reference scaling factor. For example, the reference scaling factor is scale_factor, the first first-class scaling factor is scale_factor × 0.90, and the second first-class scaling factor is scale_factor × 0.95.

[0087] For example, the number of second-type scaling factors is one, and the second-type scaling factor is equal to the reference scaling factor. For example, the reference scaling factor is scale_factor, and the second-type scaling factor is scale_factor × 1.00.

[0088] For example, the number of third-type scaling factors can be at least one, such as 1, 2, 3, etc. Taking two third-type scaling factors as an example, there is no limit to the number of third-type scaling factors, as long as the third-type scaling factor is greater than the reference scaling factor. For example, the reference scaling factor is scale_factor, the first third-type scaling factor is scale_factor × 1.05, and the second third-type scaling factor is scale_factor × 1.10.

[0089] In summary, five candidate scaling factors were obtained: scale_factor × 0.90, scale_factor × 0.95, scale_factor × 1.00, scale_factor × 1.05, and scale_factor × 1.10.

[0090] Step 504: Obtain candidate template images of the target control at K scales.

[0091] In one example, for each candidate scaling factor, the initial template image of the target control is scaled based on that candidate scaling factor to obtain a candidate template image corresponding to that candidate scaling factor.

[0092] For a candidate scaling factor of scale_factor × 0.90, the initial template image of the target control is scaled based on the candidate scaling factor of scale_factor × 0.90 to obtain a candidate template image corresponding to the candidate scaling factor of scale_factor × 0.90. This candidate template image is denoted as candidate template image A.

[0093] For a candidate scaling factor of scale_factor × 0.95, the initial template image of the target control is scaled based on the candidate scaling factor of scale_factor × 0.95 to obtain a candidate template image corresponding to the candidate scaling factor of scale_factor × 0.95. This candidate template image is denoted as candidate template image B.

[0094] For a candidate scaling factor of scale_factor × 1.00, the initial template image of the target control is scaled based on the candidate scaling factor of scale_factor × 1.00 to obtain a candidate template image corresponding to the candidate scaling factor of scale_factor × 1.00. This candidate template image is denoted as candidate template image C.

[0095] For a candidate scaling factor of scale_factor ×1.05, the initial template image of the target control is scaled based on the candidate scaling factor of scale_factor ×1.05 to obtain a candidate template image corresponding to the candidate scaling factor of scale_factor ×1.05. This candidate template image is denoted as candidate template image D.

[0096] For a candidate scaling factor of scale_factor × 1.10, the initial template image of the target control is scaled based on the candidate scaling factor of scale_factor × 1.10 to obtain a candidate template image corresponding to the candidate scaling factor of scale_factor × 1.10. This candidate template image is denoted as candidate template image E.

[0097] For example, see Table 1, which shows examples of candidate template images at K scales.

[0098] Table 1 Step 505: Determine K confidence scores based on the K candidate template images at the target control and the image to be identified. The image to be identified can be a screenshot of the user interface.

[0099] For example, during automated testing, the current device can be the device currently running the test, displaying a user interface. A screenshot of this user interface can be taken, and this screenshot can then be used as the image to be recognized.

[0100] In one example, for each candidate template image of the target control, a sliding window is traversed on the image to be recognized according to a preset step size (which can be configured according to actual needs). For example, when moving horizontally, the center of the sliding window is moved 2 pixels to the right to obtain a new sliding window position, and when moving vertically, the center of the sliding window is moved 3 pixels downward to obtain a new sliding window position. This results in multiple sliding window positions on the image to be recognized. The size of the sub-image at each sliding window position is the same as the size of the candidate template image, or the size of the sub-image at each sliding window position is different from the size of the candidate template image.

[0101] Then, the similarity corresponding to each sliding window position is determined. This similarity is the similarity between the candidate template image and the sub-image of the image to be identified at that sliding window position, i.e., the similarity between the two images, which can be cosine similarity, distance similarity, etc. For example, the similarity between the candidate template image and the sub-image at sliding window position 1 is determined; the similarity between the candidate template image and the sub-image at sliding window position 2 is determined; and so on, the similarity between the candidate template image and the sub-image at each sliding window position can be determined, thus obtaining the matching metric matrix of the image to be identified. This matching metric matrix can include the similarity corresponding to each sliding window position of the image to be identified.

[0102] Then, the matching metric matrix of the image to be recognized is normalized, such as normalizing the similarity corresponding to each sliding window position in the matching metric matrix to the range of 0 to 1.

[0103] Based on this, the maximum similarity in the normalized matching metric matrix can be determined as the confidence score, that is, the confidence score corresponding to the candidate template image is obtained.

[0104] For example, a sliding window is traversed on the image to be recognized according to a preset step size to obtain multiple sliding window positions on the image. Based on candidate template image A, the similarity corresponding to each sliding window position is determined, and the matching metric matrix is ​​normalized. The maximum similarity in the normalized matching metric matrix is ​​determined as the confidence score 'a' corresponding to candidate template image A. Similarly, based on candidate template image B, the similarity corresponding to each sliding window position is determined, and the matching metric matrix is ​​normalized. The maximum similarity in the normalized matching metric matrix is ​​determined as the confidence score 'b' corresponding to candidate template image B. Finally, based on candidate template image C, the similarity corresponding to each sliding window position is determined, and the matching metric matrix is ​​normalized. The maximum similarity in the matching metric matrix is ​​determined as the confidence score c corresponding to the candidate template image C; based on the candidate template image D, the similarity corresponding to each sliding window position is determined, and the matching metric matrix is ​​normalized. The maximum similarity in the normalized matching metric matrix is ​​determined as the confidence score d corresponding to the candidate template image D; based on the candidate template image E, the similarity corresponding to each sliding window position is determined, and the matching metric matrix is ​​normalized. The maximum similarity in the normalized matching metric matrix is ​​determined as the confidence score e corresponding to the candidate template image E. That is, a total of 5 confidence scores can be obtained.

[0105] Step 506: Select the first candidate threshold from the configured candidate threshold set as the current candidate threshold. The candidate thresholds in the candidate threshold set can be sorted from largest to smallest.

[0106] For example, the candidate threshold set can include multiple candidate thresholds, all of which are greater than 0 and less than 1. For instance, the candidate threshold set may include candidate thresholds 0.80, 0.70, and 0.60, respectively. That is, the three candidate thresholds in the candidate threshold set can be sorted from largest to smallest.

[0107] Step 507: Iterate through each confidence score in order.

[0108] During the traversal of confidence scores, if there is a confidence score greater than the current candidate threshold, proceed to step 508; if there is no confidence score greater than the current candidate threshold, proceed to step 509.

[0109] For example, the candidate template images are arranged in the order of candidate template image A, candidate template image B, candidate template image C, candidate template image D and candidate template image E, corresponding to the candidate scaling factor in ascending order. The confidence scores are arranged in the order of confidence score a, confidence score b, confidence score c, confidence score d and confidence score e, corresponding to the candidate scaling factor in ascending order.

[0110] First, iterate through confidence score a. If confidence score a is greater than the current candidate threshold, the iteration process ends, and confidence scores b, c, d, and e are no longer iterated. Step 508 is executed based on confidence score a. If confidence score a is not greater than the current candidate threshold, iterate through confidence score b. If confidence score b is greater than the current candidate threshold, the iteration process ends, and confidence scores c, d, and e are no longer iterated. Step 508 is executed based on confidence score b. If confidence score b is not greater than the current candidate threshold, iterate through confidence score c. If confidence score c is greater than the current candidate threshold, the iteration process ends, and confidence scores d and e are no longer iterated. Step 508 is executed based on confidence score c. If confidence score c is not greater than the current candidate threshold, iterate through confidence score d. If the confidence score d is greater than the current candidate threshold, the traversal process ends, and the confidence score e is no longer traversed. Step 508 is executed based on the confidence score d. If the confidence score d is not greater than the current candidate threshold, the traversal of the confidence score e continues. If the confidence score e is greater than the current candidate threshold, the traversal process ends, and step 508 is executed based on the confidence score e. If the confidence score e is not greater than the current candidate threshold, i.e., there is no confidence score greater than the current candidate threshold, step 509 is executed.

[0111] Step 508: If a confidence score greater than the current candidate threshold is encountered, the third position is determined based on the matching result corresponding to the confidence score (such as the position of the sliding window in the image to be identified).

[0112] For example, the confidence score (e.g., confidence score a) represents the similarity (i.e., maximum similarity) between a candidate template image (e.g., candidate template image A) and a certain sliding window position of the object to be identified. That is, the confidence score corresponds to a certain sliding window position of the object to be identified, and this sliding window position is the matching result corresponding to the confidence score. This sliding window position (i.e., the sub-image / sub-region corresponding to this sliding window position in the object to be identified) is the third position of the target control in the user interface. This third position represents the control position matched with the multi-scale image template. Thus, the multi-scale image template matching process is completed, the third position of the target control in the user interface is obtained, and the accurate positioning of the target control is achieved.

[0113] Step 509: If none of the K confidence scores are greater than the current candidate threshold (i.e., all confidence scores have been traversed), then determine whether the current candidate threshold is the last candidate threshold in the candidate threshold set. If yes, that is, the current candidate threshold is the last candidate threshold in the candidate threshold set, then proceed to step 510; if no, that is, the current candidate threshold is not the last candidate threshold in the candidate threshold set, then proceed to step 511.

[0114] Step 510: Determine if the third-level positioning has failed. For example, if the current candidate threshold is the last candidate threshold in the candidate threshold set, it means that the K confidence scores are not greater than all candidate thresholds in the candidate threshold set; therefore, the third-level positioning has failed. For example, in the case of a third-level positioning failure, directly downgrade to the fourth-level positioning, that is, record the failure log and immediately enter the fourth-level positioning to avoid timeout waiting.

[0115] Step 511: If the current candidate threshold is not the last candidate threshold in the candidate threshold set, then iterate through the next candidate threshold after the current candidate threshold as the current candidate threshold, and return to execute step 507. Iterate through each confidence score in order, and repeat the above steps until a confidence score greater than the current candidate threshold is found. Determine the third position based on the matching result corresponding to the confidence score, or until the current candidate threshold is not the last candidate threshold in the candidate threshold set.

[0116] In summary, we can see that we can perform similarity comparisons on each of the five candidate template images to obtain five confidence scores for each candidate image. Furthermore, we employ a confidence gradient strategy (i.e., a gradient confidence threshold matching strategy): Round 1: The candidate threshold is set to 0.80. If the confidence score of any candidate template image reaches this threshold, the match is successful, and the target control's third position in the user interface is determined based on the matching result. If the confidence scores of all candidate template images do not reach this threshold, the match fails, and a second round is performed. Round 2: The candidate threshold is set to 0.70. If the confidence score of any candidate template image reaches this threshold, the match is successful, and the target control's third position in the user interface is determined based on the matching result. If the confidence scores of all candidate template images do not reach this threshold, the match fails, and a third round is performed. The third round: The candidate threshold is set to 0.60. If the confidence score of any candidate template image reaches this candidate threshold, the match is successful, and the third position of the target control in the user interface is determined based on the matching result. If the confidence score of all candidate template images does not reach this candidate threshold, the match fails, that is, the third-level positioning fails, and it is directly downgraded to the fourth-level positioning.

[0117] The reason for using a gradient threshold instead of a fixed threshold in the above approach is that the visual similarity of the target control varies slightly in different states (such as being pressed, disabled, or obscured by a semi-transparent overlay). A fixed threshold would lead to missed detections, but a threshold that is too low would result in false matches. The gradient strategy strikes a balance between the two.

[0118] In the above method, the third position can be determined based on the matching result corresponding to the confidence score (such as the position of the sliding window in the image to be identified). For example, the sliding window position can be determined as the third position of the target control in the user interface. Alternatively, considering that the matching result is the position in the screenshot coordinate system (pixels), it can be divided by the scale_factor to convert it back to the device logical coordinates, thereby obtaining the third position of the target control in the user interface, and thus obtaining the accurate control position of the target control in the user interface.

[0119] Fourth, regarding screen ratio coordinate positioning methods. For example, the target control can be positioned at a fourth level based on screen ratio coordinates. If the fourth-level positioning is successful, the fourth position of the target control in the user interface can be determined, which can represent the control position that matches the screen ratio coordinates.

[0120] In one example, positioning a control based on screen ratio coordinates involves using coordinate information to locate the target control. This maps the relative position of the target control within a baseline interface to the absolute coordinates of the current target screen, thus achieving adaptation to devices with different resolutions. For instance, by recording the relative position of the target control within a baseline interface (such as a design draft or the device screen during initial recording), i.e., its percentage coordinates relative to the screen's width and height, the actual absolute coordinates of the target control on different resolution devices are calculated using the same percentage, thereby achieving the positioning of the target control.

[0121] When all three positioning methods (first, second, and third) fail, pre-configured screen-scale coordinates are used to position the target control. The coordinates are stored as percentages of the screen's width and height (e.g., x=0.5, y=0.8 represents 50% of the screen width and 80% of the screen height), rather than absolute pixel values. This ensures that the coordinates scale proportionally with different screen resolutions, providing better adaptability than absolute coordinates.

[0122] For example, the significance of the fourth layer of positioning is to allow the test script to continue running in the face of extreme situations, rather than exiting abnormally, thereby ensuring the integrity of the entire testing process.

[0123] See Figure 6 The diagram shown is a flowchart of a screen ratio coordinate positioning method, which may include: Step 601: Obtain the screen ratio coordinates of the target control on the baseline interface.

[0124] For example, the screen ratio coordinates of the target control on the baseline interface can be obtained and stored in advance. During automated testing, if the screen ratio coordinate positioning method is required, since the screen ratio coordinates of the target control on the baseline interface have already been stored, the screen ratio coordinates of the target control on the baseline interface can be read directly.

[0125] In one example, the screen scale coordinates can include horizontal and vertical scale coordinates. The horizontal scale coordinate can be the ratio of the target control's x-coordinate to the width of the base screen, and the vertical scale coordinate can be the ratio of the target control's y-coordinate to the height of the base screen. When storing the horizontal and vertical scale coordinates, they can be stored as absolute coordinates or as percentages of the screen width and height, such as x=0.5, y=0.8 representing 50% of the screen width and 80% of the screen height.

[0126] To obtain the screen ratio coordinates of the target control on the reference screen, you can use the following method: Obtain the size of the reference screen and the relative coordinates of the target control. For example, determine the relative coordinates of the target control on the reference screen. The relative coordinates of the target control are usually represented by the coordinates of its center point or top-left corner. For example, let the width of the reference screen be denoted as W_ref, the height of the reference screen as H_ref, and the coordinates of the target control on the reference screen as (X_ref, Y_ref).

[0127] Next, calculate the proportional coordinates of the target control (normalization). For example, convert the absolute coordinates of the target control to proportional coordinates relative to the screen size, i.e., normalized coordinates, ranging from [0, 1]. For example, the proportional horizontal coordinate P_x = X_ref / W_ref, and the proportional vertical coordinate P_y = Y_ref / H_ref. These proportional coordinates are the screen proportional coordinates of the target control, independent of the specific device resolution, and can be reused across devices.

[0128] Then, the scale coordinates can be stored. For example, the calculated scale coordinates (P_x, P_y) can be associated with control identifiers (such as control name and function description) and stored in test cases or configuration files. This step ensures that when performing tests on different devices later, there is no need to re-record or manually adjust the coordinates.

[0129] In summary, the screen ratio coordinates (P_x, P_y) of the target control on the baseline interface can be obtained and pre-stored. Based on this, during automated testing, if the screen ratio coordinate positioning method is used, the screen ratio coordinates (P_x, P_y) of the target control can be directly read.

[0130] Step 602: Obtain the width and height of the user interface.

[0131] For example, during automated testing, the current device can display a user interface, and we can obtain the width and height of the user interface, i.e., obtain the screen size of the current device. For instance, we can denote the width of the user interface (i.e., the actual screen width of the current device) as W_target, and the height of the user interface (i.e., the actual screen height of the current device) as H_target.

[0132] Step 603: Determine the reference horizontal coordinate of the target control in the user interface based on the product of the horizontal proportional coordinate in the screen proportional coordinates and the width of the user interface; determine the reference vertical coordinate of the target control in the user interface based on the product of the vertical proportional coordinate in the screen proportional coordinates and the height of the user interface.

[0133] For example, the reference horizontal coordinate of the target control in the user interface (i.e., the actual horizontal coordinate of the target control) is denoted as X_target, and X_target = P_x × W_target, where P_x represents the horizontal proportional coordinate in the screen scale coordinate system, and W_target represents the width of the user interface. The reference vertical coordinate of the target control in the user interface (i.e., the actual vertical coordinate of the target control) is denoted as Y_target, and Y_target = P_y × H_target, where P_y represents the vertical proportional coordinate in the screen scale coordinate system, and H_target represents the height of the user interface.

[0134] Step 604: Based on the reference horizontal and vertical coordinates, determine the fourth position of the target control in the user interface. The fourth position may include the reference horizontal and vertical coordinates, completing the fourth-level positioning. At this point, the accurate position of the target control in the user interface is obtained, achieving accurate positioning of the target control.

[0135] In summary, the input data for control localization methods in automated testing can include the target control's resource identifier, text content, K candidate template images at different scales, and screen ratio coordinates. In the first level of localization, the target control's resource identifier can be used to locate it. If localization is successful, the target control's position is returned, and the operation is executed. If localization fails, a log is logged, and the process proceeds to the second level of localization. In the second level of localization, the target control's text content can be used to locate it. If localization is successful, the target control's position is returned, and the operation is executed. If localization fails, a log is logged, and the process proceeds to the third level of localization. In the third level of localization, K candidate template images at different scales can be used to locate the target control. If localization is successful, the target control's position is returned, and the operation is executed. If localization fails, a log is logged, and the process proceeds to the fourth level of localization. In the fourth level of localization, screen ratio coordinates can be used to locate the target control. If localization is successful, the target control's position is returned, and the operation is executed.

[0136] As can be seen from the above technical solutions, this embodiment proposes an adaptive control positioning method based on priority degradation. It achieves accurate control positioning through a hierarchical degradation approach, suitable for scenarios with frequent application iterations, diverse device resolutions, and high requirements for test script stability, especially for regression testing and functional verification of long-term operational apps such as online education, e-commerce, and instant messaging. In this method, multiple positioning strategies are sorted according to stability and accuracy, forming an ordered positioning sequence chain: resource identifier positioning - text content positioning - multi-scale image template positioning - screen ratio coordinate positioning. Control positioning must be strictly performed according to this positioning sequence chain to achieve accurate control positioning. When a high-priority positioning method fails, it automatically tries the next lower priority positioning method until control positioning is completed, ensuring both fast and accurate positioning. For multi-scale image template positioning, adaptive scale scaling and confidence gradient adjustment are introduced, enabling the entire positioning process to maintain a high success rate under various app versions and device resolutions, adapting to devices with different resolutions.

[0137] This embodiment proposes an adaptive control location method based on priority degradation. After a major app version update, the location success rate and cross-resolution image matching success rate are significantly improved, the script repair workload is significantly reduced, and the test interruption rate due to location failure is significantly reduced. A four-layer priority degradation control location strategy chain is adopted: resource identifier → text content → adaptive image matching → screen ratio coordinates, forming a complete and ordered strategy degradation chain. Each layer automatically triggers the next layer upon failure, and the entire process is transparent to the outside world. An automatic DPI scaling factor detection mechanism is employed. By capturing a reference control of known size at runtime, the scaling factor is dynamically calculated based on the ratio of actual pixel size to logical coordinates, eliminating the need for manual configuration and adapting to DPI differences across multiple platforms. Multi-scale candidate templates and gradient confidence threshold image matching are used. Based on the detected scaling factor, multiple candidate scaled versions within a ±10% range are generated and matched sequentially with gradient thresholds of 0.80, 0.70, and 0.60, balancing accuracy and recall. The image matching coordinates are converted from pixel-based to logical coordinates. The matching result coordinates are divided by the scaling factor to restore the device's logical coordinates, ensuring the accuracy of click positions on high-DPI screens.

[0138] Based on the same concept as the above method, this application proposes a control positioning device for automated testing. See [link to relevant documentation]. Figure 7 The diagram shown is a structural schematic of the device, which includes: The first positioning module 71 is used to perform a first-level positioning of the target control based on the resource identifier of the target control; if the first-level positioning is successful, the first position of the target control in the user interface is determined, where the first position represents the position of the control that matches the resource identifier. The second positioning module 72 is used to perform a second-level positioning of the target control based on the text content of the target control when the first-level positioning fails; and to determine the second position of the target control in the user interface when the second-level positioning is successful, wherein the second position represents the position of the control that matches the text content. The third positioning module 73 is used to perform third-level positioning of the target control based on a multi-scale image template when the second-level positioning fails; and to determine the third position of the target control in the user interface when the third-level positioning is successful, wherein the third position represents the control position that matches the multi-scale image template. The fourth positioning module 74 is used to perform fourth-level positioning of the target control based on screen ratio coordinates when the third-level positioning fails; and to determine the fourth position of the target control in the user interface when the fourth-level positioning is successful, wherein the fourth position represents the control position that matches the screen ratio coordinates.

[0139] In one example, when the third positioning module 73 performs third-level positioning of the target control based on multi-scale image templates, it specifically performs the following steps: determining K confidence scores based on K candidate template images of the target control at K scales and the image to be identified, where K is greater than 1, and the image to be identified is a screenshot of the user interface; selecting the first candidate threshold from the configured candidate threshold set as the current candidate threshold; wherein, the multiple candidate thresholds in the candidate threshold set are sorted from largest to smallest; sequentially traversing each confidence score; if a confidence score greater than the current candidate threshold is encountered, determining the third position based on the matching result corresponding to that confidence score; if none of the K confidence scores are greater than the current candidate threshold, determining whether the current candidate threshold is the last candidate threshold in the candidate threshold set; if yes, determining that the third-level positioning has failed; if no, traversing the next candidate threshold after the current candidate threshold as the current candidate threshold, and returning to perform the operation of sequentially traversing each confidence score.

[0140] In one example, when the third positioning module 73 acquires K candidate template images of the target control at different scales, it specifically performs the following steps: acquiring a reference scaling factor, which is determined based on the pixel height of the current device and the pixel height of the template device; wherein, the current device is a device used to display the user interface, and the template device is a device used to acquire the initial template image of the target control; acquiring K candidate scaling factors, which include a first type of scaling factor, a second type of scaling factor, and a third type of scaling factor; the first type of scaling factor is less than the reference scaling factor, the second type of scaling factor is equal to the reference scaling factor, and the third type of scaling factor is greater than the reference scaling factor; for each candidate scaling factor, scaling the initial template image of the target control based on the candidate scaling factor to obtain a candidate template image corresponding to the candidate scaling factor.

[0141] In one example, for each candidate template image, the third localization module 73, when determining the confidence score based on the candidate template image and the image to be identified, specifically performs the following: traversing a sliding window on the image to be identified according to a preset step size; determining the similarity corresponding to each sliding window position, wherein the similarity is the similarity between the candidate template image and the sub-image at that sliding window position in the image to be identified; normalizing the matching metric matrix of the image to be identified, wherein the matching metric matrix includes the similarity corresponding to each sliding window position of the image to be identified; and determining the maximum similarity in the normalized matching metric matrix as the confidence score.

[0142] In one example, when the first positioning module 71 performs first-level positioning of the target control based on the resource identifier of the target control, it specifically performs the following: obtaining the resource identifier of the target control from an automated test script, the automated test script being used for automated testing of the user interface, the resource identifier being a unique identifier of the target control; querying the interface control tree through the resource identifier of the target control to perform first-level positioning of the target control; wherein, the interface control tree includes multiple nodes, for each node corresponding to a control, and the node storing the resource identifier of the control and the position of the control; wherein, if the target node of the interface control tree includes the resource identifier of the target control, then the position stored in the target node is the first position of the target control in the user interface; if the node of the interface control tree does not include the resource identifier of the target control, then the first-level positioning fails.

[0143] In one example, the second positioning module 72, when performing a second-level positioning of the target control based on the text content of the target control, specifically involves: obtaining the text content of the target control from an automated test script; the text content includes text attributes or content descriptions; the text attributes are the text displayed to the user on the user interface by the target control, and the content descriptions are the descriptive text of the target control for accessibility functions; querying the interface control tree through the text content of the target control to perform a second-level positioning of the target control; the interface control tree includes multiple nodes, each node corresponding to a control, and the node stores the text content of the control and the position of the control; wherein, if the text content of the target node in the interface control tree matches the text content of the target control, the position stored in the target node is the second position; if the text content of each node in the interface control tree does not match the text content of the target control, the second-level positioning fails.

[0144] In one example, when the fourth positioning module 74 performs fourth-level positioning of the target control based on screen ratio coordinates, it is specifically used to: obtain the screen ratio coordinates corresponding to the target control on the reference interface, the screen ratio coordinates including horizontal ratio coordinates and vertical ratio coordinates; wherein, the horizontal ratio coordinate is the ratio of the horizontal coordinate of the target control on the reference interface to the width of the reference interface, and the vertical ratio coordinate is the ratio of the vertical coordinate of the target control on the reference interface to the height of the reference interface; obtain the width and height of the user interface; determine the reference horizontal coordinate of the target control on the user interface based on the product of the horizontal ratio coordinate and the width of the user interface; determine the reference vertical coordinate of the target control on the user interface based on the product of the vertical ratio coordinate and the height of the user interface; and determine the fourth position of the target control on the user interface based on the reference horizontal coordinate and the reference vertical coordinate, the fourth position including the reference horizontal coordinate and the reference vertical coordinate.

[0145] Based on the same concept as the above method, this application proposes an electronic device, see [link to previous application]. Figure 8 As shown, the electronic device includes a processor 81 and a machine-readable storage medium 82, the machine-readable storage medium 82 storing machine-executable instructions that can be executed by the processor 81; the processor 81 is used to execute the machine-executable instructions to implement the control positioning method in the automated testing disclosed in the above example of this application.

[0146] Based on the same application concept as the above method, this application embodiment also provides a machine-readable storage medium storing a plurality of computer instructions, which, when executed by a processor, can implement the control positioning method in the automated testing disclosed in the above examples of this application.

[0147] The aforementioned machine-readable storage medium can be any electronic, magnetic, optical, or other physical storage device that can contain or store information, such as executable instructions, data, etc. For example, machine-readable storage media can be: RAM (Random Access Memory), volatile memory, non-volatile memory, flash memory, storage drives (such as hard disk drives), solid-state drives, any type of storage disk (such as optical discs, DVDs, etc.), or similar storage media, or combinations thereof.

[0148] Based on the same concept as the methods described above, this application also provides a computer program product, which includes a computer program. When executed by a processor, the computer program implements the control positioning method in automated testing disclosed in the examples above.

[0149] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, embodiments of this application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0150] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A control location method in automated testing, characterized in that, include: The target control is located at the first level based on its resource identifier; If the first-level positioning is successful, the first position of the target control in the user interface is determined, where the first position represents the position of the control that matches the resource identifier; if the first-level positioning fails, the target control is positioned in a second level based on its text content. If the second-level positioning is successful, the second position of the target control in the user interface is determined, where the second position represents the position of the control that matches the text content; if the second-level positioning fails, the target control is positioned in a third level based on a multi-scale image template. If the third-level positioning is successful, the target control is determined to have a third position in the user interface, where the third position represents the position of the control that matches the multi-scale image template; if the third-level positioning fails, the target control is positioned to have a fourth-level positioning based on screen ratio coordinates. If the fourth-level positioning is successful, the fourth position of the target control in the user interface is determined, where the fourth position represents the control position that matches the screen ratio coordinates.

2. The method according to claim 1, characterized in that, The first-level positioning of the target control based on the resource identifier of the target control includes: Obtain the resource identifier of the target control from the automated test script, which is used to perform automated testing of the user interface, and the resource identifier is a unique identifier of the target control; The target control is located in the interface control tree by querying the resource identifier of the target control; wherein, the interface control tree includes multiple nodes, and for each node, the node corresponds to a control, and the node stores the resource identifier of the control and the position of the control; If the target node of the interface control tree includes the resource identifier of the target control, then the location where the target node is stored is the first position of the target control in the user interface; if the node of the interface control tree does not include the resource identifier of the target control, then the first-level positioning fails.

3. The method according to claim 1, characterized in that, The second-level positioning of the target control based on its text content includes: The text content of the target control is obtained from the automated test script. The text content includes text attributes or content descriptions. The text attributes are the text displayed to the user on the user interface, and the content descriptions are the descriptive text of the target control for accessibility functions. The target control is located in a second layer by querying the interface control tree through the text content of the target control; wherein, the interface control tree includes multiple nodes, and for each node, the node corresponds to a control, and the node stores the text content of the control and the position of the control; If the text content of the target node in the interface control tree matches the text content of the target control, the target node is stored in the second position; if the text content of each node in the interface control tree does not match the text content of the target control, the second-level positioning fails.

4. The method according to claim 1, characterized in that, The third-level localization of the target control based on multi-scale image templates includes: K confidence scores are determined based on K candidate template images at K scales of the target control and the image to be identified, where K is greater than 1, and the image to be identified is a screenshot of the user interface. Select the first candidate threshold from the configured candidate threshold set as the current candidate threshold; wherein, the candidate thresholds in the candidate threshold set are sorted from largest to smallest; The system iterates through each confidence score sequentially. If a confidence score greater than the current candidate threshold is encountered, the third position is determined based on the matching result corresponding to that confidence score. If none of the K confidence scores are greater than the current candidate threshold, then determine whether the current candidate threshold is the last candidate threshold in the candidate threshold set; if so, then determine that the third-level localization has failed. If not, then iterate through the next candidate threshold after the current candidate threshold as the current candidate threshold, and return to perform the operation of iterating through each confidence score in order.

5. The method according to claim 4, characterized in that, The process of obtaining candidate template images of the target control at K scales includes: A reference scaling factor is obtained, which is determined based on the pixel height of the current device and the pixel height of the template device; wherein, the current device is the device used to display the user interface, and the template device is the device used to obtain the initial template image of the target control; Obtain K candidate scaling factors, which include a first type of scaling factor, a second type of scaling factor, and a third type of scaling factor; the first type of scaling factor is less than the reference scaling factor, the second type of scaling factor is equal to the reference scaling factor, and the third type of scaling factor is greater than the reference scaling factor. For each candidate scaling factor, the initial template image of the target control is scaled based on the candidate scaling factor to obtain a candidate template image corresponding to the candidate scaling factor.

6. The method according to claim 4, characterized in that, For each candidate template image, the process of determining a confidence score based on the candidate template image and the image to be identified specifically includes: A sliding window is traversed on the image to be identified according to a preset step size; Determine the similarity corresponding to each sliding window position, where the similarity is the similarity between the candidate template image and the sub-image at that sliding window position in the image to be identified; The matching metric matrix of the image to be identified is normalized, and the matching metric matrix includes the similarity corresponding to each sliding window position of the image to be identified. The maximum similarity in the normalized matching metric matrix is ​​determined as the confidence score.

7. The method according to claim 1, characterized in that, The fourth-level positioning of the target control based on screen ratio coordinates includes: Obtain the screen ratio coordinates of the target control on the reference interface, the screen ratio coordinates including horizontal ratio coordinates and vertical ratio coordinates; wherein, the horizontal ratio coordinate is the ratio of the horizontal coordinate of the target control on the reference interface to the width of the reference interface, and the vertical ratio coordinate is the ratio of the vertical coordinate of the target control on the reference interface to the height of the reference interface. Obtain the width and height of the user interface; Based on the product of the horizontal proportional coordinate and the width of the user interface, the reference horizontal coordinate of the target control in the user interface is determined; based on the product of the vertical proportional coordinate and the height of the user interface, the reference vertical coordinate of the target control in the user interface is determined. Based on the reference horizontal coordinate and the reference vertical coordinate, the fourth position of the target control in the user interface is determined, wherein the fourth position includes the reference horizontal coordinate and the reference vertical coordinate.

8. A control positioning device for automated testing, characterized in that, include: The first positioning module is used to perform a first-level positioning of the target control based on the resource identifier of the target control; If the first-level positioning is successful, the first position of the target control in the user interface is determined, where the first position represents the position of the control that matches the resource identifier; The second positioning module is used to perform a second-level positioning of the target control based on the text content of the target control if the first-level positioning fails; and to determine the second position of the target control in the user interface if the second-level positioning is successful, wherein the second position represents the position of the control that matches the text content. The third positioning module is used to perform third-level positioning of the target control based on a multi-scale image template when the second-level positioning fails; and to determine the third position of the target control in the user interface when the third-level positioning is successful, wherein the third position represents the control position that matches the multi-scale image template. The fourth positioning module is used to perform fourth-level positioning of the target control based on screen ratio coordinates when the third-level positioning fails; and to determine the fourth position of the target control in the user interface when the fourth-level positioning is successful, wherein the fourth position represents the control position that matches the screen ratio coordinates.

9. The apparatus according to claim 8, characterized in that, The third positioning module, when performing third-level positioning of the target control based on a multi-scale image template, is specifically used for: K confidence scores are determined based on K candidate template images at K scales of the target control and the image to be identified, where K is greater than 1, and the image to be identified is a screenshot of the user interface. Select the first candidate threshold from the configured candidate threshold set as the current candidate threshold; wherein, the candidate thresholds in the candidate threshold set are sorted from largest to smallest; The system iterates through each confidence score sequentially. If a confidence score greater than the current candidate threshold is encountered, the third position is determined based on the matching result corresponding to that confidence score. If none of the K confidence scores are greater than the current candidate threshold, then determine whether the current candidate threshold is the last candidate threshold in the candidate threshold set; if so, then determine that the third-level localization has failed. If not, then iterate through the next candidate threshold after the current candidate threshold as the current candidate threshold, and return to perform the operation of iterating through each confidence score in order.

10. The apparatus according to claim 9, characterized in that, When the third positioning module acquires the K scale candidate template images of the target control, it is specifically used for: A reference scaling factor is obtained, which is determined based on the pixel height of the current device and the pixel height of the template device; wherein, the current device is the device used to display the user interface, and the template device is the device used to obtain the initial template image of the target control; Obtain K candidate scaling factors, which include a first type of scaling factor, a second type of scaling factor, and a third type of scaling factor; the first type of scaling factor is less than the reference scaling factor, the second type of scaling factor is equal to the reference scaling factor, and the third type of scaling factor is greater than the reference scaling factor. For each candidate scaling factor, the initial template image of the target control is scaled based on the candidate scaling factor to obtain a candidate template image corresponding to the candidate scaling factor.

11. The apparatus according to claim 9, characterized in that, For each candidate template image, the third localization module, when determining the confidence score based on the candidate template image and the image to be identified, is specifically used for: A sliding window is traversed on the image to be identified according to a preset step size; Determine the similarity corresponding to each sliding window position, where the similarity is the similarity between the candidate template image and the sub-image at that sliding window position in the image to be identified; The matching metric matrix of the image to be identified is normalized, and the matching metric matrix includes the similarity corresponding to each sliding window position of the image to be identified. The maximum similarity in the normalized matching metric matrix is ​​determined as the confidence score.

12. An electronic device, characterized in that, include: A processor and a machine-readable storage medium, the machine-readable storage medium storing machine-executable instructions that can be executed by the processor; The processor is configured to execute machine-executable instructions to implement the method of any one of claims 1-7.

Images