Element positioning method and apparatus

Abstract

This application discloses an element localization method and apparatus. The method includes: acquiring target element features of the element to be localized; acquiring candidate element features of at least one candidate element; determining text similarity, resource identifier similarity, description similarity, element type similarity, and positional similarity between the element to be localized and the candidate elements based on the target element features and the candidate element features; determining the localization score corresponding to the candidate element based on the text similarity, resource identifier similarity, description similarity, element type similarity, and positional similarity; and filtering based on the localization score and a matching threshold to determine the target matching element from at least one candidate element. This method can improve the localization accuracy of UI elements, enhance the fault tolerance for attribute changes, and thus improve the success rate of automated testing.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a method and apparatus for element positioning. Background Technology

[0002] With the increasing complexity and update frequency of application interfaces, user interface (UI) automated testing is widely used in software regression testing, compatibility testing, and continuous integration testing. UI element location is a crucial step in the UI automated testing process; only by accurately locating target UI elements such as buttons, input boxes, icons, and text controls can the automated test script execute subsequent operations.

[0003] In related technologies, UI automation testing typically uses a single attribute to locate target UI elements. However, this single-attribute location method requires high stability of the dependent attribute. When developers modify an attribute, the UI element cannot be accurately located. In other words, any change in a single attribute in these technologies can lead to UI element location failure, lacking tolerance for attribute changes and thus affecting the success rate of automated testing. Summary of the Invention

[0004] In view of this, embodiments of this application provide an element positioning method and apparatus, which aim to improve the positioning accuracy of UI elements, enhance the fault tolerance for attribute changes, and thereby improve the success rate of automated testing.

[0005] In a first aspect, embodiments of this application provide an element positioning method, the method comprising: Obtain the target element features of the element to be located, wherein the target element features include text features, resource identifier features, description features, element type features, and location features; Obtain candidate element features of at least one candidate element, wherein the candidate element features include text features, resource identifier features, description features, element type features, and position features of the candidate element; Based on the target element features and the candidate element features, the text similarity, resource identifier similarity, description similarity, element type similarity, and position similarity between the element to be located and the candidate elements are determined respectively. The location score corresponding to the candidate element is determined based on the text similarity, resource identifier similarity, description similarity, element type similarity, and location similarity. Based on the location score and matching threshold, a target matching element is determined from at least one candidate element.

[0006] Secondly, embodiments of this application provide an element positioning device, the device comprising: The first acquisition module is used to acquire the target element features of the element to be located, wherein the target element features include text features, resource identifier features, description features, element type features and location features; The second acquisition module is used to acquire candidate element features of at least one candidate element, wherein the candidate element features include text features, resource identifier features, description features, element type features and position features of the candidate element; The similarity determination module is used to determine the text similarity, resource identifier similarity, description similarity, element type similarity, and position similarity between the element to be located and the candidate elements based on the target element features and the candidate element features, respectively. The location score determination module is used to determine the location score corresponding to the candidate element based on the text similarity, the resource identifier similarity, the description similarity, the element type similarity, and the location similarity. The matching module is used to filter based on the positioning score and the matching threshold to determine the target matching element from at least one candidate element.

[0007] Thirdly, embodiments of this application provide an element positioning device, the device comprising: Memory, used to store computer programs; A processor is configured to execute the computer program to cause the device to perform the element positioning method described in the first aspect above.

[0008] Fourthly, embodiments of this application provide a computer storage medium on which a computer program is stored. When the computer program is run, a device running the computer program implements the element positioning method described in the first aspect above.

[0009] Compared with the prior art, the embodiments of this application have the following beneficial effects: This application provides an element localization method, which includes: obtaining target element features of the element to be localized, the target element features including text features, resource identifier features, description features, element type features, and position features; obtaining candidate element features of at least one candidate element, the candidate element features including text features, resource identifier features, description features, element type features, and position features of the candidate element; determining the text similarity, resource identifier similarity, description similarity, element type similarity, and position similarity between the element to be localized and the candidate element based on the target element features and the candidate element features; determining the localization score corresponding to the candidate element based on the text similarity, resource identifier similarity, description similarity, element type similarity, and position similarity; and filtering based on the localization score and a matching threshold to determine the target matching element from at least one candidate element.

[0010] Therefore, this method no longer relies solely on a single attribute for precise UI element location, but instead performs a comprehensive matching of UI elements from multiple dimensions, including text, resource identifiers, descriptions, element types, and positions. In other words, even if a certain attribute of the target UI element changes, other unchanged or minimally changed features can still participate in the matching process, thus avoiding location failure directly caused by a single attribute change and improving the accuracy of UI element location. This enhances the tolerance of the UI element location process to changes in text, resource identifiers, layout, and page structure, thereby improving the success rate of automated testing. Attached Figure Description

[0011] To more clearly illustrate the technical solutions in this embodiment or the prior art, the drawings used in the description of the embodiment or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 A flowchart illustrating an element positioning method provided in an embodiment of this application; Figure 2 A schematic diagram illustrating the determination of text similarity as provided in an embodiment of this application; Figure 3 A schematic diagram illustrating the determination of resource identifier similarity as provided in an embodiment of this application; Figure 4 A schematic diagram illustrating the determination of descriptive similarity for embodiments of this application; Figure 5 A schematic diagram illustrating the determination of element type similarity as provided in an embodiment of this application; Figure 6A schematic diagram illustrating the determination of positional similarity provided in an embodiment of this application; Figure 7 A schematic diagram illustrating the determination of similarity scores provided in an embodiment of this application; Figure 8 A schematic diagram illustrating threshold filtering provided in an embodiment of this application; Figure 9 A schematic diagram illustrating the sorting of results provided in the embodiments of this application; Figure 10 A schematic diagram illustrating the knowledge confidence update method provided in an embodiment of this application; Figure 11 A schematic diagram of the architecture of the element positioning method provided in the embodiments of this application; Figure 12 This is a schematic diagram illustrating the element information structure definition provided in the embodiments of this application; Figure 13 A schematic diagram illustrating the knowledge base query method provided in an embodiment of this application; Figure 14 A schematic diagram illustrating the weighted similarity calculation method provided in the embodiments of this application; Figure 15 This is a schematic diagram of an element positioning device provided in an embodiment of this application. Detailed Implementation

[0013] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present application.

[0014] In related technologies, UI automation testing typically uses a single attribute to locate the target UI element. For example, it can locate the target UI element by its resource identifier (resource-id), by its displayed text, by screen coordinates, or by its page hierarchy path (XPath). Here, resource-id is the resource identifier configured by the developer for the UI element, text is the content displayed on the interface, coordinates represent the UI element's position on the screen, and XPath represents the UI element's hierarchical path within the page structure.

[0015] However, single-attribute positioning methods have high requirements for the stability of the attributes they rely on. When developers modify the resource-id of a UI element, positioning based on resource-id is prone to failure; when product copy is updated or language versions change, text-based positioning is prone to failure; when different devices have different screen sizes or the UI layout is adjusted, coordinate-based positioning is prone to click misalignment or failure to hit the target element; when the page structure or DOM hierarchy changes, XPath-based positioning is prone to failure. In other words, any change in any single attribute in these technologies can lead to UI element positioning failure, lacking the ability to tolerate attribute changes, thus affecting the success rate of automated testing.

[0016] To address these issues, this application provides an element location method. This method moves beyond relying solely on a single attribute for precise UI element location; instead, it comprehensively matches UI elements across multiple dimensions, including text, resource identifiers, descriptions, element types, and positions. In other words, even if a specific attribute of the target UI element changes, other unchanged or minimally changed features can still participate in the matching process, preventing location failure due to a single attribute change and improving the accuracy of UI element location. This enhances the UI element location process's tolerance to changes in text, resource identifiers, layout, and page structure, thereby increasing the success rate of automated testing.

[0017] The specific implementation of the element positioning method and apparatus in the embodiments of this application will be described in detail below with reference to the accompanying drawings and through examples.

[0018] See Figure 1 The figure is a flowchart of an element positioning method provided in an embodiment of this application, combined with... Figure 1 As shown, it can specifically include: S101: Obtain the target element features of the element to be located. The target element features include text features, resource identifier features, description features, element type features, and location features.

[0019] The element to be located is a UI element that needs to be located during automated testing. For example, the element to be located may include interface elements such as buttons, input boxes, drop-down lists, icons, and text controls. For example, on a login page, the element to be located may be a username input box, a password input box, or a login button; on an order page, the element to be located may be a submit order button, a payment button, or a cancel button. This application does not specifically limit the element to be located.

[0020] Target element features are used to characterize the attribute information of the element to be located in multiple dimensions. Target element features can include text features, resource identifier features, description features, element type features, and location features.

[0021] Among these, text features are the text content displayed on the interface of the element to be located. Examples include text such as "Login," "Login Now," "Confirm," and "Submit Order" displayed on buttons. Text features reflect the functional meaning of UI elements and therefore serve as an important basis for element location.

[0022] A resource identifier is the resource identifier corresponding to the element to be located, such as a resource-id. For example, the resource identifier could be "com.example:id / login_button", where "com.example" represents the application package name and "id / login_button" represents the resource identifier suffix. Resource identifiers can be configured by developers for UI elements to distinguish different interface elements.

[0023] Descriptive features are semantic descriptions of the element to be located. For example, descriptive features may include a "login button," a "username input box," a "password input box," and a "submit order button." Descriptive features are used to semantically represent the function of UI elements, thereby providing auxiliary matching criteria when the text content changes.

[0024] The element type characteristic (elementType) is the control type of the element to be located. For example, the element type characteristic may include button type (BUTTON), input box type (INPUT), drop-down box type (DROPDOWN), icon type (ICON), etc. This application does not specifically limit the element type characteristic.

[0025] Positional features are the coordinates of the element to be located within the interface. For example, positional features may include click coordinates (clickX, clickY) or center coordinates (centerX, centerY). Positional features are used to characterize the distribution of the element to be located within the current or historical interfaces.

[0026] As an example, on the current login page, an element to be located can be identified. The text feature of this element can be "Log in now", the resource identifier feature can be "com.newapp:id / login_button", the description feature can be "user login button", the element type feature can be BUTTON, and the position feature can be (520, 1240). By extracting the above features of this element to be located, its target element features can be obtained.

[0027] It should be noted that the element to be located can be a single UI element in the current interface, or any one of multiple UI elements in the current interface. For multiple UI elements in the current interface, each can be used as an element to be located, and the element location method provided in this application can be executed to determine the matching relationship between each element to be located and candidate elements in the element knowledge base. This application does not impose specific limitations in this regard.

[0028] By obtaining the target element features of the element to be located in the current interface, we can obtain the real-time features of the element to be located in multiple dimensions such as text, resource identifier, description, element type and position. This provides a data foundation for subsequent multi-dimensional similarity calculation with candidate elements in the element knowledge base, avoiding reliance on a single attribute to locate UI elements.

[0029] S102: Obtain candidate element features of at least one candidate element, including text features, resource identifier features, description features, element type features, and position features of the candidate element.

[0030] Candidate elements are historical element knowledge retrieved from the element knowledge base. The element knowledge base stores element knowledge generated during historical location processes; each element knowledge can correspond to a historical UI element or a historical UI element function. For example, the element knowledge base can store information such as the text features, resource identifier features, description features, element type features, location features, expected function, page type, application package name, and knowledge confidence level of historical UI elements.

[0031] In other words, candidate elements are not UI elements identified in real time on the current screen, but rather historical element knowledge from the element knowledge base that may correspond to the element to be located. By matching the element to be located on the current screen with candidate elements in the element knowledge base, it can be determined whether the element to be located corresponds to a certain historical element knowledge or target function.

[0032] In one possible implementation, candidate elements can be queried from the element knowledge base based on the page type (pageType). The page type indicates the type of page containing the element to be located, such as a login page, homepage, settings page, order page, payment page, etc. If element knowledge matching the current page type exists in the element knowledge base, then that element knowledge can be used as a candidate element.

[0033] In another possible implementation, if no element knowledge matching the current page type exists in the element knowledge base, candidate elements can be queried from the element knowledge base based on the application package name (packageName). The application package name is used to identify the application to be tested; if element knowledge matching the current application package name exists in the element knowledge base, then the element knowledge matching the current application package name can be used as a candidate element.

[0034] As an example, the current page type is a login page, and the current application package name is "com.newapp". We can first query the element knowledge base for historical elements related to the login page type. If multiple historical elements related to login pages are found, such as "username input box", "password input box", "login button", and "register button", these historical elements can be used as candidate elements. If no historical elements related to login pages are found, we can further query the historical elements under the current application package name "com.newapp" and use the found historical elements as candidate elements.

[0035] Candidate element features are used to characterize the historical attribute information of candidate elements across multiple dimensions. Candidate element features can include text features, resource identifier features, description features, element type features, and location features. For example, a candidate element recorded in the element knowledge base is a historical login button. Its text feature could be "login", its resource identifier feature could be "com.oldapp:id / login_button", its description feature could be "login button", its element type feature could be BUTTON, and its location feature could be (500, 1200).

[0036] In one possible implementation, the candidate element features may further include knowledge confidence. Knowledge confidence represents the degree of reliability of historical positioning experience. For example, knowledge confidence can be determined based on historical positioning success rates. If the element knowledge corresponding to a candidate element has been verified multiple times and has a high positioning success rate, then the candidate element may have a high knowledge confidence; if the element knowledge corresponding to a candidate element has been verified fewer times or has had a high number of historical positioning failures, then the candidate element may have a low knowledge confidence.

[0037] For example, the knowledge confidence level can be a value between 0 and 1. A knowledge confidence level greater than or equal to 0.8 indicates that the knowledge of the element has been verified multiple times and has a high degree of reliability; a knowledge confidence level greater than or equal to 0.5 and less than 0.8 indicates that the knowledge of the element has a certain verification basis and has a moderate degree of reliability; a knowledge confidence level less than 0.5 indicates that the knowledge of the element has insufficient verification or poor historical performance and has a low degree of reliability. The above values ​​are only examples, and this application does not specifically limit the range and classification method of the knowledge confidence level.

[0038] It should be noted that the candidate elements in this application can be historical element knowledge matching the current page type in the element knowledge base, or historical element knowledge matching the current application package name in the element knowledge base, or historical element knowledge obtained by comprehensively querying based on the current page type, application package name, element function category, or other contextual information. This application does not impose specific limitations on these aspects.

[0039] By obtaining candidate element features of at least one candidate element from the element knowledge base, a historical element knowledge candidate set can be formed, enabling the element to be located in the current interface to be matched with historical element knowledge in a multi-dimensional similarity, thereby identifying the historical function or target element identity corresponding to the current element to be located.

[0040] S103: Based on the characteristics of the target element and the characteristics of the candidate elements, determine the text similarity, resource identifier similarity, description similarity, element type similarity and position similarity between the element to be located and the candidate elements, respectively.

[0041] After obtaining the features of the target element and the candidate elements, the similarity between the element to be located and the candidate element in five dimensions—text, resource identifier, description, element type, and location—can be calculated for each candidate element. The similarity score characterizes the degree of similarity between the element to be located and the candidate element in a specific dimension. For example, the similarity score can be a value between 0 and 1, with a higher value indicating a higher degree of similarity and a lower value indicating a lower degree of similarity. This application does not specifically limit the range of similarity scores.

[0042] In one possible implementation, the text similarity between the element to be located and the candidate elements can be determined based on the text features in the target element features and the text features in the candidate element features. (Refer to...) Figure 2 , Figure 2 This is a schematic diagram illustrating the determination of text similarity as provided in an embodiment of this application.

[0043] As an example, if the text features of the element to be located completely match the text features of the candidate element, then the text similarity can be determined as the first text similarity value. For example, the first text similarity value can be 1.0. For instance, if the text feature of the element to be located is "login", and the text feature of the candidate element is also "login", then it can be determined that the two texts completely match, and the text similarity is 1.0.

[0044] As another example, if the text features of the element to be located do not perfectly match the text features of the candidate element, but there is an inclusion relationship between them, then the text similarity can be determined as the second text similarity value. For example, the second text similarity value can be 0.7. For instance, if the text feature of the element to be located is "login", and the text feature of the candidate element is "login immediately", since "login immediately" contains "login", it can be determined that there is an inclusion match between the two, and the text similarity is 0.7.

[0045] As another example, if the text features of the element to be located do not perfectly match the text features of the candidate element, and there is no inclusion relationship between them, the text similarity can be determined based on the number of overlapping characters between them. For example, the number of overlapping characters between the text features of the element to be located and the candidate element can be determined, and the maximum number of characters between the text features of the element to be located and the candidate element can be determined. Then, the text similarity can be determined based on the ratio of the number of overlapping characters to the maximum number of characters. For example, if the text feature of the element to be located is "account login" and the text feature of the candidate element is "login account", and they have multiple overlapping characters, the text similarity can be determined based on the ratio of overlapping characters.

[0046] By using the above text similarity calculation method, not only can completely identical text be identified, but also inclusion relationships and character overlap relationships can be identified, so that reasonable text matching results can still be obtained when the copy changes slightly.

[0047] In one possible implementation, the resource identifier similarity between the element to be located and the candidate elements can be determined based on the resource identifier features in the target element features and the resource identifier features in the candidate element features. (Refer to...) Figure 3 , Figure 3 This is a schematic diagram illustrating the determination of resource identifier similarity as provided in an embodiment of this application.

[0048] Resource identifiers can include either the complete resource identifier or a resource identifier suffix. For example, the format of resource-id can be "package name:id / suffix", such as "com.example:id / login_button". Here, "com.example" represents the application package name, and "login_button" is the resource identifier suffix. The resource identifier suffix can reflect the functional meaning of the UI element itself.

[0049] As an example, if the matching degree between the resource identifier feature of the element to be located and the resource identifier feature of the candidate element is greater than a first preset matching degree threshold, then the resource identifier similarity can be determined as the first resource identifier similarity value. For example, the first preset matching degree threshold is used to represent a perfect match, and the first resource identifier similarity value can be 1.0. For instance, if both the resource identifier feature of the element to be located and the resource identifier feature of the candidate element are "com.example:id / login_button", then the resource identifier similarity can be determined to be 1.0.

[0050] As another example, if the matching degree between the resource identifier features of the element to be located and the resource identifier features of the candidate element is not greater than a first preset matching degree threshold, then resource identifier suffixes can be extracted from the resource identifier features of the element to be located and the resource identifier features of the candidate element, respectively. If the matching degree between the resource identifier suffix of the element to be located and the resource identifier suffix of the candidate element is greater than the first preset matching degree threshold, then the resource identifier similarity can be determined as a second resource identifier similarity value. For example, the second resource identifier similarity value can be 0.9. For instance, if the resource identifier feature of the element to be located is "com.oldapp:id / login_button" and the resource identifier feature of the candidate element is "com.newapp:id / login_button", although the complete resource identifiers are different, the resource identifier suffixes are both "login_button", therefore, the resource identifier suffixes can be determined to match, and the resource identifier similarity is 0.9.

[0051] As another example, if the matching degree between the resource identifier suffix of the element to be located and the resource identifier suffix of the candidate element is not greater than a first preset matching degree threshold, and there is an inclusion relationship between the resource identifier suffix of the element to be located and the resource identifier suffix of the candidate element, then the resource identifier similarity can be determined as a third resource identifier similarity value. For example, the third resource identifier similarity value can be 0.7. For instance, if the resource identifier suffix of the element to be located is "login_button" and the resource identifier suffix of the candidate element is "btn_login_button", since the resource identifier suffix of the candidate element includes the resource identifier suffix of the element to be located, it can be determined that there is a suffix inclusion match between the two, and the resource identifier similarity is 0.7.

[0052] The first resource identifier similarity value can be greater than the second resource identifier similarity value, and the second resource identifier similarity value can be greater than the third resource identifier similarity value. It should be noted that the above values ​​are only examples, and the first, second, and third resource identifier similarity values ​​can also be set according to the actual test scenario. This application does not impose specific limitations on them.

[0053] By using the above-mentioned resource identifier similarity calculation method, even if the application package name changes, the resource identifier prefix changes due to version upgrades, or the developers adjust the resource identifier naming method, it is still possible to identify UI elements with the same or similar functions by matching the resource identifier suffix or matching the suffix inclusion, thereby reducing the probability of location failure caused by changes in the complete resource-id.

[0054] In one possible implementation, the descriptive similarity between the element to be located and the candidate elements can be determined based on the descriptive features in the target element features and the descriptive features in the candidate element features. (Refer to...) Figure 4 , Figure 4 This is a schematic diagram illustrating the determination of descriptive similarity in an embodiment of this application.

[0055] For example, the descriptive features of the element to be located can be segmented to obtain a first set of keywords; the descriptive features of the candidate elements can be segmented to obtain a second set of keywords; the number of keywords in the intersection between the first set of keywords and the second set of keywords can be determined; the maximum number of keywords in the first set of keywords and the second set of keywords can be determined; and the description similarity can be determined based on the ratio of the number of keywords in the intersection to the maximum number of keywords.

[0056] For example, if the descriptive feature of the element to be located is "login button" and the descriptive feature of the candidate element is "user login button", keywords such as "login" and "button" can be extracted from them. Since there are overlapping keywords between the two descriptive features, it can be determined that they have a high degree of similarity in semantic description.

[0057] It should be noted that the word segmentation method for describing features can be dictionary-based, rule-based, semantic model-based keyword extraction, or other keyword extraction methods. This application does not impose any specific limitations on this method.

[0058] By using the above-described similarity calculation method, element matching can be made to not rely entirely on precise text, but can be combined with descriptive keywords for semantic matching, thereby improving adaptability in scenarios with changes in text, multilingual versions, or descriptive expressions.

[0059] In one possible implementation, the element type similarity between the element to be located and the candidate elements can be determined based on the element type features in the target element features and the element type features in the candidate element features. (Refer to...) Figure 5 , Figure 5 This is a schematic diagram illustrating the determination of element type similarity in an embodiment of this application.

[0060] As an example, if the element type feature of the element to be located completely matches the element type feature of the candidate element, then the element type similarity can be determined as the first element type similarity value. For instance, the first element type similarity value can be 1.0. For example, if the element type feature of the element to be located is BUTTON, and the element type feature of the candidate element is also BUTTON, then the element type similarity can be determined to be 1.0.

[0061] As another example, if the element type feature of the element to be located does not match the element type feature of the candidate element, the element type similarity can be determined as a second element type similarity value. For instance, the second element type similarity value can be 0.0. For example, if the element type feature of the element to be located is BUTTON and the element type feature of the candidate element is INPUT, then it can be determined that the two types do not match, and the element type similarity is 0.0.

[0062] By using the above-mentioned element type similarity calculation method, we can avoid mismatching input boxes as buttons or buttons as icons, thereby improving the accuracy of element location results.

[0063] In one possible implementation, the positional similarity between the element to be located and the candidate elements can be determined based on the positional features in the target element features and the positional features in the candidate element features. (Refer to...) Figure 6 , Figure 6 This is a schematic diagram illustrating the determination of positional similarity as provided in an embodiment of this application.

[0064] For example, the first position coordinates of the element to be located and the second position coordinates of the candidate element can be obtained; the coordinate distance between the element to be located and the candidate element can be determined based on the first position coordinates and the second position coordinates; and the position similarity can be determined based on the coordinate distance and the maximum distance.

[0065] The first position coordinate can be the click coordinate or center coordinate of the element to be located, and the second position coordinate can be the click coordinate or center coordinate of the candidate element. As an example, Manhattan distance can be used to determine the coordinate distance. Specifically, the horizontal distance can be determined based on the first x-coordinate of the first position coordinate and the second x-coordinate of the second position coordinate; the vertical distance can be determined based on the first y-coordinate of the first position coordinate and the second y-coordinate of the second position coordinate; and the coordinate distance is determined by the sum of the horizontal and vertical distances.

[0066] For example, coordinate distance can be expressed as: coordinate distance = |x1-x2|+|y1-y2|, and positional similarity can be expressed as: positional similarity = 1-coordinate distance / maximum distance; Where x1 and y1 are the x and y coordinates of the first position, respectively, and x2 and y2 are the x and y coordinates of the second position, respectively. The maximum distance can be a preset value, such as 2000 pixels. In practical applications, the maximum distance can also be dynamically set according to screen resolution, device size, page type, or test scenario; this application does not impose specific limitations on this.

[0067] As an example, if the positional features of the element to be located are (500, 1200) and the positional features of the candidate element are (520, 1240), then the coordinate distance can be |500-520|+|1200-1240|=60. If the maximum distance is 2000 pixels, then the positional similarity can be 1-60 / 2000=0.97. Therefore, although the position of the UI element has changed slightly, its positional similarity can still be determined to be high.

[0068] By employing the aforementioned method of calculating positional similarity, even with UI layout adjustments, varying screen sizes across different devices, or slight element position shifts, the matching relationship between the element to be located and candidate elements can still be determined using positional proximity. Therefore, by calculating text similarity, resource identifier similarity, description similarity, element type similarity, and positional similarity separately, the degree of similarity between the element to be located and candidate elements can be measured from multiple dimensions, avoiding the failure of element location due to changes in any single attribute.

[0069] S104: Determine the location score corresponding to the candidate element based on text similarity, resource identifier similarity, description similarity, element type similarity, and location similarity.

[0070] The location score represents the overall matching degree between a candidate element and the element to be located. A higher location score indicates that the candidate element is more likely to be the target matching element. As an example, the location score for a candidate element can be determined based on text similarity, resource identifier similarity, description similarity, element type similarity, positional similarity, and their respective weights.

[0071] The weights corresponding to each similarity score represent the importance of each dimension in the element localization process. For example, the weight for text similarity can be 0.30, the weight for resource identifier similarity can be 0.25, the weight for description similarity can be 0.20, the weight for element type similarity can be 0.10, and the weight for location similarity can be 0.15. The weight for text similarity can be relatively high to indicate that the text directly reflects the element's function; the weight for resource identifier similarity can be the second highest to indicate that resource identifiers are relatively stable element identifiers; description similarity can be used to reflect semantic description matching; location similarity can be used to reflect location proximity; and element type similarity can be used for type validation. The above weight values ​​are merely examples and can be adjusted according to the test scenario, application type, element type, or historical localization results. This application does not impose specific limitations on these adjustments.

[0072] As an example, see Figure 7 , Figure 7 The diagram provided in this application illustrates the determination of similarity scores. The similarity scores of candidate elements can be determined according to the following formula: Similarity score = text similarity × 0.30 + resource identifier similarity × 0.25 + description similarity × 0.20 + element type similarity × 0.10 + position similarity × 0.15.

[0073] It should be noted that the similarity score in this application can also be called the weighted similarity score, which is used to represent the comprehensive similarity of candidate elements across multiple feature dimensions.

[0074] As an example, the target element features corresponding to the element to be located can be: text feature "login", resource identifier feature "com.oldapp:id / login_button", description feature "login button", element type feature BUTTON, and location feature (500, 1200). The candidate element features can be: text feature "login now", resource identifier feature "com.newapp:id / login_button", description feature "user login button", element type feature BUTTON, and location feature (520, 1240). Based on the aforementioned similarity calculation method, the text similarity is determined to be 0.7, the resource identifier similarity to be 0.9, the description similarity to be, for example, 0.8, the element type similarity to be 1.0, and the location similarity to be 0.97. Therefore, the similarity score of this candidate element can be: 0.7×0.30+0.9×0.25+0.8×0.20+1.0×0.10+0.97×0.15=0.8405.

[0075] By using weighted calculations, different dimensions can participate in the overall matching according to their importance, and dimensions with higher weights can better reflect the essence of the elements, thus improving matching accuracy. When a certain attribute changes, the similarity corresponding to that attribute may decrease, but other dimensions can still contribute corresponding scores. Therefore, a change in a single attribute will not cause the overall matching to fail directly.

[0076] In one possible implementation, the candidate element features may further include knowledge confidence, and the location score corresponding to the candidate element can be determined based on the similarity score and knowledge confidence of the candidate element. For example, the location score = similarity score × knowledge confidence.

[0077] Knowledge confidence reflects the historical success rate of locating element knowledge corresponding to a candidate element. The higher the knowledge confidence, the more reliable the element knowledge has been historically verified; the lower the knowledge confidence, the less verified the element knowledge or the poorer the historical locating effect.

[0078] As an example, if a candidate element has a similarity score of 0.8405 and a knowledge confidence score of 0.9, then the corresponding location score for the candidate element could be 0.8405 × 0.9 = 0.75645. As another example, if another candidate element also has a similarity score of 0.8405, but its knowledge confidence score is 0.4, then its location score would be 0.3362. Therefore, when feature similarity is similar, candidate elements with higher historical reliability can obtain higher location scores.

[0079] By introducing knowledge confidence, elements with high historical success rates in localization can be prioritized for matching, reducing the interference of low-quality elements in the localization results and thus improving the localization success rate. Furthermore, by jointly determining the localization score using similarity score and knowledge confidence, both the current feature similarity and the reliability of historical experience can be considered.

[0080] It should be noted that in implementations that do not use knowledge confidence, the similarity score can be directly determined as the location score corresponding to the candidate element; in implementations that use knowledge confidence, the final score obtained by combining the similarity score and knowledge confidence can be determined as the location score corresponding to the candidate element. This application does not make any specific limitations on this.

[0081] S105: Filter based on location score and matching threshold to determine the target matching element from at least one candidate element.

[0082] A matching threshold is used to determine whether a candidate element meets the matching conditions. For example, the matching threshold can be 0.15. If the location score corresponding to a candidate element is greater than or equal to the matching threshold, then the candidate element is considered to have the potential to become the target matching element; if the location score corresponding to a candidate element is less than the matching threshold, then the overall matching degree between the candidate element and the element to be located is considered low, and it can be filtered out. It should be noted that the matching threshold can be set according to the test scenario, element type, historical location results, or business needs; this application does not impose specific limitations on this.

[0083] In one possible implementation, refer to Figure 8 , Figure 8 A schematic diagram of threshold filtering provided in an embodiment of this application is shown below. Figure 8 As shown, candidate elements with corresponding positioning scores greater than or equal to the matching threshold can be selected from at least one candidate element to obtain a candidate matching element set; the candidate elements in the candidate matching element set are sorted according to their respective positioning scores to obtain a sorting result; and the target matching element is determined from the candidate matching element set based on the sorting result.

[0084] For example, refer to Figure 9 , Figure 9 A schematic diagram illustrating the result sorting provided in the embodiments of this application, as shown below. Figure 9 As shown, candidate elements in the candidate matching element set can be sorted in descending order according to their positioning scores, and the candidate element with the highest positioning score in the sorted result can be determined as the target matching element. Alternatively, the sorting result can be returned to an automated testing system, which can then perform click, input, or other interface operations based on the candidate element with the highest score. This application does not impose specific limitations on this approach.

[0085] As an example, the current interface includes several candidate elements such as "Log in Now," "Register," and "Forgot Password." After multi-dimensional similarity calculation and location score calculation, the location score of "Log in Now" is 0.75645, the location score of "Register" is 0.22, and the location score of "Forgot Password" is 0.18. If the matching threshold is 0.15, all three candidate elements can be included in the candidate matching element set. After further sorting in descending order of location score, "Log in Now" is at the top of the sorting results, therefore, "Log in Now" can be identified as the target matching element.

[0086] As another example, if the positioning score of a candidate element is 0.08, which is less than the matching threshold of 0.15, the candidate element can be filtered out to prevent obviously irrelevant UI elements from entering the candidate matching element set.

[0087] Therefore, by using threshold matching and score sorting, when multiple similar UI elements exist on the current interface, the UI element with the highest overall matching degree can be selected as the target matching element, reducing the probability of false matching and improving the reliability of subsequent automated testing operations.

[0088] This application provides an element location method that, instead of relying solely on a single attribute for precise UI element location, comprehensively matches UI elements from multiple dimensions such as text, resource identifiers, descriptions, element types, and positions. In other words, even if a certain attribute of the target UI element changes, other unchanged or minimally changed features can still participate in the matching process, thus avoiding location failure directly caused by a single attribute change and improving the accuracy of UI element location. Therefore, it enhances the tolerance of the UI element location process to changes in text, resource identifiers, layout, and page structure, thereby improving the success rate of automated testing.

[0089] In one possible implementation, after filtering based on location score and matching threshold to determine the target matching element from at least one candidate element, the location result of the target matching element can be obtained, and the knowledge confidence of the corresponding candidate element can be updated based on the location result.

[0090] Location results can include successful location or failed location results. A successful location result indicates that the target matching element was successfully located by the automated testing system, and subsequent operations performed based on the target matching element are as expected, such as clicking, inputting, or performing other operations as expected. A failed location result indicates that the target matching element was not successfully located, or that operations performed based on the target matching element did not achieve the expected page state.

[0091] As an example, see Figure 10 , Figure 10 A schematic diagram of the knowledge confidence update method provided in the embodiments of this application is shown below. Figure 10 As shown, if the location result is a successful location result, the success count of the corresponding candidate element's knowledge can be increased, and the knowledge confidence of the corresponding candidate element can be improved. For example, the knowledge confidence can be increased by a first confidence adjustment value, such as 0.05, and the maximum value of the knowledge confidence can be 1.0. If the increased knowledge confidence is greater than 1.0, then the knowledge confidence can be set to 1.0.

[0092] As another example, if the location result is a location failure, the failure count of the corresponding candidate element's knowledge can be increased, and the knowledge confidence of the corresponding candidate element can be decreased. For instance, the knowledge confidence can be reduced by a second confidence adjustment value, such as by 0.1, and the minimum knowledge confidence can be 0.1. If the reduced knowledge confidence is less than 0.1, then the knowledge confidence can be set to 0.1.

[0093] It should be noted that the first confidence level adjustment value, the second confidence level adjustment value, the upper limit of knowledge confidence value, and the lower limit of knowledge confidence value can all be set according to the test scenario or historical experience, and this application does not impose specific restrictions on them.

[0094] By dynamically updating the knowledge confidence level based on the location results, the confidence level of high-quality element knowledge increases with the number of successful location attempts, while the confidence level of low-quality element knowledge decreases with the number of failed location attempts. Therefore, the element knowledge base can be continuously optimized during the automated testing process, making subsequent element location more inclined to use element knowledge with higher reliability, further improving the accuracy of element location. Furthermore, long-term use of this method can lead to the formation of a high-quality knowledge base.

[0095] In one possible implementation, refer to Figure 11 , Figure 11 This is a schematic diagram of the architecture of the element positioning method provided in the embodiments of this application, as shown below. Figure 11As shown, the element localization method provided in this application embodiment can be implemented through an element knowledge service engine (ElementKnowledgeService). The element knowledge service engine may include a feature extraction module (FeatureExtract), a similarity calculation module (SimilarityCalc), a confidence weighting module (ConfWeight), a threshold filtering module (ThresholdFilter), and a result sorting output module (ResultSort).

[0096] The feature extraction module is used to extract text features, resource identifier features, description features, element type features, and location features from the element to be located and candidate elements. As an example, see [link to example]. Figure 12 , Figure 12 This is a schematic diagram illustrating the element information structure definition provided in the embodiments of this application, such as... Figure 12 As shown, an element information structure can be defined, which can include fields such as name, text content, resource identifier, description information, element type, click coordinates, center coordinates, page type, application package name, expected function, knowledge confidence level, number of successes, and number of failures. This element information structure can uniformly represent the element to be located, candidate elements, and historical element knowledge in the element knowledge base.

[0097] For example, refer to Figure 13 , Figure 13 This is a schematic diagram of the knowledge base query method provided in this application embodiment. It can prioritize querying element knowledge corresponding to candidate elements based on page type; if no element knowledge matching the page type is found, it can query element knowledge corresponding to candidate elements based on application package name; then, a list of element knowledge can be returned. Each element knowledge can include feature information, expected function, and knowledge confidence level of historical element knowledge.

[0098] For reference Figure 14 , Figure 14 A schematic diagram of the weighted similarity calculation method provided in the embodiments of this application is shown below. Figure 14 As shown, the similarity calculation module calculates text similarity, resource identifier similarity, description similarity, element type similarity, and positional similarity for each candidate element, and calculates a similarity score based on the weights corresponding to each dimension. For example, the weight of text similarity can be 0.30, the weight of resource identifier similarity can be 0.25, the weight of description similarity can be 0.20, the weight of element type similarity can be 0.10, and the weight of positional similarity can be 0.15.

[0099] The confidence-weighted module determines the location score based on the similarity score and knowledge confidence. For example, the confidence-weighted module can multiply the similarity score by the knowledge confidence to obtain the location score corresponding to the candidate element. This module allows for priority matching of knowledge-based elements with high confidence, while knowledge-based elements with low confidence are ranked lower.

[0100] The threshold filtering module filters candidate elements based on a matching threshold, and the result sorting output module sorts the filtered candidate elements based on their location scores. For example, the matching threshold can be 0.15. The threshold filtering module can filter out candidate elements with a location score greater than or equal to 0.15, and the result sorting output module can sort the results in descending order of location score, returning a list of matching results.

[0101] Therefore, by extracting features, calculating similarity, weighting confidence, filtering thresholds, and ranking results, a multi-dimensional matching loop can be formed, enabling the element localization process to have multi-dimensional fault tolerance, comprehensive scoring, and candidate ranking capabilities.

[0102] As an example, the automated testing system needs to locate the login button on the new version's login page. The historical login button characteristics recorded in the element knowledge base are: page type is login page, application package name is "com.oldapp", text feature is "login", resource identifier feature is "com.oldapp:id / login_button", description feature is "login button", element type feature is BUTTON, position feature is (500, 1200), and knowledge confidence is 0.9.

[0103] In the current interface of the new version, a candidate button is identified with the following characteristics: text feature "Login Now", resource identifier feature "com.newapp:id / login_button", description feature "User Login Button", element type feature "BUTTON", and position feature (520, 1240). According to the method provided in this application embodiment, the text similarity is determined to be 0.7, resource identifier similarity to be 0.9, description similarity to be 0.8, element type similarity to be 1.0, and position similarity to be 0.97. Therefore, the similarity score can be calculated as 0.8405. Combined with a knowledge confidence score of 0.9, the positioning score can be determined to be 0.75645. If the matching threshold is 0.15, the candidate button passes the threshold filtering, and due to its high positioning score, it can be identified as the target matching element. Thus, even if the text changes from "Login" to "Login Now", the application package name changes from "com.oldapp" to "com.newapp", and the position coordinates change slightly, the target login button can still be located through multi-dimensional comprehensive matching.

[0104] The above are some specific implementations of the element positioning method provided in the embodiments of this application. Based on this, this application also provides a corresponding device. The device provided in the embodiments of this application will be described below from the perspective of functional modularity.

[0105] See Figure 15 The figure is a schematic diagram of the structure of an element positioning device 1500 provided in an embodiment of this application. The device 1500 may include: The first acquisition module 1501 is used to acquire the target element features of the element to be located, wherein the target element features include text features, resource identifier features, description features, element type features and location features; The second acquisition module 1502 is used to acquire candidate element features of at least one candidate element, wherein the candidate element features include text features, resource identifier features, description features, element type features and position features of the candidate element. The similarity determination module 1503 is used to determine the text similarity, resource identifier similarity, description similarity, element type similarity and position similarity between the element to be located and the candidate elements based on the target element features and the candidate element features, respectively. The location score determination module 1504 is used to determine the location score corresponding to the candidate element based on the text similarity, the resource identifier similarity, the description similarity, the element type similarity, and the location similarity. The matching module 1505 is used to filter based on the positioning score and the matching threshold to determine the target matching element from at least one candidate element.

[0106] Optionally, the positioning score determination module 1504 is specifically used for: The location score corresponding to the candidate element is determined based on the text similarity, resource identifier similarity, description similarity, element type similarity, location similarity, and their respective weights.

[0107] Optionally, the candidate element features include knowledge confidence, and the location score determination module 1504 is specifically used for: The similarity score of the candidate element is determined based on the text similarity, resource identifier similarity, description similarity, element type similarity, position similarity, and their respective weights. The location score corresponding to the candidate element is determined based on the similarity score of the candidate element and the knowledge confidence score.

[0108] Optionally, the matching module 1505 is specifically used for: From at least one candidate element, select the candidate element whose corresponding positioning score is greater than or equal to the matching threshold to obtain a candidate matching element set; Based on the positioning score corresponding to each candidate element in the candidate matching element set, the candidate elements in the candidate matching element set are sorted to obtain the sorting result; Based on the sorting results, the target matching element is determined from the set of candidate matching elements.

[0109] Optionally, the device further includes: The third acquisition module is used to acquire the location result of the target matching element; The update module is used to update the knowledge confidence of the corresponding candidate elements based on the positioning results.

[0110] Optionally, the resource identifier similarity can be determined by the following units: The first determining unit is configured to determine the resource identifier similarity as a first resource identifier similarity value if the matching degree between the resource identifier feature of the element to be located and the resource identifier feature of the candidate element is greater than a first preset matching degree threshold. An extraction unit is configured to extract resource identifier suffixes from the resource identifier features of the element to be located and the resource identifier features of the candidate element, respectively, if the matching degree between the resource identifier features of the element to be located and the resource identifier features of the candidate element is not greater than the first preset matching degree threshold. The second determining unit is configured to determine the resource identifier similarity as a second resource identifier similarity value if the matching degree between the resource identifier suffix of the element to be located and the resource identifier suffix of the candidate element is greater than a first preset matching degree threshold; the second resource identifier similarity value is less than the first resource identifier similarity. The third determining unit is configured to determine the resource identifier similarity as a third resource identifier similarity value if the matching degree between the resource identifier suffix of the element to be located and the resource identifier suffix of the candidate element is not greater than a first preset matching degree threshold, and there is an inclusion relationship between the resource identifier suffix of the element to be located and the resource identifier suffix of the candidate element; the third resource identifier similarity value is less than the second resource identifier similarity.

[0111] Optionally, the positional similarity can be determined by the following unit: The first acquisition unit is used to acquire the first position coordinates of the element to be located and the second position coordinates of the candidate element; A coordinate distance determination unit is used to determine the coordinate distance between the element to be located and the candidate element based on the first position coordinates and the second position coordinates; A location similarity determination unit is used to determine the location similarity based on the coordinate distance and the maximum distance.

[0112] This application also provides corresponding devices and computer storage media for implementing the solutions provided in this application.

[0113] The device includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to enable the device to perform the element positioning method described in any embodiment of this application.

[0114] The computer storage medium stores a computer program. When the code is run, the device running the computer program implements the element positioning method described in any embodiment of this application.

[0115] In the embodiments of this application, the terms "first" and "second" (if they exist) are used only as name identifiers and do not represent the order of first and second.

[0116] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that all or part of the steps in the methods of the above embodiments can be implemented by means of software plus a general-purpose hardware platform. Based on this understanding, the technical solution of this application can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as a read-only memory (ROM) / RAM, a magnetic disk, an optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, a server, or a network communication device such as a router) to execute the methods described in various embodiments or some parts of the embodiments of this application.

[0117] It should be noted that the various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for the device embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the description of the method embodiments. The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separate. The components indicated as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment solution according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0118] The above description is merely one specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for locating elements, characterized in that, The method includes: Obtain the target element features of the element to be located, wherein the target element features include text features, resource identifier features, description features, element type features, and location features; Obtain candidate element features of at least one candidate element, wherein the candidate element features include text features, resource identifier features, description features, element type features, and position features of the candidate element; Based on the target element features and the candidate element features, the text similarity, resource identifier similarity, description similarity, element type similarity, and position similarity between the element to be located and the candidate elements are determined respectively. The location score corresponding to the candidate element is determined based on the text similarity, resource identifier similarity, description similarity, element type similarity, and location similarity. Based on the location score and matching threshold, a target matching element is determined from at least one candidate element.

2. The method according to claim 1, characterized in that, The step of determining the location score corresponding to the candidate element based on the text similarity, resource identifier similarity, description similarity, element type similarity, and location similarity includes: The location score corresponding to the candidate element is determined based on the text similarity, resource identifier similarity, description similarity, element type similarity, location similarity, and their respective weights.

3. The method according to claim 2, characterized in that, The candidate element features include knowledge confidence. Determining the location score corresponding to the candidate element based on the text similarity, resource identifier similarity, description similarity, element type similarity, location similarity, and their respective weights includes: The similarity score of the candidate element is determined based on the text similarity, resource identifier similarity, description similarity, element type similarity, position similarity, and their respective weights. The location score corresponding to the candidate element is determined based on the similarity score of the candidate element and the knowledge confidence score.

4. The method according to claim 1, characterized in that, The step of filtering based on the positioning score and matching threshold to determine the target matching element from at least one candidate element includes: From at least one candidate element, select the candidate element whose corresponding positioning score is greater than or equal to the matching threshold to obtain a candidate matching element set; Based on the positioning score corresponding to each candidate element in the candidate matching element set, the candidate elements in the candidate matching element set are sorted to obtain the sorting result; Based on the sorting results, the target matching element is determined from the set of candidate matching elements.

5. The method according to claim 3, characterized in that, After determining the target matching element from at least one candidate element by filtering based on the location score and matching threshold, the method further includes: Obtain the location result of the target matching element; Based on the location results, update the knowledge confidence of the corresponding candidate elements.

6. The method according to claim 1, characterized in that, The resource identifier similarity can be determined in the following ways: If the matching degree between the resource identifier feature of the element to be located and the resource identifier feature of the candidate element is greater than the first preset matching degree threshold, then the resource identifier similarity is determined to be the first resource identifier similarity value. If the matching degree between the resource identifier feature of the element to be located and the resource identifier feature of the candidate element is not greater than the first preset matching degree threshold, then the resource identifier suffix is ​​extracted from the resource identifier feature of the element to be located and the resource identifier feature of the candidate element, respectively. If the matching degree between the resource identifier suffix of the element to be located and the resource identifier suffix of the candidate element is greater than a first preset matching degree threshold, then the resource identifier similarity is determined to be a second resource identifier similarity value; the second resource identifier similarity value is less than the first resource identifier similarity. If the matching degree between the resource identifier suffix of the element to be located and the resource identifier suffix of the candidate element is not greater than the first preset matching degree threshold, and there is an inclusion relationship between the resource identifier suffix of the element to be located and the resource identifier suffix of the candidate element, then the resource identifier similarity is determined to be the third resource identifier similarity value; the third resource identifier similarity value is less than the second resource identifier similarity.

7. The method according to claim 1, characterized in that, The positional similarity can be determined in the following ways: Obtain the first position coordinates of the element to be located and the second position coordinates of the candidate element; Based on the first position coordinates and the second position coordinates, determine the coordinate distance between the element to be located and the candidate element; The positional similarity is determined based on the coordinate distance and the maximum distance.

8. An element positioning device, characterized in that, The device includes: The first acquisition module is used to acquire the target element features of the element to be located, wherein the target element features include text features, resource identifier features, description features, element type features and location features; The second acquisition module is used to acquire candidate element features of at least one candidate element, wherein the candidate element features include text features, resource identifier features, description features, element type features and position features of the candidate element; The similarity determination module is used to determine the text similarity, resource identifier similarity, description similarity, element type similarity, and position similarity between the element to be located and the candidate elements based on the target element features and the candidate element features, respectively. The location score determination module is used to determine the location score corresponding to the candidate element based on the text similarity, the resource identifier similarity, the description similarity, the element type similarity, and the location similarity. The matching module is used to filter based on the positioning score and the matching threshold to determine the target matching element from at least one candidate element.

9. An element positioning device, characterized in that, The device includes: Memory, used to store computer programs; A processor for executing the computer program to cause the device to perform the element positioning method as described in any one of claims 1 to 7.

10. A computer storage medium, characterized in that, The computer storage medium stores a computer program, which, when executed by a processor, implements the element location method as described in any one of claims 1 to 7.

Images