Interface element positioning method, device and system

Abstract

The invention discloses an interface element positioning method, device and system, relates to the technical field of interface automatic testing, and at least solves the problem of positioning failure of a single element positioning mode in a dynamic interface and a complex scene. The method comprises the following steps: acquiring a to-be-tested element; determining target priorities of a plurality of preset element positioning modes; the target priority represents a previous sequence of calling each preset element positioning mode for matching; the multiple preset element positioning modes comprise first text attribute positioning, second visual positioning and third context positioning; and according to the target priority, calling each preset element positioning mode, and positioning a target element corresponding to the to-be-tested element in the interface elements to obtain the target element.

Description

Technical Field

[0001] The present application relates to the technical field of interface automated testing, and in particular to a method, device, and system for locating interface elements. Background Art

[0002] In the field of UI (User Interface) automated testing, element positioning is a core technology. Research has found that the current field of UI automated testing primarily relies on a single positioning strategy, which suffers from the following drawbacks: First, when the attributes of page elements change dynamically, if a single text matching method is used, the element to be tested is an icon button without a text label and has no recognizable text attributes, and it is impossible to intelligently switch to the corresponding positioning method to locate the element, resulting in positioning failure. Second, in complex scenarios, a single positioning strategy cannot adaptively identify the type of element to be tested and adjust the element positioning method. This requires staff to make adjustments based on the type of element being tested, increasing testing costs. Summary of the Invention

[0003] The present invention provides an interface element positioning method, device, and system to at least solve the problem of single element positioning failure in dynamic interfaces and complex scenes. The technical solution of the present invention is as follows: According to the first aspect of an embodiment of the present invention, a method for locating an interface element is provided, which obtains an element to be tested; determines the target priority of multiple preset element positioning methods; the target priority represents the order in which each preset element positioning method is called for matching; multiple preset element positioning methods: a first text attribute positioning, a second visual positioning, and a third context positioning; according to the target priority, each preset element positioning method is called to locate the target element in the interface element corresponding to the element to be tested to obtain the target element.

[0004] In one implementation, according to the target priority, each preset element positioning method is called to locate the target element corresponding to the element to be measured in the interface element, including: according to the calling sequence indicated by the target priority, the target preset positioning method corresponding to the calling sequence is called in sequence to locate the target element corresponding to the element to be measured until the target element is determined in the interface element.

[0005] The element positioning methods of the corresponding positions are called in sequence according to the preset calling sequence. When the target element corresponding to the element to be measured is determined in the interface elements, the calling of the target preset positioning method corresponding to the next calling sequence is stopped. In this way, the calling sequence of the target preset positioning method is preset according to the correlation with the element to be measured. If the target element is positioned in the first sequence, the target element can be accurately positioned without having to call all the preset positioning methods in sequence, thereby improving the efficiency of element positioning.

[0006] In another implementation, locating the target element corresponding to the element to be tested includes: when determining that the target preset positioning method is the first text attribute positioning, extracting the key attributes of the element to be tested and generating an attribute feature vector; based on the attribute feature vector, determining the first candidate element in the interface element; the first candidate element represents an element in the interface element whose similarity with the attribute feature vector of the element to be tested is higher than the first preset similarity; and determining the target element with the highest similarity from multiple first candidate elements.

[0007] The text attribute positioning method is the most efficient element positioning method. It directly compares the similarity of the text attributes of the element to be tested and the interface element to determine the target element, so as to locate the target element corresponding to the element to be tested that contains some dynamically changing text attributes.

[0008] In another implementation, a DOM tree is constructed based on the interface elements and element information associated with the interface elements; the DOM tree includes: multiple DOM tree nodes and association relationships between the multiple DOM tree nodes, and each DOM tree node represents each interface element.

[0009] In another implementation method, the target element corresponding to the element to be measured is positioned, including: when determining that the target preset positioning method is the second visual positioning, obtaining the image features of the element to be measured, and obtaining the interface image corresponding to the interface element; determining the target coordinate position of the image features in the interface image; selecting the target DOM tree node from the DOM tree according to the target coordinate position; and determining the target element corresponding to the element to be measured in the interface element according to the target DOM tree node.

[0010] By locating the coordinate position of the image of the element to be tested in the interface image and converting the coordinate position of the image into the target DOM tree node, the target element in the interface element corresponding to the target DOM tree node is located to achieve element positioning in the rich media interface scenario.

[0011] In another implementation, the target element corresponding to the element to be tested is located, including: determining that the target preset positioning method is the third context positioning, determining the target element node to be tested corresponding to the element to be tested in the DOM tree; determining a target parent node closest to the target element node to be tested from the DOM tree; generating a first path between the target element node to be tested and the target parent node; the first path represents that there are no other related nodes between the target element node and the target parent node; based on the association relationship between multiple DOM tree nodes, adding one or more related nodes to the first path to generate multiple first alternative paths; the related nodes include ancestor nodes that increase the path depth, sibling nodes that increase content constraints, and child nodes that enhance the path structure; determining the path complexity priority of multiple first alternative paths; the path complexity priority is that the priority of low path complexity is higher than the priority of high path complexity; according to the path complexity priority, the multiple first alternative paths are verified in turn, and a target path that uniquely locates the element to be tested is determined from the multiple first alternative paths; based on the target path, the target element corresponding to the element to be tested in the interface element is determined.

[0012] By analyzing the relationship between each node in the DOM tree, a unique target path that can accurately locate the element to be tested is established. The target path is used to determine the corresponding target element. Through the stability of the target path context structure, element positioning is achieved in scenarios where attributes are missing or multiple elements have similar attributes.

[0013] In another implementation, according to the calling sequence indicated by the target priority, the target preset positioning method corresponding to the calling sequence is called in sequence to locate the target element corresponding to the element to be measured until the target element is determined in the interface element. The method also includes: determining the target element type of the element to be measured, and determining the target preset positioning method based on which the element to be measured obtains the target element; and associating the target element type with the target preset positioning method and storing them in a preset knowledge base.

[0014] The elements to be tested and the target positioning methods of the target elements that are successfully positioned each time are stored in the knowledge base, providing a basis for determining the target priority for the next positioning.

[0015] In another implementation, the preset knowledge base includes a preset positioning method with the highest priority corresponding to different element types, and determines the target priority of multiple preset element positioning methods, including: determining the target element type of the element to be tested; selecting the highest priority positioning method associated with the target element type from the preset knowledge base; determining the target priority based on the highest priority positioning method; when the target element type of the element to be tested is uncertain, determining the target priority of multiple preset positioning methods as follows: the first text attribute positioning priority is higher than the second visual positioning priority, which is higher than the third context positioning priority.

[0016] The highest priority indicates that the positioning method is used most frequently in the historical positioning process of the element to be measured, indicating that the positioning method corresponding to the target element type of the element to be measured has the highest frequency of successful positioning. This positioning method is called first for positioning to improve the efficiency of element positioning.

[0017] According to a second aspect of an embodiment of the present invention, an interface element positioning device is provided, which includes: an acquisition unit, configured to: acquire an element to be tested; a matching unit, configured to: determine a target priority of multiple preset element positioning methods; the target priority represents the order in which each preset element positioning method is called for matching; multiple preset element positioning methods: a first text attribute positioning, a second visual positioning, and a third context positioning; the positioning unit is configured to: call each preset element positioning method according to the target priority, and locate the target element in the interface element corresponding to the element to be tested to obtain the target element.

[0018] According to a third aspect of an embodiment of the present invention, there is provided an interface element positioning system, which is configured to execute the electricity user classification method according to the first aspect and any possible implementation thereof.

[0019] According to a fourth aspect of an embodiment of the present invention, an interface element positioning device is provided, wherein the device is configured to execute the interface element positioning method according to the first aspect and any possible implementation thereof.

[0020] According to the fifth aspect of an embodiment of the present invention, a computer-readable storage medium is provided, on which instructions are stored. When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the interface element positioning method such as the first aspect and any possible implementation thereof.

[0021] According to the sixth aspect of an embodiment of the present disclosure, a computer program product is provided, which includes computer instructions. When the computer instructions are run on an electronic device, the electronic device executes the interface element positioning method of the above-mentioned first aspect and any possible implementation thereof.

[0022] The technical solutions provided by the embodiments of the present invention bring at least the following beneficial effects: In different or complex interfaces, the attributes of the elements to be tested are different, and the corresponding element positioning methods that can be successfully positioned are different. A variety of preset element positioning methods are used to ensure that there is an element positioning method that can accurately determine the target element in the interface elements; at the same time, before locating the target element corresponding to the element to be tested, the priorities of multiple preset element positioning methods are preset. By adjusting the priorities, the element positioning method associated with the element to be tested is preferentially called to locate the target element corresponding to the element to be tested, thereby improving the positioning efficiency. Moreover, even if the element positioning method with the highest priority fails, the element positioning method corresponding to the next level priority can be adaptively called for positioning until the target element is determined in the interface elements, thereby ensuring the robustness of element positioning.

[0023] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present application, and together with the specification are used to explain the principles of the present application, and do not constitute an improper limitation on the present application.

[0025] Figure 1 is a schematic diagram of an interface element positioning system according to an exemplary embodiment; Figure 2 is a flow chart showing a method for positioning an interface element according to an exemplary embodiment; Figure 3 is a block diagram of a device for positioning interface elements according to an exemplary embodiment; Figure 4 The figure is a schematic diagram of an interface element positioning device according to an exemplary embodiment. DETAILED DESCRIPTION

[0026] In order to enable ordinary people in the art to better understand the technical solutions of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings.

[0027] It should be noted that the terms "first," "second," and the like in the specification and claims of this application and the accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate so that the embodiments of the application described herein can be implemented in an order other than those illustrated or described herein. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of apparatus and methods consistent with certain aspects of the present application, as detailed in the appended claims.

[0028] First, before giving a detailed introduction to the interface element positioning method provided in the embodiment of the present application, a brief introduction to the application scenarios involved in the embodiment of the present application is first given.

[0029] In the field of UI automation testing, element positioning is a core technical link. Research has found that the current field of UI automation testing mainly relies on a single strategy positioning method, which has the following flaws: First, when the attributes of page elements change dynamically, if a single text matching method is used, the element to be tested is an icon button without a text label and has no recognizable text attributes, and it is impossible to intelligently switch to the corresponding positioning method to locate the element, resulting in positioning failure; second, in complex scenarios, a single positioning strategy cannot adaptively identify the type of element to be tested and adjust the element positioning method. Staff must make adjustments based on the type of element to be tested, increasing testing costs.

[0030] In response to the above problems, the present application provides an interface element positioning method. In different or complex interfaces, the attributes of the elements to be measured are different, and the corresponding element positioning methods that can be successfully positioned are different. A plurality of preset element positioning methods are adopted to ensure that there is an element positioning method that can accurately determine the target element in the interface element; at the same time, before locating the target element corresponding to the element to be measured, the priorities of the plurality of preset element positioning methods are preset. By adjusting the priorities, the element positioning method associated with the element to be measured is preferentially called to locate the target element corresponding to the element to be measured, thereby improving the positioning efficiency. Moreover, even if the element positioning method with the highest priority fails, the element positioning method corresponding to the next level priority can be adaptively called for positioning until the target element is determined in the interface element, thereby ensuring the robustness of element positioning.

[0031] Next, the implementation architecture involved in this application is briefly introduced below.

[0032] Figure 1 FIG. 1 is a schematic diagram of an interface element positioning system according to an exemplary embodiment. Figure 1 As shown, the interface element positioning system includes a text attribute positioning module 101 , a visual positioning module 102 , a context positioning module 103 , and a positioning mode control module 104 .

[0033] The text attribute positioning module 101 , the visual positioning module 102 , the context positioning module 103 , and the positioning mode control module 104 are communicatively connected with each other.

[0034] The text attribute positioning module 101 is configured to extract the key attributes of the element to be tested and generate an attribute feature vector when determining that the target preset positioning method is the first text attribute positioning; based on the attribute feature vector, determine the first candidate element in the interface element; the first candidate element represents the element in the interface element whose similarity with the attribute feature vector of the element to be tested is higher than the first preset similarity; and determine the target element with the highest similarity from multiple first candidate elements.

[0035] The visual positioning module 102 is configured to obtain the image features of the element to be tested and the interface image corresponding to the interface element when determining that the preset positioning mode of the target is the second visual positioning; determine the target coordinate position of the image features in the interface image; select the target DOM tree node from the DOM tree according to the target coordinate position; and determine the target element corresponding to the element to be tested in the interface element according to the target DOM tree node.

[0036] The context positioning module 103 is configured to determine that the target preset positioning method is the third context positioning, determine the target element node to be tested corresponding to the element to be tested in the DOM tree; determine the target parent node closest to the target element node to be tested from the DOM tree; generate a first path between the target element node to be tested and the target parent node; the first path represents that there are no other related nodes between the target element node and the target parent node; based on the association relationship between multiple DOM tree nodes, add one or more related nodes to the first path to generate multiple first alternative paths; the related nodes include ancestor nodes that increase the path depth, sibling nodes that increase content constraints, and child nodes that enhance the path structure; determine the path complexity priority of multiple first alternative paths; the path complexity priority is that the priority of low path complexity is higher than the priority of high path complexity; according to the path complexity priority, verify the multiple first alternative paths in turn, and determine the target path that uniquely locates the element to be tested from the multiple first alternative paths; based on the target path, determine the target element corresponding to the element to be tested in the interface element.

[0037] The positioning mode control module 104 is configured with priority setting, automatic switching logic control and historical successful positioning learning to optimize the positioning process.

[0038] The interface element positioning method provided in the embodiment of the present application can be applied to the aforementioned Figure 1 The interface element positioning system in the implementation architecture shown.

[0039] For ease of understanding, the interface element positioning method provided in this application is specifically introduced below with reference to the accompanying drawings.

[0040] Figure 2 is a flow chart showing a method for positioning an interface element according to an exemplary embodiment. Figure 2As shown, the interface element positioning method includes the following steps.

[0041] S21, obtaining the element to be measured.

[0042] The representation of the element to be tested is obtained from the test script or user requirements according to the test requirements.

[0043] S22, determining target priorities of multiple preset element positioning methods.

[0044] The target priority represents the order in which each preset element positioning method is called for matching.

[0045] Multiple preset element positioning methods: first text attribute positioning, second visual positioning, and third context positioning; In one embodiment, determining the target priorities of multiple preset element positioning methods also includes: when the target element type of the element to be tested is uncertain, determining the target priorities of multiple preset positioning methods to be preset as the first text attribute positioning priority being higher than the second visual positioning priority being higher than the third context positioning priority.

[0046] Specifically, when the type of the element to be tested cannot be determined, the first text attribute positioning method, which is the most efficient and simplest in principle among a variety of preset element positioning methods, is used as the first priority, and positioning is performed directly by identifying the text attributes in the element; if the first text attribute positioning method fails, the second visual positioning method is automatically called for positioning; if the second visual positioning method fails, the third context positioning method is automatically called for positioning, and the context structure of the element to be tested is used to adjust the positioning path until successful positioning.

[0047] S23 , calling various preset element positioning methods according to the target priority, and positioning the target element corresponding to the element to be tested in the interface elements to obtain the target element.

[0048] In one embodiment, according to the target priority, each preset element positioning method is called to locate the target element corresponding to the element to be measured in the interface element, including: according to the calling sequence indicated by the target priority, the target preset positioning method corresponding to the calling sequence is called in sequence to locate the target element corresponding to the element to be measured until the target element is determined in the interface element.

[0049] The calling order of the target priority includes multiple arrangement orders.

[0050] Specifically, first, the target element type of the element to be measured is determined, and the calling order of multiple preset element positioning methods is determined based on the target element type. Next, each target preset positioning method is called in sequence according to the calling order to perform positioning. If positioning is successful, the calling order of the target preset positioning method corresponding to the next calling order is stopped. In this way, the calling order of multiple preset element positioning methods is preset according to the target element type of the element to be measured. If the target element positioning method corresponding to the first calling order is used, the target element can be accurately located without having to call all the preset positioning methods in sequence, thereby improving the efficiency of element positioning.

[0051] Based on the above implementation, according to the calling sequence indicated by the target priority, the target preset positioning method corresponding to the calling sequence is called in sequence to locate the target element corresponding to the element to be measured until the target element is determined in the interface element. The method also includes: determining the target element type of the element to be measured, and determining the target preset positioning method based on which the element to be measured obtains the target element; and associating the target element type with the target preset positioning method and storing them in a preset knowledge base.

[0052] In this embodiment, after each successful positioning, the target element type and the corresponding target element positioning method are stored in a preset knowledge base to provide a basis for determining the target priority in subsequent element positioning.

[0053] Exemplarily, the target element types of the elements to be detected include: First, the type of the element to be tested is a plain text attribute element, which can be located using the first text attribute positioning method.

[0054] Secondly, the type of element to be tested is a graphic element without text, which can be positioned using the second visual positioning method.

[0055] Third, the type of element to be tested is in a complex scene, and the first text attribute positioning method or the second visual positioning method fails to locate it, so the third context positioning method is used for positioning.

[0056] Based on the above implementation, the preset knowledge base includes a preset positioning method with the highest priority corresponding to different element types.

[0057] Determining target priorities of multiple preset element positioning methods includes: determining a target element type of the element to be measured; selecting a highest priority positioning method associated with the target element type from a preset knowledge base; and determining a target priority based on the highest priority positioning method.

[0058] Based on the above steps S21 to S23, specific implementations of various preset element positioning methods are as follows.

[0059] In one embodiment, when the target preset positioning method is determined to be the first text attribute positioning, the key attributes of the element to be tested are extracted to generate an attribute feature vector; based on the attribute feature vector, the first candidate element in the interface element is determined; the first candidate element represents an element in the interface element whose similarity with the attribute feature vector of the element to be tested is higher than the first preset similarity; and the target element with the highest similarity is determined from multiple first candidate elements.

[0060] In this embodiment, a Levenshtein ratio algorithm is used to calculate the Levenshtein distances (Levenshtein distances) between multiple elements in the interface elements and the element to be measured, and a similarity score is obtained based on the Levenshtein distances.

[0061] Specifically, the elements with a similarity score greater than 0.7 with the element to be tested are determined as the first candidate elements, and the element with the highest similarity score is selected from the first candidate elements as the target element. The specific procedure is as follows:

[0062]

[0063] In one embodiment, a DOM tree is constructed based on the interface elements and element information associated with the interface elements; the DOM tree includes: multiple DOM tree nodes and association relationships between the multiple DOM tree nodes, and each DOM tree node represents each interface element.

[0064] Among them, each DOM tree node corresponding to each interface element is determined to facilitate element positioning using the second visual positioning method or the third contextual positioning method.

[0065] In one embodiment, when it is determined that the preset target positioning method is the second visual positioning, the image features of the element to be measured are obtained, and the interface image corresponding to the interface element is obtained; the target coordinate position of the image feature in the interface image is determined; according to the target coordinate position, the target DOM tree node is selected from the DOM tree; according to the target DOM tree node, the target element corresponding to the element to be measured in the interface element is determined.

[0066] Specifically, the image features of the element to be tested are obtained, as well as the interface image corresponding to the interface element. By integrating the OmniPaser V2 visual model (integrating OmniPaser V2, a visual parsing model), the coordinate position of the element to be tested in the interface image is identified, and the target DOM tree node corresponding to the coordinate position is determined. The target element corresponding to the interface element is determined through the target DOM tree node. The image coordinates are converted to DOM tree nodes, and the element in the interface element corresponding to the target DOM tree node is determined as the target element. The specific procedure is as follows: from omni_parser_v2 import VisualLocator def visual_matching(target_image_path, screen_image): locator = VisualLocator() # Load the target element screenshot target_img = Image.open(target_image_path) # Identify the target location in the screenshot positions = locator.detect(target_img, screen_image) if positions: # Get DOM nodes according to coordinates (need to combine coordinate mapping technology) return get_dom_node_by_position(positions[0]) return None In one embodiment, when the target preset positioning method is determined to be the third context positioning, the target element node to be tested corresponding to the element to be tested in the DOM tree is determined; a target parent node closest to the target element node to be tested is determined from the DOM tree; a first path between the target element node to be tested and the target parent node is generated; the first path represents that there are no other related nodes between the target element node and the target parent node; based on the association relationship between multiple DOM tree nodes, one or more related nodes are added to the first path to generate multiple first alternative paths; the related nodes include ancestor nodes that increase the path depth, sibling nodes that increase content constraints, and child nodes that enhance the path structure; the path complexity priority of the multiple first alternative paths is determined; the path complexity priority is that the priority of low path complexity is higher than the priority of high path complexity; according to the path complexity priority, the multiple first alternative paths are verified in turn, and a target path that uniquely locates the element to be tested is determined from the multiple first alternative paths; based on the target path, the target element corresponding to the element to be tested in the interface element is determined.

[0067] The generated path is an XPath or CSS path. The specific procedure is as follows:

[0068] In order to achieve the above functions, the interface element positioning device includes hardware structures and / or software modules corresponding to the execution of each function. Those skilled in the art should easily realize that, in combination with the algorithm steps of each example described in the embodiments disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0069] The present application also provides a method Figure 3 The interface element positioning device shown includes: an acquisition unit 301, a matching unit 302 and a positioning unit 303.

[0070] The acquisition unit 301 is used to acquire the element to be measured.

[0071] The matching unit 302 is used to determine the target priority of multiple preset element positioning methods; the target priority represents the order in which each preset element positioning method is called for matching; multiple preset element positioning methods: first text attribute positioning, second visual positioning and third context positioning.

[0072] The positioning unit 303 is used to call various preset element positioning methods according to the target priority, and locate the target element corresponding to the element to be detected in the interface elements to obtain the target element.

[0073] As an implementation method, the matching unit 302 is specifically configured to call each preset element positioning method according to the target priority, and locate the target element corresponding to the element to be tested in the interface element, including: calling the target preset positioning method corresponding to the calling sequence in sequence according to the calling sequence indicated by the target priority, and locating the target element corresponding to the element to be tested until the target element is determined in the interface element.

[0074] As another implementation method, the matching unit 302 is specifically configured to call the target preset positioning method corresponding to the calling sequence in sequence according to the calling sequence indicated by the target priority, and locate the target element corresponding to the element to be tested until the target element is determined in the interface element. The method also includes: determining the target element type of the element to be tested, and determining the target preset positioning method based on which the element to be tested obtains the target element; and associating the target element type with the target preset positioning method and storing them in a preset knowledge base.

[0075] As another implementation method, the matching unit 302 is specifically configured such that the preset knowledge base includes a preset positioning method with the highest priority corresponding to different element types, and determines the target priority of multiple preset element positioning methods, including: determining the target element type of the element to be measured; selecting the highest priority positioning method associated with the target element type from the preset knowledge base; and determining the target priority based on the highest priority positioning method.

[0076] As another implementation method, the matching unit 302 is specifically configured to determine the target priorities of multiple preset element positioning methods, and also includes: when the target element type of the element to be tested is uncertain, determining that the target priorities of multiple preset positioning methods are preset to the first text attribute positioning priority being higher than the second visual positioning priority being higher than the third context positioning priority.

[0077] As another implementation method, the positioning unit 303 is specifically configured to locate the target element corresponding to the element to be tested, including: when determining that the target preset positioning method is the first text attribute positioning, extracting the key attributes of the element to be tested and generating an attribute feature vector; based on the attribute feature vector, determining the first candidate element in the interface element; the first candidate element represents an element in the interface element whose similarity with the attribute feature vector of the element to be tested is higher than the first preset similarity; and determining the target element with the highest similarity from multiple first candidate elements.

[0078] As another implementation method, the positioning unit 303 is specifically configured to construct a DOM tree based on the interface elements and element information associated with the interface elements; the DOM tree includes: multiple DOM tree nodes and association relationships between multiple DOM tree nodes, and each DOM tree node represents each interface element.

[0079] As another implementation method, the positioning unit 303 is specifically configured to locate the target element corresponding to the element to be measured, including: when determining that the target preset positioning method is the second visual positioning, obtaining the image features of the element to be measured, and obtaining the interface image corresponding to the interface element; determining the target coordinate position of the image feature in the interface image; selecting the target DOM tree node from the DOM tree according to the target coordinate position; and determining the target element corresponding to the element to be measured in the interface element according to the target DOM tree node.

[0080] As another implementation method, the positioning unit 303 is specifically configured to locate the target element corresponding to the element to be tested, including: when determining that the target preset positioning method is the third context positioning, determining the target element node to be tested corresponding to the element to be tested in the DOM tree; determining a target parent node closest to the target element node to be tested from the DOM tree; generating a first path between the target element node to be tested and the target parent node; the first path represents that there are no other related nodes between the target element node and the target parent node; based on the association relationship between multiple DOM tree nodes, adding one or more related nodes to the first path to generate multiple first alternative paths; the related nodes include ancestor nodes that increase the path depth, sibling nodes that increase content constraints, and child nodes that enhance the path structure; determining the path complexity priority of multiple first alternative paths; the path complexity priority is that the priority of low path complexity is higher than the priority of high path complexity; according to the path complexity priority, verifying multiple first alternative paths in turn, and determining a target path that uniquely locates the element to be tested from multiple first alternative paths; based on the target path, determining the target element corresponding to the element to be tested in the interface element.

[0081] Regarding the device in the above embodiment, the specific manner in which each unit module performs the operation has been described in detail in the embodiment of the method, and will not be elaborated here.

[0082] Figure 4 This is a schematic diagram of an interface element positioning device provided by this application. Figure 4 The interface element positioning device 60 may include at least one first processor 601 and a memory 603 for storing processor-executable instructions. The first processor 601 is configured to execute instructions in the memory 603 to implement the interface element positioning method in the following embodiments.

[0083] In addition, the interface element positioning device 60 may further include a communication bus 602 , at least one communication interface 604 , an input device 606 , and an output device 605 .

[0084] The first processor 601 can be a processor (central processing unit, CPU), a microprocessor unit, an ASIC, or one or more integrated circuits for controlling the execution of the program of the present application.

[0085] The communication bus 602 may include a pathway for transmitting information between the aforementioned components.

[0086] The communication interface 604 uses any transceiver or other device for communicating with other devices or communication networks, such as Ethernet, radio access network (RAN), wireless local area network (WLAN), etc.

[0087] The input device 606 is used to receive input signals and the output device 605 is used to output signals.

[0088] The memory 603 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, a random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, an optical disc storage (including compact discs, laser discs, optical discs, digital versatile discs, Blu-ray discs, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but is not limited thereto. The memory may be independent and connected to the processing unit via a bus. The memory may also be integrated with the processing unit.

[0089] The memory 603 is used to store instructions for executing the solution of the present application, and the execution is controlled by the first processor 601. The first processor 601 is used to execute the instructions stored in the memory 603, thereby realizing the functions of the method of the present application.

[0090] In a specific implementation, as an embodiment, the first processor 601 may include one or more CPUs, such as Figure 4 CPU0 and CPU1 in.

[0091] In a specific implementation, as an embodiment, the interface element positioning device 60 may include multiple processors, such as Figure 4 6 and 7. Each of these processors can be a single-CPU processor or a multi-CPU processor. A processor herein can refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions).

[0092] The interface element positioning device is as follows Figure 4 The system includes a first processor 601 and a memory 603 for storing executable instructions for the first processor 601. The first processor 601 is configured to execute the executable instructions to implement the interface element positioning method according to any of the above possible implementations. The above methods can achieve the same technical effects and are not described here in detail to avoid repetition.

[0093] The present application also provides a computer-readable storage medium. When the instructions in the computer-readable storage medium are executed by a processor of an interface element positioning device or an interface element positioning apparatus, the interface element positioning device or the interface element positioning apparatus can perform the interface element positioning method described in any of the possible implementations described above. The same technical effects can be achieved, and to avoid repetition, they are not described here.

[0094] The present application also provides a computer program product, including a computer program or instructions, which is executed by a processor to implement the interface element positioning method according to any of the above possible implementations. The same technical effects can be achieved, and to avoid repetition, they are not described here.

[0095] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered as exemplary only, and the true scope and spirit of the present application are indicated by the following claims.

[0096] It should be understood that the present application is not limited to the exact structures described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A method for positioning interface elements, characterized in that: The method comprises: Get the element to be tested; Determining target priorities of multiple preset element positioning methods; the target priorities represent the order in which each preset element positioning method is called for matching; the multiple preset element positioning methods include: first, text attribute positioning, second, visual positioning, and third, context positioning; According to the target priority, each preset element positioning method is called to locate the target element corresponding to the element to be tested in the interface elements to obtain the target element.

2. The interface element positioning method according to claim 1, characterized in that: The method of calling each preset element positioning method according to the target priority to position the target element corresponding to the element to be detected in the interface elements includes: According to the calling sequence indicated by the target priority, the target preset positioning methods corresponding to the calling sequence are called in sequence to locate the target element corresponding to the element to be measured until the target element is determined in the interface elements.

3. The interface element positioning method according to claim 2, characterized in that: Positioning the target element corresponding to the element to be measured includes: When it is determined that the target preset positioning mode is the first text attribute positioning, extracting the key attributes of the element to be measured and generating an attribute feature vector; Based on the attribute feature vector, determining a first candidate element among the interface elements; the first candidate element represents an element among the interface elements whose attribute feature vector similarity with the element to be tested is higher than a first preset similarity; The target element with the highest similarity is determined from the plurality of first candidate elements.

4. The interface element positioning method according to claim 2, characterized in that: The method further comprises: A DOM tree is constructed according to the interface element and the element information associated with the interface element; the DOM tree includes: a plurality of DOM tree nodes and association relationships between the plurality of DOM tree nodes, and each DOM tree node represents each interface element.

5. The interface element positioning method according to claim 4, characterized in that: The positioning of the target element corresponding to the element to be measured includes: When it is determined that the preset positioning mode of the target is the second visual positioning, the image feature of the element to be measured is obtained, and the interface image corresponding to the interface element is obtained; Determining a target coordinate position of the image feature in the interface image; Selecting a target DOM tree node from the DOM tree according to the target coordinate position; According to the target DOM tree node, the target element corresponding to the element to be tested in the interface elements is determined.

6. The interface element positioning method according to claim 4, characterized in that: The positioning of the target element corresponding to the element to be measured includes: When it is determined that the target preset positioning mode is the third context positioning, Determine the target element node to be tested corresponding to the element to be tested in the DOM tree; Determine from the DOM tree a target parent node that is closest to the target element node to be tested; Generate a first path between the target element node to be tested and the target parent node; the first path indicates that there are no other related nodes between the target element node and the target parent node; Based on the association relationship between the multiple DOM tree nodes, one or more related nodes are added to the first path to generate multiple first candidate paths; the related nodes include ancestor nodes that increase the path depth, sibling nodes that add content constraints, and child nodes that enhance the path structure; Determining path complexity priorities of the plurality of first candidate paths; wherein the path complexity priority is such that a path with low complexity has a higher priority than a path with high complexity; According to the path complexity priority, multiple first candidate paths are verified in sequence, and a target path that uniquely locates the measured element is determined from the multiple first candidate paths; based on the target path, the target element corresponding to the element to be measured in the interface element is determined.

7. The interface element positioning method according to claim 2, characterized in that: The method further includes calling the target preset positioning methods corresponding to the calling order in sequence according to the calling order indicated by the target priority, and locating the target element corresponding to the element to be measured until the target element is determined in the interface elements: Determining the target element type of the element to be measured, and determining the target preset positioning mode according to which the element to be measured obtains the target element; The target element type and the target preset positioning method are associated and stored in a preset knowledge base.

8. The interface element positioning method according to claim 1, characterized in that: The preset knowledge base includes a preset positioning method with the highest priority corresponding to different element types. The target priority of the plurality of preset element positioning methods is determined, including: Determining the target element type of the element to be measured; Selecting the highest priority positioning method associated with the target element type from the preset knowledge base; Determining the target priority based on the highest priority positioning method; When the target element type of the element to be detected is uncertain, the target priorities of the multiple preset positioning methods are determined to be preset such that the first text attribute positioning priority is higher than the second visual positioning priority and higher than the third context positioning priority.

9. An interface element positioning device, characterized in that: The device comprises: The acquisition unit is configured to: acquire the element to be tested; The matching unit is configured to: determine target priorities of multiple preset element positioning methods; the target priorities represent the order in which the preset element positioning methods are called for matching; the multiple preset element positioning methods include: first, text attribute positioning, second, visual positioning, and third, context positioning; The positioning unit is configured to: call each preset element positioning method according to the target priority, and locate the target element corresponding to the element to be detected in the interface elements to obtain the target element.

10. An interface element positioning system, characterized in that: The interface element positioning system is configured to execute the interface element positioning method according to any one of claims 1 to 8.

Images