Interface interaction element positioning method and device, equipment, medium and program product

Abstract

The invention relates to an interface interaction element positioning method and device, equipment, a medium and a program product. The method comprises the steps of obtaining a to-be-tested page, a description text of a to-be-tested target interaction element in the to-be-tested page and an attribute value of a unique attribute; performing screenshot on the to-be-tested page to obtain a to-be-tested image; according to the description text and the to-be-tested image, a text image recognition model is called to carry out element positioning on a target interaction element in the to-be-tested image, an element positioning result is obtained, and the element positioning result comprises at least one piece of candidate positioning information for the target interaction element; and under the condition that the element positioning result comprises a plurality of pieces of candidate positioning information, calculating the distance between the position information of the attribute value of the unique attribute in the to-be-detected image and each piece of candidate positioning information, and selecting target positioning information from the plurality of pieces of candidate positioning information according to the distance to serve as a positioning result. By adopting the method, accurate interface interaction element positioning can be realized without relying on screenshot of unstructured data.

Description

Technical Field

[0001] The present application relates to the field of software development technology, and in particular to a method, device, computer equipment, computer-readable storage medium and computer program product for locating interface interactive elements. Background Art

[0002] With the development of software development technology, software testing is a key link in ensuring product quality and performance in the software development cycle. How to improve test efficiency and ensure test quality is inseparable from the construction of automated testing. In the agile development model, requirements change faster and the iteration cycle is short. The rapid iteration of requirements will lead to frequent changes in UI elements, resulting in higher UI automation maintenance costs. However, existing UI element positioning technologies, such as positioning methods based on XPATH expressions or other attributes, are difficult to cope with all situations as the front-end framework continues to evolve and the position of elements changes. On the other hand, using an image-based positioning method, the element to be located is screenshotted and then compared with the overall page screenshot to locate the element. However, this method relies on the stability of the page. Once the page changes, it is very likely that the original positioning method will become invalid, and it is very inconvenient to store unstructured image data. Summary of the invention

[0003] Based on this, it is necessary to provide an interface interaction element positioning method, device, computer equipment, computer-readable storage medium and computer program product that can achieve element positioning based on text-image recognition without relying on changes in element positions to address the above technical problems.

[0004] In a first aspect, the present application provides a method for locating an interface interactive element, comprising:

[0005] Get the page to be tested for interface interaction testing;

[0006] Obtaining the description text and the attribute value of the unique attribute of the target interactive element to be tested in the page to be tested;

[0007] Taking a screenshot of the page to be tested to obtain an image to be tested;

[0008] According to the description text and the image to be tested, calling a text image recognition model to locate a target interactive element in the image to be tested, and obtaining an element positioning result, wherein the element positioning result includes at least one candidate positioning information for the target interactive element;

[0009] In the case that the element positioning result includes multiple candidate positioning information, the distance between the position information of the attribute value of the unique attribute in the image to be tested and each of the candidate positioning information is calculated, and the target positioning information is selected from the multiple candidate positioning information according to the distance as the positioning result for the target interactive element. The positioning result is used to perform interface interaction test on the target interactive element in the page to be tested.

[0010] In one of the embodiments, calling a text image recognition model to locate a target interactive element in the image to be tested based on the description text and the image to be tested to obtain an element location result includes:

[0011] Determine the type of the target interactive element according to the description text and the image to be tested;

[0012] The text image recognition model is called according to the type to locate the target interactive element in the image to be tested, and the element positioning result is obtained.

[0013] In one embodiment, the method further comprises:

[0014] In the case where the type indicates that the target interactive element is a non-unique element, the text image recognition model and the attribute value are called to locate the target interactive element in the image to be tested to obtain the element location result.

[0015] In one embodiment, the candidate positioning information includes: vertex coordinates and text box length and width data; the distance between the position information of the attribute value of the unique attribute in the image to be tested and each of the candidate positioning information is calculated, including:

[0016] Calculate the center point according to the vertex coordinates and the length and width data of the text box to obtain candidate center point coordinates;

[0017] The distance between the attribute value center point coordinates of the location information and each candidate center point coordinate is calculated using a Euclidean distance calculation model.

[0018] In one embodiment, selecting the target positioning information from the plurality of candidate positioning information according to the distance includes:

[0019] Sorting the distances from small to large to obtain a sorting result;

[0020] The candidate positioning information corresponding to the distance with the smallest ranking in the sorting result is used as the target positioning information.

[0021] In one embodiment, the method further comprises:

[0022] In the case that the element positioning result includes a candidate positioning information, the candidate positioning information is used as the positioning result for the target interactive element, and the positioning result is used to perform an interface interaction test on the target interactive element in the page to be tested.

[0023] In a second aspect, the present application also provides an interface interactive element positioning device, comprising:

[0024] The first acquisition module is used to acquire the page to be tested for interface interaction testing;

[0025] A second acquisition module is used to acquire the description text and the attribute value of the unique attribute of the target interactive element to be tested in the page to be tested;

[0026] A screenshot module, used to take a screenshot of the page to be tested to obtain an image to be tested;

[0027] an element positioning module, configured to call a text image recognition model to locate a target interactive element in the image to be tested according to the description text and the image to be tested, and obtain an element positioning result, wherein the element positioning result includes at least one candidate positioning information for the target interactive element;

[0028] A positioning selection module is used to calculate the distance between the position information of the attribute value of the unique attribute in the image to be tested and each of the candidate positioning information when the element positioning result includes multiple candidate positioning information, and select target positioning information from the multiple candidate positioning information according to the distance as the positioning result for the target interactive element, and the positioning result is used to perform interface interaction testing on the target interactive element in the page to be tested.

[0029] In a third aspect, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0030] Get the page to be tested for interface interaction testing;

[0031] Obtaining the description text and the attribute value of the unique attribute of the target interactive element to be tested in the page to be tested;

[0032] Taking a screenshot of the page to be tested to obtain an image to be tested;

[0033] According to the description text and the image to be tested, calling a text image recognition model to locate a target interactive element in the image to be tested, and obtaining an element positioning result, wherein the element positioning result includes at least one candidate positioning information for the target interactive element;

[0034] In the case that the element positioning result includes multiple candidate positioning information, the distance between the position information of the attribute value of the unique attribute in the image to be tested and each of the candidate positioning information is calculated, and the target positioning information is selected from the multiple candidate positioning information according to the distance as the positioning result for the target interactive element. The positioning result is used to perform interface interaction test on the target interactive element in the page to be tested.

[0035] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the following steps are implemented:

[0036] Get the page to be tested for interface interaction testing;

[0037] Obtaining the description text and the attribute value of the unique attribute of the target interactive element to be tested in the page to be tested;

[0038] Taking a screenshot of the page to be tested to obtain an image to be tested;

[0039] According to the description text and the image to be tested, calling a text image recognition model to locate a target interactive element in the image to be tested, and obtaining an element positioning result, wherein the element positioning result includes at least one candidate positioning information for the target interactive element;

[0040] In the case that the element positioning result includes multiple candidate positioning information, the distance between the position information of the attribute value of the unique attribute in the image to be tested and each of the candidate positioning information is calculated, and the target positioning information is selected from the multiple candidate positioning information according to the distance as the positioning result for the target interactive element. The positioning result is used to perform interface interaction test on the target interactive element in the page to be tested.

[0041] In a fifth aspect, the present application further provides a computer program product, including a computer program, which implements the following steps when executed by a processor:

[0042] Get the page to be tested for interface interaction testing;

[0043] Obtaining the description text and the attribute value of the unique attribute of the target interactive element to be tested in the page to be tested;

[0044] Taking a screenshot of the page to be tested to obtain an image to be tested;

[0045] According to the description text and the image to be tested, calling a text image recognition model to locate a target interactive element in the image to be tested, and obtaining an element positioning result, wherein the element positioning result includes at least one candidate positioning information for the target interactive element;

[0046] In the case that the element positioning result includes multiple candidate positioning information, the distance between the position information of the attribute value of the unique attribute in the image to be tested and each of the candidate positioning information is calculated, and the target positioning information is selected from the multiple candidate positioning information according to the distance as the positioning result for the target interactive element. The positioning result is used to perform interface interaction test on the target interactive element in the page to be tested.

[0047] The above-mentioned interface interaction element positioning method, device, computer equipment, computer-readable storage medium and computer program product obtain the page to be tested for interface interaction; obtain the description text and attribute value of the unique attribute of the target interactive element to be tested in the page to be tested; take a screenshot of the page to be tested to obtain the image to be tested; based on the description text and the image to be tested, call the text image recognition model to perform element positioning on the target interactive element in the image to be tested to obtain an element positioning result, and the element positioning result includes at least one candidate positioning information for the target interactive element; when the element positioning result includes multiple candidate positioning information, calculate the distance between the position information of the attribute value of the unique attribute in the image to be tested and each candidate positioning information, select the target positioning information from the multiple candidate positioning information according to the distance, and use it as the positioning result for the target interactive element, and the positioning result is used to perform interface interaction testing on the target interactive element in the page to be tested. Therefore, by calling the text image recognition model to locate the target interactive element in the image to be tested, the element positioning result is obtained, and the relative position of the element in the page is directly used for positioning. The distance between the position information of the attribute value of the unique attribute in the image to be tested and each candidate positioning information is calculated, and the relative position of the element in the page is used for positioning, which will not bring about the problem of how to effectively store unstructured image data. And according to the distance, the unique target positioning information is accurately selected from multiple candidate positioning information as the positioning result for the target interactive element, thereby realizing efficient recognition and positioning of the target interactive element. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0049] Figure 1 A diagram of an application environment of a method for positioning an interface interaction element in an embodiment;

[0050] Figure 2 A schematic diagram of a flow chart of a method for locating an interface interaction element in an embodiment;

[0051] Figure 3 A schematic diagram of a flow chart of a method for locating an interface interaction element in an embodiment;

[0052] Figure 4 An element positioning interface diagram of an interface interaction element positioning method in one embodiment;

[0053] Figure 5 It is an element positioning principle diagram of a method for positioning an interface interaction element in one embodiment;

[0054] Figure 6 A complete flowchart of a method for locating an interface interaction element in one embodiment;

[0055] Figure 7 It is a structural block diagram of a device for positioning an interface interaction element in an embodiment;

[0056] Figure 8 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0057] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0058] The interface interaction element positioning method provided in the embodiment of the present application can be applied to Figure 1In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other network servers. The server 104 obtains the page to be tested for interface interaction testing; obtains the description text and the attribute value of the unique attribute of the target interactive element to be tested in the page to be tested; takes a screenshot of the page to be tested to obtain the image to be tested; according to the description text and the image to be tested, calls the text image recognition model to locate the target interactive element in the image to be tested, and obtains the element positioning result, and the element positioning result includes at least one candidate positioning information for the target interactive element; when the element positioning result includes multiple candidate positioning information, calculate the distance between the position information of the attribute value of the unique attribute in the image to be tested and each candidate positioning information, select the target positioning information from the multiple candidate positioning information according to the distance, as the positioning result for the target interactive element, and the positioning result is used to perform interface interaction testing on the target interactive element in the page to be tested. The terminal 102 may be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, IoT devices, and portable wearable devices. The IoT devices may be smart speakers, smart TVs, smart air conditioners, smart car devices, projection devices, etc. Portable wearable devices may be smart watches, smart bracelets, head-mounted devices, etc. Head-mounted devices may be virtual reality (VR) devices, augmented reality (AR) devices, smart glasses, etc. The server 104 may be an independent physical server, a server cluster or distributed system consisting of multiple physical servers, or a cloud server that provides cloud computing services.

[0059] In an exemplary embodiment, Figure 2 As shown, a method for positioning interface interactive elements is provided, and the method is applied to Figure 1 The server 104 in the example is used as an example to illustrate, including the following steps 202 to 210. Among them:

[0060] Step 202, obtaining a page to be tested for interface interaction testing.

[0061] The interface interaction test to be performed may be a test process for an interactive function of an interface or an interactive system of an interface, but is not limited thereto. The page to be tested may be a test page for a certain part or a certain function of the interface interaction.

[0062] In some embodiments, a page to be tested for interface interaction testing in a certain function may be obtained according to user needs to achieve real-time interactive response and improve the interactive flexibility in actual application.

[0063] Step 204: Obtain the description text and the attribute value of the unique attribute of the target interactive element to be tested in the page to be tested.

[0064] The target interactive element to be tested may be a component in the page to be tested, and the description text corresponds to the text description content of the component, for example, a deletion component, whose description text is "delete". The attribute value of the unique attribute refers to an attribute value used to uniquely identify the target interactive element to be tested, and the attribute value may be text or numbers. For example, the attribute value of the unique attribute of the target interactive element to be tested may be "test" or "123", but is not limited thereto.

[0065] In some embodiments, the description text of the target interactive element to be tested in the page to be tested can be first obtained from the interface interaction test database. If the description text is unique, there is no need to obtain the attribute value of the unique attribute; if the description text is not unique, it is necessary to obtain the attribute value of the unique attribute corresponding to the target interactive element to be tested. The attribute value of the unique attribute can be obtained through user input, but is not limited to this.

[0066] Further examples are needed: Figure 3 As shown in the figure, when the element text is "New User" and is the only element text in the page to be tested, the attribute value of the unique attribute can be left blank; when the element text is "Delete" and is not a unique element text in the page to be tested, the unique attribute value can be obtained to assist in locating the element in the page to be tested. Figure 3 The interface interaction test to be performed is "delete the test user", and the element text obtained is "delete" and the only attribute value is "test".

[0067] Step 206: Take a screenshot of the page to be tested to obtain an image to be tested.

[0068] In some embodiments, a screenshot of the page to be tested is taken to obtain an image to be tested so that the area where element positioning is required is reduced by the screenshot, thereby reducing the amount of data required to be recognized by a subsequent text image recognition model. Therefore, the execution efficiency of subsequent element positioning can be effectively improved. This embodiment does not rely on screenshots for element positioning.

[0069] In some embodiments, a screenshot of the page to be tested is taken, and the image to be tested is obtained as follows: Figure 3 as shown in .

[0070] Step 208: Based on the description text and the image to be tested, the text image recognition model is called to locate the target interactive element in the image to be tested, and an element positioning result is obtained. The element positioning result includes at least one candidate positioning information for the target interactive element.

[0071] Among them, the text image recognition model is a computer vision model. In this embodiment, the AnyText AI model is used as an example for illustration, but it is not limited to this.

[0072] In some embodiments, based on the description text and the image to be tested, a text image recognition model is called to locate the target interactive element in the image to be tested to obtain an element positioning result, including: determining the type of the target interactive element based on the description text and the image to be tested; and calling the text image recognition model to locate the target interactive element in the image to be tested according to the type to obtain an element positioning result.

[0073] Among them, the target interactive element is the interactive component that needs to interact in the interface interaction test.

[0074] In some embodiments, the number of elements, font, and the number of words in the description text of the target interactive element in the image to be tested are determined based on the description text, thereby determining the type of the target interactive element. According to the type, a text image recognition model is called to locate the target interactive element in the image to be tested. The target interactive element can be accurately located, and the text box vertex coordinates and text box length and width data of the target interactive element are obtained as the element positioning result.

[0075] In some embodiments, the method further includes: when the type indicates that the target interactive element is not a unique element, calling a text image recognition model and attribute values ​​to locate the target interactive element in the image to be tested, and obtaining an element location result.

[0076] It should be noted that when the number of elements corresponding to this type is greater than 1, it indicates that the target interactive element is not a unique element. By calling the text image recognition model and attribute values ​​to locate the target interactive element in the image to be tested, the attribute values ​​can be used to assist in locating the element position, thereby obtaining a more accurate element positioning result.

[0077] Step 210, when the element positioning result includes multiple candidate positioning information, calculate the distance between the position information of the attribute value of the unique attribute in the image to be tested and each candidate positioning information, and select the target positioning information from the multiple candidate positioning information according to the distance as the positioning result for the target interactive element. The positioning result is used to perform interface interaction testing on the target interactive element in the page to be tested.

[0078] The distance between the position information of the attribute value of the unique attribute in the image to be tested and each candidate positioning information refers to the distance between the center coordinate point of the attribute value of the unique attribute and each candidate positioning information, that is, the distance between the center coordinate point corresponding to the candidate positioning information.

[0079] In some embodiments, the candidate positioning information includes: vertex coordinates and text box length and width data; calculating the distance between the position information of the attribute value of the unique attribute in the image to be tested and each candidate positioning information, including: performing center point calculation based on the vertex coordinates and the text box length and width data to obtain the candidate center point coordinates; calculating the distance between the center point coordinates of the attribute value of the position information and each candidate center point coordinates through a Euclidean distance calculation model.

[0080] In some embodiments, the center point is calculated based on the vertex coordinates and the length and width data of the text box to obtain the candidate center point coordinates corresponding to each candidate positioning information, and then the distance between the center point coordinates of the attribute value of the location information and each candidate center point coordinate is calculated using the Euclidean distance calculation model. The specific Euclidean distance calculation model is as follows:

[0081] ;

[0082] in,( , ) is the coordinate of the center point of the attribute value, ( , ) are the candidate center point coordinates.

[0083] In some embodiments, Figure 4 As shown, the distances d1, d2, and d3 between the coordinates of the center point of the attribute value of the location information and the coordinates of each candidate center point are calculated respectively by using the Euclidean distance calculation model.

[0084] In some embodiments, target positioning information is selected from multiple candidate positioning information based on distance, including: sorting the distances from small to large to obtain a sorting result; and taking the candidate positioning information corresponding to the smallest distance in the sorting result as the target positioning information.

[0085] In some embodiments, Figure 4 As shown, by comparing the distances between the coordinates of the center point of the attribute value and the coordinates of each candidate center point, the closest distance is obtained as d2, and the candidate positioning information corresponding to d2 is used as the target positioning information. The target interactive element that is truly associated with and corresponds to the attribute value of the unique attribute "test" can be accurately and efficiently determined.

[0086] In some embodiments, the coordinates of the nearest candidate center point can be determined more orderly and quickly as target positioning information through sorting, and the Euclidean distance calculation model can be used to accurately locate elements, thereby improving the stability of the automated script and using the relative position of the element in the page for positioning, rather than relying on screenshots of unstructured data. Therefore, there will be no problem of how to effectively store unstructured image data.

[0087] In the above-mentioned interface interaction element positioning method, a page to be tested for interface interaction testing is obtained; description text and attribute value of a unique attribute of a target interactive element to be tested in the page to be tested are obtained; a screenshot of the page to be tested is taken to obtain an image to be tested; based on the description text and the image to be tested, a text image recognition model is called to perform element positioning on the target interactive element in the image to be tested to obtain an element positioning result, and the element positioning result includes at least one candidate positioning information for the target interactive element; when the element positioning result includes multiple candidate positioning information, the distance between the position information of the attribute value of the unique attribute in the image to be tested and each candidate positioning information is calculated, and the target positioning information is selected from the multiple candidate positioning information according to the distance as the positioning result for the target interactive element, and the positioning result is used to perform interface interaction testing on the target interactive element in the page to be tested. Therefore, by calling the text image recognition model to locate the target interactive element in the image to be tested, the element positioning result is obtained, and the relative position of the element in the page is directly used for positioning. The distance between the position information of the attribute value of the unique attribute in the image to be tested and each candidate positioning information is calculated, and the relative position of the element in the page is used for positioning, which will not bring about the problem of how to effectively store unstructured image data. And according to the distance, the unique target positioning information is accurately selected from multiple candidate positioning information as the positioning result for the target interactive element, thereby realizing efficient recognition and positioning of the target interactive element.

[0088] In an exemplary embodiment, Figure 5 As shown, the method for locating an interface interaction element further includes step 502. Among them:

[0089] Step 502: When the element positioning result includes a candidate positioning information, the candidate positioning information is used as the positioning result for the target interactive element, and the positioning result is used to perform interface interaction test on the target interactive element in the page to be tested.

[0090] Among them, interface interaction testing can be an automated testing process.

[0091] In some embodiments, the method further includes: when the element positioning result includes a candidate positioning information, using the candidate positioning information as the positioning result for the target interactive element, and the positioning result is used to perform interface interaction testing on the target interactive element in the page to be tested.

[0092] In some embodiments, when the element positioning result includes a candidate positioning information, it means that there is only one target interactive element in the current image to be tested, so there is no need to calculate the distance between the candidate positioning information and the position information of the attribute value of the unique attribute. Therefore, the candidate positioning information is directly used as the positioning result for the target interactive element, thereby achieving efficient identification and positioning of the target interactive element.

[0093] In order to understand the solution of this application more clearly, Figure 6 The details are as follows:

[0094] In some embodiments, Figure 6 In the method, a test image P is obtained by taking a screenshot of the test page, and then the description text A of the target interactive element to be tested in the test image P is obtained. Then, it is determined whether the description text A exists uniquely in the test image P. If the description text A exists uniquely, the description text and the test image are input into the text image recognition model, i.e., AnyText AI model. If the description text A exists uniquely, that is, when the element positioning result includes multiple candidate positioning information, the attribute value B of the unique attribute of the test target interactive element is obtained, and the description text A, the test image P and the attribute value B of the unique attribute are input into the text image recognition model for element positioning, and the position information of each candidate positioning information corresponding to the description text A and the position information of the attribute value B are obtained. Then, the distance between the position information of the attribute value of the unique attribute in the test image and each candidate positioning information is calculated, and the candidate positioning information with the closest distance is selected from the multiple candidate positioning information according to the distance as the target positioning information, and as the positioning result for the target interactive element. The positioning result is used to perform interface interaction test on the target interactive element in the test page.

[0095] In some embodiments, it is necessary to illustrate that the image P to be tested, the description text A and the attribute value of the unique attribute are sent to the computer vision model. This method takes the AnyText AI model as an example to obtain the coordinates of the input text in the screenshot ( , , , ),in( , ) is the text box vertex coordinate of the upper left corner of the text box, , The length and width data of the text box.

[0096] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0097] Based on the same inventive concept, the embodiment of the present application also provides an interface interaction element positioning device for implementing the interface interaction element positioning method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more embodiments of the interface interaction element positioning device provided below can refer to the limitations of the interface interaction element positioning method above, and will not be repeated here.

[0098] In an exemplary embodiment, Figure 7 As shown, a device for positioning an interface interactive element is provided, including: a first acquisition module 701, a second acquisition module 702, a screenshot module 703, an element positioning module 704 and a positioning selection module 705, wherein:

[0099] The first acquisition module 701 is used to acquire a page to be tested for interface interaction testing;

[0100] The second acquisition module 702 is used to acquire the description text and the attribute value of the unique attribute of the target interactive element to be tested in the page to be tested;

[0101] A screenshot module 703 is used to take a screenshot of the page to be tested to obtain an image to be tested;

[0102] An element positioning module 704 is used to call a text image recognition model to locate a target interactive element in the image to be tested according to the description text and the image to be tested, and obtain an element positioning result, wherein the element positioning result includes at least one candidate positioning information for the target interactive element;

[0103] The positioning selection module 705 is used to calculate the distance between the position information of the attribute value of the unique attribute in the image to be tested and each candidate positioning information when the element positioning result includes multiple candidate positioning information, and select the target positioning information from the multiple candidate positioning information according to the distance as the positioning result for the target interactive element. The positioning result is used to perform interface interaction testing on the target interactive element in the page to be tested.

[0104] In some embodiments, the element positioning module 704 is further used to determine the type of the target interactive element according to the description text and the image to be tested; and to call the text image recognition model according to the type to locate the target interactive element in the image to be tested, to obtain an element positioning result.

[0105] In some embodiments, the element positioning module 704 is further used to call the text image recognition model and attribute values ​​to locate the target interactive element in the image to be tested when the type indicates that the target interactive element is not a unique element, so as to obtain an element positioning result.

[0106] In some embodiments, the candidate positioning information includes: vertex coordinates and text box length and width data; the positioning selection module 705 is also used to calculate the center point based on the vertex coordinates and text box length and width data to obtain the candidate center point coordinates; and calculate the distance between the attribute value center point coordinates of the location information and each candidate center point coordinate through the Euclidean distance calculation model.

[0107] In some embodiments, the positioning selection module 705 is further used to sort the distances from small to large to obtain a sorting result; and use the candidate positioning information corresponding to the smallest distance in the sorting result as the target positioning information.

[0108] In some embodiments, the device also includes: a second positioning selection module, which is used to use the candidate positioning information as the positioning result for the target interactive element when the element positioning result includes a candidate positioning information, and the positioning result is used to perform interface interaction testing on the target interactive element in the page to be tested.

[0109] In the above-mentioned interface interaction element positioning device, a page to be tested for interface interaction test is obtained; description text and attribute value of a unique attribute of a target interaction element to be tested in the page to be tested are obtained; a screenshot of the page to be tested is taken to obtain an image to be tested; based on the description text and the image to be tested, a text image recognition model is called to perform element positioning on the target interaction element in the image to be tested to obtain an element positioning result, and the element positioning result includes at least one candidate positioning information for the target interaction element; when the element positioning result includes multiple candidate positioning information, the distance between the position information of the attribute value of the unique attribute in the image to be tested and each candidate positioning information is calculated, and the target positioning information is selected from the multiple candidate positioning information according to the distance as the positioning result for the target interaction element, and the positioning result is used to perform interface interaction test on the target interaction element in the page to be tested. Therefore, by calling the text image recognition model to locate the target interactive element in the image to be tested, the element positioning result is obtained, and the relative position of the element in the page is directly used for positioning. The distance between the position information of the attribute value of the unique attribute in the image to be tested and each candidate positioning information is calculated, and the relative position of the element in the page is used for positioning, which will not bring about the problem of how to effectively store unstructured image data. And according to the distance, the unique target positioning information is accurately selected from multiple candidate positioning information as the positioning result for the target interactive element, thereby realizing efficient recognition and positioning of the target interactive element.

[0110] Each module in the above-mentioned interface interactive element positioning device can be implemented in whole or in part by software, hardware or a combination thereof. Each of the above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory in the computer device in the form of software, so that the processor can call and execute the corresponding operations of each of the above modules.

[0111] In an exemplary embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as shown in FIG. Figure 8As shown. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface, the display unit and the input device are connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and the external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, near field communication (NFC) or other technologies. When the computer program is executed by the processor, a method for positioning an interface interaction element is implemented. The display unit of the computer device is used to form a visually visible picture, which can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device shell, or an external keyboard, touchpad or mouse.

[0112] Those skilled in the art will understand that Figure 8 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0113] In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps of the above-mentioned interface interaction element positioning method when executing the computer program.

[0114] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned interface interaction element positioning method are implemented.

[0115] In one embodiment, a computer program product is provided, including a computer program, which implements the steps of the above-mentioned interface interaction element positioning method when executed by a processor.

[0116] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0117] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0118] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0119] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A method for locating an interface interactive element, characterized in that: The method comprises: Get the page to be tested for interface interaction testing; Obtaining the description text and the attribute value of the unique attribute of the target interactive element to be tested in the page to be tested; Taking a screenshot of the page to be tested to obtain an image to be tested; According to the description text and the image to be tested, calling a text image recognition model to locate a target interactive element in the image to be tested, and obtaining an element positioning result, wherein the element positioning result includes at least one candidate positioning information for the target interactive element; In the case that the element positioning result includes multiple candidate positioning information, the distance between the position information of the attribute value of the unique attribute in the image to be tested and each of the candidate positioning information is calculated, and the target positioning information is selected from the multiple candidate positioning information according to the distance as the positioning result for the target interactive element. The positioning result is used to perform interface interaction test on the target interactive element in the page to be tested.

2. The method according to claim 1, characterized in that: The step of calling a text image recognition model to locate a target interactive element in the image to be tested according to the description text and the image to be tested, and obtaining an element location result includes: Determine the type of the target interactive element according to the description text and the image to be tested; The text image recognition model is called according to the type to locate the target interactive element in the image to be tested, and the element positioning result is obtained.

3. The method according to claim 2, characterized in that The method further comprises: In the case where the type indicates that the target interactive element is a non-unique element, the text image recognition model and the attribute value are called to locate the target interactive element in the image to be tested to obtain the element location result.

4. The method according to claim 1, characterized in that: The candidate positioning information includes: vertex coordinates and text box length and width data; the distance between the position information for calculating the attribute value of the unique attribute in the image to be tested and each of the candidate positioning information includes: Calculate the center point according to the vertex coordinates and the length and width data of the text box to obtain candidate center point coordinates; The distance between the attribute value center point coordinates of the location information and each candidate center point coordinate is calculated using a Euclidean distance calculation model.

5. The method according to claim 1, characterized in that The selecting the target positioning information from the plurality of candidate positioning information according to the distance comprises: Sorting the distances from small to large to obtain a sorting result; The candidate positioning information corresponding to the distance with the smallest ranking in the sorting result is used as the target positioning information.

6. The method according to claim 1, characterized in that The method further comprises: In the case that the element positioning result includes a candidate positioning information, the candidate positioning information is used as the positioning result for the target interactive element, and the positioning result is used to perform an interface interaction test on the target interactive element in the page to be tested.

7. An interface interactive element positioning device, characterized in that: The device comprises: The first acquisition module is used to acquire the page to be tested for interface interaction testing; A second acquisition module is used to acquire the description text and the attribute value of the unique attribute of the target interactive element to be tested in the page to be tested; A screenshot module, used to take a screenshot of the page to be tested to obtain an image to be tested; an element positioning module, configured to call a text image recognition model to locate a target interactive element in the image to be tested according to the description text and the image to be tested, and obtain an element positioning result, wherein the element positioning result includes at least one candidate positioning information for the target interactive element; A positioning selection module is used to calculate the distance between the position information of the attribute value of the unique attribute in the image to be tested and each of the candidate positioning information when the element positioning result includes multiple candidate positioning information, and select target positioning information from the multiple candidate positioning information according to the distance as the positioning result for the target interactive element, and the positioning result is used to perform interface interaction testing on the target interactive element in the page to be tested.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images